@solidxai/core 0.1.11-beta.4 → 0.1.11-beta.6

package/docs/module-package-import-handoff.md

@@ -0,0 +1,691 @@
+# Module Package Import Handoff
+
+## Purpose
+
+This document captures the custom SolidX module import/export work completed across `solid-core-module` and `solid-core-ui`, plus the sample `contact-management` module created for end-to-end testing.
+
+The goal of this feature set is:
+
+- allow a user to import a SolidX module package from the module metadata list view
+- preview and validate the archive before copying files into `solid-api` and `solid-ui`
+- run a guided post-import flow for restart check, build, and seed
+- support packaging/exporting an existing module into a `.sldx` archive
+
+This is a custom implementation and intentionally does not reuse the older import/export controllers or UI flows.
+
+---
+
+## Scope Summary
+
+Implemented areas:
+
+- backend import/export endpoints in `solid-core-module`
+- backend archive parsing, validation, extraction, placement, build, and seed orchestration
+- frontend import dialog and multi-step progress UI in `solid-core-ui`
+- stock `SolidListView` header button wiring for module metadata list view
+- global error handling change from redirect to toast for `/admin` and `/auth`
+- sample `contact-management` module with matching metadata, API, and UI structure
+- archive validation support for macOS Finder zip artifacts and top-level wrapper folders
+- fix for imported archive staging polluting Nest build input
+
+Partially complete / still rough:
+
+- backend restart detection UX is fragile because importing into `solid-ui/src` can trigger Vite restart and disrupt the in-flight UI
+- the ping/restart step needs a more resilient experience
+- consuming projects still need to refresh their installed `@solidxai/core` / `@solidxai/core-ui` package builds after source changes if they are not linked live
+
+---
+
+## Archive Format
+
+### Extension
+
+- `.sldx`
+
+### Expected logical structure
+
+At minimum the archive must contain:
+
+- `manifest.json`
+- `solid-api/src/<module-name>/<module-name>.module.ts`
+- `solid-api/src/<module-name>/metadata/<module-name>-metadata.json`
+- `solid-ui/src/<module-name>/<module-name>.ui-module.ts`
+
+Example:
+
+```text
+manifest.json
+solid-api/
+  src/
+    contact-management/
+      contact-management.module.ts
+      metadata/
+        contact-management-metadata.json
+solid-ui/
+  src/
+    contact-management/
+      contact-management.ui-module.ts
+```
+
+### Supported archive variations
+
+The validator/importer now supports:
+
+- a flat archive with `manifest.json`, `solid-api`, and `solid-ui` at the root
+- an archive wrapped in a single top-level folder, for example `contact-management/...`
+- macOS Finder zip metadata like `__MACOSX/...` and `._*`, which are ignored during validation
+
+### Manifest expectations
+
+The manifest format currently expects:
+
+```json
+{
+  "schemaVersion": "1.0",
+  "packageType": "solidx-module",
+  "module": {
+    "name": "contact-management",
+    "displayName": "Contact Management",
+    "description": "..."
+  },
+  "contents": {
+    "metadataPath": "solid-api/src/contact-management/metadata/contact-management-metadata.json",
+    "apiModulePath": "solid-api/src/contact-management/contact-management.module.ts",
+    "uiModulePath": "solid-ui/src/contact-management/contact-management.ui-module.ts"
+  },
+  "postImport": {
+    "recommendedSteps": ["restart", "build", "seed"]
+  },
+  "checksums": {
+    "...": "sha256..."
+  }
+}
+```
+
+### Current validation markers
+
+Blocking validations:
+
+- archive extension must be `.sldx`
+- `manifest.json` must exist
+- `manifest.json` must be valid JSON
+- `manifest.packageType` must be `solidx-module`
+- `manifest.schemaVersion` must be `1.0`
+- `manifest.module.name` must be present
+- `manifest.contents.*` must exactly match the expected conventional paths
+- required files from `solid-api` and `solid-ui` must be physically present in the archive
+- metadata JSON must be valid JSON
+- metadata `moduleMetadata.name` must match `manifest.module.name` when present
+- archive paths must be safe, no absolute paths or traversal (`..`)
+
+Non-blocking warnings:
+
+- missing checksums
+- local conflicts where the target module already exists
+
+---
+
+## Backend Changes In `solid-core-module`
+
+### New files
+
+- `src/controllers/module-package.controller.ts`
+- `src/services/module-package.service.ts`
+- `src/dtos/confirm-module-package-import.dto.ts`
+- `src/dtos/run-module-package-build.dto.ts`
+- `src/dtos/run-module-package-seed.dto.ts`
+
+### Registration changes
+
+Registered into:
+
+- `src/solid-core.module.ts`
+- `src/index.ts`
+
+### New endpoints
+
+Defined in `src/controllers/module-package.controller.ts`:
+
+- `POST /module-packages/import/validate`
+- `GET /module-packages/export/:moduleName`
+- `POST /module-packages/import/:id/confirm`
+- `GET /module-packages/import/:id/status`
+- `POST /module-packages/import/:id/build`
+- `POST /module-packages/import/:id/seed`
+
+### Runtime behavior
+
+#### `validate`
+
+`validateUpload()` in `src/services/module-package.service.ts`:
+
+- accepts uploaded `.sldx` archive via Multer
+- creates a transaction working directory
+- moves uploaded file into the working directory
+- reads archive entries using `unzip -Z1`
+- extracts the archive
+- normalizes root folder handling
+- ignores macOS zip metadata entries
+- validates manifest and required marker files
+- computes preview payload for frontend
+- writes transaction status JSON
+
+#### `confirm`
+
+`confirmImport()`:
+
+- checks validation state and conflict state
+- resolves extracted archive root
+- copies `solid-api/src/<module>` into consuming app `solid-api/src/<module>`
+- copies `solid-ui/src/<module>` into consuming app `solid-ui/src/<module>`
+- updates status to `awaiting_restart`
+- records step output
+
+Important later fix:
+
+- after successful copy, `extractDirPath` is deleted and nulled so extracted temp contents do not linger
+
+#### `build`
+
+`runBuild()`:
+
+- can build `solid-api`
+- can build `solid-ui`
+- records combined command output into transaction status
+- sets status to `build_succeeded` or `build_failed`
+- does not throw away partial results
+
+#### `seed`
+
+`runSeed()`:
+
+- resolves `ModuleMetadataSeederService` from the registered seeders in `SolidRegistry`
+- seeds only the imported module name
+- can optionally seed global metadata
+- stores success/failure output in transaction status
+
+### Export behavior
+
+`exportModulePackage()`:
+
+- validates that existing module paths exist locally
+- stages a package under temporary export working directory
+- copies module folders from both `solid-api` and `solid-ui`
+- generates `manifest.json`
+- adds checksums for the required marker files
+- zips the staged folder into `.sldx`
+
+### Archive handling improvements added during debugging
+
+Added logic in `src/services/module-package.service.ts`:
+
+- `detectArchiveRootPrefix()`
+- `normalizeArchiveEntries()`
+- `toArchiveEntryPath()`
+- `isMacOsMetadataEntry()`
+
+These were added because Finder-created archives often looked like:
+
+- `contact-management/...`
+- `__MACOSX/...`
+
+Without this normalization the validator incorrectly reported a missing root manifest.
+
+### Latest build-scope fix
+
+Original issue:
+
+- module package working directories were stored under `solid-api/media-uploads/module-package-imports/...`
+- the extracted tree contained `solid-ui/src/...` TypeScript files
+- Nest build picked those files up and tried to compile them while building `solid-api`
+
+User-observed example:
+
+- `nest build` tried compiling extracted `contact-management.ui-module.ts` and `ContactRosterPage.tsx`
+
+Source-level fix now in `src/services/module-package.service.ts`:
+
+- module package runtime data no longer stages under `solid-api/media-uploads`
+- new runtime root:
+  - `../.solidx-runtime/module-package-imports`
+  - `../.solidx-runtime/module-package-exports`
+- optional override:
+  - `SOLIDX_MODULE_PACKAGE_RUNTIME_DIR`
+
+This is implemented via:
+
+- `getModulePackageRuntimeRoot()`
+- `getModulePackageImportsRoot()`
+- `getModulePackageExportsRoot()`
+
+This prevents future archive staging from contaminating app source trees or TypeScript build inputs.
+
+---
+
+## Frontend Changes In `solid-core-ui`
+
+### Module package dialog
+
+Main UI file:
+
+- `src/components/core/module/ModulePackageDialog.tsx`
+
+Styles:
+
+- `src/components/core/module/ModulePackageDialog.css`
+
+The dialog provides:
+
+- `.sldx` upload via `react-dropzone`
+- preview step before import
+- archive summary and file tree
+- conflict awareness and overwrite checkbox
+- guided multi-step progress UX
+- spinner, warning, success, and error states for each step
+- final summary text after flow completion
+
+### Step flow in UI
+
+The UI models four steps:
+
+1. Upload and place files
+2. Wait for service restart
+3. Build solid-api and solid-ui
+4. Seed module metadata
+
+`pingBackendWithRetry()`:
+
+- polls `/api/ping`
+- currently retries `40` times with `1500ms` delay
+
+Visual state types:
+
+- `pending`
+- `running`
+- `success`
+- `warning`
+- `error`
+
+### RTK Query API additions
+
+Added to `src/redux/api/moduleApi.ts`:
+
+- `validateModulePackageImport`
+- `confirmModulePackageImport`
+- `getModulePackageImportStatus`
+- `runModulePackageBuild`
+- `runModulePackageSeed`
+- `exportModulePackage`
+
+Important frontend bug fix:
+
+The backend responses are wrapped in `{ data: ... }`.
+
+The module-package endpoints needed:
+
+- `transformResponse: (response) => response?.data ?? response`
+
+Without this, the preview dialog appeared empty even though backend validation was successful.
+
+This was the root cause of the earlier state where:
+
+- network response looked valid in DevTools
+- `validation.valid` was `true`
+- but the preview UI still showed attention state / empty sections
+
+### Stock list header button wiring
+
+The final intended entry point is not the earlier custom `ModuleListViewData` toolbar.
+
+The actual live route the user uses is:
+
+- `/admin/core/solid-core/module-metadata/list`
+
+That route resolves into stock list view behavior and uses seeded metadata.
+
+Final wiring:
+
+- `solid-core-module/src/seeders/seed-data/solid-core-metadata.json`
+- `solid-core-ui/src/helpers/registry.ts`
+- `solid-core-ui/src/components/core/extension/solid-core/moduleMetadata/list/ModuleImportListHeaderAction.tsx`
+
+Seeded `headerButtons` config for `moduleMetadata-list-view`:
+
+- label: `Import Module`
+- action: `ModuleImportListHeaderAction`
+- openInPopup: `true`
+- icon: `si-upload`
+- closable: `false`
+- popupWidth: `min(1180px, calc(100vw - 32px))`
+- customComponentIsSystem: `true`
+
+Extension registration:
+
+- `registerExtensionComponent("ModuleImportListHeaderAction", ModuleImportListHeaderAction, ExtensionComponentTypes.listHeaderAction)`
+
+Extension component behavior:
+
+- renders `ModulePackageImportContent`
+- closes the popup via `closePopup`
+- refreshes registered list views via `listViewRegistry` after successful import
+
+### Note about older custom path still present
+
+There is older custom import/export wiring still present in:
+
+- `src/components/core/module/ModuleListViewData.tsx`
+
+That was an earlier attempt before the user clarified the stock list view route. It is not the authoritative final entry point for the module metadata list view.
+
+It still contains:
+
+- direct dialog state
+- export action handling
+- extra button logic
+
+Future cleanup decision:
+
+- either delete this path if unused
+- or keep it intentionally if another page still consumes `ModuleListViewData`
+
+### Global error behavior change
+
+Changed in:
+
+- `src/routes/AppEventListener.tsx`
+
+Previous behavior:
+
+- `AppEvents.GlobalError` redirected to dedicated error pages for 500/network failures
+
+Current behavior:
+
+- no redirect
+- show toast only
+- only active when current layout path starts with `/admin` or `/auth`
+
+Dev-mode enhancement:
+
+- checks `VITE_SOLIDX_ENV`
+- if dev/development, toast includes detailed error payload
+- can include status, message, response body, and stack
+- dev toasts are sticky and longer-lived
+
+Production-style behavior:
+
+- only summary toast
+
+This uses the existing toast SOP via:
+
+- `src/redux/features/toastSlice.ts`
+
+---
+
+## Seed Metadata Changes In `solid-core-module`
+
+The module metadata list view was updated in:
+
+- `src/seeders/seed-data/solid-core-metadata.json`
+
+Relevant change:
+
+- `moduleMetadata-list-view` now has a `headerButtons` entry for the import action
+
+This is what makes the standard list route surface the new header action without custom route-level UI replacement.
+
+---
+
+## Sample Module: `contact-management`
+
+Created at:
+
+- `contact-management/`
+
+Purpose:
+
+- provide a realistic module package to manually zip into `.sldx`
+- exercise both API and UI module conventions
+- include custom layout support rather than only CRUD defaults
+
+### Included structure
+
+At minimum:
+
+- `contact-management/manifest.json`
+- `contact-management/solid-api/src/contact-management/contact-management.module.ts`
+- `contact-management/solid-api/src/contact-management/metadata/contact-management-metadata.json`
+- `contact-management/solid-ui/src/contact-management/contact-management.ui-module.ts`
+
+Also created:
+
+- API entities, DTOs, repositories, controller, service
+- UI custom roster page and CSS
+- multiple contact-book related models
+
+The module models are centered around:
+
+- contacts
+- contact companies
+- contact activities / interactions
+
+### Metadata alignment work
+
+The sample files were tightened so metadata and generated code conventions match more closely:
+
+- DTO validators aligned to field definitions
+- date-like fields moved to `Date` with date validators where needed
+- inverse relation DTO shapes added for consistency
+- removed over-strict assumptions like `IsEmail` or `IsUrl` where not dictated by metadata
+- manifest checksums added for required marker files
+
+### Archive file
+
+Current generated archive for testing:
+
+- `contact-management/contact-management.sldx`
+
+It was rebuilt after earlier manifest and macOS zip issues.
+
+---
+
+## Debugging Timeline And Fixes
+
+### 1. Header action not visible
+
+Initial issue:
+
+- import button was added in custom module list view logic
+- user was actually testing the stock list route from seeded metadata
+
+Fix:
+
+- moved to `headerButtons` metadata + extension component pattern
+
+### 2. Finder zip reported missing root manifest
+
+Initial issue:
+
+- macOS archive added top-level folder and `__MACOSX`
+- validator looked only for strict root `manifest.json`
+
+Fix:
+
+- support root prefix normalization
+- ignore `__MACOSX` and `._*`
+
+### 3. Preview showed empty state even when backend validation was valid
+
+Initial issue:
+
+- RTK Query endpoints returned wrapper response instead of `.data`
+
+Fix:
+
+- added `transformResponse` unwrapping for module package endpoints
+
+### 4. Consuming app appeared stale
+
+Observed issue:
+
+- code changes existed in package source and installed package tree
+- browser still showed stale behavior
+
+Likely reason:
+
+- running Vite dev server had not restarted after package updates
+
+Recommended action:
+
+- restart consuming app dev server
+- hard refresh browser
+
+### 5. Imported archive staging broke Nest build
+
+Observed issue:
+
+- extracted archive inside `solid-api/media-uploads/module-package-imports/...`
+- `nest build` traversed extracted `solid-ui` TS/TSX files
+
+Fixes:
+
+- source-level fix in `solid-core-module` to move runtime staging to `../.solidx-runtime`
+- defensive app-side fix in consuming app `solid-library-management/solid-api/tsconfig.build.json` to exclude `media-uploads`
+
+The defensive app-side fix was added because earlier imported archives already existed there and would continue breaking builds until excluded or removed.
+
+---
+
+## Current Known Limitations
+
+### Restart / ping step UX
+
+The import itself works and places files correctly.
+
+However:
+
+- when files are copied into `solid-ui/src`, Vite may restart
+- the browser UI can reload
+- the restart check/ping flow can be interrupted
+
+This is why the user saw:
+
+- import worked
+- files/folders were created
+- ping flow did not feel reliable
+
+This is a UX/product gap, not primarily an archive validation issue.
+
+Potential future improvements:
+
+- persist transaction key in query param or local storage and resume status polling after reload
+- use backend status endpoint more aggressively after UI restart
+- decouple frontend app restart from post-import orchestration
+- run build/seed from backend asynchronously and let UI reconnect
+
+### Package refresh in consuming projects
+
+Source changes in `solid-core-module` and `solid-core-ui` do not automatically become active in consuming apps if those apps depend on local tarballs instead of live workspace links.
+
+Another agent should verify, per consuming app:
+
+- which `@solidxai/core` tarball is installed
+- which `@solidxai/core-ui` tarball is installed
+- whether a consuming app restart is still needed
+
+### Older `ModuleListViewData` implementation
+
+This still exists and may confuse future agents if they search for import UI code first.
+
+The final authoritative stock-list integration is:
+
+- metadata `headerButtons`
+- `ModuleImportListHeaderAction`
+
+Not:
+
+- the earlier button inside `ModuleListViewData`
+
+---
+
+## Files Most Important For Continuation
+
+### Backend
+
+- `solid-core-module/src/services/module-package.service.ts`
+- `solid-core-module/src/controllers/module-package.controller.ts`
+- `solid-core-module/src/dtos/confirm-module-package-import.dto.ts`
+- `solid-core-module/src/dtos/run-module-package-build.dto.ts`
+- `solid-core-module/src/dtos/run-module-package-seed.dto.ts`
+- `solid-core-module/src/seeders/seed-data/solid-core-metadata.json`
+- `solid-core-module/src/solid-core.module.ts`
+- `solid-core-module/src/index.ts`
+
+### Frontend
+
+- `solid-core-ui/src/components/core/module/ModulePackageDialog.tsx`
+- `solid-core-ui/src/components/core/module/ModulePackageDialog.css`
+- `solid-core-ui/src/components/core/extension/solid-core/moduleMetadata/list/ModuleImportListHeaderAction.tsx`
+- `solid-core-ui/src/redux/api/moduleApi.ts`
+- `solid-core-ui/src/helpers/registry.ts`
+- `solid-core-ui/src/routes/AppEventListener.tsx`
+- `solid-core-ui/src/redux/features/toastSlice.ts`
+
+### Sample/test assets
+
+- `contact-management/manifest.json`
+- `contact-management/contact-management.sldx`
+- `contact-management/solid-api/src/contact-management/...`
+- `contact-management/solid-ui/src/contact-management/...`
+
+### Consuming app defensive build fix
+
+- `solid-library-management/solid-api/tsconfig.build.json`
+
+---
+
+## Suggested Next Steps For The Next Agent
+
+1. Stabilize the restart experience.
+
+Possible direction:
+
+- on successful confirm/import, persist transaction key
+- if Vite reloads, reopen import progress view and call status endpoint
+- continue with build/seed after reconnect
+
+2. Decide whether build and seed should be:
+
+- UI-driven sequential calls after restart
+- or backend-queued follow-up steps with resumable status polling
+
+3. Decide whether `ModuleListViewData.tsx` import/export code should be removed or retained.
+
+4. Verify consuming apps are actually using the latest local `@solidxai/core` and `@solidxai/core-ui` builds.
+
+5. Consider adding richer validation markers later:
+
+- checksum verification instead of warning-only presence
+- optional version compatibility checks
+- optional manifest-declared dependencies
+
+6. If the experience should survive browser reloads, add transaction restoration support in the UI.
+
+---
+
+## Current Truth Snapshot
+
+At the point this handoff note was written:
+
+- custom backend controller/service for module package flow exists
+- custom frontend dialog exists
+- module metadata stock list view header button exists
+- preview-before-import exists
+- archive root wrapper and macOS zip handling exists
+- import works and copies files correctly
+- build and seed endpoints exist
+- global error redirects were replaced with toasts for `/admin` and `/auth`
+- sample `contact-management` module and `.sldx` archive exist
+- the `solid-api/media-uploads` build contamination bug has been fixed at source level in `solid-core-module`
+- a defensive `media-uploads` exclusion was also added in `solid-library-management/solid-api/tsconfig.build.json`
+
+The biggest remaining problem is not archive structure anymore. It is the restart/resume UX once `solid-ui` changes trigger frontend/dev-server churn.