@solidxai/core 0.1.11-beta.11 → 0.1.11-beta.12

package/docs/datasource-introspection-implementation-plan.md

@@ -0,0 +1,306 @@
+# Datasource Introspection Implementation Plan
+
+## Current Status
+- [x] Review the handwritten UX note for datasource introspection.
+- [x] Inspect how SolidX currently represents models in metadata JSON.
+- [x] Inspect the three superclass patterns used by generated entities.
+- [x] Inspect migration placement conventions via `solidctl`.
+- [x] Compare DDL SQL files against generated entities and metadata JSON.
+  - Detailed findings are captured in [datasource-introspection-ddl-analysis.md](/Users/harishpatel/Code/javascript/solid-core-module/docs/datasource-introspection-ddl-analysis.md).
+  - The only blocked subset is the missing `MOB3PO_*` amextxn DDL coverage.
+
+## Scope We Are Planning For
+- [x] Given a datasource and a target module, inspect legacy database tables and generate SolidX model metadata JSON.
+- [x] Keep per-session mapping options in frontend React state and send them to dedicated backend endpoints.
+- [x] Treat the generated metadata JSON as the primary output of introspection.
+- [x] Leave the rest of the code asset generation to existing SolidX flows such as `refresh model`.
+- [x] Force introspection-created legacy mappings onto `synchronize = false`.
+- [x] Add a migration generation path that is template-based rather than TypeORM diff-based.
+
+## Findings From The Current Analysis
+
+### 1. `legacyTableType` is already the right switch
+- [x] `ModelMetadata` already persists `legacyTableType` and supports `none`, `existing_id`, and `generated_id`.
+- [x] `CreateModelMetadataDto` already accepts `legacyTableType`.
+- [x] `ModelMetadataHelperService.getSystemFieldsMetadata()` already derives system fields from `legacyTableType`.
+- [x] This means the new introspection flow should emit the correct `legacyTableType` instead of inventing a parallel concept.
+
+### 2. The three superclass rules are now clear
+- [x] `CommonEntity` is the Solid-native case.
+  - Use when the table follows standard SolidX system columns like `id`, `created_at`, `updated_at`, `deleted_at`, `published_at`, `created_by_id`, and `updated_by_id`.
+  - Metadata should use `legacyTableType = none`.
+- [x] `LegacyCommonEntityWithExistingId` is the legacy-table-with-existing-key case.
+  - Use when the legacy table already has a stable key strategy and SolidX should not add a generated `ss_id`.
+  - SolidX system fields are expected with the `ss_` prefix.
+  - Metadata should use `legacyTableType = existing_id`.
+- [x] `LegacyCommonEntityWithGeneratedId` is the legacy-table-needing-surrogate-id case.
+  - Use when the legacy table does not have the SolidX-style surrogate key we want for framework operations.
+  - SolidX adds generated `ss_id` plus the other `ss_` system fields.
+  - Metadata should use `legacyTableType = generated_id`.
+
+### 3. System fields are not guessed, they are already encoded
+- [x] For `none`, SolidX expects system fields such as `id`, `createdAt`, `updatedAt`, `deletedAt`, `deletedTracker`, `publishedAt`, `localeName`, `defaultEntityLocaleId`, `createdBy`, and `updatedBy`.
+- [x] For legacy tables, the same logical system fields are mapped to `ss_*` column names.
+- [x] For `generated_id`, `id` is included and maps to `ss_id`.
+- [x] For `existing_id`, the generated `id` system field is omitted.
+- [x] The introspection implementation should reuse these existing rules instead of hardcoding them a second time.
+
+### 4. Entity-vs-metadata audit shows existing inconsistencies
+- [x] I compared entity superclass usage against model metadata entries in `mswipe-erp-solidx`.
+- [x] Sample audit result:
+  - Total models checked: `114`
+  - Matches between entity superclass and metadata `legacyTableType`: `81`
+  - Mismatches: `33`
+- [x] The mismatches are mostly cases where the entity clearly extends a legacy superclass but metadata still falls back to implicit `none`.
+- [x] Example mismatches observed:
+  - `peoples/employee` extends `LegacyCommonEntityWithExistingId`, but metadata does not stamp `existing_id`.
+  - `mswipe-masters/itemMaster` extends `LegacyCommonEntityWithGeneratedId`, but metadata does not stamp `generated_id`.
+  - `frms/frmsLead` extends `LegacyCommonEntityWithGeneratedId`, but metadata does not stamp `generated_id`.
+- [x] Conclusion:
+  - Existing metadata JSON is directionally useful, but not always fully normalized.
+  - The introspection feature should treat superclass/legacy strategy rules as explicit logic, not as something to infer from historical metadata files alone.
+
+### 5. What current metadata JSON tells us about the target output
+- [x] A model metadata entry minimally needs:
+  - `singularName`
+  - `pluralName`
+  - `displayName`
+  - `description`
+  - `dataSource`
+  - `dataSourceType`
+  - `tableName`
+  - `enableAuditTracking`
+  - `enableSoftDelete`
+  - `draftPublishWorkflow`
+  - `internationalisation`
+  - `isChild`
+  - `fields`
+  - optional `userKeyFieldUserKey`
+  - optional `legacyTableType`
+- [x] A field metadata entry minimally needs:
+  - `name`
+  - `displayName`
+  - `type`
+  - `ormType`
+  - `required`
+  - `unique`
+  - `index`
+  - `private`
+  - `columnName`
+  - optional `max`
+  - optional `defaultValue`
+  - optional `isPrimaryKey`
+  - optional `isUserKey`
+- [x] This aligns well with a backend introspection service that produces normalized DTO-shaped JSON rather than source files directly.
+
+### 6. DDL-to-JSON comparison status
+- [x] The DDL-vs-entity-vs-metadata comparison has now been completed for the available exports.
+- [ ] We still need to validate the missing `MOB3PO_*` amextxn tables once their DDL is available.
+- [x] The completed comparison covered:
+  - column name preservation
+  - nullable to `required` mapping
+  - key/index mapping
+  - `varchar` length mapping
+  - numeric precision/scale mapping
+  - date/time type mapping
+  - defaults
+  - foreign-key relation opportunities
+  - whether a table should be `none`, `existing_id`, or `generated_id`
+
+## Proposed Backend Design
+
+### Controller Layer
+- [ ] Add a dedicated controller for datasource introspection endpoints inside `solid-core-module`.
+- [ ] Keep it separate from generic CRUD metadata endpoints because this workflow is file-backed, session-driven, and safety-sensitive.
+
+### Service Layer
+- [ ] Add an orchestration service, for example `DatasourceIntrospectionService`, responsible for:
+  - loading datasource configuration
+  - validating `synchronize = false`
+  - selecting the correct database-engine implementation
+  - normalizing introspection results into SolidX model metadata DTOs
+  - writing metadata JSON updates
+  - optionally generating template migrations
+
+### Engine Interface
+- [ ] Define a shared interface for engine-specific implementations, for example:
+  - `listTables()`
+  - `describeTable(tableName)`
+  - `listColumns(tableName)`
+  - `listPrimaryKeys(tableName)`
+  - `listIndexes(tableName)`
+  - `listForeignKeys(tableName)`
+  - `inferLegacyTableType(tableDescriptor, options)`
+- [ ] Implement providers for:
+  - `MssqlDatasourceIntrospectionProvider`
+  - `MysqlDatasourceIntrospectionProvider`
+  - `PostgresDatasourceIntrospectionProvider`
+- [ ] Keep database-specific SQL isolated to these providers.
+
+### Metadata Mapping Layer
+- [ ] Add a mapper layer that converts engine-neutral table/column descriptors into `CreateModelMetadataDto`-shaped objects.
+- [ ] This mapper should own:
+  - table-name to model-name conversion
+  - column-name to field-name conversion
+  - `ormType` mapping
+  - Solid field type mapping
+  - `required`, `unique`, `index`, `defaultValue`, and `max` derivation
+  - `userKeyFieldUserKey` selection
+  - `legacyTableType` resolution
+
+### File Writer Layer
+- [ ] Add a file writer that updates the module metadata JSON file on disk.
+- [ ] This should support:
+  - append new model metadata into an existing module JSON
+  - optionally create a new module JSON if the user chose “new module”
+  - stable formatting and ordering so diffs stay readable
+
+### Migration Template Layer
+- [ ] Add a template-based migration generator for introspected models.
+- [ ] This generator should not rely on TypeORM schema diffing.
+- [ ] It should instead emit migrations that add only the SolidX superclass columns implied by:
+  - `legacyTableType = existing_id`
+  - `legacyTableType = generated_id`
+  - possibly `legacyTableType = none` if we ever support generating Solid-native tables from this flow
+- [ ] Generated files must land under the same folder convention used by `solidctl`:
+  - `solid-api/src/<module>/migrations/<datasource>/`
+
+## Proposed Frontend Design
+
+### Primary UX Direction
+- [ ] Add datasource introspection as a focused workflow rather than a generic list view.
+- [ ] Entry point should sit close to module creation/editing because the user is mapping tables into a module.
+- [ ] Follow the note’s flow:
+  - choose new module or existing module
+  - choose datasource
+  - list tables in datasource
+  - indicate whether each table is already mapped
+  - if mapped, route into the model form view
+  - if not mapped, open a mapping UI for columns and settings
+
+### Session Settings
+- [ ] Keep mapping-session options in React state and send them with each introspection request.
+- [ ] Likely session options:
+  - target module
+  - datasource
+  - naming conventions override
+  - audit tracking default
+  - default user key strategy
+  - relation inference toggle
+  - include/exclude tables
+  - preferred `legacyTableType` override
+  - migration generation toggle
+
+### Sync Safety Guard
+- [ ] Before allowing table mapping, fetch datasource details and validate that synchronize is disabled.
+- [ ] If synchronize is `true`, block the flow with a clear validation message.
+- [ ] This validation should be enforced in both UI and backend.
+
+## Legacy Table Strategy Rules For Introspection
+
+### `legacyTableType = none`
+- [ ] Use only when the datasource/table is intended to behave like a Solid-native table.
+- [ ] For the legacy-database mapping flow, this should probably be rare.
+
+### `legacyTableType = existing_id`
+- [ ] Use when the table already has a stable business or primary key that SolidX can rely on.
+- [ ] Typical indicators:
+  - a meaningful single-column key already exists
+  - the table already has a durable row identity that should remain authoritative
+  - we do not want to add `ss_id`
+
+### `legacyTableType = generated_id`
+- [ ] Use when the table does not offer a suitable SolidX-compatible surrogate key.
+- [ ] Typical indicators:
+  - composite or awkward primary keys
+  - no stable numeric surrogate key
+  - we still need a generated SolidX framework identity via `ss_id`
+
+### Important Note
+- [ ] The final decision logic should be explicit and reviewable.
+- [ ] Do not quietly infer superclass choice from incomplete historical metadata JSON.
+
+## Type Mapping Rules To Confirm Once DDL Arrives
+- [ ] MSSQL:
+  - `varchar` / `nvarchar` -> `shortText` or `longText` depending on length
+  - `text` -> `longText`
+  - `bit` -> `boolean`
+  - `int` / `bigint` / `numeric` / `decimal` -> numeric Solid field types
+  - `date` / `datetime` / `datetime2` / `time` -> date/time Solid field types
+- [ ] MySQL:
+  - confirm handling of `tinyint(1)` for boolean-like columns
+  - confirm `text` family mapping thresholds
+- [ ] PostgreSQL:
+  - confirm `text`, `json`, `jsonb`, `timestamp`, `timestamptz`, `uuid`
+- [ ] Cross-engine:
+  - precision/scale preservation in `ormType`-driven fields
+  - default value normalization
+  - nullability to `required`
+
+## Proposed API Shape
+
+### Discovery Endpoints
+- [ ] `GET /datasource-introspection/datasources`
+- [ ] `GET /datasource-introspection/datasources/:name/tables`
+- [ ] `GET /datasource-introspection/datasources/:name/tables/:tableName`
+
+### Mapping Endpoints
+- [ ] `POST /datasource-introspection/preview-model`
+  - returns the normalized model metadata JSON payload before writing
+- [ ] `POST /datasource-introspection/apply-model`
+  - writes model metadata JSON into the chosen module
+- [ ] `POST /datasource-introspection/generate-migration`
+  - creates the template migration for superclass/system columns
+
+### Safety / Validation
+- [ ] Every write endpoint should verify:
+  - datasource exists
+  - module exists or can be created
+  - `synchronize` is `false`
+  - target table is not already mapped unless overwrite is explicitly requested
+
+## Proposed Execution Phases
+
+### Phase 1: Audit And Foundations
+- [ ] Finalize DDL-vs-JSON comparison once SQL files are added.
+- [ ] Write down exact superclass decision rules with examples.
+- [ ] Decide the canonical naming-conversion rules for model and field names.
+- [ ] Decide how to pick `userKeyFieldUserKey`.
+
+### Phase 2: Backend Read Path
+- [ ] Implement datasource introspection controller and orchestration service.
+- [ ] Implement engine interface and MSSQL provider first.
+- [ ] Add table and column discovery endpoints.
+- [ ] Add normalize-to-metadata preview endpoint.
+
+### Phase 3: Frontend Flow
+- [ ] Add the new introspection entry point in the module workflow.
+- [ ] Build datasource selector and table browser.
+- [ ] Show mapped vs unmapped tables.
+- [ ] Add preview/mapping canvas for a selected table.
+- [ ] Add synchronize guardrail messaging.
+
+### Phase 4: Write Path
+- [ ] Persist generated model metadata JSON into the module metadata file.
+- [ ] Support “add to existing module”.
+- [ ] Support “create new module”.
+- [ ] Prevent duplicate model/table mappings.
+
+### Phase 5: Migration Templates
+- [ ] Generate migration stubs for superclass/system columns.
+- [ ] Place migrations under the datasource-specific module folder.
+- [ ] Ensure generation works cleanly with the existing `solidctl migration` conventions.
+
+### Phase 6: Validation And Hardening
+- [ ] Validate generated metadata by running the downstream SolidX refresh flow.
+- [ ] Test against at least one MSSQL datasource first.
+- [ ] Then add MySQL and PostgreSQL coverage.
+- [ ] Add regression checks for relation inference, key detection, and field-type mapping.
+
+## Immediate Next Steps
+- [ ] Wait for the DDL SQL files to be added under `mswipe-erp-solidx`.
+- [ ] Once ready, perform a table-by-table comparison between:
+  - DDL SQL
+  - generated TypeORM entity
+  - module metadata JSON
+- [ ] Use that comparison to lock the exact mapping rules before implementation begins.

package/docs/grouping-enhancements.md

@@ -0,0 +1,89 @@
+# Grouping & Aggregation Enhancements (Code Review Summary)
+
+This document explains the recent changes to grouping/aggregation in the CRUD helper/service, why they were made, and how to use them (with examples on `PincodeMaster`).
+
+## What Changed and Why
+- **Dedicated grouping pipeline**: Grouping no longer reuses the record-level query (which had `SELECT entity.*`, pagination, and default order). A separate path builds clean group queries to avoid SQL errors and ensure correct counts.
+- **Multiple group-by fields**: The one-field limit was removed. You can now group on multiple fields (including relations) in the requested order.
+- **Relation-safe grouping**: Group-by fields can traverse many-to-one relations (e.g., `state.name`, `city.name`). The helper reuses existing joins (from filters) or adds the necessary joins and aliases.
+- **Date granularity**: Grouping supports `day`, `week`, `month`, `year` granularities in a DB-aware way (Postgres, MySQL/MariaDB, SQL Server).
+- **Aggregates**: Supports a core, DB-agnostic set: `count`, `count_distinct`, `sum`, `avg`, `min`, `max`. If `aggregates` is omitted, it defaults to `count(*)`.
+- **Group sorting/pagination**: Sorting and pagination are applied to group rows (not entity rows). Record pagination is kept separate for non-grouped queries.
+- **Ordered group names**: Group names follow the order of `groupBy` fields. Relation values and date/grouped values are included in sequence.
+- **Formatted date group labels**: You can add an optional format specifier to a date groupBy field: `field:granularity:format`. Supported formats: `MMM`, `MMMM`, `YYYY`, `YYYY-MM`, `YYYY-MM-DD` (defaults to the raw value if omitted).
+- **Count of groups**: Group counts are computed without pagination interference, so `meta.totalRecords` reflects total groups.
+- **DTO update**: `BasicFilterDto` now includes optional `aggregates?: string[]`.
+
+## Caveats
+- `populateGroup` is **not** supported when grouping on relation fields (e.g., `state.name`, `city.name`). Use it for scalar group-by fields only; otherwise fetch group metadata and then retrieve records in a separate call with the group key.
+- Sorting works for group keys without extra colons (e.g., `state.name`) and for aggregate aliases (e.g., `id_max`). For date bucket group keys with granularity/format (`createdAt:month:YYYY`), the sort parser treats the last segment as the order; results may vary by driver and may not sort as expected in all cases.
+
+## Usage Examples (PincodeMaster)
+Assume `PincodeMaster` has many-to-one `state` and `city` relations and a `createdAt` timestamp.
+
+### 1) Group by State and City
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&groupBy[1]=city.name
+```
+Returns groups for every state/city combination with default `count(*)` aggregate.
+
+### 2) Group by Relation + Filter on Relation
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&filters[state][name][$eq]=Maharashtra
+```
+Groups by state name but only for rows where state is Maharashtra.
+
+### 3) Multiple Group Fields (Relation + Scalar)
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&groupBy[1]=city.name&groupBy[2]=pincode
+```
+Group names are ordered: state → city → pincode.
+
+### 4) Group by Date with Granularity (Month)
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=createdAt:month
+```
+Groups by month (driver-aware), default group labels are raw date buckets.
+
+### 5) Date Granularity with Formatting
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=createdAt:month:MMM
+```
+Group labels use short month names (Jan, Feb, …). For full names: `createdAt:month:MMMM`. Other formats: `YYYY`, `YYYY-MM`, `YYYY-MM-DD`.
+
+### 6) Aggregates (Count Distinct)
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&aggregates[0]=pincode:count_distinct
+```
+Shows distinct pincodes per state.
+
+### 7) Aggregates (Multiple)
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&groupBy[1]=city.name&aggregates[0]=id:count&aggregates[1]=id:count_distinct
+```
+Returns both total rows and distinct IDs per state/city group.
+
+### 8) Date Granularity + Relations + Aggregates
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=createdAt:year&groupBy[1]=state.name&aggregates[0]=pincode:count_distinct
+```
+Distinct pincodes per state, per year.
+
+### 9) Filters with Grouping (Many-to-One)
+```
+GET /api/pincode-master?offset=0&limit=200&groupBy[0]=state.name&groupBy[1]=city.name&filters[state][name][$eq]=Nagaland
+```
+Groups only rows where state.name = Nagaland.
+
+### 10) Group Sorting and Pagination
+- Sorting applies to the group rows. Example:
+```
+GET /api/pincode-master?offset=0&limit=50&groupBy[0]=state.name&sort[0]=state.name:ASC
+```
+- Pagination (`offset/limit`) limits the number of groups returned; total group count remains in `meta.totalRecords`.
+
+## Notes and Behavior
+- If `aggregates` is omitted, `COUNT(*)` is added automatically.
+- Group names reflect `groupBy` order and apply formatting when specified.
+- Group queries reuse joins from filters when possible; otherwise, they create necessary joins for relation paths.
+- Non-grouped find behavior remains unchanged.

package/docs/java-spring/README.md

@@ -0,0 +1,3 @@
+# Java Spring Docs
+
+- [solid-core-module Deep Dive Report](./solid-core-module-deep-dive-report.md)