@meshery/schemas 0.8.106 → 0.8.108

package/README.md

@@ -124,10 +124,35 @@ schemas/
   constructs/
     <schema-version>/               # e.g., v1beta1
       <construct>/                  # e.g., model, component
-        <construct>.json            # Schema definition for the construct (noun)
-        openapi.yml                 # API operations (verbs: create, update, delete)
-        <construct>_template.json   # Generated JSON template from schema
-        <construct>_template.yaml   # Generated YAML template from schema
+        api.yml                     # Index file: references all subschemas + defines API endpoints
+        <construct>.yaml            # Subschema: data model definition for the construct
+        <other_subschema>.yaml      # Additional subschemas (optional)
+        templates/                  # Manually defined templates directory
+          <construct>_template.json # JSON template from schema
+          <construct>_template.yaml # YAML template from schema
+          <variant>_template.json   # Additional variant templates (optional)
+  
+  typescript/                       # TypeScript source and generated files
+    index.ts                        # Manually maintained - public API surface
+    generated/                      # Auto-generated (do NOT commit)
+      <schema-version>/
+        <construct>/
+          <Construct>.d.ts          # TypeScript type definitions
+          <Construct>Schema.ts      # OpenAPI schema as JS object
+    rtk/                            # RTK Query client configurations
+      cloud.ts
+      meshery.ts
+  
+  dist/                             # Built distribution (do NOT commit)
+    index.js, index.d.ts
+    cloudApi.js, mesheryApi.js
+    constructs/                     # Built schema exports (renamed from 'generated')
+      <schema-version>/<construct>/<Construct>Schema.js
+  
+  models/                           # Auto-generated Go code (do NOT commit)
+    <schema-version>/
+      <construct>/
+        <construct>.go
 ```
 
 ### 🧠 Explanation
@@ -135,9 +160,203 @@ schemas/
 * **`constructs/`** – Holds schemas for various versions.
   * **`<schema-version>/`** – Represents a version (e.g., `v1alpha2`, `v1beta1`).
     * **`<construct>/`** – A directory to contain all files for any given construct like `pattern`, `component`, etc.
-      * **`<construct>.json`** – Defines the **data model (noun)** for the construct.
-      * **`openapi.yml`** – Contains a package manifest of all schema files with respect to any given construct (used by openapi generator) AND defines **API operations** (verbs).
-      * **Templates** – `*_template.json` and `*_template.yaml` are auto-generated examples with resolved references and defaults.
+      * **`api.yml`** – The **index file** for the construct. This file:
+        1. References all subschemas (via `$ref`) to bundle them together
+        2. Defines all **API endpoints** (REST operations: GET, POST, PUT, DELETE) for the construct
+        3. Serves as the entry point for code generation tools (oapi-codegen, openapi-typescript)
+      * **`<construct>.yaml`** – A **subschema** that defines the data model (noun) for the construct. Contains the schema properties, types, and validation rules.
+      * **Other `.yaml` files** – Additional subschemas can be defined in separate files (e.g., `model_core.yml`, `component_metadata.yml`) and referenced from `api.yml`.
+      * **`templates/`** – A subdirectory containing manually defined template files. You can add as many different templates here for different variants, use cases, or configurations. Templates are example instances of the schema with default or sample values.
+        * `<construct>_template.json` / `<construct>_template.yaml` – Default templates in JSON/YAML format.
+        * Additional variant templates can be added (e.g., `<construct>_minimal_template.json`, `<construct>_full_template.yaml`) for different use cases.
+
+---
+
+## Naming Conventions
+
+- Property names
+  - Use camelCase for property fields (e.g., `schemaVersion`, `displayName`, `componentsCount`).
+  - Identifier fields use lowerCamelCase with "Id" suffix (e.g., `modelId`, `registrantId`, `categoryId`).
+  - Enums use lowercase words (e.g., `enabled`, `ignored`, `duplicate`).
+  - Use singular nouns for object names (e.g., `model`, `component`, `design`).
+
+- OpenAPI schema names
+  - PascalCase nouns under `components/schemas` (e.g., `Model`, `Component`).
+  - Files/folders are lowercase: `api.yml` (index), `<construct>.yaml` (subschemas), `templates/<construct>_template.(json|yaml)`.
+
+- Endpoints and operations
+  - Paths are under `/api` with kebab-case , plural nouns (e.g., `/api/workspaces`, `/api/environments`).
+  - Path params are camelCase  (e.g., `{subscriptionId}`, `{connectionId}`).
+  - Non-CRUD actions append a verb segment (e.g., `.../register`, `.../export`, `.../cancel`); legacy lowerCamelCase may appear (e.g., `.../upgradePreview`).
+  - `operationId` is camelCase VerbNoun (e.g., `registerMeshmodels`).
+
+- Versioning
+  - `schemaVersion` uses group/version (e.g., `models.meshery.io/v1beta1`, `components.meshery.io/v1beta1`).
+  - Version strings follow k8s-style (`v1`, `v1alpha1`, `v1beta1`); semver fields use standard SemVer.
+
+---
+
+## 🔧 Go Helper Files
+
+While Go structs are auto-generated from schemas, you often need to add **custom methods** to make these structs compatible with databases, implement interfaces, or add utility functions. This is done through manually created **helper files**.
+
+### When to Create Helper Files
+
+Create a helper file (`*_helper.go` or `helpers.go`) in the generated package when you need:
+
+1. **SQL Driver Compatibility** - Implement `database/sql/driver.Scanner` and `driver.Valuer` interfaces
+2. **Entity Interface Implementation** - Implement the `entity.Entity` interface for database operations
+3. **GORM Table Names** - Define custom table names via `TableName()` method
+4. **Utility Methods** - Add helper functions for serialization, validation, or business logic
+5. **Type Conversions** - Add methods to convert between related types
+
+### Helper File Location
+
+```
+models/
+├── core/
+│   ├── core.go                    # Auto-generated (do NOT edit)
+│   ├── helpers.go                 # Manual: utility functions
+│   ├── datatype_map.go            # Manual: Map type with SQL driver methods
+│   └── datatype_null_time.go      # Manual: NullTime with SQL driver methods
+├── v1beta1/
+│   ├── model/
+│   │   ├── model.go               # Auto-generated (do NOT edit)
+│   │   └── model_helper.go        # Manual: Entity interface, TableName, etc.
+│   ├── component/
+│   │   ├── component.go           # Auto-generated (do NOT edit)
+│   │   └── component_helper.go    # Manual: Entity interface, TableName, etc.
+│   └── category/
+│       ├── category.go            # Auto-generated (do NOT edit)
+│       └── category_helper.go     # Manual: Entity interface, TableName, etc.
+```
+
+### SQL Driver Interface Implementation
+
+To store complex types in SQL databases, implement `Scan` and `Value` methods:
+
+```go
+// helpers.go - This is NOT autogenerated
+package mypackage
+
+import (
+    "database/sql/driver"
+    "encoding/json"
+    
+    "github.com/meshery/schemas/models/core"
+)
+
+// Scan implements sql.Scanner interface for reading from database
+func (m *MyComplexType) Scan(value interface{}) error {
+    mapVal := core.Map{}
+    err := mapVal.Scan(value)
+    if err != nil {
+        return err
+    }
+    return core.MapToStruct(mapVal, m)
+}
+
+// Value implements driver.Valuer interface for writing to database
+func (m MyComplexType) Value() (driver.Value, error) {
+    mapVal, err := core.StructToMap(m)
+    if err != nil {
+        return nil, err
+    }
+    return core.Map(mapVal).Value()
+}
+```
+
+### Entity Interface Implementation
+
+For structs that need database CRUD operations, implement the `entity.Entity` interface:
+
+```go
+// component_helper.go - This is NOT autogenerated
+package component
+
+import (
+    "fmt"
+    
+    "github.com/gofrs/uuid"
+    "github.com/meshery/meshkit/database"
+    "github.com/meshery/meshkit/models/meshmodel/entity"
+    "gorm.io/gorm/clause"
+)
+
+// TableName returns the database table name for GORM
+func (c ComponentDefinition) TableName() string {
+    return "component_definition_dbs"
+}
+
+// Type returns the entity type identifier
+func (c ComponentDefinition) Type() entity.EntityType {
+    return entity.ComponentDefinition
+}
+
+// GenerateID generates a new UUID for the entity
+func (c *ComponentDefinition) GenerateID() (uuid.UUID, error) {
+    return uuid.NewV4()
+}
+
+// GetID returns the entity's ID
+func (c ComponentDefinition) GetID() uuid.UUID {
+    return c.Id
+}
+
+// GetEntityDetail returns a human-readable description
+func (c *ComponentDefinition) GetEntityDetail() string {
+    return fmt.Sprintf("type: %s, name: %s, model: %s", 
+        c.Type(), c.DisplayName, c.Model.Name)
+}
+
+// Create inserts the entity into the database
+func (c *ComponentDefinition) Create(db *database.Handler, hostID uuid.UUID) (uuid.UUID, error) {
+    c.Id, _ = c.GenerateID()
+    err := db.Omit(clause.Associations).Create(&c).Error
+    return c.Id, err
+}
+
+// UpdateStatus updates the entity's status in the database
+func (c *ComponentDefinition) UpdateStatus(db *database.Handler, status entity.EntityStatus) error {
+    return nil
+}
+```
+
+### Core Utility Types
+
+The `models/core/` package provides reusable types with built-in SQL compatibility:
+
+| Type | Purpose | Use Case |
+|------|---------|----------|
+| `core.Map` | `map[string]any` with SQL support | Storing JSON objects in database |
+| `core.NullTime` | Nullable time with JSON/YAML support | Optional timestamp fields |
+| `core.Time` | Time wrapper with custom formatting | Required timestamp fields |
+
+### Example: Using Core Types
+
+```go
+// In your helper file
+package mypackage
+
+import "github.com/meshery/schemas/models/core"
+
+// For nullable timestamps (e.g., deleted_at)
+type MyStruct struct {
+    DeletedAt core.NullTime `json:"deleted_at" gorm:"column:deleted_at"`
+}
+
+// For JSON metadata stored as blob
+type MyStruct struct {
+    Metadata core.Map `json:"metadata" gorm:"type:bytes;serializer:json"`
+}
+```
+
+### Important Notes
+
+1. **File Header Comment**: Always add `// This is not autogenerated.` at the top of helper files
+2. **Same Package**: Helper files must be in the same package as the generated code
+3. **Do Commit Helper Files**: Unlike generated `.go` files, helper files ARE committed to the repository
+4. **Mutex for Creation**: Use sync.Mutex when implementing `Create()` to prevent race conditions
 
 ---
 
@@ -145,11 +364,41 @@ schemas/
 
 Meshery supports **automated code generation** from schemas for:
 
-* **Go**: Strongly-typed models for backend.
-* **TypeScript**: Interfaces and types for frontend use.
-* **RTK Query**: Clients generated from OpenAPI for use with Redux.
+* **Go**: Strongly-typed models for backend → `models/<version>/<package>/`
+* **TypeScript Types**: Interfaces and type definitions → `typescript/generated/<version>/<package>/<Package>.d.ts`
+* **TypeScript Schemas**: OpenAPI schemas as const JS objects → `typescript/generated/<version>/<package>/<Package>Schema.ts`
+* **RTK Query**: Clients generated from OpenAPI for use with Redux → `typescript/rtk/`
 * **JSON/YAML**: Templates with defaults and resolved references.
 
+### TypeScript Schema Exports
+
+Each construct's OpenAPI schema is exported as a const JavaScript object for runtime use:
+
+```typescript
+// Import from main index
+import {
+  ModelDefinitionV1Beta1OpenApiSchema,
+  ComponentDefinitionV1Beta1OpenApiSchema,
+  DesignDefinitionV1Beta1OpenApiSchema,
+} from "@meshery/schemas";
+
+// Or import individual schemas directly
+import ModelSchema from "@meshery/schemas/dist/constructs/v1beta1/model/ModelSchema";
+import ComponentSchema from "@meshery/schemas/dist/constructs/v1beta1/component/ComponentSchema";
+```
+
+### TypeScript Type Namespaces
+
+Types are organized by version in namespaces:
+
+```typescript
+import { v1beta1, v1alpha1 } from "@meshery/schemas";
+
+const component: v1beta1.Component = { /* ... */ };
+const model: v1beta1.Model = { /* ... */ };
+const design: v1beta1.Design = { /* ... */ };
+```
+
 ---
 
 ## 🚀 Unified Build: One Command for Everything
@@ -158,6 +407,7 @@ Use the following command to perform the **entire schema-driven generation workf
 
 ```bash
 make build
+npm run build  # Build TypeScript distribution with tsup
 ```
 
 ### 🔧 What `make build` does:
@@ -170,10 +420,16 @@ make build
    
 2. **Generates:**
 
-   * Golang structs
-   * TypeScript types
-   * JSON & YAML templates
-   * RTK Query clients
+   * Golang structs → `models/`
+   * TypeScript type definitions (`.d.ts`) → `typescript/generated/`
+   * TypeScript schema exports (`*Schema.ts`) → `typescript/generated/`
+   * RTK Query clients → `typescript/rtk/`
+
+3. **After `npm run build`:**
+
+   * Builds distribution files → `dist/`
+   * Creates CJS and ESM bundles
+   * Generates declaration files
 
 > ⚠️ This is the recommended way to stay in sync with schema changes.
 
@@ -233,7 +489,7 @@ generate_schema_models <construct> <schema-version> [<openapi-file>]
 
 generate_schema_models "capability" "v1alpha1"
 generate_schema_models "category" "v1beta1"
-generate_schema_models "pattern" "v1beta1" "schemas/constructs/v1beta1/design/openapi.yml"
+generate_schema_models "pattern" "v1beta1" "schemas/constructs/v1beta1/design/api.yml"
 ```
 
 This maps to Go packages like:
@@ -255,7 +511,7 @@ npx @redocly/cli join schemas/base_cloud.yml \
      "${v1beta1}/component/${merged_construct}" \
      "${v1beta1}/model/${merged_construct}" \
      ... \
-     -o schemas/merged_openapi.yml \
+     -o _openapi_build/merged_openapi.yml \
      --prefix-tags-with-info-prop title \
      --prefix-components-with-info-prop title
 ```
@@ -444,19 +700,34 @@ make build
 Or validate a single file:
 
 ```bash
-npx @redocly/cli lint schemas/constructs/v1beta1/pattern/openapi.yml
+npx @redocly/cli lint schemas/constructs/v1beta1/pattern/api.yml
 ```
 
 ---
 
 ## ✅ Summary
 
-| Task                    | Command                 |
-| ----------------------- | ----------------------- |
-| Generate everything     | `make build`            |
-| Generate Go code only   | `make golang-generate`  |
-| Generate TS + templates | `make generate-types`   |
-| Lint OpenAPI            | `npx @redocly/cli lint` |
+| Task                         | Command                 |
+| ---------------------------- | ----------------------- |
+| Generate everything          | `make build`            |
+| Build TypeScript dist        | `npm run build`         |
+| Generate Go code only        | `make golang-generate`  |
+| Generate TS types + schemas  | `make generate-ts`      |
+| Lint OpenAPI                 | `npx @redocly/cli lint` |
+
+### Importing Schemas
+
+```typescript
+// Via namespaces (types)
+import { v1beta1 } from "@meshery/schemas";
+const model: v1beta1.Model = { /* ... */ };
+
+// Via schema exports (runtime)
+import { ModelDefinitionV1Beta1OpenApiSchema } from "@meshery/schemas";
+
+// Direct schema import
+import ModelSchema from "@meshery/schemas/dist/constructs/v1beta1/model/ModelSchema";
+```
 
 ---