@hamak/smart-data-dico 1.0.0

package/README.md

@@ -0,0 +1,292 @@
+# Smart Data Dictionary
+
+A collaborative data dictionary management system for modeling, documenting, and governing your organization's data landscape. Built on a microkernel plugin architecture for modularity, with file-based persistence (YAML/Git) designed for both technical and non-technical users.
+
+## Core Concepts
+
+### Packages
+
+Packages organize entities into logical groups reflecting your system architecture (microservices, modules, bounded contexts). They are hierarchical -- packages can contain sub-packages, forming a tree that mirrors your application structure.
+
+- Navigable via nested URLs: `/packages/eShop/ordering/entities/Order`
+- Each package has a dashboard with entity counts, relationship counts, and metadata
+- Package names follow kebab-case convention
+
+### Entities & Attributes
+
+Entities are the primary modeling unit -- they represent data structures (tables, documents, events, value objects). Each entity has:
+
+- **Attributes** with types (string, number, integer, boolean, datetime, enum, object, array), constraints (min/max length, pattern, precision), and metadata
+- **Nested attributes** for object and array types (e.g. `shippingAddress.postalCode`)
+- **Status lifecycle**: draft -> submitted -> approved (or returned)
+- **UUID-based identity** for stable cross-references
+
+### Relationships
+
+Relationships connect entities across packages. They are stored at the package level and define:
+
+- **Source and target** entity (by UUID) with cardinality (one/many)
+- **Navigation property names** for path traversal
+- **Metadata** on the relationship itself
+
+### Stereotypes
+
+Stereotypes define metadata schemas that apply to specific element types (entity, attribute, or package). When assigned, they auto-populate required and optional metadata fields, ensuring consistency.
+
+**Predefined stereotypes:**
+
+| Stereotype | Applies To | Required Fields |
+|-----------|-----------|-----------------|
+| Aggregate Root | Entity | bounded-context |
+| Reference Data | Entity | cacheable (flag) |
+| Domain Event | Entity | event-source |
+| Value Object | Entity | immutable (flag) |
+| PII | Attribute | pii-category |
+| Indexed | Attribute | - |
+| Deprecated | Attribute | deprecated-since |
+
+**Metadata types**: string, number, boolean, date, **flag** (semantic boolean rendered as toggle), **rule** (text + severity: info/warning/error).
+
+### Perspectives
+
+A Perspective captures a business view over a subset of the data model scoped to a business process (e.g. "Ordering", "Billing"). It is the core differentiating concept of the application.
+
+**How it works:**
+
+1. Select **root entities** -- the starting points of the business process
+2. The system performs **BFS traversal** through relationships to discover all reachable entities
+3. Each entity is identified by its **relationship path from root** (e.g. `Order/orderItems/product`)
+4. The same entity reached via different paths is treated as separate usages
+
+**Path grammar** -- uniform `/` navigation for all levels:
+```
+Order                              -- root entity
+Order/totalAmount                  -- attribute on root
+Order/orderItems                   -- relationship -> OrderItem
+Order/orderItems/quantity          -- attribute on OrderItem
+Order/orderItems/product/sku       -- deep chain -> attribute
+Order/shippingAddress/postalCode   -- attribute via relationship
+Order/billingAddress/postalCode    -- same attribute, different path
+```
+
+**PerspectiveNode** annotations on any path:
+- **`traverse: false`** -- frontier node (include but stop BFS here)
+- **`exclude: true`** -- prune this path entirely
+- **`metadata`** -- path-scoped annotations (not overrides -- new metadata specific to this perspective usage)
+
+### Review Workflow
+
+Entities go through a status lifecycle for team governance:
+
+```
+draft --> submitted --> approved
+                   \-> returned (with comments) --> submitted
+```
+
+- **Comments** can target specific attributes (e.g. "Missing validation rule on discountRate")
+- **Status transitions** are role-gated (editors submit, admins approve/return)
+- Comments are stored as sidecar YAML files per entity
+
+### Quality Metrics
+
+The quality dashboard tracks documentation completeness:
+
+- **Description coverage** -- % of entities/attributes with descriptions
+- **Metadata compliance** -- % of entities with correct stereotype metadata
+- **Relationship coverage** -- % of entities with at least one relationship
+- **Per-entity scoring** -- weighted composite (30% description, 30% attribute descriptions, 20% relationships, 20% stereotype compliance)
+- **Drill-down** from package level to individual entity gaps
+
+## User Journeys
+
+### Journey 1: Model a New Domain (Data Architect)
+
+> *Alice documents the data model for an e-commerce platform.*
+
+1. **Create packages** -- `eShop > Ordering`, `Catalog`, `Billing` via the package dashboard
+2. **Define entities** -- `Order`, `OrderLine`, `Customer` with typed attributes and constraints
+3. **Define relationships** -- `Order` to `OrderLine` (one-to-many) at the package level
+4. **Assign stereotypes** -- Mark `Order` as `aggregate-root`, auto-populating `bounded-context` metadata
+5. **Add metadata** -- Flag `Customer.email` as PII, add business rules
+6. **Visualize** -- Open the Cytoscape.js interactive graph, verify the model, save the layout
+7. **Save & publish** -- Save (commit to workspace), publish (push to shared)
+
+### Journey 2: Define a Perspective (Business Analyst)
+
+> *Bob captures the billing-specific view of the data model.*
+
+1. **Create perspective** -- "Billing" with `Invoice` and `Payment` as root entities
+2. **Review resolved paths** -- BFS walks relationships, discovers `Order`, `Customer`, `LineItem`
+3. **Set frontiers** -- Mark `Product` as frontier (`traverse: false`) -- include but don't follow further
+4. **Exclude paths** -- Exclude `Order/audit` (not relevant to billing)
+5. **Add annotations** -- On `Order/billingAddress/postalCode`, add validation rule "Must match payment country"
+6. **Visualize** -- Graph overlay highlights perspective members, dims others
+
+### Journey 3: Review & Approve (Data Steward)
+
+> *Carol reviews changes submitted by the team.*
+
+1. **See pending entities** -- Order is in `submitted` status
+2. **Add comments** -- "Missing description on id field" targeting specific attribute
+3. **Return for revision** -- Status changes to `returned`, author gets notification
+4. **Approve** -- After fixes, approve. Status moves to `approved`
+5. **Check quality** -- Quality dashboard shows 90% for order-service, up from 80%
+
+### Journey 4: Explore & Discover (Developer)
+
+> *Dave needs to understand the data model for implementation.*
+
+1. **Browse** -- Navigate `/packages/order-service` to see the dashboard with stats
+2. **Search** -- Search "order" returns 27 results across entities, attributes, metadata, relationships, packages
+3. **Filter** -- Faceted search: filter by type=relationship, service=order-service, stereotype=aggregate-root
+4. **Impact analysis** -- Check "Impact" tab on Order: 2 relationships, 1 perspective reference
+5. **Export** -- Export order-service as JSON Schema for code generation, or Markdown for documentation
+
+### Journey 5: Version & Collaborate (Team Lead)
+
+> *Eve manages versioning aligned with application releases.*
+
+1. **Create workspace** -- Creates `workspace/billing-v2` (git branch with friendly name)
+2. **Monitor status** -- Navbar shows "MAIN 8" (branch + unsaved count), dropdown shows ahead/behind
+3. **Save & publish** -- Save page shows changed files, commit with message, push to remote
+4. **Switch workspaces** -- Workspaces page lists branches, switch with one click
+5. **Merge** -- Merge page selects source workspace, previews diff, merges
+
+## Architecture
+
+### Monorepo
+
+```
+smart-data-dico/
+  backend/              Express + TypeScript (controllers > services > models)
+  frontend/             React 18 + Vite + TypeScript + Tailwind CSS + DaisyUI
+  data-dictionaries/    YAML entity files, relationships, perspectives, stereotypes
+  Dockerfile            Multi-stage production build
+  docker-compose.yml    One-command deployment
+```
+
+### Microkernel Plugin Architecture
+
+The frontend uses `@hamak/app-framework` with 11 plugins:
+
+| Plugin | Responsibility |
+|--------|---------------|
+| **store** | Redux state management (10 domain slices) |
+| **shell** | Layout, theming (synced with DaisyUI) |
+| **auth** | JWT authentication with Auth0 |
+| **data-dictionary** | Entity, package, relationship CRUD |
+| **visualization** | Interactive Cytoscape.js graph |
+| **search** | Full-text and faceted search |
+| **version-control** | Save/publish, history |
+| **perspective** | Perspective management and resolution |
+| **remote-fs** | File operations via `@hamak/filesystem-server-impl` |
+| **remote-git** | Git operations via `@hamak/ui-remote-git-fs-impl` |
+| **notification** | Toast notifications |
+
+### Data Model
+
+```
+Package
+  +-- Entity (uuid, name, description, stereotype, status, attributes[], metadata[])
+  |     +-- Attribute (uuid, name, type, required, constraints, metadata[])
+  +-- Relationship (uuid, source, target, cardinality, metadata[])
+  +-- Sub-packages
+
+Stereotype (id, name, appliesTo, metadataDefinitions[])
+Perspective (uuid, name, rootEntities[], nodes[], maxDepth)
+  +-- PerspectiveNode (path, traverse?, exclude?, metadata[])
+  +-- ResolvedNode (entityUuid, path, hopDistance, isRoot, isFrontier)
+```
+
+### Persistence
+
+```
+data-dictionaries/
+  microservices/{package}/
+    {uuid}_{name}.yaml          -- entity files
+    {uuid}.comments.yaml        -- review comments (sidecar)
+    relationships.yaml          -- package-level relationships
+    metadata.yaml               -- package metadata
+  perspectives/{uuid}.yaml      -- perspective definitions
+  stereotypes.yaml              -- stereotype definitions
+  diagrams/{id}.json            -- saved diagram layouts
+```
+
+All files are git-tracked for versioning, branching, and collaboration.
+
+### API Surface
+
+| Area | Endpoints |
+|------|-----------|
+| Entities | CRUD on `/api/services/:service/entities/:entity` |
+| Packages | CRUD on `/api/packages`, hierarchy, path navigation |
+| Relationships | CRUD on `/api/packages/:pkg/relationships` |
+| Stereotypes | CRUD on `/api/stereotypes` |
+| Perspectives | CRUD + resolve + graph on `/api/perspectives` |
+| Search | `GET /api/search?q=...&type=...&service=...&stereotype=...` |
+| Impact | `GET /api/entities/:uuid/impact` |
+| Review | Submit/approve/return + comments on entities |
+| Import | `POST /api/import/json-schema`, `POST /api/import/sql-ddl` |
+| Export | `GET /api/export/json-schema/:service`, `GET /api/export/markdown/:service` |
+| Quality | `GET /api/quality/report` |
+| Git | Full git operations via `/api/git/dictionaries/*` (framework routes) |
+| Version | Commit, history, revert via `/api/commit`, `/api/history` |
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js v18+
+- npm
+
+### Quick Start
+
+```bash
+# Install dependencies
+cd backend && npm install
+cd ../frontend && npm install
+
+# Start (two terminals)
+cd backend && npm run dev     # port 3001
+cd frontend && npm run dev    # port 3000
+```
+
+- Frontend: http://localhost:3000
+- Backend API: http://localhost:3001
+- API docs: http://localhost:3001/api-docs
+
+### Docker
+
+```bash
+docker-compose up
+# App available at http://localhost:3001
+```
+
+### Dev Credentials
+
+| User | Password | Role |
+|------|----------|------|
+| admin | admin123 | ADMIN (full access) |
+| editor | editor123 | EDITOR (create/edit) |
+| viewer | viewer123 | VIEWER (read-only) |
+
+### Configuration Profiles
+
+Set via `PROFILE` environment variable:
+
+| Profile | Auth | Git | Use Case |
+|---------|------|-----|----------|
+| **local** | None | Local only | Single user, desktop |
+| **team** | Basic/JWT | Remote | Small team sharing a repo |
+| **server** | Auth0/SSO | Remote | Organization-wide deployment |
+
+## Technologies
+
+| Layer | Stack |
+|-------|-------|
+| Backend | Node.js, Express, TypeScript (ESM), YAML, Git |
+| Frontend | React 18, Vite, TypeScript, Tailwind CSS, DaisyUI, Redux |
+| Framework | @hamak/app-framework (microkernel, DI, plugins) |
+| Visualization | Cytoscape.js (dagre + fcose layouts) |
+| Auth | JWT + Auth0 (mock mode for dev) |
+| Deployment | Docker (multi-stage build) |

package/backend/package.json

@@ -0,0 +1,51 @@
+{
+  "name": "data-dictionary-backend",
+  "version": "0.1.0",
+  "description": "Backend for Data Dictionary Management System",
+  "type": "module",
+  "main": "dist/server.js",
+  "scripts": {
+    "start": "node dist/server.js",
+    "dev": "nodemon --exec tsx src/server.ts",
+    "build": "tsc",
+    "test": "jest",
+    "lint": "eslint . --ext .ts",
+    "test:coverage": "jest --coverage"
+  },
+  "dependencies": {
+    "@hamak/filesystem-server-api": "^0.5.1",
+    "@hamak/filesystem-server-impl": "^0.5.1",
+    "@hamak/filesystem-server-spi": "^0.5.1",
+    "@hamak/shared-utils": "^0.5.1",
+    "@hamak/ui-remote-git-fs-backend": "file:../../app-framework/packages/ui-remote-git-fs/backend/ui-remote-git-fs-backend",
+    "@types/jsonwebtoken": "^9.0.9",
+    "cors": "^2.8.5",
+    "dotenv": "^16.3.1",
+    "express": "^4.18.2",
+    "jsonschema": "^1.4.1",
+    "jsonwebtoken": "^9.0.2",
+    "swagger-jsdoc": "^6.2.8",
+    "swagger-ui-express": "^5.0.1",
+    "yaml": "^2.3.1"
+  },
+  "devDependencies": {
+    "@jest/globals": "^30.3.0",
+    "@types/cors": "^2.8.13",
+    "@types/express": "^4.17.17",
+    "@types/jest": "^29.5.3",
+    "@types/node": "^20.17.50",
+    "@types/supertest": "^6.0.3",
+    "@types/swagger-jsdoc": "^6.0.4",
+    "@types/swagger-ui-express": "^4.1.8",
+    "@typescript-eslint/eslint-plugin": "^6.0.0",
+    "@typescript-eslint/parser": "^6.0.0",
+    "eslint": "^8.45.0",
+    "jest": "^29.6.2",
+    "nodemon": "^3.0.1",
+    "supertest": "^7.1.1",
+    "ts-jest": "^29.1.1",
+    "ts-node": "^10.9.1",
+    "tsx": "^4.21.0",
+    "typescript": "^5.1.6"
+  }
+}

package/backend/src/__tests__/integration/api.test.ts

@@ -0,0 +1,149 @@
+import { NextFunction, Request, Response } from 'express';
+import request from 'supertest';
+
+import { UserRole } from '../../middleware/auth.js';
+import app from '../../server.js';
+
+// Extend Request type to include user property
+declare global {
+  namespace Express {
+    interface Request {
+      user?: {
+        id: string;
+        role: string;
+      };
+    }
+  }
+}
+
+// Mock auth middleware to bypass authentication
+jest.mock('../../middleware/auth', () => ({
+  UserRole: {
+    ADMIN: 'admin',
+    EDITOR: 'editor',
+    VIEWER: 'viewer',
+  },
+  authenticate: jest.fn().mockImplementation((_roles: string[]) => (req: Request, _res: Response, next: NextFunction) => {
+    req.user = { id: 'test-user', role: 'admin' };
+    next();
+  }),
+}));
+
+// Mock JWT auth to bypass token verification
+jest.mock('../../middleware/jwtAuth', () => ({
+  verifyToken: jest.fn().mockImplementation((req: Request, _res: Response, next: NextFunction) => {
+    req.user = { id: 'test-user', role: 'admin' };
+    next();
+  }),
+  authorizeJwt: jest.fn().mockImplementation((_roles: string[]) => (req: Request, _res: Response, next: NextFunction) => {
+    req.user = { id: 'test-user', role: 'admin' };
+    next();
+  }),
+}));
+
+// Mock services (uses __mocks__ directory auto-mocks)
+jest.mock('../../services/dictionaryService');
+jest.mock('../../services/entityService');
+jest.mock('../../services/serviceService');
+jest.mock('../../services/versionService');
+jest.mock('../../utils/logger');
+
+describe('API Integration Tests', () => {
+  describe('Health & Status', () => {
+    it('should return health status', async () => {
+      const response = await request(app).get('/health');
+      expect(response.status).toBe(200);
+      expect(response.body).toHaveProperty('status', 'ok');
+    });
+  });
+
+  describe('Service Endpoints', () => {
+    it('GET /api/services should return service list', async () => {
+      const response = await request(app).get('/api/services');
+      expect(response.status).toBe(200);
+      expect(response.body).toHaveProperty('data');
+      expect(Array.isArray(response.body.data)).toBe(true);
+      expect(response.body.data).toContain('user-service');
+    });
+
+    it('GET /api/services/:service/entities should return entities', async () => {
+      const response = await request(app).get('/api/services/user-service/entities');
+      expect(response.status).toBe(200);
+    });
+
+    it('GET /api/services/:service/entities/:entity should return entity schema', async () => {
+      const response = await request(app).get('/api/services/user-service/entities/User');
+      expect(response.status).toBe(200);
+    });
+
+    it('POST /api/services/:service/entities should create entity', async () => {
+      const response = await request(app)
+        .post('/api/services/user-service/entities')
+        .send({
+          uuid: 'a38d1597-cc4f-4934-bb08-c876c023f693',
+          name: 'NewEntity',
+          description: 'A new entity',
+          attributes: [{ uuid: 'b49e2608-dd5f-4045-aa09-d464c234e694', name: 'id', description: 'ID', type: 'string', required: true }],
+        });
+      expect(response.status).toBe(201);
+    });
+
+    it('PUT /api/services/:service/entities/:entity should update entity', async () => {
+      const response = await request(app)
+        .put('/api/services/user-service/entities/User')
+        .send({
+          uuid: 'a38d1597-cc4f-4934-bb08-c876c023f693',
+          name: 'User',
+          description: 'Updated user',
+          attributes: [{ uuid: 'b49e2608-dd5f-4045-aa09-d464c234e694', name: 'id', description: 'ID', type: 'string', required: true }],
+        });
+      expect(response.status).toBe(200);
+    });
+
+    it('DELETE /api/services/:service/entities/:entity should delete entity', async () => {
+      const response = await request(app).delete('/api/services/user-service/entities/User');
+      expect(response.status).toBe(200);
+    });
+  });
+
+  describe('Search & Graph', () => {
+    it('GET /api/search should return results', async () => {
+      const response = await request(app).get('/api/search?q=user');
+      expect(response.status).toBe(200);
+    });
+
+    it('GET /api/graph/:service should return graph data', async () => {
+      const response = await request(app).get('/api/graph/user-service');
+      expect(response.status).toBe(200);
+    });
+  });
+
+  describe('Version Control', () => {
+    it('POST /api/commit should commit changes', async () => {
+      const response = await request(app)
+        .post('/api/commit')
+        .send({ message: 'Test commit' });
+      expect(response.status).toBe(200);
+    });
+
+    it('GET /api/history should return commit history', async () => {
+      const response = await request(app).get('/api/history');
+      expect(response.status).toBe(200);
+    });
+
+    it('POST /api/revert should revert to commit', async () => {
+      const response = await request(app)
+        .post('/api/revert')
+        .send({ commitHash: 'mock-commit-1' });
+      expect(response.status).toBe(200);
+    });
+  });
+
+  describe('Relationship Endpoints', () => {
+    it('GET /api/packages/:packageName/relationships should return relationships', async () => {
+      const response = await request(app).get('/api/packages/user-service/relationships');
+      expect(response.status).toBe(200);
+      expect(response.body).toHaveProperty('data');
+    });
+  });
+});

package/backend/src/__tests__/setup.ts

@@ -0,0 +1,24 @@
+import dotenv from 'dotenv';
+
+// Load environment variables for testing
+dotenv.config({ path: '.env.test' });
+
+// Mock logger to prevent console output during tests
+jest.mock('../utils/logger', () => ({
+  logger: {
+    info: jest.fn(),
+    error: jest.fn(),
+    warn: jest.fn(),
+    debug: jest.fn(),
+  },
+}));
+
+// Global test setup
+beforeAll(() => {
+  // Setup any global test requirements here
+});
+
+// Global test teardown
+afterAll(() => {
+  // Clean up any global test resources here
+});

package/backend/src/__tests__/utils/testUtils.ts

@@ -0,0 +1,76 @@
+import { Express } from 'express';
+import request from 'supertest';
+import fs from 'fs';
+import path from 'path';
+
+/**
+ * Creates a test client for making HTTP requests to the Express app
+ * @param app Express application
+ * @returns Supertest instance
+ */
+export const createTestClient = (app: Express) => {
+  return request(app);
+};
+
+/**
+ * Creates a temporary test directory for file operations
+ * @param dirPath Directory path to create
+ */
+export const createTestDirectory = (dirPath: string): void => {
+  if (!fs.existsSync(dirPath)) {
+    fs.mkdirSync(dirPath, { recursive: true });
+  }
+};
+
+/**
+ * Removes a test directory and all its contents
+ * @param dirPath Directory path to remove
+ */
+export const removeTestDirectory = (dirPath: string): void => {
+  if (fs.existsSync(dirPath)) {
+    fs.rmSync(dirPath, { recursive: true, force: true });
+  }
+};
+
+/**
+ * Creates a test YAML file with the given content
+ * @param filePath Path to create the file
+ * @param content Content to write to the file
+ */
+export const createTestYamlFile = (filePath: string, content: string): void => {
+  const dir = path.dirname(filePath);
+  if (!fs.existsSync(dir)) {
+    fs.mkdirSync(dir, { recursive: true });
+  }
+  fs.writeFileSync(filePath, content);
+};
+
+/**
+ * Generates a random entity for testing
+ * @param overrides Properties to override in the generated entity
+ * @returns A test entity
+ */
+export const generateTestEntity = (overrides = {}) => {
+  return {
+    id: `test-entity-${Date.now()}`,
+    name: 'TestEntity',
+    description: 'A test entity for unit tests',
+    microservice: 'test-service',
+    version: '1.0.0',
+    attributes: [
+      {
+        name: 'id',
+        description: 'Primary identifier',
+        type: 'string',
+        required: true,
+      },
+      {
+        name: 'name',
+        description: 'Entity name',
+        type: 'string',
+        required: true,
+      },
+    ],
+    ...overrides,
+  };
+};