@asaidimu/anansi 1.6.5 → 2.0.0

package/README.md

@@ -1,5 +1,51 @@
 # Anansi Schema Evolution Platform
 
+[![npm version](https://img.shields.io/npm/v/@asaidimu/anansi.svg)](https://www.npmjs.com/package/@asaidimu/anansi)
+[![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE.md)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/asaidimu/data-model/main.yml?branch=main&label=build)](https://github.com/asaidimu/data-model/actions)
+
+A comprehensive toolkit for advanced data modelling, schema evolution, and adaptive persistence management in complex enterprise systems.
+
+---
+
+## Table of Contents
+
+- [Why the Name "Anansi"?](#why-the-name-anansi)
+- [Theoretical Foundation](#theoretical-foundation)
+- [Overview & Features](#overview--features)
+  - [Detailed Description](#detailed-description)
+  - [Key Features](#key-features)
+- [Installation & Setup](#installation--setup)
+  - [Prerequisites](#prerequisites)
+  - [Installation Steps](#installation-steps)
+  - [Configuration](#configuration)
+- [Usage Documentation](#usage-documentation)
+  - [Defining a Schema](#defining-a-schema)
+  - [Ephemeral Persistence](#ephemeral-persistence)
+  - [Schema Registry (Git-backed)](#schema-registry-git-backed)
+  - [Schema Migration](#schema-migration)
+  - [Type Generation](#type-generation)
+  - [Documentation Generation](#documentation-generation)
+- [Project Architecture](#project-architecture)
+  - [Directory Structure](#directory-structure)
+  - [Core Components](#core-components)
+  - [Data Flow](#data-flow)
+  - [Extension Points](#extension-points)
+- [Development & Contributing](#development--contributing)
+  - [Development Setup](#development-setup)
+  - [Available Scripts](#available-scripts)
+  - [Testing](#testing)
+  - [Contributing Guidelines](#contributing-guidelines)
+  - [Issue Reporting](#issue-reporting)
+- [Additional Information](#additional-information)
+  - [Troubleshooting](#troubleshooting)
+  - [FAQ](#faq)
+  - [Changelog](#changelog)
+  - [License](#license)
+  - [Acknowledgments](#acknowledgments)
+
+---
+
 ## Why the Name "Anansi"?
 
 Named after the legendary Akan trickster god of West Africa, our platform embodies Anansi's core attributes:
@@ -13,92 +59,731 @@ The name reflects our philosophical approach: transformative, intelligent, and d
 
 A mathematically rigorous framework for managing data model complexity, grounded in advanced theoretical principles of system evolution and distributed computing.
 
-## Overview
+## Overview & Features
+
+### Detailed Description
+
+Anansi is a comprehensive platform built to tackle the challenges of modern enterprise data management. It provides a principled approach to schema design, evolution, and system integration, moving beyond simple CRUD operations to offer a robust, version-controlled environment for your data models. By treating schemas as living, evolving constructs, Anansi enables systematic transformations while preserving system-wide data integrity. Its in-memory capabilities, backed by a persistent Git-enabled registry, ensure both high performance and reliable versioning for collaborative development and deployment. Anansi is ideal for organizations navigating complex, rapidly evolving, and distributed system architectures.
+
+### Key Conceptual Components
+
+- 👷 **Schema Registry**: Inspired by the intricate webs of knowledge, this component offers comprehensive metadata tracking and explicit dependency management for all your data schemas.
+- 🧠 **Theoretical Migration Framework**: Provides formal methods for schema evolution, guaranteeing atomic transformations and supporting bidirectional migrations to ensure data consistency across versions.
+- 🌐 **Architectural Abstractions**: Enables decoupled schema representation, cross-system consistency models, and adaptive persistence strategies to integrate seamlessly into diverse architectural landscapes.
+
+### Key Features
+
+- **Schema Definition & Validation**: Define complex data structures using a rich `SchemaDefinition` interface, complete with fields, nested schemas, constraints, and indexes. Leverage a powerful, auto-generated validation SDK to ensure data integrity against your defined schemas.
+- **Schema Evolution & Migration**: Manage schema changes over time with a robust `MigrationEngine`. Define schema changes and data transformations to seamlessly evolve your data models forward and backward, preserving data consistency.
+- **Flexible Persistence Adapters**: Interact with various data stores through a unified `Persistence` interface. Includes an in-memory ephemeral persistence layer for rapid prototyping and a production-ready PocketBase adapter with built-in retry mechanisms and error categorization.
+- **Schema Registry with Version Control**: Store, manage, and version your schemas using a `SchemaRegistry` that can operate purely in-memory (backed by `LightningFS`) or integrate with Git for distributed version control, branching, and tagging.
+- **Code Generation & Utilities**: Automatically generate TypeScript types (`schemaToTypes`) and human-readable documentation (`docgen`) directly from your `SchemaDefinition` files. Includes essential utilities for cryptographic hashing, deep merging, and JSON Patch operations.
+- **Event-driven Data Operations**: Subscribe to granular persistence events (e.g., `create:success`, `update:failed`) and define triggers or scheduled tasks to automate workflows around data changes.
+
+## Installation & Setup
+
+### Prerequisites
+
+-   **Node.js**: Version 18.x or higher.
+-   **Bun**: Recommended for faster installation and script execution. (Alternatively, npm/yarn can be used).
+-   **TypeScript**: For developing with Anansi and its generated types.
+-   **PocketBase** (Optional): If using the PocketBase persistence adapter, a running PocketBase instance is required.
+
+### Installation Steps
+
+Install Anansi into your project using Bun (recommended) or npm:
+
+```bash
+# Using Bun
+bun add @asaidimu/anansi
+
+# Using npm
+npm install @asaidimu/anansi
+```
+
+### Configuration
+
+#### Ephemeral Persistence (In-memory)
+
+For local development and testing, you can use the ephemeral persistence layer which requires no external configuration:
+
+```typescript
+import { createEphemeralPersistence } from '@asaidimu/anansi';
+
+// Define your predicate map (if using custom constraints)
+const myPredicates = {
+  isPositive: ({ data, field, arguments: min }) => data[field] > min,
+};
+
+// Create a new in-memory persistence instance
+const persistence = createEphemeralPersistence({}, myPredicates); // No functionMap needed for basic use
+
+// Now you can create and manage collections in memory
+```
+
+#### Git-backed Schema Registry
+
+To enable persistent, version-controlled schema management, you'll configure `createGitSchemaRegistry` with a remote Git repository (e.g., GitHub, Gitea). This will require authentication credentials.
+
+```typescript
+import { createGitSchemaRegistry, createGithubRepository } from '@asaidimu/anansi';
+
+// Example for GitHub
+const remoteRepo = await createGithubRepository({
+  username: process.env.GITHUB_USERNAME!,
+  password: process.env.GITHUB_PAT!, // Personal Access Token
+  repository: 'my-anansi-schemas',
+  create: true, // Auto-create if not exists
+});
+
+const gitRegistry = await createGitSchemaRegistry('/my-schema-registry', {
+  remote: remoteRepo,
+  author: { name: 'Anansi Bot', email: 'anansi-bot@example.com' },
+  // proxy: 'https://cors.isomorphic-git.org', // Uncomment if encountering CORS issues
+});
+
+await gitRegistry.init(); // Initialize the local Git repository and sync with remote
+```
+
+#### PocketBase Persistence
+
+To use PocketBase, you need to provide the PocketBase URL and optionally an auth token:
+
+```typescript
+import { createPocketBasePersistence } from '@asaidimu/anansi/sdk/pocketbase'; // Note: path to specific SDK
+import { createSchemaMigrationHelper } from '@asaidimu/anansi/lib/schema/helpers'; // Note: direct import for helper
+
+const pocketBasePersistence = createPocketBasePersistence({
+  url: 'http://127.0.0.1:8090', // Your PocketBase instance URL
+  // authToken: 'YOUR_POCKETBASE_AUTH_TOKEN', // Optional: if auth is required
+  env: 'development', // 'development' for migrations/rollbacks, 'production' for read-only
+});
+
+// Use pocketBasePersistence like any other Anansi persistence instance
+```
+
+## Usage Documentation
+
+### Defining a Schema
+
+Schemas are defined using the `SchemaDefinition` interface. Here's a basic example:
+
+```typescript
+import { SchemaDefinition } from '@asaidimu/anansi';
+
+const UserSchema: SchemaDefinition = {
+  name: "User",
+  version: "1.0.0",
+  description: "Represents a user in the system",
+  fields: {
+    id: { name: "id", type: "string", required: true, description: "Unique user ID" },
+    email: {
+      name: "email",
+      type: "string",
+      required: true,
+      constraints: [{
+        name: "isEmail",
+        predicate: "isEmailFormat", // Assumes 'isEmailFormat' is in your predicateMap
+        parameters: /^[^\s@]+@[^\s@]+\.[^\s@]+$/,
+        errorMessage: "Must be a valid email address."
+      }],
+      description: "User's email address"
+    },
+    status: {
+      name: "status",
+      type: "enum",
+      values: ["active", "inactive", "pending"],
+      default: "pending",
+      description: "User's current status"
+    },
+    profile: {
+      name: "profile",
+      type: "object",
+      schema: { id: "UserProfile" }, // References a nested schema
+      required: false,
+      description: "User's profile details"
+    },
+    roles: {
+      name: "roles",
+      type: "set",
+      itemsType: "string",
+      description: "Set of unique roles assigned to the user"
+    }
+  },
+  nestedSchemas: {
+    UserProfile: {
+      name: "UserProfile",
+      fields: {
+        firstName: { name: "firstName", type: "string", required: true },
+        lastName: { name: "lastName", type: "string", required: true },
+        age: { name: "age", type: "number", required: false, default: 18 }
+      }
+    }
+  },
+  indexes: [
+    { name: "emailIndex", fields: ["email"], type: "unique", description: "Ensures unique email addresses" }
+  ],
+  constraints: [
+    {
+      name: "ageConstraint",
+      operator: "and",
+      rules: [
+        { name: "minAge", predicate: "min", field: "profile.age", parameters: 18 },
+        { name: "maxAge", predicate: "max", field: "profile.age", parameters: 120 }
+      ]
+    }
+  ],
+  mock: (faker) => ({
+    id: faker.string.uuid(),
+    email: faker.internet.email(),
+    status: faker.helpers.arrayElement(["active", "inactive", "pending"]),
+    profile: {
+      firstName: faker.person.firstName(),
+      lastName: faker.person.lastName(),
+      age: faker.number.int({ min: 18, max: 99 })
+    },
+    roles: faker.helpers.arrayElements(["admin", "user", "guest"], { min: 1, max: 3 })
+  })
+};
+```
+
+### Ephemeral Persistence
+
+Interact with data using the in-memory persistence layer:
+
+```typescript
+import { createEphemeralPersistence, SchemaDefinition } from '@asaidimu/anansi';
+
+const myPredicates = {
+  isEmailFormat: ({ data, field, arguments: regex }) => regex.test(data[field]),
+  min: ({ data, field, arguments: minValue }) => data[field] >= minValue,
+  max: ({ data, field, arguments: maxValue }) => data[field] <= maxValue,
+};
+
+const persistence = createEphemeralPersistence({}, myPredicates);
+
+// Assume UserSchema is defined as above
+const usersCollection = await persistence.createCollection<typeof UserSchema>({
+  name: UserSchema.name,
+  version: UserSchema.version,
+  fields: UserSchema.fields,
+  nestedSchemas: UserSchema.nestedSchemas,
+  constraints: UserSchema.constraints,
+  indexes: UserSchema.indexes,
+});
+
+// 🚀 Create
+const newUser = await usersCollection.create({
+  data: {
+    id: 'user123',
+    email: 'test@example.com',
+    status: 'active',
+    profile: { firstName: 'John', lastName: 'Doe', age: 30 },
+    roles: ['admin', 'user'],
+  },
+});
+console.log('Created user:', newUser);
+
+// ⚡ Read
+const activeUsers = await usersCollection.read({
+  query: { filters: { status: { $eq: 'active' } } },
+});
+console.log('Active users:', activeUsers);
+
+// 🔄 Update
+const updatedUsers = await usersCollection.update({
+  query: { email: { $eq: 'test@example.com' } },
+  data: { status: 'inactive' },
+});
+console.log('Updated users:', updatedUsers);
+
+// 🗑️ Delete
+const deletedCount = await usersCollection.delete({
+  query: { id: { $eq: 'user123' } },
+});
+console.log('Deleted users count:', deletedCount);
+
+// ✨ Validate
+const invalidUser = { id: 'invalid', email: 'bad-email' };
+const validationResult = usersCollection.validate(invalidUser);
+console.log('Validation issues:', validationResult.issues);
+```
+
+### Schema Registry (Git-backed)
+
+Manage your schemas persistently with Git integration.
+
+```typescript
+import {
+  createGitSchemaRegistry,
+  createGithubRepository,
+  SchemaDefinition,
+} from '@asaidimu/anansi';
+import { UserSchema } from './my-schemas'; // Assuming UserSchema is defined
+
+// Configure remote repository (e.g., GitHub)
+const githubRepo = await createGithubRepository({
+  username: 'your-github-username',
+  password: 'your-github-pat', // Use a Personal Access Token
+  repository: 'anansi-schemas',
+  create: true, // Create the repo if it doesn't exist
+});
+
+// Create Git-backed schema registry
+const registry = await createGitSchemaRegistry('/anansi-schemas', {
+  remote: githubRepo,
+  author: { name: 'Anansi Bot', email: 'bot@anansi.com' },
+});
+
+// Initialize the registry (clones/creates local repo, syncs)
+await registry.init();
+
+// 👷 Create a schema
+await registry.create({ schema: UserSchema });
+console.log(`Schema '${UserSchema.name}' created/updated in registry.`);
+
+// 📋 List all schemas
+const allSchemas = await registry.list();
+console.log('All schemas:', allSchemas);
+
+// 🔍 Retrieve a schema definition
+const retrievedSchema = await registry.schema({ name: 'User' });
+console.log('Retrieved User schema:', retrievedSchema?.version);
+
+// 🔄 Update a schema
+const updatedUserSchema: SchemaDefinition = {
+  ...UserSchema,
+  version: "1.1.0",
+  fields: {
+    ...UserSchema.fields,
+    phone: { name: "phone", type: "string", required: false, description: "User's phone number" }
+  }
+};
+await registry.update({ schema: updatedUserSchema });
+console.log(`Schema '${updatedUserSchema.name}' updated to v${updatedUserSchema.version}.`);
+
+// ⚡ Sync local changes with remote Git repository
+await registry.sync();
+console.log('Registry synced with remote Git repository.');
+
+// 🗑️ Delete a schema
+// await registry.delete({ name: 'User' });
+// console.log(`Schema 'User' deleted.`);
+```
+
+### Schema Migration
+
+Apply and roll back schema changes, including data transformations.
+
+```typescript
+import { createEphemeralPersistence, createSchemaMigrationHelper, SchemaDefinition, DataTransform } from '@asaidimu/anansi';
+
+// Assume UserSchema is defined as above
+const oldUserSchema: SchemaDefinition = {
+  ...UserSchema,
+  version: "1.0.0", // Original version
+  fields: {
+    id: { name: "id", type: "string", required: true },
+    oldEmail: { name: "oldEmail", type: "string", required: false } // Field to be migrated
+  },
+  nestedSchemas: {}
+};
+
+const persistence = createEphemeralPersistence({}, {});
+const usersCollection = await persistence.createCollection<any>({
+  name: oldUserSchema.name,
+  version: oldUserSchema.version,
+  fields: oldUserSchema.fields,
+  nestedSchemas: oldUserSchema.nestedSchemas
+});
+
+// Add some dummy data with oldEmail
+await usersCollection.create({
+  data: [
+    { id: 'u1', oldEmail: 'user1@old.com', status: 'active' },
+    { id: 'u2', oldEmail: 'user2@old.com', status: 'inactive' }
+  ]
+});
+
+// Define the migration
+const addEmailMigration = (h: ReturnType<typeof createSchemaMigrationHelper>) => {
+  // Define forward transformation: oldEmail -> email
+  const forwardTransform: DataTransform<any, any>['forward'] = (data) => ({
+    ...data,
+    email: data.oldEmail,
+    oldEmail: undefined // Remove old field
+  });
+
+  // Define backward transformation: email -> oldEmail
+  const backwardTransform: DataTransform<any, any>['backward'] = (data) => ({
+    ...data,
+    oldEmail: data.email,
+    email: undefined // Remove new field
+  });
+
+  h.addField('email', { name: 'email', type: 'string', required: true, description: 'New email field' });
+  h.removeField('oldEmail'); // Deprecate/remove old field
+
+  return { forward: forwardTransform, backward: backwardTransform };
+};
+
+// ➡️ Perform a dry run migration
+console.log('\n--- Dry Run Migration (Forward) ---');
+const { newSchema: dryRunSchema, dataPreview } = await usersCollection.migrate(
+  'Migrate oldEmail to email',
+  addEmailMigration,
+  true // dryRun = true
+);
+console.log('Dry run new schema version:', dryRunSchema.version);
+const previewRecords = await new Response(dataPreview).json(); // Read from stream
+console.log('Dry run data preview:', previewRecords);
+
+// 🚀 Apply the actual migration
+console.log('\n--- Applying Migration (Forward) ---');
+await usersCollection.migrate(
+  'Migrate oldEmail to email',
+  addEmailMigration,
+  false // dryRun = false
+);
+console.log('Migration applied. Current schema version:', usersCollection.schema().version);
+const migratedData = await usersCollection.read({});
+console.log('Migrated data:', migratedData);
+
+// ⏪ Rollback the migration (if supported by persistence)
+console.log('\n--- Rolling Back Migration ---');
+await usersCollection.rollback(undefined, false); // Rollback to previous version
+console.log('Rolled back. Current schema version:', usersCollection.schema().version);
+const rolledBackData = await usersCollection.read({});
+console.log('Rolled back data:', rolledBackData);
+```
+
+### Type Generation
+
+Generate TypeScript types from your schema definitions.
+
+```typescript
+import { schemaToTypes, SchemaDefinition } from '@asaidimu/anansi';
+
+const ProductSchema: SchemaDefinition = {
+  name: "Product",
+  version: "1.0.0",
+  fields: {
+    id: { name: "id", type: "string", required: true },
+    name: { name: "name", type: "string", required: true },
+    price: { name: "price", type: "number", required: true },
+    currency: { name: "currency", type: "enum", values: ["USD", "EUR", "GBP"] },
+    details: {
+      name: "details",
+      type: "object",
+      schema: { id: "ProductDetails" }
+    }
+  },
+  nestedSchemas: {
+    ProductDetails: {
+      name: "ProductDetails",
+      fields: {
+        weight: { name: "weight", type: "number" },
+        dimensions: {
+          name: "dimensions",
+          type: "object",
+          schema: { id: "ProductDimensions" }
+        }
+      }
+    },
+    ProductDimensions: {
+      name: "ProductDimensions",
+      fields: {
+        length: { name: "length", type: "number" },
+        width: { name: "width", type: "number" },
+        height: { name: "height", type: "number" }
+      }
+    }
+  }
+};
+
+const generatedTypes = schemaToTypes(ProductSchema);
+console.log(generatedTypes);
+
+/*
+// Output will be similar to:
+export type ProductDimensions = {
+  length?: number;
+  width?: number;
+  height?: number;
+};
+
+export type ProductDetails = {
+  weight?: number;
+  dimensions?: ProductDimensions;
+};
+
+export type ProductCurrency = "USD" | "EUR" | "GBP";
+
+export type Product = {
+  id: string;
+  name: string;
+  price: number;
+  currency: ProductCurrency;
+  details?: string | ProductDetails; // "string" for concrete schemas reference
+};
+
+export enum ProductIndexNames {
+  ...
+}
+*/
+```
+
+### Documentation Generation
+
+Generate markdown documentation for your schemas.
+
+```typescript
+import { docgen, SchemaDefinition } from '@asaidimu/anansi';
+import { faker } from '@faker-js/faker';
+
+// Assume ProductSchema is defined as above
+
+const productDoc = docgen(ProductSchema, { faker });
+console.log(productDoc);
+
+/*
+// Output will be similar to:
+# Product Schema (Version 1.0.0)
+
+## Metadata
+- **Dependencies:** None
+- **Created:** 2024-01-01T00:00:00.000Z
+
+## Fields
+
+| Name     | Type   | Required | Default | Description        | Deprecated | Unique | Constraints |
+|----------|--------|----------|---------|--------------------|------------|--------|-------------|
+| id       | string | Yes      | `None`  |                    | No         | No     | 0           |
+| name     | string | Yes      | `None`  |                    | No         | No     | 0           |
+| price    | number | Yes      | `None`  |                    | No         | No     | 0           |
+| currency | enum   | Yes      | `"USD"` |                    | No         | No     | 0           |
+| details  | object | No       | `None`  |                    | No         | No     | 0           |
+
+### Nested Schema: ProductDetails
+
+#### weight (number)
+
+**Required:** No
+
+#### dimensions (object)
+
+##### length (number)
+
+**Required:** No
+
+##### width (number)
+
+**Required:** No
+
+##### height (number)
+
+**Required:** No
+
+## Indexes
+
+| Name | Type   | Fields  | Unique | Order | Partial Condition | Description |
+|------|--------|---------|--------|-------|-------------------|-------------|
+| ...  | ...    | ...     | ...    | ...   | ...               | ...         |
+
+## Constraints
+### Schema-level Constraints
+...
+
+## Migrations
+
+| ID | Description | Status | Changes |
+|----|-------------|--------|---------|
+| ...| ...         | ...    | ...     |
+
+## Example Data
+```json
+{
+  "id": "e221b3a4-c5d6-7890-a1b2-c3d4e5f67890",
+  "name": "Ergonomic Widget",
+  "price": 99.99,
+  "currency": "USD",
+  "details": {
+    "weight": 0.5,
+    "dimensions": {
+      "length": 10,
+      "width": 5,
+      "height": 2
+    }
+  }
+}
+*/
+```
+
+## Project Architecture
+
+Anansi is structured to provide a modular and extensible platform for data model management.
+
+### Directory Structure
+
+```
+.
+├── src/
+│   ├── lib/                  # Core libraries for persistence, registry, migration, schema
+│   │   ├── persistence/      # In-memory and adapter-based data persistence (EmphemeralCollection)
+│   │   ├── registry/         # Schema Registry (LightningFS + Git integration)
+│   │   ├── migration/        # Schema migration engine (MigrationEngine)
+│   │   └── schema/           # Schema validation, helpers, and utilities
+│   ├── sdk/                  # Specific SDK implementations (e.g., PocketBase adapter, static validators)
+│   ├── types/                # Core TypeScript interfaces (SchemaDefinition, Persistence, Migration, etc.)
+│   └── tools/                # General utilities (crypto, merge, patch, typegen, docgen, validator, version)
+├── docs/                     # VitePress documentation site
+├── tests/                    # Unit and integration tests
+├── public/                   # Public assets for UI (e.g., Vite SVG)
+├── index.ts                  # Main entry point for the Anansi library
+├── package.json              # Project metadata and dependencies
+├── dist.package.json         # Package.json for distributed npm package
+├── vitest.config.ts          # Vitest configuration for testing
+├── vite.config.ts            # Vite configuration for UI development
+└── tsconfig.json             # TypeScript configuration
+```
+
+### Core Components
+
+-   **`SchemaDefinition` (`src/types/schema-definition.ts`)**: The central contract defining the structure of data models, including fields, nested schemas, constraints, and migrations.
+-   **`Persistence` (`src/types/persistence.ts`)**: An abstract interface for all data storage operations (create, read, update, delete, subscribe), designed for pluggable backends.
+    -   **`EmphemeralCollection` (`src/lib/persistence/collection.ts`)**: An in-memory implementation of `PersistenceCollection` for rapid development.
+    -   **PocketBase Adapter (`src/sdk/pocketbase/index.tsx`)**: A concrete `Persistence` implementation for PocketBase, handling schema-driven collection management, migrations, and eventing.
+-   **`MigrationEngine` (`src/lib/migration/index.ts`)**: Manages the evolution of schemas over time, applying schema changes and data transformations in a controlled, versioned manner.
+-   **`SchemaRegistry` (`src/lib/registry/registry.ts`)**: The core component for storing and versioning schema definitions. It uses `LightningFS` for in-memory storage.
+    -   **`createGitSchemaRegistry` (`src/lib/registry/git-registry.ts`)**: A factory function that wraps `SchemaRegistry` with `isomorphic-git` to provide persistent, distributed version control for schemas.
+-   **`Validators` (`src/tools/validator.ts`, `src/lib/schema/validators.ts`)**: Provides utilities for validating data against `SchemaDefinition` rules and for validating the schema definitions themselves.
+-   **`Type Generators` (`src/tools/typegen.ts`)**: Automatically generates TypeScript type definitions from `SchemaDefinition` files, ensuring strict type safety across your codebase.
+-   **`Documentation Generator` (`src/tools/docgen.ts`)**: Generates human-readable Markdown documentation for schemas, including fields, indexes, constraints, and example mock data.
+
+### Data Flow
+
+1.  **Schema Definition**: Developers define data models using the `SchemaDefinition` interface.
+2.  **Schema Registration**: Schemas are added to the `SchemaRegistry` (either in-memory or Git-backed) for version control and discovery.
+3.  **Persistence Layer**: A `Persistence` instance (e.g., `EphemeralPersistence`, `PocketBasePersistence`) is initialized with optional predicates and functions for query and validation.
+4.  **Collection Interaction**: Developers interact with collections (e.g., `usersCollection.create()`, `usersCollection.read()`) through the `PersistenceCollection` interface.
+5.  **Validation**: All data operations trigger internal validation against the collection's `SchemaDefinition` using the `createStandardSchemaValidator`.
+6.  **Schema Evolution**: When data models change, `SchemaChanges` are defined, and the `MigrationEngine` applies these changes to both the schema definition and existing data.
+7.  **Synchronization (Git)**: For Git-backed registries, changes are committed and pushed, enabling collaborative schema evolution and traceability.
+
+### Extension Points
+
+-   **Custom Persistence Adapters**: Implement the `Persistence` and `PersistenceCollection` interfaces to integrate Anansi with any new data storage backend.
+-   **Custom Predicates**: Define custom validation logic (e.g., `isEmailFormat`, `isStrongPassword`) and provide them to `createEphemeralPersistence` or any other persistence implementation.
+-   **Data Transforms**: Write custom `forward` and `backward` transformation functions for migrations, enabling complex data shape changes between schema versions.
+-   **Schema Hints**: Extend `InputHint` and `SchemaHint` to guide UI generation or other tooling based on schema metadata.
+
+## Development & Contributing
+
+### Development Setup
+
+To set up the project for local development:
+
+1.  **Clone the repository:**
+    ```bash
+    git clone https://github.com/asaidimu/data-model.git anansi
+    cd anansi
+    ```
+2.  **Install dependencies using Bun (recommended):**
+    ```bash
+    bun install
+    ```
+    If you don't have Bun, you can use npm:
+    ```bash
+    npm install
+    ```
+
+### Available Scripts
+
+The `package.json` includes several scripts for development workflows:
 
-A comprehensive platform for solving complex enterprise data management challenges through a principled approach to schema design, evolution, and system integration.
+-   `bun ci`: Installs dependencies.
+-   `bun clean`: Removes the `dist` directory.
+-   `bun prebuild`: Cleans and runs `./.sync-package.ts`.
+-   `bun build`: Compiles TypeScript files to `dist/` for CJS and ESM formats, generates declaration files, and minifies.
+-   `bun build:watch`: Runs `build` in watch mode for continuous compilation.
+-   `bun postbuild`: Copies `README.md`, `LICENSE.md`, and `dist.package.json` into the `dist` directory.
+-   `bun test`: Runs unit and integration tests using Vitest.
+-   `bun test:ci`: Runs Vitest tests in CI mode (runs once, exits).
+-   `bun test:debug`: Runs Vitest with debugger attached.
+-   `bun docs:dev`: Starts the VitePress development server for documentation.
+-   `bun docs:build`: Builds the static VitePress documentation site.
+-   `bun ui:dev`: Starts the Vite development server for the example UI.
+-   `bun docs:preview`: Previews the built documentation site.
 
-## Core Philosophical Principles
+### Testing
 
-### 1. Theoretical Precision
-- Modeled schema transformations
-- Rigorous state transition modeling
-- Formal methods for ensuring data consistency
+Anansi uses [Vitest](https://vitest.dev/) for its test suite.
 
-### 2. Holistic System Design
-- View schemas as living, evolving constructs
-- Provide a unified approach to data model management
-- Support complex enterprise architectural needs
+To run all tests:
+```bash
+bun test
+```
 
-### 3. Systematic Evolution Strategy
-- Mathematically defined migration paths
-- Preservation of system-wide data integrity
-- Controlled, traceable schema transformations
+To run tests in CI mode (non-interactive):
+```bash
+bun test:ci
+```
 
-## Key Conceptual Components
+Tests include coverage checks and are configured to run in a `happy-dom` environment with `fake-indexeddb` for browser-like filesystem operations (`LightningFS`).
 
-- 👷 **Schema Registry**
-  - Inspired by the intricate webs of knowledge
-  - Comprehensive metadata tracking
-  - Explicit dependency management
+### Contributing Guidelines
 
-- 🧠 **Theoretical Migration Framework**
-  - Formal methods for schema evolution
-  - Atomic transformation guarantees
-  - Bidirectional migration support
+We welcome contributions! Please follow these guidelines:
 
-- 🌐 **Architectural Abstractions**
-  - Decoupled schema representation
-  - Cross-system consistency models
-  - Adaptive persistence strategies
+1.  **Fork** the repository and **clone** your fork.
+2.  Create a **new branch** for your feature or bug fix: `git checkout -b feature/my-new-feature` or `bugfix/fix-some-bug`.
+3.  Make your changes, ensuring code adheres to existing style and conventions.
+4.  Write or update **tests** for your changes to ensure proper functionality and maintain test coverage.
+5.  Ensure all tests pass (`bun test`).
+6.  Use **semantic commit messages** (e.g., `feat: add new feature`, `fix: resolve bug`). This project uses `semantic-release`.
+7.  **Open a Pull Request** to the `main` branch of the upstream repository.
 
-## Theoretical Strengths
+### Issue Reporting
 
-The library is built on advanced computational and systems theory:
-- Temporal consistency modeling
-- Causal dependency tracking
-- Atomic migration operation design
-- Abstract representation of complex data relationships
+For bugs, feature requests, or questions, please open an issue on our [GitHub Issues page](https://github.com/asaidimu/data-model/issues).
 
-## Target Domains
+## Additional Information
 
-Ideal for organizations facing:
-- Highly complex enterprise architectures
-- Rapidly evolving business requirements
-- Multi-service, distributed system challenges
-- Strict regulatory and data integrity requirements
+### Troubleshooting
 
-## Conceptual Benefits
+-   **`Buffer is not defined` error**: If running in a browser environment, ensure that `window.Buffer = Buffer;` is included as done in `src/lib/registry/registry.ts`.
+-   **CORS issues with `isomorphic-git` or PocketBase**: If interacting with remote Git repositories or PocketBase from a browser, you might need a CORS proxy. Configure `createGitSchemaRegistry` or `createPocketBasePersistence` with a `proxy` URL.
+-   **Migration `TRANSFORM_ERROR`**: Ensure your `DataTransform` functions are correctly defined and handle all expected input shapes. For remote transforms, verify the URL and module export.
+-   **Git `fastForwardOnly` errors**: When performing `git pull` or `git push`, if `fastForwardOnly` is enabled and conflicts exist, the operation might fail. Consider resolving conflicts manually or disable `fastForwardOnly` if acceptable for your workflow.
 
-- ✅ Sound schema management
-- ✅ Systematic approach to system evolution
-- ✅ Comprehensive architectural flexibility
-- ✅ Rigorous data integrity preservation
-- ✅ Advanced theoretical foundations
+### FAQ
 
-## Architectural Approach
+-   **What is Anansi primarily designed for?**
+    Anansi is designed for managing complex enterprise data models, focusing on schema evolution, data integrity, and flexible persistence across distributed systems. It's a comprehensive toolkit, not just a simple ORM or validation library.
+-   **How does Anansi handle data transformations during migrations?**
+    Anansi uses `DataTransform` objects within migrations, which contain explicit `forward` and `backward` functions. These functions are executed on data streams to transform data shapes as the schema evolves.
+-   **Is Anansi production-ready?**
+    Yes, Anansi is built with production use cases in mind, emphasizing theoretical rigor, data integrity, and extensible architecture. The PocketBase adapter and Git-backed schema registry provide production-grade capabilities for persistence and version control.
+-   **Can I use Anansi with other databases?**
+    Yes, Anansi is designed with a pluggable `Persistence` interface. You can create custom adapters for any database or data source by implementing this interface.
 
-Rather than a simple tool, this is a comprehensive library that provides:
-- Conceptual frameworks
-- Theoretical models
-- Architectural guidelines
-- Reference implementations
+### Changelog
 
-## Research and Exploration
+For a detailed history of changes, features, and bug fixes, please refer to the [CHANGELOG.md](CHANGELOG.md) file.
 
-The platform is a living research initiative exploring:
-- Advanced schema management techniques
-- Distributed system design principles
-- Theoretical computer science applications in enterprise architecture
+### License
 
-## Collaboration and Research
+This project is licensed under the MIT License. See the [LICENSE.md](LICENSE.md) file for details.
 
-In the spirit of Anansi's collaborative wisdom, we invite:
-- Academic researchers
-- Enterprise architects
-- Systems design professionals
+### Acknowledgments
 
-To explore, challenge, and extend the theoretical foundations of enterprise system evolution.
+Anansi draws inspiration from and builds upon several foundational technologies and concepts:
 
-## Documentation and Exploration
+-   **SQL Pragmatism**: For the approach to schema definition, constraints, and indexing.
+-   **`isomorphic-git`**: For enabling Git operations in diverse JavaScript environments.
+-   **`LightningFS`**: For providing a performant in-memory filesystem abstraction.
+-   **PocketBase**: For its powerful real-time backend capabilities, integrated via a dedicated persistence adapter.
+-   **`@faker-js/faker`**: For robust mock data generation capabilities.
+-   **`@standard-schema/spec`**: For providing a standardized schema validation specification.
 
-Coming soon...
+We are grateful to the creators and maintainers of these projects for their invaluable contributions to the open-source ecosystem.