@noego/proper 0.0.3 → 0.0.5

package/readme.md

@@ -1,29 +1,36 @@
 ## SQL Proper
 
-**SQL Proper** is a lightweight and straightforward SQL migration tool for both **MySQL/MariaDB** (via `mysql2`) and **SQLite**. It allows you to:
+**SQL Proper** is a lightweight and straightforward SQL migration tool for both **MySQL/MariaDB** (via `mysql2`) and **SQLite**, with:
 
-- Easily run migrations (`up`) or roll them back (`down`)
-- Reset your database to a clean state
-- Check migration status
-- Create new migration files
-- Seamlessly manage your migration process via a simple **CLI** or by **importing** the library in your Node.js project
+- A simple **CLI** for application developers
+- A focused **programmatic API** for Node.js
+- A small, testable **framework core** for contributors and advanced users
+
+This README is written primarily for **developers integrating or extending** SQL Proper. If you only need usage details, you can skim **Quick Start (CLI)** and **Usage in Code (Import)**.
 
 ---
 
-## Quick Start (CLI)
+## Installation
 
-Below is a fast-track on **how to work** with SQL Proper. Skip to [Configuration](#configuration) to learn how to connect to your database.
+Add SQL Proper to your project:
 
-1. **Install** SQL Proper globally or locally:
-   ```bash
-   # Globally (optional)
-   npm install -g @noego/proper
+```bash
+# Globally (optional, for CLI-only usage)
+npm install -g @noego/proper
 
-   # Or locally to your project:
-   npm install --save @noego/proper
-   ```
+# Or locally to your project (recommended)
+npm install --save @noego/proper
+```
+
+For TypeScript projects, the emitted `.d.ts` files from the build are included in the published package; you don’t need to install separate type packages.
 
-2. **Initialize a config file** (creates `proper.json` by default):
+---
+
+## Quick Start (CLI)
+
+Below is a fast-track on **how to work** with SQL Proper from the command line. Skip to [Configuration](#configuration) to learn how to connect to your database or to [Programmatic API](#programmatic-api) for using it from code.
+
+1. **Initialize a config file** (creates `proper.json` by default):
    ```bash
    proper init
    ```
@@ -44,33 +51,33 @@ Below is a fast-track on **how to work** with SQL Proper. Skip to [Configuration
 
    Also create a `migrations/` directory in your project root to hold `.up.sql` and `.down.sql` files. File names follow `<timestamp>_<name>.up.sql` and `<timestamp>_<name>.down.sql`.
 
-3. **Create a new migration** (generates timestamped `.up.sql` & `.down.sql` files):
+2. **Create a new migration** (generates timestamped `.up.sql` & `.down.sql` files):
    ```bash
    proper create --name "create_users_table"
    ```
    Edit those `.sql` files and add your migration scripts.
 
-4. **Apply migrations**:
+3. **Apply migrations**:
    ```bash
    proper up
    ```
    - Applies all **pending** migrations.
    - Use `--increment <number>` to limit the count, e.g. `proper up --increment 1`.
     
-5. **Roll back migrations**:
+4. **Roll back migrations**:
    ```bash
    proper down
    ```
    - Rolls back the **latest** applied migrations.
    - Use `--increment <number>` to limit the count, e.g. `proper down --increment 1`.
     
-6. **Reset your database**:
+5. **Reset your database**:
    ```bash
    proper reset
    ```
    - Rolls back **all** migrations, then re-applies them.
     
-7. **Check status**:
+6. **Check status**:
    ```bash
    proper status
    ```
@@ -117,9 +124,9 @@ proper up --config path/to/my-custom.json
 
 ---
 
-## Usage in Code (Import)
+## Programmatic API
 
-If you prefer integrating migrations in your Node.js scripts, import SQL Proper:
+If you prefer integrating migrations directly in your Node.js services or test harnesses, import SQL Proper:
 
 ```ts
 import { MigrationRunnerFactory } from "@noego/proper";
@@ -127,6 +134,10 @@ import { MigrationRunnerFactory } from "@noego/proper";
 (async () => {
   // Suppose "myconfig.json" is your config file
   const runner = await MigrationRunnerFactory.create("myconfig.json");
+
+  // Or pass an existing connection (useful for testing or connection pooling):
+  // const runner = await MigrationRunnerFactory.create("myconfig.json", existingConnection);
+
   await runner.setup();
 
   // Apply pending migrations
@@ -144,15 +155,213 @@ import { MigrationRunnerFactory } from "@noego/proper";
 })();
 ```
 
-**Key methods**:
-- `setup()`: Prepares DB environment (creates migration table if missing).
-- `getMigrations()`: Loads all migrations from the configured folder.
+### Public exports
+
+From `@noego/proper`:
+
+- `MigrationRunnerFactory`
+  - `create(configPath: string, connection?: any)`: create a runner backed by config, optionally reusing an existing DB connection (see examples below).
+  - `createEmpty(configPath: string)`: create a runner without connecting to a database (used by `init`, `create`).
+- `MigrationConfig`: config model used by the framework.
+- `MigrationRunner` (alias of `MySQLMigrationRunner`): convenience export for MySQL-specific runner.
+
+From `@noego/proper/runners`:
+
+- `BaseSQLRunner`, `SQLRunner`, `SQLiteRunner`: low-level database runners for advanced integrations and testing.
+
+These are implemented in the `framework/` directory and re-exported via `framework/lib/runner.ts`. Check the generated `.d.ts` files in `lib/` and `bin/` for the exact TypeScript shapes in a given release.
+
+### Core runner API
+
+The `MigrationRunner` instances returned from `MigrationRunnerFactory.create` expose:
+
+- `setup()`: Prepare DB environment (creates migration table if missing).
+- `getMigrations()`: Load all migrations from the configured folder.
 - `getPendingMigrations()` / `getCompletedMigrations()`: Filtered migration sets.
-- `migrate(nodes, forward)`: Executes migrations (forward = `true` for `up`).
-- `reset()`: Rolls back everything, then reapplies them.
-- `createMigration(name)`: Scaffolds `<timestamp>_<name>.up.sql` and `.down.sql`.
-- `init(configPath)`: Creates a default `proper.json` if missing.
-- `close()`: Closes the DB connection.
+- `migrate(nodes, forward)`: Execute migrations (`forward = true` for `up`, `false` for `down`).
+- `reset()`: Roll back everything, then reapply them (used by CLI `reset`).
+- `createMigration(name)`: Scaffold `<timestamp>_<name>.up.sql` and `.down.sql` files.
+- `init(configPath)`: Create a default `proper.json` if missing.
+- `query(sql: string, params?)`: Execute read-only queries (used by CLI `query`).
+- `close()`: Close the DB connection (no-op if you provided the connection).
+
+---
+
+## Testing with In-Memory Databases
+
+For test isolation and faster test execution, you can pass an existing database connection to Proper. This is particularly useful when working with SQLite `:memory:` databases where you want multiple tests to share the same in-memory instance.
+
+**Why This Matters:** When using SQLite `:memory:` databases for testing, the database only exists as long as the connection is open. By creating the connection yourself and passing it to Proper, you can:
+
+1. **Create** the `:memory:` connection once
+2. **Run migrations** using Proper with that connection
+3. **Run your tests** using the same connection with a fully migrated schema
+
+Without this capability, you'd have to let Proper create the connection, but then you couldn't access that same in-memory database from your tests (it would be lost when Proper closes). Connection injection solves this by letting you control the lifecycle.
+
+### Example: Using a Shared `:memory:` Connection
+
+```ts
+import { MigrationRunnerFactory } from "@noego/proper";
+import * as sqlite from "sqlite";
+import * as sqlite3 from "sqlite3";
+
+// Create a single :memory: connection that all tests will share
+const memoryDb = await sqlite.open({
+  filename: ':memory:',
+  driver: sqlite3.Database
+});
+
+// Pass the connection to Proper
+const runner = await MigrationRunnerFactory.create("test-config.json", memoryDb);
+
+// Setup and run migrations on the shared connection
+await runner.setup();
+const pending = await runner.getPendingMigrations();
+await runner.migrate(pending, true);
+
+// Now your tests can use the same memoryDb instance
+// All queries will run against the same in-memory database
+```
+
+### Example: Test Setup with Connection Injection
+
+```ts
+import { describe, beforeAll, afterAll, test } from '@jest/globals';
+
+describe('Database Tests', () => {
+  let db;
+  let runner;
+
+  beforeAll(async () => {
+    // Step 1: Create the :memory: database connection
+    // This connection will persist for all tests in this suite
+    db = await sqlite.open({
+      filename: ':memory:',
+      driver: sqlite3.Database
+    });
+
+    // Step 2: Pass the connection to Proper and run migrations
+    // Proper will use YOUR connection instead of creating its own
+    runner = await MigrationRunnerFactory.create("proper.json", db);
+    await runner.setup();
+    await runner.migrate(await runner.getPendingMigrations(), true);
+
+    // Step 3: Now your tests can use the same 'db' connection
+    // The schema is fully migrated and ready for testing
+  });
+
+  afterAll(async () => {
+    await runner.close();
+  });
+
+  test('should query migrated database', async () => {
+    // Use the same db connection that was migrated by Proper
+    // This works because we created the connection ourselves
+    const result = await db.get('SELECT * FROM users WHERE id = ?', [1]);
+    expect(result).toBeDefined();
+  });
+});
+```
+
+---
+
+## Seeding
+
+SQL Proper supports a powerful seeding feature that allows you to populate your database with initial data (reference data, test users, etc.). Seeds can be SQL files or JavaScript/TypeScript modules.
+
+### Configuration
+
+Configure seeds in your `proper.json`:
+
+```json
+{
+  "seeds": {
+    "migrationsDir": "database/seed/migrations",
+    "dataDir": "database/seed/data",
+    "list": ["muscles", "exercises", "equipments", "user"]
+  }
+}
+```
+
+- **migrationsDir**: Directory containing seed implementation files (`.sql`, `.ts`, or `.js`).
+- **dataDir**: (Optional) Directory containing JSON data files for seeds.
+- **list**: Default list of seed names to run when no names are provided to the CLI.
+
+### CLI Usage
+
+Run seeds via the CLI:
+
+```bash
+# Run default seeds defined in config
+proper seed
+
+# Run specific seeds
+proper seed muscles exercises
+
+# Seed rollback (runs down migration for seeds)
+proper seed --action down muscles
+
+# Transactional mode
+proper seed --transactional seed   # Wrap each seed in a transaction
+proper seed --transactional runner # Wrap all seeds in one transaction
+```
+
+### Seed Implementations
+
+Seeds can be implemented in two ways:
+
+**1. SQL Pair** (`<name>.up.sql` and `<name>.down.sql`) in `migrationsDir`.
+   - Simple raw SQL execution.
+   - Good for static reference data.
+
+**2. TypeScript/JavaScript Module** (`<name>.ts` or `<name>.js`) in `migrationsDir`.
+   - Exports `up` and `down` functions.
+   - Receives a `runner` and a `context` object.
+   - `context` contains:
+     - `data`: Loaded JSON data matching the seed name (from `dataDir`).
+     - `dialect`: The current database dialect (`sql`, `sqlite`, etc.).
+     - `log`: A logger function.
+
+**Example TS Seed (`database/seed/migrations/user.ts`):**
+
+```ts
+import type { IMigrationRunner, SeedContext } from "@noego/proper";
+
+export async function up(runner: IMigrationRunner, ctx: SeedContext) {
+  const users = ctx.data as any[]; // Data loaded from database/seed/data/user.json
+  
+  for (const user of users) {
+    await runner.query("INSERT INTO users (email, name) VALUES (?, ?)", [user.email, user.name]);
+    ctx.log?.(`Created user ${user.email}`);
+  }
+}
+
+export async function down(runner: IMigrationRunner, ctx: SeedContext) {
+  const users = ctx.data as any[];
+  for (const user of users) {
+    await runner.query("DELETE FROM users WHERE email = ?", [user.email]);
+  }
+}
+```
+
+### Programmatic API
+
+Run seeds from your code (great for tests):
+
+```ts
+import { MigrationRunnerFactory, runSeedsWithRunner, loadMigrationConfig } from "@noego/proper";
+
+// Load config
+const config = loadMigrationConfig("proper.json");
+const runner = await MigrationRunnerFactory.create("proper.json");
+
+// Run seeds
+await runSeedsWithRunner(runner, config, 'up', {
+  names: ['muscles', 'exercises'], // Optional: override list
+  validate: true, // Optional: validate JSON data against schema
+});
+```
 
 ---
 
@@ -199,19 +408,66 @@ SQL_PASSWORD=supersecret proper up --config proper.json
 
 ---
 
-## Development
+## Project Layout (for contributors)
+
+This is a small, framework-style project. The main pieces are:
+
+- `cli.ts`: CLI entrypoint (compiled to `bin/cli.js` and published as the `proper` binary).
+- `index.ts`: Library entrypoint, exporting `MigrationRunnerFactory`, `MigrationConfig`, and `MigrationRunner`.
+- `framework/`: Core framework classes:
+  - `MigrationCLI.ts`: CLI parser and command wiring.
+  - `MigrationRunner.ts`: main migration runner + factory.
+  - `MigrationConfig.ts`: config reading/validation.
+  - `MigrationDirectoryReader.ts`: migration discovery from the filesystem.
+  - `MigrationStatus.ts`, `MigrationNode.ts`, `MigrationFilter.ts`: migration model, status helpers, and filtering logic.
+  - `SQLRunner.ts`: MySQL and SQLite runner implementations.
+  - `errors.ts`: domain-specific error types (used heavily by CLI and tests).
+- `framework/lib/runner.ts`: re-exports low-level SQL runners for consumers (`@noego/proper/runners`).
+- `helper/`: small utilities (e.g. string helpers).
+- `tests/`: Jest-based unit tests for the framework and CLI.
+- `example/`: runnable MySQL + migrations example (with `docker-compose` and `Makefile`).
+
+Build artifacts are written to:
+
+- `bin/`: compiled CLI and public library entrypoints.
+- `lib/`: compiled runner exports (`runners` subpath).
+
+The package’s `"files"` array in `package.json` ensures only `bin/`, `lib/`, and `readme.md` are published.
+
+---
+
+## Development & Contributing
+
+Requirements:
+
+- Node.js 16+ (LTS recommended)
+- npm (or a compatible package manager)
 
-- Node: 16+ recommended
-- Install: `npm install`
-- Build: `npm run build` (emits to `bin/` via tsup)
-- Test: `npm test`
+Local workflow:
+
+- Install dependencies: `npm install`
+- Run tests: `npm test`
+- Build the package: `npm run build` (emits to `bin/` and `lib/` via `tsup`)
 - Run CLI from source (TypeScript): `npx tsx cli.ts --config sqlite/proper.json up`
 
+Test tooling:
+
+- Jest with `ts-jest` and `jest-extended` (see `jest.config.ts`).
+- Core framework tests in `tests/unit/framework`.
+- CLI tests in `tests/unit/cli.test.ts`.
+
 Debugging in VS Code:
+
 - Use the provided `.vscode/launch.json`:
-  - “Jest: Current File” runs the open test file
-  - “Jest: All Tests” runs the full suite
-  - “Jest: CLI Test File” runs only `tests/unit/cli.test.ts`
+  - “Jest: Current File” runs the open test file.
+  - “Jest: All Tests” runs the full suite.
+  - “Jest: CLI Test File” runs only `tests/unit/cli.test.ts`.
+
+Contribution guidelines (informal for now):
+
+- Keep the public API surface (`index.ts`, `framework/lib/runner.ts`, CLI flags) backward compatible where possible.
+- Add or update tests in `tests/unit` when changing framework or CLI behavior.
+- Prefer small, focused abstractions in `framework/` over adding logic in the CLI.
 
 ---