@toiroakr/lines-db 0.1.0 → 0.2.0

package/README.md

@@ -0,0 +1,359 @@
+# lines-db
+
+A data management library that treats JSONL (JSON Lines) files as tables. Perfect for managing application seed data and testing.
+
+## Features
+
+- 📝 Load JSONL files as database tables
+- ✅ **CLI tools for validation and data migration**
+- 🔄 Automatic schema inference
+- 📦 **JSON column support** with automatic serialization/deserialization
+- ✅ Built-in validation using StandardSchema (Valibot, Zod, etc.)
+- 🎯 **Automatic type inference from table names**
+- 🔄 **Bidirectional schema transformations**
+- 💾 **Auto-sync to JSONL files**
+- 🛡️ Type-safe with TypeScript
+- Node.js 22.5+ support
+
+## VS Code Extension
+
+A VS Code extension is available that provides syntax highlighting and validation for JSONL files with schema support.
+
+[![VS Code Marketplace](https://img.shields.io/visual-studio-marketplace/v/toiroakr.lines-db-vscode?label=VS%20Code%20Marketplace&logo=visual-studio-code)](https://marketplace.visualstudio.com/items?itemName=toiroakr.lines-db-vscode)
+
+[Install from VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=toiroakr.lines-db-vscode)
+
+## Installation
+
+```bash
+npm install @toiroakr/lines-db
+# or
+pnpm add @toiroakr/lines-db
+```
+
+## CLI Usage
+
+### Setting Up Schemas
+
+Create schema files alongside your JSONL files:
+
+**Directory structure:**
+
+```
+data/
+  ├── users.jsonl
+  ├── users.schema.ts
+  ├── products.jsonl
+  └── products.schema.ts
+```
+
+**Example schema (users.schema.ts):**
+
+```typescript
+import * as v from 'valibot';
+import { defineSchema } from '@toiroakr/lines-db';
+
+export const schema = defineSchema(
+  v.object({
+    id: v.pipe(v.number(), v.integer(), v.minValue(1)),
+    name: v.pipe(v.string(), v.minLength(1)),
+    age: v.pipe(v.number(), v.integer(), v.minValue(0), v.maxValue(150)),
+    email: v.pipe(v.string(), v.email()),
+  }),
+);
+export default schema;
+```
+
+**Supported validation libraries:**
+
+- Any library implementing [StandardSchema](https://standardschema.dev/)
+
+### Validate JSONL Files
+
+Validate your JSONL files against their schemas:
+
+```bash
+npx lines-db validate <path>
+```
+
+**Example:**
+
+```bash
+# Validate all JSONL files in ./data directory
+npx lines-db validate ./data
+
+# Validate a specific file
+npx lines-db validate ./data/users.jsonl
+
+# Verbose output
+npx lines-db validate ./data --verbose
+```
+
+This command will:
+
+- For directories: Find all `.jsonl` files in the directory
+- For files: Validate the specified `.jsonl` file
+- Load corresponding `.schema.ts` files
+- Validate each record against the schema
+- Report validation errors with detailed messages
+
+### Migrate Data
+
+Transform data in JSONL files with validation:
+
+```bash
+npx lines-db migrate <file> <transform> [options]
+```
+
+**Example:**
+
+```bash
+# Update all ages by adding 1
+npx lines-db migrate ./data/users.jsonl "(row) => ({ ...row, age: row.age + 1 })"
+
+# Migrate with filter
+npx lines-db migrate ./data/users.jsonl "(row) => ({ ...row, active: true })" --filter "{ age: (age) => age > 18 }"
+
+# Save transformed data on error
+npx lines-db migrate ./data/users.jsonl "(row) => ({ ...row, age: row.age + 1 })" --errorOutput ./migrated.jsonl
+```
+
+**Options:**
+
+- `--filter, -f <expr>` - Filter expression to select rows
+- `--errorOutput, -e <path>` - Save transformed data to file if migration fails
+- `--verbose, -v` - Show detailed error messages
+
+The migration runs in a transaction and validates all transformed rows before committing.
+
+## TypeScript Usage
+
+### Generate Types
+
+Generate TypeScript types from your schemas for type-safe database access:
+
+```bash
+npx lines-db generate <dataDir>
+```
+
+**Example:**
+
+```bash
+# Generate types (creates ./data/db.ts by default)
+npx lines-db generate ./data
+```
+
+**Add to package.json:**
+
+```json
+"scripts": {
+  "db:validate": "lines-db validate ./data",
+  "db:generate": "lines-db generate ./data"
+}
+```
+
+### Quick Start
+
+**1. Create a JSONL file (./data/users.jsonl):**
+
+```jsonl
+{"id":1,"name":"Alice","age":30,"email":"alice@example.com"}
+{"id":2,"name":"Bob","age":25,"email":"bob@example.com"}
+{"id":3,"name":"Charlie","age":35,"email":"charlie@example.com"}
+```
+
+**2. Use in TypeScript:**
+
+```typescript
+import { LinesDB } from '@toiroakr/lines-db';
+
+const db = LinesDB.create({ dataDir: './data' });
+await db.initialize();
+
+// Find all users
+const users = db.find('users');
+console.log(users); // [{ id: 1, name: "Alice", ... }, ...]
+
+// Find a specific user
+const user = db.findOne('users', { id: 1 });
+console.log(user); // { id: 1, name: "Alice", age: 30, ... }
+
+// Find with conditions
+const adults = db.find('users', { age: (age) => age >= 30 });
+
+await db.close();
+```
+
+### Using Generated Types
+
+After running `npx lines-db generate ./data`:
+
+```typescript
+import { LinesDB } from '@toiroakr/lines-db';
+import { config } from './data/db.js';
+
+const db = LinesDB.create(config);
+await db.initialize();
+
+// ✨ Type is automatically inferred!
+const users = db.find('users');
+
+// ✨ Type-safe operations
+db.insert('users', {
+  id: 10,
+  name: 'Alice',
+  age: 30,
+  email: 'alice@example.com',
+});
+
+await db.close();
+```
+
+### Core API
+
+**Query Operations:**
+
+- `find(table, where?)` - Find all matching records
+- `findOne(table, where?)` - Find a single record
+- `query(sql, params?)` - Execute raw SQL query
+
+**Modify Operations:**
+
+- `insert(table, data)` - Insert a single record
+- `update(table, data, where)` - Update matching records
+- `delete(table, where)` - Delete matching records
+
+**Batch Operations:**
+
+- `batchInsert(table, data[])` - Insert multiple records
+- `batchUpdate(table, updates[])` - Update multiple records
+- `batchDelete(table, where)` - Delete multiple records
+
+**Transaction & Schema:**
+
+- `transaction(fn)` - Execute operations in a transaction
+- `getSchema(table)` - Get table schema
+- `getTableNames()` - Get all table names
+
+**Where Conditions:**
+
+```typescript
+// Simple equality
+db.find('users', { age: 30 });
+
+// Multiple conditions (AND)
+db.find('users', { age: 30, name: 'Alice' });
+
+// Advanced conditions
+db.find('users', {
+  age: (age) => age > 25,
+  name: (name) => name.startsWith('A'),
+});
+```
+
+### JSON Columns
+
+Objects and arrays are automatically handled as JSON columns:
+
+```typescript
+db.insert('orders', {
+  id: 1,
+  items: [{ name: 'Laptop', quantity: 1 }],
+  metadata: { source: 'web' },
+});
+
+const order = db.findOne('orders', { id: 1 });
+console.log(order.items[0].name); // "Laptop"
+```
+
+### Schema Transformations
+
+When your schema transforms data types (e.g., parsing date strings into Date objects), you need to provide a backward transformation to save data back to JSONL files.
+
+**Why?** JSONL files store strings like `"2024-01-01"`, but your app works with `Date` objects. You need to convert both ways.
+
+**Example:**
+
+```typescript
+import * as v from 'valibot';
+import { defineSchema } from '@toiroakr/lines-db';
+
+const eventSchema = v.pipe(
+  v.object({
+    id: v.number(),
+    // Transform: string → Date (when reading)
+    date: v.pipe(
+      v.string(),
+      v.isoDate(),
+      v.transform((str) => new Date(str)),
+    ),
+  }),
+);
+
+// Provide backward transformation: Date → string (when writing)
+export const schema = defineSchema(eventSchema, (output) => ({
+  ...output,
+  date: output.date.toISOString(), // Convert Date back to string
+}));
+```
+
+**In your JSONL file (events.jsonl):**
+
+```jsonl
+{
+  "id": 1,
+  "date": "2024-01-01T00:00:00.000Z"
+}
+```
+
+**In your TypeScript code:**
+
+```typescript
+const event = db.findOne('events', { id: 1 });
+console.log(event.date instanceof Date); // true
+console.log(event.date.getFullYear()); // 2024
+```
+
+### Transactions
+
+Operations outside transactions are auto-synced:
+
+```typescript
+db.insert('users', { id: 10, name: 'Alice', age: 30 });
+// ↑ Automatically synced to users.jsonl
+```
+
+Batch operations with transactions:
+
+```typescript
+await db.transaction(async (tx) => {
+  tx.insert('users', { id: 10, name: 'Alice', age: 30 });
+  tx.update('users', { age: 31 }, { id: 1 });
+  // All changes synced atomically on commit
+});
+```
+
+## Configuration
+
+```typescript
+interface DatabaseConfig {
+  dataDir: string; // Directory containing JSONL files
+}
+
+const db = LinesDB.create({ dataDir: './data' });
+```
+
+## Type Mapping
+
+| JSON Type        | Column Type | SQLite Storage |
+| ---------------- | ----------- | -------------- |
+| number (integer) | INTEGER     | INTEGER        |
+| number (float)   | REAL        | REAL           |
+| string           | TEXT        | TEXT           |
+| boolean          | INTEGER     | INTEGER        |
+| object           | JSON        | TEXT           |
+| array            | JSON        | TEXT           |
+
+## License
+
+MIT

package/assets/logo.svg

@@ -0,0 +1,57 @@
+<svg viewBox="0 0 400 400" xmlns="http://www.w3.org/2000/svg">
+  <!-- Database icon with background centered -->
+  <g transform="translate(200, 200) scale(0.24)">
+    <g transform="translate(-512, -512)">
+      <!-- Database background from original SVG -->
+      <path transform="translate(-184.32, -184.32), scale(87.04)" fill="#7ed0ec" d="M9.166.33a2.25 2.25 0 00-2.332 0l-5.25 3.182A2.25 2.25 0 00.5 5.436v5.128a2.25 2.25 0 001.084 1.924l5.25 3.182a2.25 2.25 0 002.332 0l5.25-3.182a2.25 2.25 0 001.084-1.924V5.436a2.25 2.25 0 00-1.084-1.924L9.166.33z" strokewidth="0"/>
+
+      <!-- Database layers -->
+      <path d="M117 608.4v178.5c1.5 93.7 155.7 169.5 395 169.5s393.4-75.8 395-169.5V608.4H117z" fill="#1565c0"/>
+      <path d="M907 607.7c0 99.4-154.8 180-395 180s-395-80.6-395-180 154.8-180 395-180 395 80.5 395 180z" fill="#FFFFFF"/>
+      <path d="M117 428.4v158.5c1.5 93.7 155.7 179.5 395 179.5s393.4-85.8 395-179.5V428.4H117z" fill="#1565c0"/>
+      <path d="M907 427.7c0 99.4-154.8 180-395 180s-395-80.6-395-180 154.8-180 395-180 395 80.5 395 180z" fill="#FFFFFF"/>
+      <path d="M117 248.4v158.5c1.5 93.7 155.7 179.5 395 179.5s393.4-85.8 395-179.5V248.4H117z" fill="#1565c0"/>
+      <path d="M907 247.7c0 99.4-154.8 180-395 180s-395-80.6-395-180 154.8-180 395-180 395 80.5 395 180z" fill="#1976d2"/>
+    </g>
+  </g>
+
+  <!-- LinesDB text overlapping database bottom -->
+  <g transform="translate(200, 315)">
+    <!-- White outline for text -->
+    <text x="-140" y="12" text-anchor="middle"
+          font-family="'Fira Code', 'JetBrains Mono', Consolas, monospace"
+          font-size="70" font-weight="500"
+          fill="none" stroke="#ffffff" stroke-width="5" stroke-linejoin="round">
+      {
+    </text>
+    <text x="0" y="8" text-anchor="middle"
+          font-family="'Segoe UI', 'Helvetica Neue', Arial, sans-serif"
+          font-size="56" font-weight="bold"
+          fill="none" stroke="#ffffff" stroke-width="5" stroke-linejoin="round">
+      LinesDB
+    </text>
+    <text x="140" y="12" text-anchor="middle"
+          font-family="'Fira Code', 'JetBrains Mono', Consolas, monospace"
+          font-size="70" font-weight="500"
+          fill="none" stroke="#ffffff" stroke-width="5" stroke-linejoin="round">
+      }
+    </text>
+
+    <!-- Actual text -->
+    <text x="-140" y="12" text-anchor="middle"
+          font-family="'Fira Code', 'JetBrains Mono', Consolas, monospace"
+          font-size="70" font-weight="500" fill="#999999">
+      {
+    </text>
+    <text x="0" y="8" text-anchor="middle"
+          font-family="'Segoe UI', 'Helvetica Neue', Arial, sans-serif"
+          font-size="56" font-weight="bold" fill="#444444">
+      LinesDB
+    </text>
+    <text x="140" y="12" text-anchor="middle"
+          font-family="'Fira Code', 'JetBrains Mono', Consolas, monospace"
+          font-size="70" font-weight="500" fill="#999999">
+      }
+    </text>
+  </g>
+</svg>

package/bin/cli.js

@@ -80,7 +80,7 @@ var TypeGenerator = class {
 		return `// Auto-generated by lines-db
 // Do not edit this file manually
 
-${imports.length > 0 ? `${imports.join("\n")}\n` : ""}import type { DatabaseConfig${imports.length > 0 ? ", InferOutput" : ""} } from 'lines-db';
+${imports.length > 0 ? `${imports.join("\n")}\n` : ""}import type { DatabaseConfig${imports.length > 0 ? ", InferOutput" : ""} } from '@toiroakr/lines-db';
 import { fileURLToPath } from 'node:url';
 import { dirname } from 'node:path';
 
@@ -202,6 +202,18 @@ var JsonlReader = class {
 //#endregion
 //#region src/schema-loader.ts
 var SchemaLoader = class {
+	/**
+	* Check if a schema file exists for a table
+	*/
+	static async hasSchema(jsonlPath) {
+		const schemaPath = join(dirname(jsonlPath), `${basename(jsonlPath, ".jsonl")}.schema.ts`);
+		try {
+			await access(schemaPath);
+			return true;
+		} catch {
+			return false;
+		}
+	}
 	/**
 	* Load a validation schema file for a table
 	* Requires ${tableName}.schema.ts to exist alongside the JSONL file
@@ -262,15 +274,26 @@ var Validator = class {
 		const jsonlFiles = (await readdir(dirPath, { withFileTypes: true })).filter((entry) => entry.isFile() && entry.name.endsWith(".jsonl")).map((entry) => join(dirPath, entry.name));
 		if (jsonlFiles.length === 0) throw new Error(`No JSONL files found in directory: ${dirPath}`);
 		const allErrors = [];
-		for (const file of jsonlFiles) {
+		const allWarnings = [];
+		const filesWithSchema = [];
+		for (const file of jsonlFiles) if (await SchemaLoader.hasSchema(file)) filesWithSchema.push(file);
+		else {
+			const tableName = basename(file, ".jsonl");
+			allWarnings.push(`Skipping validation for '${tableName}': schema file not found`);
+		}
+		for (const file of filesWithSchema) {
 			const result = await this.validateFile(file);
 			allErrors.push(...result.errors);
+			allWarnings.push(...result.warnings);
+		}
+		if (filesWithSchema.length > 0) {
+			const fkErrors = await this.validateForeignKeys(dirPath, filesWithSchema);
+			allErrors.push(...fkErrors);
 		}
-		const fkErrors = await this.validateForeignKeys(dirPath, jsonlFiles);
-		allErrors.push(...fkErrors);
 		return {
 			valid: allErrors.length === 0,
-			errors: allErrors
+			errors: allErrors,
+			warnings: allWarnings
 		};
 	}
 	/**
@@ -346,7 +369,8 @@ var Validator = class {
 		}
 		return {
 			valid: errors.length === 0,
-			errors
+			errors,
+			warnings: []
 		};
 	}
 };
@@ -354,8 +378,6 @@ var Validator = class {
 //#endregion
 //#region src/runtime.ts
 function detectRuntime() {
-	if (typeof globalThis !== "undefined" && "Bun" in globalThis && typeof globalThis.Bun !== "undefined") return "bun";
-	if (typeof globalThis !== "undefined" && "Deno" in globalThis && typeof globalThis.Deno !== "undefined") return "deno";
 	if (typeof process !== "undefined" && process.versions && process.versions.node) return "node";
 	return "unknown";
 }
@@ -364,43 +386,13 @@ const RUNTIME = detectRuntime();
 //#endregion
 //#region src/sqlite-adapter.ts
 /**
-* Create a SQLite database instance based on the runtime environment
+* Create a SQLite database instance for Node.js
 */
 function createDatabase(path = ":memory:") {
-	if (RUNTIME === "bun") return createBunDatabase(path);
-	else if (RUNTIME === "node" || RUNTIME === "deno") return createNodeDatabase(path);
+	if (RUNTIME === "node") return createNodeDatabase(path);
 	else throw new Error(`Unsupported runtime: ${RUNTIME}`);
 }
 /**
-* Create a Bun SQLite database
-*/
-function createBunDatabase(path) {
-	const { Database } = __require("bun:sqlite");
-	const db = new Database(path);
-	return {
-		prepare(sql) {
-			const stmt = db.prepare(sql);
-			return {
-				run(...params) {
-					return stmt.run(...params);
-				},
-				get(...params) {
-					return stmt.get(...params);
-				},
-				all(...params) {
-					return stmt.all(...params);
-				}
-			};
-		},
-		exec(sql) {
-			db.exec(sql);
-		},
-		close() {
-			db.close();
-		}
-	};
-}
-/**
 * Create a Node.js SQLite database
 */
 function createNodeDatabase(path) {
@@ -1212,7 +1204,7 @@ function runInSandbox(expression, context = {}) {
 	return runInNewContext(expression, sandbox, { timeout: 1e3 });
 }
 const program = new Command();
-program.name("lines-db").description("Database utilities for JSONL files").version("1.0.0");
+program.name("@toiroakr/lines-db").description("Database utilities for JSONL files").version("1.0.0");
 program.command("generate").description("Generate TypeScript type definitions from schema files").argument("<dataDir>", "Directory containing JSONL and schema files").action(async (dataDir) => {
 	try {
 		await new TypeGenerator({ dataDir }).generate();
@@ -1225,6 +1217,10 @@ program.command("generate").description("Generate TypeScript type definitions fr
 program.command("validate").description("Validate JSONL file(s) against schema").argument("<path>", "File or directory path to validate").option("-v, --verbose", "Show verbose error output", false).action(async (path, options) => {
 	try {
 		const result = await new Validator({ path }).validate();
+		if (result.warnings.length > 0) {
+			for (const warning of result.warnings) console.warn(styleText("yellow", `⚠ ${warning}`));
+			console.log("");
+		}
 		if (result.valid) {
 			console.log("✓ All records are valid");
 			process.exit(0);

package/dist/index.cjs

@@ -30,8 +30,6 @@ node_url = __toESM(node_url);
 
 //#region src/runtime.ts
 function detectRuntime() {
-	if (typeof globalThis !== "undefined" && "Bun" in globalThis && typeof globalThis.Bun !== "undefined") return "bun";
-	if (typeof globalThis !== "undefined" && "Deno" in globalThis && typeof globalThis.Deno !== "undefined") return "deno";
 	if (typeof process !== "undefined" && process.versions && process.versions.node) return "node";
 	return "unknown";
 }
@@ -40,43 +38,13 @@ const RUNTIME = detectRuntime();
 //#endregion
 //#region src/sqlite-adapter.ts
 /**
-* Create a SQLite database instance based on the runtime environment
+* Create a SQLite database instance for Node.js
 */
 function createDatabase(path = ":memory:") {
-	if (RUNTIME === "bun") return createBunDatabase(path);
-	else if (RUNTIME === "node" || RUNTIME === "deno") return createNodeDatabase(path);
+	if (RUNTIME === "node") return createNodeDatabase(path);
 	else throw new Error(`Unsupported runtime: ${RUNTIME}`);
 }
 /**
-* Create a Bun SQLite database
-*/
-function createBunDatabase(path) {
-	const { Database } = require("bun:sqlite");
-	const db = new Database(path);
-	return {
-		prepare(sql) {
-			const stmt = db.prepare(sql);
-			return {
-				run(...params) {
-					return stmt.run(...params);
-				},
-				get(...params) {
-					return stmt.get(...params);
-				},
-				all(...params) {
-					return stmt.all(...params);
-				}
-			};
-		},
-		exec(sql) {
-			db.exec(sql);
-		},
-		close() {
-			db.close();
-		}
-	};
-}
-/**
 * Create a Node.js SQLite database
 */
 function createNodeDatabase(path) {
@@ -214,6 +182,18 @@ var JsonlWriter = class {
 //#endregion
 //#region src/schema-loader.ts
 var SchemaLoader = class {
+	/**
+	* Check if a schema file exists for a table
+	*/
+	static async hasSchema(jsonlPath) {
+		const schemaPath = (0, node_path.join)((0, node_path.dirname)(jsonlPath), `${(0, node_path.basename)(jsonlPath, ".jsonl")}.schema.ts`);
+		try {
+			await (0, node_fs_promises.access)(schemaPath);
+			return true;
+		} catch {
+			return false;
+		}
+	}
 	/**
 	* Load a validation schema file for a table
 	* Requires ${tableName}.schema.ts to exist alongside the JSONL file
@@ -989,7 +969,7 @@ var TypeGenerator = class {
 		return `// Auto-generated by lines-db
 // Do not edit this file manually
 
-${imports.length > 0 ? `${imports.join("\n")}\n` : ""}import type { DatabaseConfig${imports.length > 0 ? ", InferOutput" : ""} } from 'lines-db';
+${imports.length > 0 ? `${imports.join("\n")}\n` : ""}import type { DatabaseConfig${imports.length > 0 ? ", InferOutput" : ""} } from '@toiroakr/lines-db';
 import { fileURLToPath } from 'node:url';
 import { dirname } from 'node:path';
 
@@ -1054,15 +1034,26 @@ var Validator = class {
 		const jsonlFiles = (await (0, node_fs_promises.readdir)(dirPath, { withFileTypes: true })).filter((entry) => entry.isFile() && entry.name.endsWith(".jsonl")).map((entry) => (0, node_path.join)(dirPath, entry.name));
 		if (jsonlFiles.length === 0) throw new Error(`No JSONL files found in directory: ${dirPath}`);
 		const allErrors = [];
-		for (const file of jsonlFiles) {
+		const allWarnings = [];
+		const filesWithSchema = [];
+		for (const file of jsonlFiles) if (await SchemaLoader.hasSchema(file)) filesWithSchema.push(file);
+		else {
+			const tableName = (0, node_path.basename)(file, ".jsonl");
+			allWarnings.push(`Skipping validation for '${tableName}': schema file not found`);
+		}
+		for (const file of filesWithSchema) {
 			const result = await this.validateFile(file);
 			allErrors.push(...result.errors);
+			allWarnings.push(...result.warnings);
+		}
+		if (filesWithSchema.length > 0) {
+			const fkErrors = await this.validateForeignKeys(dirPath, filesWithSchema);
+			allErrors.push(...fkErrors);
 		}
-		const fkErrors = await this.validateForeignKeys(dirPath, jsonlFiles);
-		allErrors.push(...fkErrors);
 		return {
 			valid: allErrors.length === 0,
-			errors: allErrors
+			errors: allErrors,
+			warnings: allWarnings
 		};
 	}
 	/**
@@ -1138,7 +1129,8 @@ var Validator = class {
 		}
 		return {
 			valid: errors.length === 0,
-			errors
+			errors,
+			warnings: []
 		};
 	}
 };