git-sqlite-vfs 0.0.2 → 0.0.6

package/README.md

@@ -1,56 +1,91 @@
 # git-sqlite-vfs
 
-A Git-Versioned SQLite Database via a Custom Virtual File System (VFS).
+A Git-Versioned SQLite Database powered by a custom native Virtual File System (VFS).
 
-This project bridges the mathematical robustness of SQLite's B-Tree engine with the distributed version control capabilities of Git, neutralizing the fundamental friction between binary databases and text-based source control.
+By combining the robustness of native SQLite, Drizzle ORM, and libSQL with the distributed tracking power of Git, `git-sqlite-vfs` enables you to version, diff, and merge your application's database exactly like your source code. It works out-of-the-box in both **Node.js** and **Deno**.
 
-## The Architecture
+## Architecture
 
-By default, standard monolithic SQLite databases undergo "cascading byte shifts" during standard operations (e.g., page splits, rebalancing). This destroys Git's ability to efficiently delta-compress the binary, causing massive repository bloat.
+Traditional SQLite databases are stored as a single flat file, making them difficult to version control because a 1-byte insertion can trigger a cascading byte shift across the entire file, rendering delta-compression useless.
 
-**The GitVFS Sharding Engine:**
-We solve this by replacing the POSIX I/O layer with a custom SQLite Virtual File System (VFS) written in C. Instead of writing to a single `.db` file, `gitvfs` dynamically shards the database into isolated, deterministic 4KB hexadecimal `.bin` pages (e.g., `.db/pages/0A/1B/0A1B2C.bin`). 
+This package dynamically loads a specialized SQLite C Extension that overrides the default VFS. It shards your database into 4KB deterministic binary pages inside a targeted directory (e.g. `.my-db`).
 
-Because changes are mathematically isolated to specific physical files, Git's `xdelta` sliding window algorithm achieves near-perfect binary compression. Operations like `VACUUM` naturally trigger `xTruncate`, unlinking dead pages and shrinking the physical directory footprint.
+When you merge branches, our custom `git-merge-sqlitevfs` driver is natively hooked into Git's conflict resolution pipeline to properly reconcile binary B-Tree page conflicts, ensuring absolute data integrity!
 
-## The Custom Merge Strategy
+## Installation
 
-Standard Git auto-merges (`ort`) operate on a file-by-file basis. Merging isolated binary pages from divergent branches silently corrupts the mathematical integrity of a B-Tree graph.
+Install the VFS package alongside your libSQL and Drizzle tools:
 
-This package provides a **Native Git Merge Strategy** (`git-merge-sqlitevfs`) that elevates the merge context from the file level to the database level. When Git encounters a branch merge, it delegates the entire operation to our C executable:
-1. `git-merge-sqlitevfs` uses `git archive` to safely reconstruct `MERGE_HEAD` and the Ancestor database states without index-lock collisions.
-2. It uses `ATTACH DATABASE` to instantly mount all three branches (Local, Remote, Ancestor) into a single unified SQLite VDBE engine.
-3. Using the `EXCEPT` operator, it calculates full-row tuples and structural DDL schema diffs instantly.
-4. It performs a true mathematically sound 3-Way Logical Merge—resolving schema evolutions, propagating insertions/deletions, and mitigating exact row-level conflicts (preferring `HEAD`)—then stages the physically reconciled `.bin` pages back to Git.
+```bash
+npm install git-sqlite-vfs @libsql/client drizzle-orm
+```
 
-## Usage
+## Git Setup
 
-Install the package via npm (requires `better-sqlite3` and `make`):
+To enable Git versioning and binary merging, you must configure your repository to use the custom merge driver. We provide a convenient CLI to wire everything up:
 
 ```bash
-npm install git-sqlite-vfs
+npx git-sqlite-setup --vfs-dir .my-db
 ```
 
-Initialize your version-controlled connection in Node.js:
+This will:
+1. Register `git-merge-sqlitevfs` as a custom Git merge driver in your local `.git/config`.
+2. Create or append to a `.gitattributes` file in your repo root to route all files in `.my-db/*` through the custom merge driver.
+
+## Usage (Isomorphic)
+
+The VFS works transparently in both Node.js and Deno. By using the `bootstrapGitVFS()` method before you initialize `@libsql/client`, the native memory spaces are perfectly synchronized.
 
-```javascript
-const GitSQLite = require('git-sqlite-vfs');
+### Using `@libsql/client` with Drizzle ORM
 
-// Configure Git optimizations and register the VFS merge driver
-GitSQLite.setupGit();
+```typescript
+import { createClient } from '@libsql/client'; // In Deno: 'npm:@libsql/client/node'
+import { drizzle } from 'drizzle-orm/libsql';
+import { sqliteTable, integer, text } from 'drizzle-orm/sqlite-core';
+import { bootstrapGitVFS } from 'git-sqlite-vfs';
 
-// Open a connection. Our Node wrapper automatically loads the C extension 
-// and routes the URI query via better-sqlite3.
-const db = GitSQLite.open('.db');
+// 1. Load the native extension process-wide and set the VFS directory
+await bootstrapGitVFS({ dir: '.my-db' });
 
-// Execute standard SQL natively
-db.exec("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT);");
-db.exec("INSERT INTO users (name) VALUES ('Alice');");
+// 2. Initialize your database connection using a standard file: URL
+const client = createClient({
+    url: 'file:.my-db/local.db'
+});
 
-const row = db.prepare("SELECT * FROM users WHERE name = ?").get('Alice');
-console.log(row.name); // 'Alice'
+// 3. Wrap with Drizzle ORM
+const db = drizzle(client);
 
-db.close();
+// 4. Define your schema
+const users = sqliteTable('users', {
+    id: integer('id').primaryKey(),
+    name: text('name')
+});
+
+// 5. Query natively! All I/O is safely intercepted by the Git VFS.
+await db.insert(users).values({ id: 1, name: 'Alice' });
+const allUsers = await db.select().from(users);
+
+console.log(allUsers);
 ```
 
-Because the underlying files are flawlessly tracked, you can seamlessly branch, commit, and `git reset --hard HEAD~1` to time travel instantly!
+## Preventing Git Bloat
+
+Because SQLite usually zeroes out deleted data pages rather than shrinking the file, you might accumulate "zombie" `.bin` pages in your repository over time. To ensure the Git VFS automatically garbage-collects these abandoned chunks, you must configure SQLite to run `FULL` auto-vacuuming and use `DELETE` journaling.
+
+Run these PRAGMAs once when initializing your database connection:
+
+```sql
+PRAGMA auto_vacuum = FULL;
+PRAGMA journal_mode = DELETE;
+```
+
+Or run `VACUUM;` periodically. When SQLite explicitly shrinks the database file, the underlying VFS `xTruncate` routine will physically `unlink()` the out-of-bounds `.bin` shards, keeping your Git tracking history perfectly compressed!
+
+## Compatibility
+
+- **Node.js**: v22.5+ (using the new `node:sqlite` API internally) or fallback to `better-sqlite3`.
+- **Deno**: Supported automatically (loads the extension dynamically via `jsr:@db/sqlite`).
+
+## License
+
+ISC

package/bin/cli.js

@@ -0,0 +1,53 @@
+#!/usr/usr/bin/env node
+
+import path from 'node:path';
+import { parseArgs } from 'node:util';
+import { configureGitIntegration } from '../index.js';
+
+const options = {
+    'repo-dir': {
+        type: 'string',
+        short: 'r',
+    },
+    'vfs-dir': {
+        type: 'string',
+        short: 'v',
+    },
+    help: {
+        type: 'boolean',
+        short: 'h',
+    },
+};
+
+const { values, positionals } = parseArgs({ options, allowPositionals: true });
+
+if (values.help) {
+    console.log(`
+Usage: git-sqlite-setup [options]
+
+Configure the current Git repository to use the git-sqlite-vfs merge driver.
+This natively intercepts merge conflicts on your SQLite B-Tree binary shards.
+
+Options:
+  -r, --repo-dir <path>   Path to the Git repository (default: current working directory)
+  -v, --vfs-dir <path>    The VFS shard directory to apply the merge driver to (default: .db)
+  -h, --help              Show this help message
+`);
+    process.exit(0);
+}
+
+const repoDir = values['repo-dir'] ? path.resolve(values['repo-dir']) : process.cwd();
+const vfsDir = values['vfs-dir'] || '.db';
+
+console.log(`Configuring Git Integration...`);
+console.log(`Repository: ${repoDir}`);
+console.log(`VFS Target Directory: ${vfsDir}`);
+
+try {
+    await configureGitIntegration({ repoDir, vfsDir });
+    console.log(`\nSuccessfully configured the SQLite VFS merge driver!`);
+    console.log(`Git will now use the custom C merge driver for conflicts inside: ${vfsDir}/*`);
+} catch (err) {
+    console.error(`\nFailed to configure Git integration:`, err.message);
+    process.exit(1);
+}

package/c/Makefile

@@ -1,38 +1,53 @@
-CC = gcc
-CFLAGS = -Wall -Wextra -g -O2 -std=c99 -D_POSIX_C_SOURCE=200809L
-LDFLAGS = -lsqlite3
+CC ?= gcc
+CFLAGS = -Wall -Wextra -g -O2 -std=c99 -D_POSIX_C_SOURCE=200809L -I.
+LDFLAGS = 
 
-# On macOS, loadable extensions must dynamically look up SQLite symbols
 UNAME_S := $(shell uname -s)
 ifeq ($(UNAME_S),Darwin)
 	SHARED_LDFLAGS = -undefined dynamic_lookup
+	EXT = dylib
+	EXE = 
+else ifeq ($(OS),Windows_NT)
+	SHARED_LDFLAGS = 
+	EXT = dll
+	EXE = .exe
 else
 	SHARED_LDFLAGS = 
+	EXT = so
+	EXE = 
 endif
 
 SRC_DIR = .
 OUT_DIR = output
 
-all: $(OUT_DIR)/gitvfs_test $(OUT_DIR)/git-merge-sqlitevfs $(OUT_DIR)/gitvfs.so
+all: $(OUT_DIR)/gitvfs_test$(EXE) $(OUT_DIR)/git-merge-sqlitevfs$(EXE) $(OUT_DIR)/gitvfs.$(EXT)
+
+$(SRC_DIR)/sqlite3.c:
+	node download-sqlite.cjs
+
+$(SRC_DIR)/sqlite3.h: $(SRC_DIR)/sqlite3.c
+$(SRC_DIR)/sqlite3ext.h: $(SRC_DIR)/sqlite3.c
+
+$(OUT_DIR)/sqlite3.o: $(SRC_DIR)/sqlite3.c $(SRC_DIR)/sqlite3.h | $(OUT_DIR)
+	$(CC) -g -O2 -c $(SRC_DIR)/sqlite3.c -o $(OUT_DIR)/sqlite3.o
 
-$(OUT_DIR)/gitvfs.so: $(SRC_DIR)/gitvfs.c $(SRC_DIR)/gitvfs.h | $(OUT_DIR)
-	$(CC) $(CFLAGS) -fPIC -shared -DCOMPILE_SQLITE_EXTENSION $(SRC_DIR)/gitvfs.c -o $(OUT_DIR)/gitvfs.so $(SHARED_LDFLAGS)
+$(OUT_DIR)/gitvfs.$(EXT): $(SRC_DIR)/gitvfs.c $(SRC_DIR)/gitvfs.h $(SRC_DIR)/sqlite3.h $(SRC_DIR)/sqlite3ext.h | $(OUT_DIR)
+	$(CC) $(CFLAGS) -fPIC -shared -DCOMPILE_SQLITE_EXTENSION $(SRC_DIR)/gitvfs.c -o $(OUT_DIR)/gitvfs.$(EXT) $(SHARED_LDFLAGS)
 
-$(OUT_DIR)/gitvfs.o: $(SRC_DIR)/gitvfs.c $(SRC_DIR)/gitvfs.h | $(OUT_DIR)
+$(OUT_DIR)/gitvfs.o: $(SRC_DIR)/gitvfs.c $(SRC_DIR)/gitvfs.h $(SRC_DIR)/sqlite3.h $(SRC_DIR)/sqlite3ext.h | $(OUT_DIR)
 	$(CC) $(CFLAGS) -c $(SRC_DIR)/gitvfs.c -o $(OUT_DIR)/gitvfs.o
 
-$(OUT_DIR)/main.o: $(SRC_DIR)/main.c $(SRC_DIR)/gitvfs.h | $(OUT_DIR)
+$(OUT_DIR)/main.o: $(SRC_DIR)/main.c $(SRC_DIR)/gitvfs.h $(SRC_DIR)/sqlite3.h | $(OUT_DIR)
 	$(CC) $(CFLAGS) -c $(SRC_DIR)/main.c -o $(OUT_DIR)/main.o
 
-$(OUT_DIR)/gitvfs_test: $(OUT_DIR)/main.o $(OUT_DIR)/gitvfs.o | $(OUT_DIR)
-	$(CC) $(CFLAGS) $(OUT_DIR)/main.o $(OUT_DIR)/gitvfs.o -o $(OUT_DIR)/gitvfs_test $(LDFLAGS)
+$(OUT_DIR)/gitvfs_test$(EXE): $(OUT_DIR)/main.o $(OUT_DIR)/gitvfs.o $(OUT_DIR)/sqlite3.o | $(OUT_DIR)
+	$(CC) $(CFLAGS) $(OUT_DIR)/main.o $(OUT_DIR)/gitvfs.o $(OUT_DIR)/sqlite3.o -o $(OUT_DIR)/gitvfs_test$(EXE) $(LDFLAGS)
 
-$(OUT_DIR)/git-merge-sqlitevfs: $(SRC_DIR)/git-merge-sqlitevfs.c $(OUT_DIR)/gitvfs.o | $(OUT_DIR)
-	$(CC) $(CFLAGS) $(SRC_DIR)/git-merge-sqlitevfs.c $(OUT_DIR)/gitvfs.o -o $(OUT_DIR)/git-merge-sqlitevfs $(LDFLAGS)
+$(OUT_DIR)/git-merge-sqlitevfs$(EXE): $(SRC_DIR)/git-merge-sqlitevfs.c $(OUT_DIR)/gitvfs.o $(OUT_DIR)/sqlite3.o $(SRC_DIR)/sqlite3.h | $(OUT_DIR)
+	$(CC) $(CFLAGS) $(SRC_DIR)/git-merge-sqlitevfs.c $(OUT_DIR)/gitvfs.o $(OUT_DIR)/sqlite3.o -o $(OUT_DIR)/git-merge-sqlitevfs$(EXE) $(LDFLAGS)
 
 $(OUT_DIR):
-	mkdir -p $(OUT_DIR)
+	node -e "const fs=require('fs'); if (!fs.existsSync('$(OUT_DIR)')) fs.mkdirSync('$(OUT_DIR)');"
 
 clean:
-	rm -rf $(OUT_DIR)
-	rm -rf .db .git
+	node -e "const fs=require('fs'); fs.rmSync('$(OUT_DIR)', {recursive: true, force: true}); fs.rmSync('.db', {recursive: true, force: true}); fs.rmSync('.git', {recursive: true, force: true}); fs.rmSync('sqlite3.c', {force: true}); fs.rmSync('sqlite3.h', {force: true}); fs.rmSync('sqlite3ext.h', {force: true});"

package/c/download-sqlite.cjs

@@ -0,0 +1,17 @@
+const fs = require('fs');
+const path = require('path');
+const { execSync } = require('child_process');
+
+if (!fs.existsSync('sqlite3.c') || !fs.existsSync('sqlite3.h') || !fs.existsSync('sqlite3ext.h')) {
+    console.log('Downloading SQLite source...');
+    try {
+        execSync('curl -L -O https://www.sqlite.org/2024/sqlite-autoconf-3450200.tar.gz', { stdio: 'inherit' });
+        execSync('tar -xzf sqlite-autoconf-3450200.tar.gz', { stdio: 'inherit' });
+        fs.copyFileSync('sqlite-autoconf-3450200/sqlite3.c', 'sqlite3.c');
+        fs.copyFileSync('sqlite-autoconf-3450200/sqlite3.h', 'sqlite3.h');
+        fs.copyFileSync('sqlite-autoconf-3450200/sqlite3ext.h', 'sqlite3ext.h');
+    } catch (e) {
+        console.error('Failed to download or extract SQLite:', e.message);
+        process.exit(1);
+    }
+}