gh-as-db 1.1.1 → 1.3.0

package/README.md

@@ -11,14 +11,19 @@ Use a private GitHub repository as a database for your application. `gh-as-db` p
 - 🔐 **Secure**: Designed for private repositories using Personal Access Tokens (PAT).
 - 🚀 **Performance**: Built-in in-memory caching and auto-indexing for fast local queries.
 - 🛡️ **Concurrency**: Optimistic locking using Git SHAs to prevent data loss.
+- 🔄 **Retry & Rate Limits**: Automatic retries with exponential backoff for transient errors and GitHub rate limits.
+- 🔗 **Transactions**: Group multiple operations into a single atomic Git commit.
+- 📁 **Sharding**: One-file-per-document storage strategy for massive collections.
 - 🧩 **Middleware**: Extensible hooks for data validation or transformation.
+- 🔀 **Validation**: Pluggable schema validation support (Zod, etc.).
+- 🌐 **Edge Ready**: Fully compatible with Vercel Edge and Cloudflare Workers.
 - ⌨️ **CLI Tool**: Initialize and manage your "database" from the terminal.
 - 📦 **TypeScript**: Fully typed for a great developer experience.
 
 ## Installation
 
 ```bash
-npm install gh-as-db
+pnpm add gh-as-db
 ```
 
 ## Quick Start
@@ -58,6 +63,15 @@ const results = await users.find({
     { field: 'name', operator: 'eq', value: 'John Doe' }
   ]
 });
+
+// Use Transactions for atomic multi-collection updates
+await db.transaction(async (tx) => {
+  const usersTx = tx.collection('users');
+  const logsTx = tx.collection('logs');
+  
+  await usersTx.update('1', { name: 'Updated Name' });
+  await logsTx.create({ id: 'log-1', action: 'Update user' });
+}, 'Atomic update of user and logs');
 ```
 
 ## API Reference
@@ -72,6 +86,7 @@ const db = new GitHubDB({
   owner: string;       // Repository owner
   repo: string;        // Repository name
   cacheTTL?: number;   // Optional: Cache TTL in ms. Default is 0 (strict consistency).
+  retry?: RetryConfig | false; // Optional: Retry config, or false to disable. See below.
 });
 ```
 
@@ -131,6 +146,98 @@ const users = db.collection<User>('users', {
 });
 ```
 
+### Schema Validation
+
+You can use any validation library (Zod, Valibot, etc.) by providing a `validator` object.
+
+```typescript
+import { z } from 'zod';
+
+const userSchema = z.object({
+  id: z.string(),
+  name: z.string().min(3),
+  email: z.string().email(),
+});
+
+const users = db.collection<User>('users', {
+  validator: {
+    validate: (data) => userSchema.parseAsync(data),
+  }
+});
+```
+
+### Transactions & Batching
+
+Transactions allow you to group multiple operations across different collections into a **single Git commit**. This reduces API calls and ensures that either all operations succeed or none are committed.
+
+```typescript
+const commitSha = await db.transaction(async (tx) => {
+  const posts = tx.collection('posts');
+  const count = tx.collection('stats');
+
+  await posts.create({ id: 'p1', title: 'New Post' });
+  await count.update('total', { value: 101 });
+}, 'Create post and increment counter');
+```
+
+### Storage Strategies (Sharding)
+
+By default, `gh-as-db` stores the entire collection in a single JSON file (`name.json`). For large collections, you can use the `sharded` strategy, which stores **one file per document** (`name/id.json`).
+
+```typescript
+const users = db.collection<User>('users', {
+  strategy: 'sharded'
+});
+```
+
+**Why use Sharding?**
+- 🚀 **Performance**: `findById` reads only the specific file, which is much faster than loading a massive JSON array.
+- 📈 **Scalability**: Avoids GitHub's file size limits and reduces merge conflicts.
+- 🧹 **Cleanliness**: Better organization for repositories with thousands of documents.
+
+### Retry & Rate Limit Handling
+
+All GitHub API calls are automatically retried on transient errors (429 rate limits, 500/502/503 server errors) with exponential backoff. If GitHub returns a `Retry-After` header, it is respected.
+
+```typescript
+const db = new GitHubDB({
+  accessToken: process.env.GITHUB_TOKEN,
+  owner: 'your-username',
+  repo: 'my-data-repo',
+  retry: {
+    maxRetries: 3,   // Default: 3
+    baseDelay: 1000, // Default: 1000ms
+    maxDelay: 10000, // Default: 10000ms
+  }
+});
+```
+
+To disable retries entirely:
+
+```typescript
+const db = new GitHubDB({
+  // ...
+  retry: false,
+});
+```
+
+Non-transient errors (401, 403, 404) are thrown immediately without retrying. Concurrency conflicts (409) still throw `ConcurrencyError` immediately. If rate limit retries are exhausted, a `RateLimitError` is thrown.
+
+```typescript
+import { RateLimitError, ConcurrencyError } from 'gh-as-db';
+
+try {
+  await users.create({ id: '1', name: 'Alice' });
+} catch (error) {
+  if (error instanceof RateLimitError) {
+    console.log('Rate limited, retry after:', error.retryAfter);
+  }
+  if (error instanceof ConcurrencyError) {
+    console.log('Conflict, re-read and retry');
+  }
+}
+```
+
 ## CLI Usage
 
 `gh-as-db` comes with a CLI tool to help you manage your repository.
@@ -159,6 +266,7 @@ For small projects, side-projects, or internal tools, setting up a database serv
 - **Write-Through**: Updates the local cache immediately after a write, preventing 404s during redirects.
 - **Indexing**: Automatic in-memory indexing on all fields makes querying fast even as data grows.
 - **Optimistic Concurrency**: Uses Git SHAs to ensure that you don't overwrite changes made by another client.
+- **Automatic Retries**: Transient failures and rate limits are handled transparently with exponential backoff.
 
 ## License
 

package/dist/core/types.d.ts

@@ -1,8 +1,15 @@
+export interface RetryConfig {
+    maxRetries?: number;
+    baseDelay?: number;
+    maxDelay?: number;
+}
 export interface GitHubDBConfig {
     accessToken: string;
     owner: string;
     repo: string;
+    branch?: string;
     cacheTTL?: number;
+    retry?: RetryConfig | false;
 }
 export type Schema = Record<string, any>;
 export type MiddlewareOperation = "create" | "update" | "read" | "delete";
@@ -38,13 +45,32 @@ export declare class ConcurrencyError extends Error {
     readonly path: string;
     constructor(path: string);
 }
+export declare class RateLimitError extends Error {
+    readonly retryAfter?: number | undefined;
+    constructor(retryAfter?: number | undefined);
+}
 export interface StorageResponse<T> {
     data: T;
     sha: string;
 }
+export interface Validator<T> {
+    validate: (data: unknown) => Promise<T> | T;
+}
+export interface CommitChange {
+    path: string;
+    content: any;
+}
+export type StorageStrategy = "single-file" | "sharded";
 export interface IStorageProvider {
     testConnection(): Promise<boolean>;
     exists(path: string): Promise<boolean>;
     readJson<T>(path: string): Promise<StorageResponse<T>>;
     writeJson<T>(path: string, content: T, message: string, sha?: string): Promise<string>;
+    deleteFile(path: string, message: string, sha: string): Promise<void>;
+    listDirectory(path: string): Promise<{
+        path: string;
+        sha: string;
+        type: "file" | "dir";
+    }[]>;
+    commit(changes: CommitChange[], message: string): Promise<string>;
 }

package/dist/core/types.js

@@ -6,3 +6,11 @@ export class ConcurrencyError extends Error {
         this.name = "ConcurrencyError";
     }
 }
+export class RateLimitError extends Error {
+    retryAfter;
+    constructor(retryAfter) {
+        super(`GitHub API rate limit exceeded${retryAfter ? `. Retry after ${retryAfter}s` : ""}`);
+        this.retryAfter = retryAfter;
+        this.name = "RateLimitError";
+    }
+}

package/dist/index.d.ts

@@ -1,4 +1,5 @@
 export { GitHubDB } from "./ui/github-db.js";
 export { Collection } from "./ui/collection.js";
+export { Transaction } from "./ui/transaction.js";
 export * from "./core/types.js";
-export declare const version = "1.0.1";
+export declare const version = "1.3.0";

package/dist/index.js

@@ -1,4 +1,5 @@
 export { GitHubDB } from "./ui/github-db.js";
 export { Collection } from "./ui/collection.js";
+export { Transaction } from "./ui/transaction.js";
 export * from "./core/types.js";
-export const version = "1.0.1";
+export const version = "1.3.0";

package/dist/infrastructure/github-storage.d.ts

@@ -1,4 +1,4 @@
-import { GitHubDBConfig, IStorageProvider, StorageResponse } from "../core/types.js";
+import { GitHubDBConfig, IStorageProvider, StorageResponse, CommitChange } from "../core/types.js";
 import { ICacheProvider } from "./cache-provider.js";
 export declare class GitHubStorageProvider implements IStorageProvider {
     private config;
@@ -7,8 +7,16 @@ export declare class GitHubStorageProvider implements IStorageProvider {
     private staleCache;
     private readonly DEFAULT_TTL;
     constructor(config: GitHubDBConfig, cache?: ICacheProvider);
+    private retryWithBackoff;
     testConnection(): Promise<boolean>;
     exists(path: string): Promise<boolean>;
     readJson<T>(path: string): Promise<StorageResponse<T>>;
     writeJson<T>(path: string, content: T, message: string, sha?: string): Promise<string>;
+    commit(changes: CommitChange[], message: string): Promise<string>;
+    deleteFile(path: string, message: string, sha: string): Promise<void>;
+    listDirectory(path: string): Promise<{
+        path: string;
+        sha: string;
+        type: "file" | "dir";
+    }[]>;
 }

package/dist/infrastructure/github-storage.js

@@ -1,5 +1,5 @@
 import { Octokit } from "@octokit/rest";
-import { ConcurrencyError, } from "../core/types.js";
+import { ConcurrencyError, RateLimitError, } from "../core/types.js";
 import { MemoryCacheProvider } from "./cache-provider.js";
 export class GitHubStorageProvider {
     config;
@@ -14,12 +14,58 @@ export class GitHubStorageProvider {
         });
         this.cache = cache || new MemoryCacheProvider();
     }
+    async retryWithBackoff(fn) {
+        if (this.config.retry === false)
+            return fn();
+        const { maxRetries = 3, baseDelay = 1000, maxDelay = 10000, } = this.config.retry ?? {};
+        let lastError;
+        for (let attempt = 0; attempt <= maxRetries; attempt++) {
+            try {
+                return await fn();
+            }
+            catch (error) {
+                lastError = error;
+                const status = error?.status;
+                // Non-retryable: 409 (concurrency), 4xx client errors (except 429)
+                if (status === 409)
+                    throw error;
+                if (status && status < 500 && status !== 429)
+                    throw error;
+                // Errors without a status code (non-HTTP errors) — rethrow
+                if (!status)
+                    throw error;
+                // Last attempt — don't wait, just break
+                if (attempt === maxRetries)
+                    break;
+                // Calculate delay
+                let delay;
+                if (status === 429 && error.response?.headers?.["retry-after"]) {
+                    delay =
+                        parseInt(error.response.headers["retry-after"], 10) * 1000;
+                }
+                else {
+                    delay = baseDelay * Math.pow(2, attempt);
+                    // Add jitter to avoid thundering herd on simultaneous retries
+                    delay = delay * (0.5 + Math.random() * 0.5);
+                }
+                // Cap all delays (including Retry-After) against maxDelay
+                delay = Math.min(delay, maxDelay);
+                await new Promise((resolve) => setTimeout(resolve, delay));
+            }
+        }
+        // Exhausted retries on 429 → RateLimitError
+        if (lastError.status === 429) {
+            const retryAfter = lastError.response?.headers?.["retry-after"];
+            throw new RateLimitError(retryAfter ? parseInt(retryAfter, 10) : undefined);
+        }
+        throw lastError;
+    }
     async testConnection() {
         try {
-            await this.octokit.repos.get({
+            await this.retryWithBackoff(() => this.octokit.repos.get({
                 owner: this.config.owner,
                 repo: this.config.repo,
-            });
+            }));
             return true;
         }
         catch (error) {
@@ -31,11 +77,11 @@ export class GitHubStorageProvider {
             return true;
         }
         try {
-            await this.octokit.repos.getContent({
+            await this.retryWithBackoff(() => this.octokit.repos.getContent({
                 owner: this.config.owner,
                 repo: this.config.repo,
                 path,
-            });
+            }));
             return true;
         }
         catch (error) {
@@ -54,19 +100,25 @@ export class GitHubStorageProvider {
         // 2. Check stale cache for conditional request
         const stale = this.staleCache.get(path);
         try {
-            const response = await this.octokit.repos.getContent({
+            const response = await this.retryWithBackoff(() => this.octokit.repos.getContent({
                 owner: this.config.owner,
                 repo: this.config.repo,
                 path,
                 headers: stale ? { "if-none-match": `"${stale.sha}"` } : {},
-            });
+            }));
             if (Array.isArray(response.data)) {
                 throw new Error("Path is a directory, not a file");
             }
             if (!("content" in response.data) || !("sha" in response.data)) {
                 throw new Error("No content or SHA in response");
             }
-            const content = Buffer.from(response.data.content, "base64").toString("utf-8");
+            const binary = atob(response.data.content.replace(/\s/g, ""));
+            const bytes = new Uint8Array(binary.length);
+            for (let i = 0; i < binary.length; i++) {
+                bytes[i] = binary.charCodeAt(i);
+            }
+            const decoder = new TextDecoder("utf-8");
+            const content = decoder.decode(bytes);
             const result = {
                 data: JSON.parse(content),
                 sha: response.data.sha,
@@ -96,11 +148,11 @@ export class GitHubStorageProvider {
             }
             else {
                 try {
-                    const existing = await this.octokit.repos.getContent({
+                    const existing = await this.retryWithBackoff(() => this.octokit.repos.getContent({
                         owner: this.config.owner,
                         repo: this.config.repo,
                         path,
-                    });
+                    }));
                     if (!Array.isArray(existing.data) && "sha" in existing.data) {
                         internalSha = existing.data.sha;
                     }
@@ -112,15 +164,20 @@ export class GitHubStorageProvider {
                 }
             }
         }
+        // Note: internalSha is resolved before the retry wrapper. If a write fails
+        // with a 5xx after GitHub has already committed the file (e.g. network timeout),
+        // the retry will reuse the stale sha and may fail with a 409.
         try {
-            const response = await this.octokit.repos.createOrUpdateFileContents({
+            const response = await this.retryWithBackoff(() => this.octokit.repos.createOrUpdateFileContents({
                 owner: this.config.owner,
                 repo: this.config.repo,
                 path,
                 message,
-                content: Buffer.from(JSON.stringify(content, null, 2)).toString("base64"),
+                content: btoa(Array.from(new TextEncoder().encode(JSON.stringify(content, null, 2)))
+                    .map((b) => String.fromCharCode(b))
+                    .join("")),
                 sha: internalSha,
-            });
+            }));
             const newSha = response.data.content?.sha || "";
             const result = { data: content, sha: newSha };
             const ttl = this.config.cacheTTL ?? this.DEFAULT_TTL;
@@ -135,4 +192,117 @@ export class GitHubStorageProvider {
             throw error;
         }
     }
+    async commit(changes, message) {
+        if (changes.length === 0) {
+            throw new Error("No changes to commit");
+        }
+        const branch = this.config.branch || "main";
+        try {
+            // 1. Get the current head SHA
+            const { data: ref } = await this.retryWithBackoff(() => this.octokit.git.getRef({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                ref: `heads/${branch}`,
+            }));
+            const latestCommitSha = ref.object.sha;
+            // 2. Get the tree SHA of the latest commit
+            const { data: latestCommit } = await this.retryWithBackoff(() => this.octokit.git.getCommit({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                commit_sha: latestCommitSha,
+            }));
+            const baseTreeSha = latestCommit.tree.sha;
+            // 3. Create a new tree with multiple files
+            const treeEntries = changes.map((change) => {
+                if (change.content === null) {
+                    return {
+                        path: change.path,
+                        mode: "100644",
+                        type: "blob",
+                        sha: null,
+                    };
+                }
+                return {
+                    path: change.path,
+                    mode: "100644",
+                    type: "blob",
+                    content: JSON.stringify(change.content, null, 2),
+                };
+            });
+            const { data: newTree } = await this.retryWithBackoff(() => this.octokit.git.createTree({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                base_tree: baseTreeSha,
+                tree: treeEntries,
+            }));
+            // 4. Create a new commit
+            const { data: newCommit } = await this.retryWithBackoff(() => this.octokit.git.createCommit({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                message,
+                tree: newTree.sha,
+                parents: [latestCommitSha],
+            }));
+            // 5. Update the reference
+            await this.retryWithBackoff(() => this.octokit.git.updateRef({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                ref: `heads/${branch}`,
+                sha: newCommit.sha,
+            }));
+            // 6. Update cache for all involved files
+            const ttl = this.config.cacheTTL ?? this.DEFAULT_TTL;
+            for (const entry of newTree.tree) {
+                if (entry.path && entry.sha) {
+                    const change = changes.find((c) => c.path === entry.path);
+                    if (change) {
+                        const result = { data: change.content, sha: entry.sha };
+                        this.cache.set(entry.path, result, ttl);
+                        this.staleCache.set(entry.path, result);
+                    }
+                }
+            }
+            return newCommit.sha;
+        }
+        catch (error) {
+            if (error.status === 409) {
+                throw new ConcurrencyError("batch-commit");
+            }
+            throw error;
+        }
+    }
+    async deleteFile(path, message, sha) {
+        await this.retryWithBackoff(() => this.octokit.repos.deleteFile({
+            owner: this.config.owner,
+            repo: this.config.repo,
+            path,
+            message,
+            sha,
+        }));
+        this.cache.delete(path);
+        this.staleCache.delete(path);
+    }
+    async listDirectory(path) {
+        try {
+            const response = await this.retryWithBackoff(() => this.octokit.repos.getContent({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                path,
+            }));
+            if (!Array.isArray(response.data)) {
+                throw new Error("Path is not a directory");
+            }
+            return response.data.map((item) => ({
+                path: item.path,
+                sha: item.sha,
+                type: item.type,
+            }));
+        }
+        catch (error) {
+            if (error.status === 404) {
+                return [];
+            }
+            throw error;
+        }
+    }
 }

package/dist/infrastructure/transaction-storage.d.ts

@@ -0,0 +1,25 @@
+import { CommitChange, IStorageProvider, StorageResponse } from "../core/types.js";
+/**
+ * A decorator for IStorageProvider that buffers writes in memory.
+ * Used during transactions to group multiple operations into a single commit.
+ */
+export declare class TransactionStorageProvider implements IStorageProvider {
+    private readonly baseStorage;
+    private pendingChanges;
+    constructor(baseStorage: IStorageProvider);
+    testConnection(): Promise<boolean>;
+    exists(path: string): Promise<boolean>;
+    readJson<T>(path: string): Promise<StorageResponse<T>>;
+    writeJson<T>(path: string, content: T, _message: string, _sha?: string): Promise<string>;
+    deleteFile(path: string, _message: string, _sha: string): Promise<void>;
+    listDirectory(path: string): Promise<{
+        path: string;
+        sha: string;
+        type: "file" | "dir";
+    }[]>;
+    commit(changes: CommitChange[], message: string): Promise<string>;
+    /**
+     * Returns all pending changes in this transaction.
+     */
+    getChanges(): CommitChange[];
+}

package/dist/infrastructure/transaction-storage.js

@@ -0,0 +1,54 @@
+/**
+ * A decorator for IStorageProvider that buffers writes in memory.
+ * Used during transactions to group multiple operations into a single commit.
+ */
+export class TransactionStorageProvider {
+    baseStorage;
+    pendingChanges = new Map();
+    constructor(baseStorage) {
+        this.baseStorage = baseStorage;
+    }
+    async testConnection() {
+        return this.baseStorage.testConnection();
+    }
+    async exists(path) {
+        if (this.pendingChanges.has(path)) {
+            return true;
+        }
+        return this.baseStorage.exists(path);
+    }
+    async readJson(path) {
+        const pending = this.pendingChanges.get(path);
+        if (pending !== undefined) {
+            return {
+                data: pending,
+                sha: "transaction-pending-sha",
+            };
+        }
+        return this.baseStorage.readJson(path);
+    }
+    async writeJson(path, content, _message, _sha) {
+        this.pendingChanges.set(path, content);
+        return "transaction-pending-sha";
+    }
+    async deleteFile(path, _message, _sha) {
+        this.pendingChanges.set(path, null);
+    }
+    async listDirectory(path) {
+        // For now, we delegate to base storage.
+        // In sharded mode, we might need to merge with pending changes.
+        return this.baseStorage.listDirectory(path);
+    }
+    async commit(changes, message) {
+        return this.baseStorage.commit(changes, message);
+    }
+    /**
+     * Returns all pending changes in this transaction.
+     */
+    getChanges() {
+        return Array.from(this.pendingChanges.entries()).map(([path, content]) => ({
+            path,
+            content,
+        }));
+    }
+}

package/dist/ui/collection.d.ts

@@ -1,14 +1,22 @@
-import { IStorageProvider, Middleware, QueryOptions, Schema } from "../core/types.js";
+import { IStorageProvider, Middleware, QueryOptions, Schema, StorageStrategy, Validator } from "../core/types.js";
 export declare class Collection<T extends Schema> {
     readonly name: string;
     private readonly storage;
-    private readonly middleware;
     private lastSha;
     private indexer;
     private items;
     private dataLoaded;
-    constructor(name: string, storage: IStorageProvider, middleware?: Middleware<T>[]);
+    private middleware;
+    private validator?;
+    private strategy;
+    private shas;
+    constructor(name: string, storage: IStorageProvider, middlewareOrOptions?: Middleware<T>[] | {
+        middleware?: Middleware<T>[];
+        validator?: Validator<T>;
+        strategy?: StorageStrategy;
+    });
     private get path();
+    private getItemPath;
     create(item: T): Promise<T>;
     find(queryOrPredicate?: ((item: T) => boolean) | QueryOptions<T>): Promise<T[]>;
     private applyQueryOptions;

package/dist/ui/collection.js

@@ -3,21 +3,42 @@ import { Indexer } from "../core/indexer.js";
 export class Collection {
     name;
     storage;
-    middleware;
     lastSha;
     indexer = new Indexer();
     items = [];
     dataLoaded = false;
-    constructor(name, storage, middleware = []) {
+    middleware;
+    validator;
+    strategy;
+    shas = new Map(); // path -> sha for sharded mode
+    constructor(name, storage, middlewareOrOptions) {
         this.name = name;
         this.storage = storage;
-        this.middleware = middleware;
+        if (Array.isArray(middlewareOrOptions)) {
+            this.middleware = middlewareOrOptions;
+            this.strategy = "single-file";
+        }
+        else {
+            this.middleware = middlewareOrOptions?.middleware || [];
+            this.validator = middlewareOrOptions?.validator;
+            this.strategy = middlewareOrOptions?.strategy || "single-file";
+        }
     }
     get path() {
-        return `${this.name}.json`;
+        return this.strategy === "sharded" ? `${this.name}/` : `${this.name}.json`;
+    }
+    getItemPath(id) {
+        return this.strategy === "sharded"
+            ? `${this.name}/${id}.json`
+            : `${this.name}.json`;
     }
     async create(item) {
         let finalItem = item;
+        // 1. Validation
+        if (this.validator) {
+            finalItem = await this.validator.validate(finalItem);
+        }
+        // 2. Middleware
         const context = {
             collection: this.name,
             operation: "create",
@@ -27,6 +48,22 @@ export class Collection {
                 finalItem = await mw.beforeSave(finalItem, context);
             }
         }
+        if (this.strategy === "sharded") {
+            const itemPath = this.getItemPath(finalItem.id);
+            const sha = await this.storage.writeJson(itemPath, finalItem, `Create item ${finalItem.id} in ${this.name}`, this.shas.get(itemPath));
+            this.shas.set(itemPath, sha);
+            if (this.dataLoaded) {
+                const index = this.items.findIndex((i) => i.id === finalItem.id);
+                if (index > -1) {
+                    this.items[index] = finalItem;
+                }
+                else {
+                    this.items.push(finalItem);
+                }
+                this.indexer.add(finalItem);
+            }
+            return finalItem;
+        }
         let items = [];
         if (this.dataLoaded) {
             items = [...this.items];
@@ -92,9 +129,21 @@ export class Collection {
             }
             return this.items;
         }
-        const response = await this.storage.readJson(this.path);
-        this.lastSha = response.sha;
-        items = response.data;
+        if (this.strategy === "sharded") {
+            const files = await this.storage.listDirectory(this.name);
+            items = await Promise.all(files
+                .filter((f) => f.type === "file" && f.path.endsWith(".json"))
+                .map(async (file) => {
+                const response = await this.storage.readJson(file.path);
+                this.shas.set(file.path, response.sha);
+                return response.data;
+            }));
+        }
+        else {
+            const response = await this.storage.readJson(this.path);
+            this.lastSha = response.sha;
+            items = response.data;
+        }
         items = await Promise.all(items.map(async (item) => {
             let currentItem = item;
             for (const mw of this.middleware) {
@@ -177,6 +226,31 @@ export class Collection {
             const results = this.indexer.query("id", id);
             return results && results.length > 0 ? results[0] : null;
         }
+        if (this.strategy === "sharded") {
+            try {
+                const itemPath = this.getItemPath(id);
+                const response = await this.storage.readJson(itemPath);
+                this.shas.set(itemPath, response.sha);
+                let finalItem = response.data;
+                const context = {
+                    collection: this.name,
+                    operation: "read",
+                };
+                for (const mw of this.middleware) {
+                    if (mw.afterRead) {
+                        finalItem = await mw.afterRead(finalItem, context);
+                    }
+                }
+                // If data is not fully loaded, we don't update this.items yet to avoid partial consistency
+                // But if someone calls find() later it will load everything.
+                return finalItem;
+            }
+            catch (error) {
+                if (error.status === 404)
+                    return null;
+                throw error;
+            }
+        }
         const items = await this.find();
         return items.find((item) => item.id === id) || null;
     }
@@ -189,6 +263,11 @@ export class Collection {
         const originalItem = items[index];
         items[index] = { ...items[index], ...updates };
         let finalItem = items[index];
+        // 1. Validation
+        if (this.validator) {
+            finalItem = await this.validator.validate(finalItem);
+        }
+        // 2. Middleware
         const context = {
             collection: this.name,
             operation: "update",
@@ -199,14 +278,29 @@ export class Collection {
             }
         }
         items[index] = finalItem;
-        try {
-            this.lastSha = await this.storage.writeJson(this.path, items, `Update item ${id} in ${this.name}`, this.lastSha);
+        if (this.strategy === "sharded") {
+            const itemPath = this.getItemPath(id);
+            try {
+                const sha = await this.storage.writeJson(itemPath, finalItem, `Update item ${id} in ${this.name}`, this.shas.get(itemPath));
+                this.shas.set(itemPath, sha);
+            }
+            catch (error) {
+                if (error instanceof ConcurrencyError) {
+                    this.dataLoaded = false;
+                }
+                throw error;
+            }
         }
-        catch (error) {
-            if (error instanceof ConcurrencyError) {
-                this.dataLoaded = false;
+        else {
+            try {
+                this.lastSha = await this.storage.writeJson(this.path, items, `Update item ${id} in ${this.name}`, this.lastSha);
+            }
+            catch (error) {
+                if (error instanceof ConcurrencyError) {
+                    this.dataLoaded = false;
+                }
+                throw error;
             }
-            throw error;
         }
         if (this.dataLoaded) {
             this.items = items;
@@ -222,14 +316,29 @@ export class Collection {
         }
         const itemToDelete = items[index];
         const filtered = items.filter((_, i) => i !== index);
-        try {
-            this.lastSha = await this.storage.writeJson(this.path, filtered, `Delete item ${id} from ${this.name}`, this.lastSha);
+        if (this.strategy === "sharded") {
+            const itemPath = this.getItemPath(id);
+            const sha = this.shas.get(itemPath);
+            if (!sha) {
+                // We need the SHA to delete. If we don't have it, we must fetch it.
+                const response = await this.storage.readJson(itemPath);
+                await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, response.sha);
+            }
+            else {
+                await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, sha);
+            }
+            this.shas.delete(itemPath);
         }
-        catch (error) {
-            if (error instanceof ConcurrencyError) {
-                this.dataLoaded = false;
+        else {
+            try {
+                this.lastSha = await this.storage.writeJson(this.path, filtered, `Delete item ${id} from ${this.name}`, this.lastSha);
+            }
+            catch (error) {
+                if (error instanceof ConcurrencyError) {
+                    this.dataLoaded = false;
+                }
+                throw error;
             }
-            throw error;
         }
         if (this.dataLoaded) {
             this.items = filtered;

package/dist/ui/github-db.d.ts

@@ -1,5 +1,6 @@
-import { GitHubDBConfig, IStorageProvider, Middleware, Schema } from "../core/types.js";
+import { GitHubDBConfig, IStorageProvider, Middleware, Schema, StorageStrategy, Validator } from "../core/types.js";
 import { Collection } from "./collection.js";
+import { Transaction } from "./transaction.js";
 export declare class GitHubDB {
     readonly config: GitHubDBConfig;
     readonly storage: IStorageProvider;
@@ -7,5 +8,8 @@ export declare class GitHubDB {
     connect(): Promise<boolean>;
     collection<T extends Schema>(name: string, options?: {
         middleware?: Middleware<T>[];
+        validator?: Validator<T>;
+        strategy?: StorageStrategy;
     }): Collection<T>;
+    transaction(fn: (tx: Transaction) => Promise<void>, message?: string): Promise<string>;
 }

package/dist/ui/github-db.js

@@ -1,5 +1,6 @@
 import { GitHubStorageProvider } from "../infrastructure/github-storage.js";
 import { Collection } from "./collection.js";
+import { Transaction } from "./transaction.js";
 export class GitHubDB {
     config;
     storage;
@@ -20,6 +21,11 @@ export class GitHubDB {
         return this.storage.testConnection();
     }
     collection(name, options = {}) {
-        return new Collection(name, this.storage, options.middleware);
+        return new Collection(name, this.storage, options);
+    }
+    async transaction(fn, message = "Transaction commit") {
+        const tx = new Transaction(this.storage);
+        await fn(tx);
+        return tx.commit(message);
     }
 }

package/dist/ui/transaction.d.ts

@@ -0,0 +1,23 @@
+import { IStorageProvider, Middleware, Schema, Validator } from "../core/types.js";
+import { Collection } from "./collection.js";
+/**
+ * Handles atomic batch operations across multiple collections.
+ */
+export declare class Transaction {
+    private readonly baseStorage;
+    private txStorage;
+    constructor(baseStorage: IStorageProvider);
+    /**
+     * Returns a collection instance bound to this transaction.
+     * Operations on this collection will be buffered until the transaction is committed.
+     */
+    collection<T extends Schema>(name: string, options?: {
+        middleware?: Middleware<T>[];
+        validator?: Validator<T>;
+    }): Collection<T>;
+    /**
+     * Commits all buffered changes in the transaction to GitHub in a single commit.
+     * @internal
+     */
+    commit(message: string): Promise<string>;
+}

package/dist/ui/transaction.js

@@ -0,0 +1,31 @@
+import { TransactionStorageProvider } from "../infrastructure/transaction-storage.js";
+import { Collection } from "./collection.js";
+/**
+ * Handles atomic batch operations across multiple collections.
+ */
+export class Transaction {
+    baseStorage;
+    txStorage;
+    constructor(baseStorage) {
+        this.baseStorage = baseStorage;
+        this.txStorage = new TransactionStorageProvider(baseStorage);
+    }
+    /**
+     * Returns a collection instance bound to this transaction.
+     * Operations on this collection will be buffered until the transaction is committed.
+     */
+    collection(name, options = {}) {
+        return new Collection(name, this.txStorage, options);
+    }
+    /**
+     * Commits all buffered changes in the transaction to GitHub in a single commit.
+     * @internal
+     */
+    async commit(message) {
+        const changes = this.txStorage.getChanges();
+        if (changes.length === 0) {
+            return "";
+        }
+        return this.txStorage.commit(changes, message);
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "gh-as-db",
-  "version": "1.1.1",
+  "version": "1.3.0",
   "description": "Use a private GitHub repository as a database for your application.",
   "type": "module",
   "main": "./dist/index.js",
@@ -20,12 +20,6 @@
     "README.md",
     "LICENSE"
   ],
-  "scripts": {
-    "test": "vitest",
-    "test:run": "vitest run",
-    "build": "tsc",
-    "prepublishOnly": "npm run build"
-  },
   "keywords": [
     "github",
     "database",
@@ -60,5 +54,10 @@
     "chalk": "^5.4.1",
     "commander": "^13.1.0",
     "enquirer": "^2.4.1"
+  },
+  "scripts": {
+    "test": "vitest",
+    "test:run": "vitest run",
+    "build": "tsc"
   }
-}
+}