gh-as-db 1.3.0 → 2.0.0

package/CHANGELOG.md

@@ -0,0 +1,51 @@
+# Changelog
+
+## [2.0.0] — 2026-07-17
+
+### Breaking Changes
+
+- **`transaction()` return type**: `Promise<string>` → `Promise<string | null>`. Empty transactions now return `null` instead of `""`. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Collection name validation**: Names must match `[A-Za-z0-9._-]` (max 255 chars). Names with spaces, slashes, or special characters now throw. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Item ID validation**: IDs are validated on `create()`, `findById()`, `update()`, and `delete()`. IDs must match `[A-Za-z0-9._-]` (max 255 chars). ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Unknown filter operators now throw** instead of silently skipping. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`"in"` filter operator requires an array value** — non-array values now throw. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Invalid pagination throws**: negative/infinite `offset` or `limit` now throws. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Sharded items: id is now immutable** — `update()` with a different id on sharded collections throws. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`create()` no longer silently swallows transient read errors** — prevents data loss from overwriting entire collections. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+
+### Added
+
+- **Path traversal protection**: Collection names, item IDs, and commit paths validated against `../`, null bytes, and path separators. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Prototype pollution hardening**: `__proto__`, `constructor`, and `prototype` keys rejected in filter fields and update payloads. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Network error retry**: ENOTFOUND, ECONNRESET, ETIMEDOUT, ECONNREFUSED, EPIPE, and other network errors are retried with exponential backoff. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Secondary rate limit detection**: 403 responses with `Retry-After` header or secondary rate limit message are retried. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`Retry-After` HTTP-date parsing**: Falls back to exponential backoff on malformed values. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Real ETag capture** for conditional `GET` requests (`If-None-Match`). ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`CorruptedContentError`**: Explicit error for malformed JSON or binary content. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`toJSON()` on `GitHubDB` and `GitHubStorageProvider`** — accessToken redacted for safe logging. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **CLI: sharded collection detection** — `list` and `inspect` commands detect directory-based (sharded) collections. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **CLI: detailed connection test** — categorized failure reasons (auth, not-found, rate-limit, network). ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+
+### Fixed
+
+- **`afterRead` middleware now applies on all read paths** — including cached, indexed, and fast-path reads. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Items stored as raw data** — `afterRead` transformations are no longer persisted to storage. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`update()` works on immutable copies** — no longer mutates in-memory items in-place. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Sharded upsert uses `indexer.update()`** (not bare `add()`) — fixes stale index entries. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`deleteFile` wraps 409 → `ConcurrencyError`**; **`commit()` maps 422 → `ConcurrencyError`**. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **`exists()` only trusts fresh cache**; stale cache bounded to 1000 entries (LRU). ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **Transaction isolation**: `listDirectory` merges pending creates/deletes; `exists()` handles pending deletes. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+- **CLI error messages genericized** — no raw API errors leaked to terminal. ([#5](https://github.com/Almyk/gh-as-db/pull/5))
+
+### Security
+
+- Path traversal hardening (CWE-22)
+- Prototype pollution hardening (CWE-1321)
+- accessToken redaction in `toJSON()` / logging
+- CLI error message sanitization
+
+---
+
+## [1.3.0] — 2025-12-03
+
+- Initial public release

package/README.md

@@ -44,6 +44,7 @@ const db = new GitHubDB({
 });
 
 // Access the 'users' collection
+// Note: avoid logging or serializing the db instance — the access token is stored on db.config
 const users = db.collection<User>('users');
 
 // Create a new user
@@ -85,7 +86,7 @@ const db = new GitHubDB({
   accessToken: string; // GitHub PAT with 'repo' scope
   owner: string;       // Repository owner
   repo: string;        // Repository name
-  cacheTTL?: number;   // Optional: Cache TTL in ms. Default is 0 (strict consistency).
+  cacheTTL?: number;   // Optional: Cache TTL in ms. Default is 0 (strict consistency — always validates with GitHub). Set to a positive value (e.g. 5000) to enable in-memory caching for performance. Note: cacheTTL > 0 may return stale data across instances until the TTL expires.
   retry?: RetryConfig | false; // Optional: Retry config, or false to disable. See below.
 });
 ```
@@ -195,6 +196,8 @@ const users = db.collection<User>('users', {
 - 📈 **Scalability**: Avoids GitHub's file size limits and reduces merge conflicts.
 - 🧹 **Cleanliness**: Better organization for repositories with thousands of documents.
 
+**Important**: In sharded mode, item `id` values cannot be changed via `update()`. To change an id, delete the item and re-create it with the new id.
+
 ### Retry & Rate Limit Handling
 
 All GitHub API calls are automatically retried on transient errors (429 rate limits, 500/502/503 server errors) with exponential backoff. If GitHub returns a `Retry-After` header, it is respected.
@@ -221,7 +224,7 @@ const db = new GitHubDB({
 });
 ```
 
-Non-transient errors (401, 403, 404) are thrown immediately without retrying. Concurrency conflicts (409) still throw `ConcurrencyError` immediately. If rate limit retries are exhausted, a `RateLimitError` is thrown.
+Non-transient errors (401, 404, and permission-related 403) are thrown immediately. GitHub secondary rate limits (abuse-detection 403 responses) are retried automatically with backoff, just like 429 rate limits. Concurrency conflicts (409) still throw `ConcurrencyError` immediately. If rate limit retries are exhausted, a `RateLimitError` is thrown.
 
 ```typescript
 import { RateLimitError, ConcurrencyError } from 'gh-as-db';
@@ -262,12 +265,18 @@ For small projects, side-projects, or internal tools, setting up a database serv
 
 ## Performance
 
-- **Consistent Caching**: Uses **Conditional GET** (`If-None-Match`) to ensure data is always up-to-date even across multiple instances (e.g., serverless), while minimizing API costs.
+- **Consistent Caching**: With the default `cacheTTL: 0`, every read validates against GitHub using **Conditional GET** (`If-None-Match`) to ensure data is always up-to-date even across multiple instances (e.g., serverless), while minimizing API costs. Set `cacheTTL > 0` for additional in-memory caching with a possible staleness window.
 - **Write-Through**: Updates the local cache immediately after a write, preventing 404s during redirects.
 - **Indexing**: Automatic in-memory indexing on all fields makes querying fast even as data grows.
 - **Optimistic Concurrency**: Uses Git SHAs to ensure that you don't overwrite changes made by another client.
 - **Automatic Retries**: Transient failures and rate limits are handled transparently with exponential backoff.
 
+## Security
+
+- **Token handling**: The GitHub PAT is stored on `db.config.accessToken`. Avoid logging, serializing, or exposing the `db` object. The library's `toJSON()` method redacts the token if you need to serialize the config for debugging.
+- **Path safety**: Collection names and item IDs are validated against path traversal attacks. Only alphanumeric characters, dots, hyphens, and underscores are permitted.
+- **Input validation**: Query operators are validated and unknown operators throw errors. Pagination parameters are range-checked.
+
 ## License
 
 MIT

package/dist/core/types.d.ts

@@ -49,9 +49,14 @@ export declare class RateLimitError extends Error {
     readonly retryAfter?: number | undefined;
     constructor(retryAfter?: number | undefined);
 }
+export declare class CorruptedContentError extends Error {
+    readonly path: string;
+    constructor(path: string, cause: Error);
+}
 export interface StorageResponse<T> {
     data: T;
     sha: string;
+    etag?: string;
 }
 export interface Validator<T> {
     validate: (data: unknown) => Promise<T> | T;
@@ -70,7 +75,7 @@ export interface IStorageProvider {
     listDirectory(path: string): Promise<{
         path: string;
         sha: string;
-        type: "file" | "dir";
+        type: "file" | "dir" | "symlink" | "submodule";
     }[]>;
     commit(changes: CommitChange[], message: string): Promise<string>;
 }

package/dist/core/types.js

@@ -14,3 +14,12 @@ export class RateLimitError extends Error {
         this.name = "RateLimitError";
     }
 }
+export class CorruptedContentError extends Error {
+    path;
+    constructor(path, cause) {
+        super(`Corrupted or malformed content at ${path}: ${cause.message}`);
+        this.path = path;
+        this.name = "CorruptedContentError";
+        this.cause = cause;
+    }
+}

package/dist/core/validation.d.ts

@@ -0,0 +1,3 @@
+export declare function validateName(name: string, context?: string): void;
+export declare function validateId(id: string): void;
+export declare function validatePath(path: string, context?: string): void;

package/dist/core/validation.js

@@ -0,0 +1,35 @@
+const VALID_NAME_RE = /^[A-Za-z0-9._-]+$/;
+export function validateName(name, context = "name") {
+    if (!name || typeof name !== "string") {
+        throw new Error(`Invalid ${context}: must be a non-empty string`);
+    }
+    if (name === "." || name === "..") {
+        throw new Error(`Invalid ${context}: "${name}" is a reserved name`);
+    }
+    if (name.includes("/") || name.includes("\\") || name.includes("\0")) {
+        throw new Error(`Invalid ${context}: "${name}" contains path separators or control characters`);
+    }
+    if (!VALID_NAME_RE.test(name)) {
+        throw new Error(`Invalid ${context}: "${name}" contains invalid characters`);
+    }
+    if (name.length > 255) {
+        throw new Error(`Invalid ${context}: exceeds maximum length of 255 characters`);
+    }
+}
+export function validateId(id) {
+    validateName(id, "id");
+}
+export function validatePath(path, context = "path") {
+    if (!path || typeof path !== "string") {
+        throw new Error(`Invalid ${context}: must be a non-empty string`);
+    }
+    if (path.startsWith("/")) {
+        throw new Error(`Invalid ${context}: "${path}" must not start with /`);
+    }
+    if (path.includes("..")) {
+        throw new Error(`Invalid ${context}: "${path}" contains path traversal ("..")`);
+    }
+    if (path.includes("\0")) {
+        throw new Error(`Invalid ${context}: "${path}" contains null byte`);
+    }
+}

package/dist/index.d.ts

@@ -2,4 +2,4 @@ export { GitHubDB } from "./ui/github-db.js";
 export { Collection } from "./ui/collection.js";
 export { Transaction } from "./ui/transaction.js";
 export * from "./core/types.js";
-export declare const version = "1.3.0";
+export declare const version = "2.0.0";

package/dist/index.js

@@ -2,4 +2,4 @@ export { GitHubDB } from "./ui/github-db.js";
 export { Collection } from "./ui/collection.js";
 export { Transaction } from "./ui/transaction.js";
 export * from "./core/types.js";
-export const version = "1.3.0";
+export const version = "2.0.0";

package/dist/infrastructure/github-storage.d.ts

@@ -1,4 +1,4 @@
-import { GitHubDBConfig, IStorageProvider, StorageResponse, CommitChange } from "../core/types.js";
+import { RetryConfig, GitHubDBConfig, IStorageProvider, StorageResponse, CommitChange } from "../core/types.js";
 import { ICacheProvider } from "./cache-provider.js";
 export declare class GitHubStorageProvider implements IStorageProvider {
     private config;
@@ -6,9 +6,18 @@ export declare class GitHubStorageProvider implements IStorageProvider {
     private cache;
     private staleCache;
     private readonly DEFAULT_TTL;
+    private readonly MAX_STALE_ENTRIES;
     constructor(config: GitHubDBConfig, cache?: ICacheProvider);
+    private setStaleCache;
+    private parseRetryAfter;
+    private isSecondaryRateLimit;
     private retryWithBackoff;
     testConnection(): Promise<boolean>;
+    testConnectionDetailed(): Promise<{
+        ok: boolean;
+        reason?: "auth" | "not-found" | "rate-limit" | "network" | "unknown";
+        error?: Error;
+    }>;
     exists(path: string): Promise<boolean>;
     readJson<T>(path: string): Promise<StorageResponse<T>>;
     writeJson<T>(path: string, content: T, message: string, sha?: string): Promise<string>;
@@ -17,6 +26,16 @@ export declare class GitHubStorageProvider implements IStorageProvider {
     listDirectory(path: string): Promise<{
         path: string;
         sha: string;
-        type: "file" | "dir";
+        type: "file" | "dir" | "symlink" | "submodule";
     }[]>;
+    toJSON(): {
+        config: {
+            accessToken: string;
+            owner: string;
+            repo: string;
+            branch?: string;
+            cacheTTL?: number;
+            retry?: RetryConfig | false;
+        };
+    };
 }

package/dist/infrastructure/github-storage.js

@@ -1,12 +1,14 @@
 import { Octokit } from "@octokit/rest";
-import { ConcurrencyError, RateLimitError, } from "../core/types.js";
+import { ConcurrencyError, CorruptedContentError, RateLimitError, } from "../core/types.js";
 import { MemoryCacheProvider } from "./cache-provider.js";
+import { validatePath } from "../core/validation.js";
 export class GitHubStorageProvider {
     config;
     octokit;
     cache;
     staleCache = new Map();
     DEFAULT_TTL = 0; // Default to 0 for consistency
+    MAX_STALE_ENTRIES = 1000;
     constructor(config, cache) {
         this.config = config;
         this.octokit = new Octokit({
@@ -14,6 +16,37 @@ export class GitHubStorageProvider {
         });
         this.cache = cache || new MemoryCacheProvider();
     }
+    setStaleCache(path, entry) {
+        if (this.staleCache.size >= this.MAX_STALE_ENTRIES) {
+            const oldest = this.staleCache.keys().next().value;
+            if (oldest)
+                this.staleCache.delete(oldest);
+        }
+        this.staleCache.set(path, entry);
+    }
+    parseRetryAfter(value) {
+        const num = Number(value);
+        if (Number.isFinite(num) && num > 0) {
+            return num * 1000;
+        }
+        const date = Date.parse(value);
+        if (Number.isFinite(date)) {
+            return Math.max(0, date - Date.now());
+        }
+        return NaN;
+    }
+    isSecondaryRateLimit(error) {
+        const headers = error?.response?.headers || {};
+        if (headers["retry-after"])
+            return true;
+        const message = error?.message || "";
+        if (message.includes("secondary rate limit") ||
+            message.includes("abuse detection") ||
+            message.includes("abuse mechanism")) {
+            return true;
+        }
+        return false;
+    }
     async retryWithBackoff(fn) {
         if (this.config.retry === false)
             return fn();
@@ -29,19 +62,40 @@ export class GitHubStorageProvider {
                 // Non-retryable: 409 (concurrency), 4xx client errors (except 429)
                 if (status === 409)
                     throw error;
-                if (status && status < 500 && status !== 429)
-                    throw error;
-                // Errors without a status code (non-HTTP errors) — rethrow
-                if (!status)
-                    throw error;
+                if (status && status < 500 && status !== 429) {
+                    if (status === 403 && this.isSecondaryRateLimit(error)) {
+                        // Fall through to retry logic below
+                    }
+                    else {
+                        throw error;
+                    }
+                }
+                // Retry status-less errors that appear to be network errors
+                if (!status) {
+                    const code = error?.code;
+                    const isNetworkError = code === "ENOTFOUND" ||
+                        code === "ECONNRESET" ||
+                        code === "ETIMEDOUT" ||
+                        code === "EAI_AGAIN" ||
+                        code === "ECONNREFUSED" ||
+                        code === "EPIPE" ||
+                        !error?.response;
+                    if (!isNetworkError) {
+                        throw error;
+                    }
+                    // Fall through to retry logic below
+                }
                 // Last attempt — don't wait, just break
                 if (attempt === maxRetries)
                     break;
                 // Calculate delay
                 let delay;
                 if (status === 429 && error.response?.headers?.["retry-after"]) {
-                    delay =
-                        parseInt(error.response.headers["retry-after"], 10) * 1000;
+                    const retryAfter = error.response.headers["retry-after"];
+                    delay = this.parseRetryAfter(retryAfter);
+                    if (!Number.isFinite(delay) || delay < 0) {
+                        delay = baseDelay * Math.pow(2, attempt);
+                    }
                 }
                 else {
                     delay = baseDelay * Math.pow(2, attempt);
@@ -72,8 +126,30 @@ export class GitHubStorageProvider {
             return false;
         }
     }
+    async testConnectionDetailed() {
+        try {
+            await this.retryWithBackoff(() => this.octokit.repos.get({
+                owner: this.config.owner,
+                repo: this.config.repo,
+            }));
+            return { ok: true };
+        }
+        catch (error) {
+            const status = error?.status;
+            if (status === 401)
+                return { ok: false, reason: "auth", error };
+            if (status === 404)
+                return { ok: false, reason: "not-found", error };
+            if (status === 429)
+                return { ok: false, reason: "rate-limit", error };
+            if (!status || error?.code === "ENOTFOUND" || error?.code === "ECONNRESET" || error?.code === "ETIMEDOUT") {
+                return { ok: false, reason: "network", error };
+            }
+            return { ok: false, reason: "unknown", error };
+        }
+    }
     async exists(path) {
-        if (this.cache.get(path) || this.staleCache.has(path)) {
+        if (this.cache.get(path)) {
             return true;
         }
         try {
@@ -104,7 +180,11 @@ export class GitHubStorageProvider {
                 owner: this.config.owner,
                 repo: this.config.repo,
                 path,
-                headers: stale ? { "if-none-match": `"${stale.sha}"` } : {},
+                headers: stale
+                    ? {
+                        "if-none-match": stale.etag || `"${stale.sha}"`,
+                    }
+                    : {},
             }));
             if (Array.isArray(response.data)) {
                 throw new Error("Path is a directory, not a file");
@@ -112,21 +192,36 @@ export class GitHubStorageProvider {
             if (!("content" in response.data) || !("sha" in response.data)) {
                 throw new Error("No content or SHA in response");
             }
-            const binary = atob(response.data.content.replace(/\s/g, ""));
-            const bytes = new Uint8Array(binary.length);
-            for (let i = 0; i < binary.length; i++) {
-                bytes[i] = binary.charCodeAt(i);
+            let content;
+            try {
+                const binary = atob(response.data.content.replace(/\s/g, ""));
+                const bytes = new Uint8Array(binary.length);
+                for (let i = 0; i < binary.length; i++) {
+                    bytes[i] = binary.charCodeAt(i);
+                }
+                const decoder = new TextDecoder("utf-8");
+                content = decoder.decode(bytes);
+            }
+            catch (e) {
+                throw new CorruptedContentError(path, e);
+            }
+            let parsed;
+            try {
+                parsed = JSON.parse(content);
+            }
+            catch (e) {
+                throw new CorruptedContentError(path, e);
             }
-            const decoder = new TextDecoder("utf-8");
-            const content = decoder.decode(bytes);
+            const etag = response.headers?.etag || undefined;
             const result = {
-                data: JSON.parse(content),
+                data: parsed,
                 sha: response.data.sha,
+                etag,
             };
             // Update both caches
             const ttl = this.config.cacheTTL ?? this.DEFAULT_TTL;
             this.cache.set(path, result, ttl);
-            this.staleCache.set(path, result);
+            this.setStaleCache(path, result);
             return result;
         }
         catch (error) {
@@ -178,11 +273,14 @@ export class GitHubStorageProvider {
                     .join("")),
                 sha: internalSha,
             }));
-            const newSha = response.data.content?.sha || "";
+            const newSha = response.data.content?.sha;
+            if (!newSha) {
+                throw new Error(`writeJson succeeded but returned no SHA for ${path}`);
+            }
             const result = { data: content, sha: newSha };
             const ttl = this.config.cacheTTL ?? this.DEFAULT_TTL;
             this.cache.set(path, result, ttl);
-            this.staleCache.set(path, result);
+            this.setStaleCache(path, result);
             return newSha;
         }
         catch (error) {
@@ -196,6 +294,9 @@ export class GitHubStorageProvider {
         if (changes.length === 0) {
             throw new Error("No changes to commit");
         }
+        for (const change of changes) {
+            validatePath(change.path, "commit path");
+        }
         const branch = this.config.branch || "main";
         try {
             // 1. Get the current head SHA
@@ -258,27 +359,41 @@ export class GitHubStorageProvider {
                     if (change) {
                         const result = { data: change.content, sha: entry.sha };
                         this.cache.set(entry.path, result, ttl);
-                        this.staleCache.set(entry.path, result);
+                        this.setStaleCache(entry.path, result);
                     }
                 }
             }
+            for (const change of changes) {
+                if (change.content === null) {
+                    this.cache.delete(change.path);
+                    this.staleCache.delete(change.path);
+                }
+            }
             return newCommit.sha;
         }
         catch (error) {
-            if (error.status === 409) {
+            if (error.status === 409 || error.status === 422) {
                 throw new ConcurrencyError("batch-commit");
             }
             throw error;
         }
     }
     async deleteFile(path, message, sha) {
-        await this.retryWithBackoff(() => this.octokit.repos.deleteFile({
-            owner: this.config.owner,
-            repo: this.config.repo,
-            path,
-            message,
-            sha,
-        }));
+        try {
+            await this.retryWithBackoff(() => this.octokit.repos.deleteFile({
+                owner: this.config.owner,
+                repo: this.config.repo,
+                path,
+                message,
+                sha,
+            }));
+        }
+        catch (error) {
+            if (error.status === 409) {
+                throw new ConcurrencyError(path);
+            }
+            throw error;
+        }
         this.cache.delete(path);
         this.staleCache.delete(path);
     }
@@ -305,4 +420,12 @@ export class GitHubStorageProvider {
             throw error;
         }
     }
+    toJSON() {
+        return {
+            config: {
+                ...this.config,
+                accessToken: "[redacted]",
+            },
+        };
+    }
 }

package/dist/infrastructure/transaction-storage.d.ts

@@ -15,7 +15,7 @@ export declare class TransactionStorageProvider implements IStorageProvider {
     listDirectory(path: string): Promise<{
         path: string;
         sha: string;
-        type: "file" | "dir";
+        type: "file" | "dir" | "symlink" | "submodule";
     }[]>;
     commit(changes: CommitChange[], message: string): Promise<string>;
     /**

package/dist/infrastructure/transaction-storage.js

@@ -13,13 +13,18 @@ export class TransactionStorageProvider {
     }
     async exists(path) {
         if (this.pendingChanges.has(path)) {
-            return true;
+            return this.pendingChanges.get(path) !== null;
         }
         return this.baseStorage.exists(path);
     }
     async readJson(path) {
         const pending = this.pendingChanges.get(path);
         if (pending !== undefined) {
+            if (pending === null) {
+                throw Object.assign(new Error(`File not found in transaction: ${path}`), {
+                    status: 404,
+                });
+            }
             return {
                 data: pending,
                 sha: "transaction-pending-sha",
@@ -35,9 +40,30 @@ export class TransactionStorageProvider {
         this.pendingChanges.set(path, null);
     }
     async listDirectory(path) {
-        // For now, we delegate to base storage.
-        // In sharded mode, we might need to merge with pending changes.
-        return this.baseStorage.listDirectory(path);
+        const baseList = await this.baseStorage.listDirectory(path);
+        const basePaths = new Set(baseList.map((e) => e.path));
+        const merged = [...baseList];
+        for (const [pendingPath, content] of this.pendingChanges) {
+            if (!pendingPath.startsWith(path))
+                continue;
+            if (content === null) {
+                // Deletion — remove from list
+                continue;
+            }
+            if (!basePaths.has(pendingPath)) {
+                // New file — add to list
+                merged.push({
+                    path: pendingPath,
+                    sha: "transaction-pending-sha",
+                    type: "file",
+                });
+            }
+        }
+        // Filter out deleted paths
+        return merged.filter((entry) => {
+            const pending = this.pendingChanges.get(entry.path);
+            return pending !== undefined ? pending !== null : true;
+        });
     }
     async commit(changes, message) {
         return this.baseStorage.commit(changes, message);

package/dist/ui/cli/commands.js

@@ -2,6 +2,7 @@ import enquirer from "enquirer";
 import chalk from "chalk";
 import { Octokit } from "@octokit/rest";
 import { GitHubStorageProvider } from "../../infrastructure/github-storage.js";
+import { validateName } from "../../core/validation.js";
 export async function initCommand() {
     console.log(chalk.yellow("\n--- Database Initialization ---\n"));
     try {
@@ -29,14 +30,21 @@ export async function initCommand() {
             accessToken: answers.accessToken,
         });
         process.stdout.write(chalk.blue("Verifying connection... "));
-        const success = await storage.testConnection();
-        if (success) {
+        const result = await storage.testConnectionDetailed();
+        if (result.ok) {
             console.log(chalk.green("Success!"));
             console.log(chalk.dim("\nYou can now use these credentials in your application configuration."));
         }
         else {
-            console.log(chalk.red("Failed."));
-            console.log(chalk.red("Please check your token permissions and repository details."));
+            console.log(chalk.red(`Failed — ${result.reason || "unknown error"}`));
+            if (result.reason === "auth")
+                console.log(chalk.red("Check your token and permissions."));
+            else if (result.reason === "not-found")
+                console.log(chalk.red("Check that the owner and repo are correct."));
+            else if (result.reason === "rate-limit")
+                console.log(chalk.red("You've been rate limited. Try again later."));
+            else
+                console.log(chalk.red("Please check your token permissions and repository details."));
         }
     }
     catch (error) {
@@ -52,6 +60,8 @@ export async function listCollectionsCommand() {
         console.log(chalk.red("Error: Environment variables GH_DB_OWNER, GH_DB_REPO, and GH_DB_TOKEN must be set."));
         return;
     }
+    validateName(owner, "owner");
+    validateName(repo, "repo");
     const octokit = new Octokit({ auth });
     try {
         const { data } = await octokit.repos.getContent({
@@ -60,9 +70,13 @@ export async function listCollectionsCommand() {
             path: "",
         });
         if (Array.isArray(data)) {
-            const collections = data
-                .filter((item) => item.name.endsWith(".json"))
+            const jsonCollections = data
+                .filter((item) => item.type === "file" && item.name.endsWith(".json"))
                 .map((item) => item.name.replace(".json", ""));
+            const dirCollections = data
+                .filter((item) => item.type === "dir")
+                .map((item) => `${item.name}/ (sharded)`);
+            const collections = [...jsonCollections, ...dirCollections];
             if (collections.length === 0) {
                 console.log(chalk.dim("No collections found in the repository."));
             }
@@ -74,7 +88,7 @@ export async function listCollectionsCommand() {
         }
     }
     catch (error) {
-        console.error(chalk.red(`Error fetching collections: ${error.message}`));
+        console.error(chalk.red(`Error fetching collections: GitHub API error`));
     }
 }
 export async function inspectCollectionCommand(name) {
@@ -86,23 +100,37 @@ export async function inspectCollectionCommand(name) {
         console.log(chalk.red("Error: Environment variables GH_DB_OWNER, GH_DB_REPO, and GH_DB_TOKEN must be set."));
         return;
     }
+    validateName(name, "collection name");
     const storage = new GitHubStorageProvider({
         owner,
         repo,
         accessToken: auth,
     });
     try {
-        const path = `${name}.json`;
-        if (!(await storage.exists(path))) {
+        const singlePath = `${name}.json`;
+        const shardedPath = `${name}`;
+        let isSingleFile = await storage.exists(singlePath);
+        let isSharded = !isSingleFile ? await storage.exists(shardedPath) : false;
+        if (!isSingleFile && !isSharded) {
             console.log(chalk.red(`Collection "${name}" does not exist.`));
             return;
         }
-        const response = await storage.readJson(path);
-        console.log(JSON.stringify(response.data, null, 2));
-        console.log(chalk.dim(`\nTotal records: ${response.data.length}`));
-        console.log(chalk.dim(`Current SHA: ${response.sha}`));
+        if (isSingleFile) {
+            const response = await storage.readJson(singlePath);
+            console.log(JSON.stringify(response.data, null, 2));
+            console.log(chalk.dim(`\nTotal records: ${response.data.length}`));
+            console.log(chalk.dim(`Current SHA: ${response.sha}`));
+        }
+        else {
+            const files = await storage.listDirectory(shardedPath);
+            const jsonFiles = files.filter((f) => f.type === "file" && f.path.endsWith(".json"));
+            console.log(chalk.dim(`\nSharded collection with ${jsonFiles.length} documents:`));
+            for (const file of jsonFiles) {
+                console.log(chalk.cyan(` • ${file.path}`));
+            }
+        }
     }
     catch (error) {
-        console.error(chalk.red(`Error inspecting collection: ${error.message}`));
+        console.error(chalk.red(`Error inspecting collection: GitHub API error`));
     }
 }

package/dist/ui/cli/index.js

@@ -13,7 +13,7 @@ console.log(boxen(chalk.bold.blue("gh-as-db CLI"), {
 program
     .name("gh-as-db")
     .description("CLI to manage your GitHub-based database")
-    .version("0.1.0");
+    .version("2.0.0");
 program
     .command("init")
     .description("Initialize connection and verify repository")

package/dist/ui/collection.js

@@ -1,5 +1,6 @@
 import { ConcurrencyError, } from "../core/types.js";
 import { Indexer } from "../core/indexer.js";
+import { validateName, validateId } from "../core/validation.js";
 export class Collection {
     name;
     storage;
@@ -14,6 +15,7 @@ export class Collection {
     constructor(name, storage, middlewareOrOptions) {
         this.name = name;
         this.storage = storage;
+        validateName(name);
         if (Array.isArray(middlewareOrOptions)) {
             this.middleware = middlewareOrOptions;
             this.strategy = "single-file";
@@ -28,6 +30,7 @@ export class Collection {
         return this.strategy === "sharded" ? `${this.name}/` : `${this.name}.json`;
     }
     getItemPath(id) {
+        validateId(id);
         return this.strategy === "sharded"
             ? `${this.name}/${id}.json`
             : `${this.name}.json`;
@@ -49,18 +52,21 @@ export class Collection {
             }
         }
         if (this.strategy === "sharded") {
+            validateId(finalItem.id);
             const itemPath = this.getItemPath(finalItem.id);
             const sha = await this.storage.writeJson(itemPath, finalItem, `Create item ${finalItem.id} in ${this.name}`, this.shas.get(itemPath));
             this.shas.set(itemPath, sha);
             if (this.dataLoaded) {
                 const index = this.items.findIndex((i) => i.id === finalItem.id);
                 if (index > -1) {
+                    const oldItem = this.items[index];
                     this.items[index] = finalItem;
+                    this.indexer.update(oldItem, finalItem);
                 }
                 else {
                     this.items.push(finalItem);
+                    this.indexer.add(finalItem);
                 }
-                this.indexer.add(finalItem);
             }
             return finalItem;
         }
@@ -77,7 +83,8 @@ export class Collection {
                 }
             }
             catch (error) {
-                // If file doesn't exist, start with empty array
+                if (error?.status !== 404)
+                    throw error;
             }
         }
         items.push(finalItem);
@@ -115,19 +122,38 @@ export class Collection {
                 queryOrPredicate.filters[0].operator === "eq" &&
                 this.indexer.hasIndex(queryOrPredicate.filters[0].field)) {
                 const filter = queryOrPredicate.filters[0];
-                const results = this.indexer.query(filter.field, filter.value);
-                if (results !== null) {
+                const rawResults = this.indexer.query(filter.field, filter.value);
+                if (rawResults !== null) {
+                    let results = (await Promise.all(rawResults.map(async (item) => {
+                        let current = item;
+                        for (const mw of this.middleware) {
+                            if (mw.afterRead)
+                                current = await mw.afterRead(current, context);
+                        }
+                        return current;
+                    })));
+                    if (queryOrPredicate) {
+                        results = this.applyQueryOptions(results, queryOrPredicate);
+                    }
                     return results;
                 }
             }
             // Fallback to full scan of current in-memory items
+            let cachedItems = (await Promise.all(this.items.map(async (item) => {
+                let current = item;
+                for (const mw of this.middleware) {
+                    if (mw.afterRead)
+                        current = await mw.afterRead(current, context);
+                }
+                return current;
+            })));
             if (typeof queryOrPredicate === "function") {
-                return this.items.filter(queryOrPredicate);
+                return cachedItems.filter(queryOrPredicate);
             }
             if (queryOrPredicate) {
-                return this.applyQueryOptions(this.items, queryOrPredicate);
+                return this.applyQueryOptions(cachedItems, queryOrPredicate);
             }
-            return this.items;
+            return cachedItems;
         }
         if (this.strategy === "sharded") {
             const files = await this.storage.listDirectory(this.name);
@@ -144,6 +170,7 @@ export class Collection {
             this.lastSha = response.sha;
             items = response.data;
         }
+        const rawItems = items;
         items = await Promise.all(items.map(async (item) => {
             let currentItem = item;
             for (const mw of this.middleware) {
@@ -153,10 +180,10 @@ export class Collection {
             }
             return currentItem;
         }));
-        // Build index and store in-memory items
+        // Build index and store RAW items (before afterRead) in memory
         if (!this.dataLoaded) {
-            this.items = items;
-            this.indexer.build(items, items.length > 0 ? Object.keys(items[0]) : []);
+            this.items = rawItems;
+            this.indexer.build(rawItems, rawItems.length > 0 ? Object.keys(rawItems[0]) : []);
             this.dataLoaded = true;
         }
         if (typeof queryOrPredicate === "function") {
@@ -169,10 +196,20 @@ export class Collection {
     }
     applyQueryOptions(items, options) {
         let result = [...items];
+        const PROTOTYPE_KEYS = new Set(["__proto__", "constructor", "prototype"]);
+        const VALID_OPERATORS = new Set([
+            "eq", "neq", "gt", "gte", "lt", "lte", "contains", "in",
+        ]);
         // Apply filters
         if (options.filters) {
             for (const filter of options.filters) {
                 result = result.filter((item) => {
+                    if (PROTOTYPE_KEYS.has(filter.field)) {
+                        return true;
+                    }
+                    if (!VALID_OPERATORS.has(filter.operator)) {
+                        throw new Error(`Unknown filter operator: "${filter.operator}". Supported: ${[...VALID_OPERATORS].join(", ")}`);
+                    }
                     const val = item[filter.field];
                     switch (filter.operator) {
                         case "eq":
@@ -192,10 +229,12 @@ export class Collection {
                                 return val.includes(filter.value);
                             }
                             return (typeof val === "string" && val.includes(filter.value));
-                        case "in":
-                            return Array.isArray(filter.value) && filter.value.includes(val);
-                        default:
-                            return true;
+                        case "in": {
+                            if (!Array.isArray(filter.value)) {
+                                throw new Error(`"in" operator requires an array value, got ${typeof filter.value}`);
+                            }
+                            return filter.value.includes(val);
+                        }
                     }
                 });
             }
@@ -217,14 +256,33 @@ export class Collection {
         // Apply pagination
         if (options.pagination) {
             const { limit, offset = 0 } = options.pagination;
+            if (typeof offset !== "number" || offset < 0 || !Number.isFinite(offset)) {
+                throw new Error(`Invalid pagination offset: ${offset}`);
+            }
+            if (limit !== undefined && (typeof limit !== "number" || limit < 0 || !Number.isFinite(limit))) {
+                throw new Error(`Invalid pagination limit: ${limit}`);
+            }
             result = result.slice(offset, limit ? offset + limit : undefined);
         }
         return result;
     }
     async findById(id) {
+        validateId(id);
         if (this.dataLoaded && this.indexer.hasIndex("id")) {
             const results = this.indexer.query("id", id);
-            return results && results.length > 0 ? results[0] : null;
+            if (results && results.length > 0) {
+                let item = results[0];
+                const context = {
+                    collection: this.name,
+                    operation: "read",
+                };
+                for (const mw of this.middleware) {
+                    if (mw.afterRead)
+                        item = await mw.afterRead(item, context);
+                }
+                return item;
+            }
+            return null;
         }
         if (this.strategy === "sharded") {
             try {
@@ -255,13 +313,20 @@ export class Collection {
         return items.find((item) => item.id === id) || null;
     }
     async update(id, updates) {
-        const items = await this.find();
+        validateId(id);
+        const items = [...(await this.find())];
         const index = items.findIndex((item) => item.id === id);
         if (index === -1) {
             throw new Error(`Item with id ${id} not found in ${this.name}`);
         }
         const originalItem = items[index];
-        items[index] = { ...items[index], ...updates };
+        const safeUpdates = {};
+        for (const key of Object.keys(updates)) {
+            if (key !== "__proto__" && key !== "constructor" && key !== "prototype") {
+                safeUpdates[key] = updates[key];
+            }
+        }
+        items[index] = { ...items[index], ...safeUpdates };
         let finalItem = items[index];
         // 1. Validation
         if (this.validator) {
@@ -278,7 +343,11 @@ export class Collection {
             }
         }
         items[index] = finalItem;
+        validateId(id);
         if (this.strategy === "sharded") {
+            if (updates.id !== undefined && updates.id !== id) {
+                throw new Error(`Cannot change id of a sharded item. Delete and re-create instead.`);
+            }
             const itemPath = this.getItemPath(id);
             try {
                 const sha = await this.storage.writeJson(itemPath, finalItem, `Update item ${id} in ${this.name}`, this.shas.get(itemPath));
@@ -309,6 +378,7 @@ export class Collection {
         return items[index];
     }
     async delete(id) {
+        validateId(id);
         const items = await this.find();
         const index = items.findIndex((item) => item.id === id);
         if (index === -1) {
@@ -316,22 +386,40 @@ export class Collection {
         }
         const itemToDelete = items[index];
         const filtered = items.filter((_, i) => i !== index);
+        validateId(id);
         if (this.strategy === "sharded") {
             const itemPath = this.getItemPath(id);
-            const sha = this.shas.get(itemPath);
-            if (!sha) {
-                // We need the SHA to delete. If we don't have it, we must fetch it.
-                const response = await this.storage.readJson(itemPath);
-                await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, response.sha);
+            try {
+                const sha = this.shas.get(itemPath);
+                if (!sha) {
+                    const response = await this.storage.readJson(itemPath);
+                    await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, response.sha);
+                }
+                else {
+                    await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, sha);
+                }
+                this.shas.delete(itemPath);
             }
-            else {
-                await this.storage.deleteFile(itemPath, `Delete item ${id} from ${this.name}`, sha);
+            catch (error) {
+                if (error instanceof ConcurrencyError) {
+                    this.dataLoaded = false;
+                    this.shas.delete(itemPath);
+                }
+                throw error;
             }
-            this.shas.delete(itemPath);
         }
         else {
             try {
-                this.lastSha = await this.storage.writeJson(this.path, filtered, `Delete item ${id} from ${this.name}`, this.lastSha);
+                if (filtered.length === 0) {
+                    const fileExists = await this.storage.exists(this.path);
+                    if (fileExists) {
+                        const existing = await this.storage.readJson(this.path);
+                        await this.storage.deleteFile(this.path, `Delete item ${id} from ${this.name}`, existing.sha);
+                    }
+                }
+                else {
+                    this.lastSha = await this.storage.writeJson(this.path, filtered, `Delete item ${id} from ${this.name}`, this.lastSha);
+                }
             }
             catch (error) {
                 if (error instanceof ConcurrencyError) {

package/dist/ui/github-db.d.ts

@@ -11,5 +11,12 @@ export declare class GitHubDB {
         validator?: Validator<T>;
         strategy?: StorageStrategy;
     }): Collection<T>;
-    transaction(fn: (tx: Transaction) => Promise<void>, message?: string): Promise<string>;
+    transaction(fn: (tx: Transaction) => Promise<void>, message?: string): Promise<string | null>;
+    toJSON(): {
+        config: {
+            owner: string;
+            repo: string;
+            accessToken: string;
+        };
+    };
 }

package/dist/ui/github-db.js

@@ -28,4 +28,13 @@ export class GitHubDB {
         await fn(tx);
         return tx.commit(message);
     }
+    toJSON() {
+        return {
+            config: {
+                owner: this.config.owner,
+                repo: this.config.repo,
+                accessToken: "[redacted]",
+            },
+        };
+    }
 }

package/dist/ui/transaction.d.ts

@@ -19,5 +19,5 @@ export declare class Transaction {
      * Commits all buffered changes in the transaction to GitHub in a single commit.
      * @internal
      */
-    commit(message: string): Promise<string>;
+    commit(message: string): Promise<string | null>;
 }

package/dist/ui/transaction.js

@@ -24,7 +24,7 @@ export class Transaction {
     async commit(message) {
         const changes = this.txStorage.getChanges();
         if (changes.length === 0) {
-            return "";
+            return null;
         }
         return this.txStorage.commit(changes, message);
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "gh-as-db",
-  "version": "1.3.0",
+  "version": "2.0.0",
   "description": "Use a private GitHub repository as a database for your application.",
   "type": "module",
   "main": "./dist/index.js",
@@ -18,7 +18,8 @@
   "files": [
     "dist",
     "README.md",
-    "LICENSE"
+    "LICENSE",
+    "CHANGELOG.md"
   ],
   "keywords": [
     "github",