mongofire 6.5.1

package/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+All notable changes to MongoFire are documented here.
+
+## [6.5.0] — 2026-03-08
+
+### Fixed — Critical
+
+- **Batch upload data corruption** — operations in the same upload batch could be incorrectly acknowledged, causing sync state to diverge silently. Each operation is now acknowledged individually with its correct status
+- **Atlas connection pool leak** — a failed Atlas health check left the underlying connection open, leaking pool connections and causing every subsequent sync to fail with a connection error. Connection is now fully released on failure
+- **Delete overwrites concurrent update** — a queued delete could silently overwrite a document that another device had updated in the meantime. This is now detected and surfaced as a `conflict` event instead of data loss
+- **Date fields corrupted on sync** — `Date` values in synced documents were being converted to plain ISO strings, breaking date-range queries (`$gt`, `$lt`) on the receiving device. BSON types are now preserved correctly across sync
+- **`syncOwner` error grants access to all data** — if the `syncOwner` function threw an error, MongoFire silently fell back to syncing all owners. Now the sync is aborted and an `error` event is emitted, preventing unintended data access
+
+### Fixed — Medium
+
+- **Docker / cloned VM device ID collision** — instances sharing the same MAC address and hostname (e.g. Docker replicas, cloned VMs) generated identical device IDs, causing changes from those devices to be silently skipped on other instances. Device IDs are now always unique
+- **`updateOne` hook could track wrong document** — the change tracking hook re-read the document after the update using the original filter, which could match a different document in a concurrent write scenario. The hook now locks on the specific document ID before the update runs
+- **Bootstrap restart from scratch on failure** — if a bootstrap sync was interrupted (crash, network drop), the next start re-downloaded all documents from the beginning. Bootstrap now resumes from the last completed checkpoint
+- **Sync state and device registry created on Atlas** — two local-only internal collections were being created on Atlas unnecessarily, consuming storage and triggering index maintenance. They are now created on local only
+
+### Fixed — Minor
+
+- **Shared internal object could be accidentally mutated** — an internal configuration object was shared across calls; mutating it in one place could silently affect all subsequent calls. It is now immutable
+- **Multiple app instances share one signal handler** — only the first MongoFire instance received `SIGINT`/`SIGTERM` cleanup. Each instance now manages its own handler and removes it on `stop()`, preventing memory leaks in test environments
+- **`require('mongofire/plugin')` API inconsistency** — the direct plugin import had a different call signature to `mongofire.plugin()`. Both now work the same way, with a `.factory()` helper added for parity
+- **Collection name with special characters causes key collision** — two different collections could produce identical internal keys, leading to metadata corruption. Collection names are now validated at startup with a clear error message
+
+### Fixed — Security
+
+- **Hardware fingerprint stored on Atlas** — a derived MAC address value was being stored in Atlas, accessible to anyone with database read access. This constituted an unnecessary hardware fingerprint. The field has been removed
+- **Arbitrary collection names accepted** — collection names were not validated, allowing names that could interfere with MongoFire's internal collections. Names are now validated at startup: no special characters that cause key collisions, no reserved prefixes
+- **Oversized documents cause cryptic failure** — documents close to MongoDB's 16MB limit failed with an unhelpful low-level error. MongoFire now checks document size before writing and throws a clear, actionable message
+- **Manual `sync()` calls not rate-limited** — a runaway loop calling `sync()` in rapid succession could hammer Atlas with back-to-back requests. Rapid successive calls are now throttled automatically
+
+### Fixed — Performance
+
+- **Checksum computed on every document** — a cryptographic checksum was calculated for every document on every sync cycle regardless of whether verification was enabled. It is now skipped by default and only computed when `MONGOFIRE_VERIFY_REMOTE=1`
+- **Bulk update tracking loaded all IDs into memory** — tracking changes for a large `updateMany` operation materialised the full list of matching IDs in memory (100MB+ for millions of documents). The hook now uses a streaming cursor instead
+- **Too many database round-trips during upload** — uploading a large backlog of pending changes required many more database queries than necessary. Batch size has been increased significantly, reducing round-trips by ~60%
+
+### Fixed — Reliability
+
+- **Conflicts were silently swallowed** — when a version conflict was detected during upload, it was recorded internally but never surfaced to the application. MongoFire now emits a `conflict` event with structured data so you can respond to it
+- **Same-millisecond changes could be missed** — the delta sync cursor used only a timestamp, so two changes with the exact same timestamp could result in one being skipped on the next sync. The cursor now uses a compound position that eliminates this gap
+
+### Added
+
+- **`conflict` event** — emitted when a version conflict is detected during upload, with `{ collection, docId, localVersion, remoteVersion, op }` payload. See Events section in the README
+- **`ConflictData` TypeScript interface** — fully typed payload for the `conflict` event
+- **`mongofire/plugin` factory export** — `require('mongofire/plugin').factory(name, opts)` matches the `mongofire.plugin()` signature for direct use without the singleton
+- **`MONGOFIRE_COLLECTION_CONCURRENCY` env var** — configure how many collections sync in parallel at runtime (default: 4)
+- **Startup validation for collection names** — invalid names fail immediately with a clear message instead of silently corrupting data at sync time
+
+### Changed
+
+- `syncInterval` default is now `30000`ms (polling mode) or `5000`ms (when `realtime: true`)
+- `clean()` default changed from 30 days → **7 days**, matching the Atlas-side TTL so both sides stay consistent
+- `plugin()` `concurrency` option is now actually used (was accepted but ignored in previous versions; default: `8`)
+- Device IDs now include random bytes — existing device records are preserved on startup, so this only affects new installations
+
+---
+
+## [6.2.0] — 2026-03-08
+
+### Fixed — Critical
+- **`start()` concurrent safety** — multiple simultaneous `start()` calls now share one init promise instead of racing
+- **Bootstrap re-trigger bug** — an empty collection no longer forces a full re-bootstrap of all collections
+- **Silent change tracking errors** — errors in the Mongoose hooks are now logged instead of swallowed
+- **Realtime sync not working** — change stream pipeline fix; was silently delivering zero events on most Atlas clusters
+
+### Fixed — Medium
+- **`deleteMany` OOM risk** — plugin now streams and batches docs before deletion; removes 10,000-doc silent cap
+- **Session not forwarded in `updateOne` and `deleteOne` hooks** — reads now occur within the same transaction context
+
+### Added
+- Full TypeScript declarations (`types/index.d.ts`) with typed events, config, and result interfaces
+- `require('mongofire/plugin')` subpath export
+- Max retry limit (10 attempts) for permanently failing operations
+
+---
+
+## [6.1.0] — initial release
+
+- Offline-first sync with Local MongoDB + Atlas
+- Mongoose plugin with hooks for save, update, delete operations
+- Bootstrap + delta oplog sync
+- Automatic conflict resolution (version vector + timestamp + deviceId tiebreaker)
+- Real-time sync via Atlas Change Streams with polling fallback
+- `npx mongofire init / status / clean` CLI
+- Exponential backoff retry with jitter
+- TTL index auto-cleanup of old sync records
+- Multi-tenant `syncOwner` support

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,292 @@
+# 🔥 MongoFire
+
+> **Offline-first MongoDB sync** — Local + Atlas feel like ONE database.  
+> Automatic conflict resolution, Mongoose plugin, zero boilerplate.
+
+[![npm version](https://img.shields.io/npm/v/mongofire)](https://www.npmjs.com/package/mongofire)
+[![Node.js](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+---
+
+## What it does
+
+MongoFire keeps a **local MongoDB** and **MongoDB Atlas** in sync automatically.
+
+- Your app reads/writes to local MongoDB — always fast, works offline
+- MongoFire tracks every change and uploads to Atlas when online
+- Downloads remote changes from Atlas in the background
+- If you go offline, changes queue up and sync when you reconnect
+- Conflicts resolved automatically (last-writer-wins with version vectors)
+
+---
+
+## Installation
+
+```bash
+npm install mongofire
+```
+
+**Peer dependencies** (install the ones you use):
+
+```bash
+npm install mongodb mongoose
+```
+
+---
+
+## Quick Start
+
+### 1. Init config files
+
+```bash
+npx mongofire init
+```
+
+This creates:
+- `.env` — MongoDB connection strings
+- `mongofire.config.js` — which collections to sync
+- `mongofire.js` — app entry point
+
+### 2. Fill in `.env`
+
+```env
+ATLAS_URI=mongodb+srv://user:pass@cluster0.xxxxx.mongodb.net/
+LOCAL_URI=mongodb://127.0.0.1:27017
+DB_NAME=myapp
+```
+
+### 3. Start sync in your app
+
+```js
+// CommonJS
+const mongofire = require('mongofire');
+const config    = require('./mongofire.config');
+
+await mongofire.start(config);
+
+// ESM
+import mongofire from 'mongofire';
+import config from './mongofire.config.js';
+
+await mongofire.start(config);
+```
+
+### 4. Add the plugin to your Mongoose schema
+
+```js
+const mongofire = require('mongofire');
+
+const UserSchema = new mongoose.Schema({
+  name: String,
+  email: String,
+  userId: mongoose.Types.ObjectId,
+});
+
+// Track changes on the 'users' collection
+// ownerField: field used for per-user data isolation (multi-tenant)
+UserSchema.plugin(mongofire.plugin('users', { ownerField: 'userId' }));
+
+const User = mongoose.model('User', UserSchema);
+```
+
+---
+
+## Config Options
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `collections` | `string[]` | *required* | Collection names to sync |
+| `localUri` | `string` | `mongodb://localhost:27017` | Local MongoDB URI |
+| `atlasUri` | `string` | `null` | Atlas connection string |
+| `dbName` | `string` | `'mongofire'` | Database name |
+| `syncInterval` | `number` | `30000` (polling) / `5000` (realtime) | Polling interval in ms |
+| `batchSize` | `number` | `200` | Docs per upload/download batch |
+| `syncOwner` | `string\|fn` | `'*'` | Owner key for multi-tenant filtering. If a function, throwing will **abort** the sync to prevent unintended data access |
+| `realtime` | `boolean` | `false` | Use Atlas Change Streams for instant sync |
+| `onSync` | `function` | `null` | Called after each sync cycle |
+| `onError` | `function` | `null` | Called on sync errors |
+
+---
+
+## Events
+
+```js
+mongofire.on('ready',           ()  => console.log('MongoFire started'));
+mongofire.on('online',          ()  => console.log('Atlas connected'));
+mongofire.on('offline',         ()  => console.log('Working locally'));
+mongofire.on('sync',            (r) => console.log('Sync result:', r));
+mongofire.on('conflict',        (c) => console.warn('Conflict detected:', c));
+mongofire.on('realtimeStarted', ()  => console.log('Change streams active'));
+mongofire.on('stopped',         ()  => console.log('Shut down cleanly'));
+mongofire.on('error',           (e) => console.error('Sync error:', e));
+```
+
+### `conflict` event
+
+Emitted when a local write conflicts with a concurrent remote change. Use this to notify users or trigger a re-fetch:
+
+```js
+mongofire.on('conflict', ({ collection, docId, localVersion, remoteVersion, op }) => {
+  console.warn(`Conflict on ${collection}/${docId}`);
+  console.warn(`  Local was at version ${localVersion}, Atlas is at ${remoteVersion}`);
+  // Fetch the latest remote doc and re-apply your changes if needed
+});
+```
+
+---
+
+## API
+
+### `mongofire.start(config)` → `Promise<MongoFire>`
+Connect and start background sync. Concurrent calls are safe — all await the same init.
+
+### `mongofire.stop(timeoutMs?)` → `Promise<void>`
+Flush pending ops, wait for active sync, close all connections. Default timeout: 10 seconds.
+
+### `mongofire.sync(type?)` → `Promise<SyncResult>`
+Manually trigger a sync. `type` can be `'required'` (default) or `'all'`. Rapid successive calls are throttled automatically.
+
+### `mongofire.status()` → `Promise<SyncStatus>`
+Get pending op counts and online/realtime status.
+
+### `mongofire.clean(days?)` → `Promise<number>`
+Delete old sync records older than `days` days (default: **7**). Returns count of deleted records.
+
+### `mongofire.plugin(collectionName, options?)`
+Returns a Mongoose schema plugin. Options:
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `ownerField` | `string` | `null` | Dot-path to owner key field (e.g. `'userId'` or `'org._id'`) |
+| `batchSize` | `number` | `200` | Batch size for bulk operations |
+| `concurrency` | `number` | `8` | Concurrent change tracking calls per batch |
+
+---
+
+## Using the plugin directly
+
+If you prefer not to use the MongoFire singleton, import the plugin directly:
+
+```js
+// Raw Mongoose plugin
+const mongofirePlugin = require('mongofire/plugin');
+UserSchema.plugin(mongofirePlugin, { collection: 'users', ownerField: 'userId' });
+
+// Or use the factory helper (same signature as mongofire.plugin())
+const { factory } = require('mongofire/plugin');
+UserSchema.plugin(factory('users', { ownerField: 'userId' }));
+```
+
+---
+
+## Real-Time Sync
+
+Enable instant sync via MongoDB Atlas Change Streams:
+
+```js
+await mongofire.start({
+  // ...
+  realtime: true,   // requires Atlas cluster or local replica set
+});
+```
+
+If Change Streams are unavailable, MongoFire automatically falls back to polling — no crash, no config needed.
+
+---
+
+## Multi-Tenant Usage
+
+```js
+// Sync only data belonging to a specific user
+await mongofire.start({
+  collections: ['notes', 'tasks'],
+  syncOwner: () => currentUser.id,  // dynamic — re-evaluated on each sync cycle
+  // ...
+});
+```
+
+> **Note:** If `syncOwner` throws, the sync is **aborted** and an `error` event is emitted. This prevents unintended access to other users' data.
+
+---
+
+## CLI
+
+```bash
+# Create config files in current project
+npx mongofire init
+
+# Force overwrite existing config
+npx mongofire init --force
+
+# Check pending sync status
+npx mongofire status
+
+# Delete old sync records
+npx mongofire clean
+npx mongofire clean --days=7
+```
+
+---
+
+## TypeScript
+
+Full TypeScript support is included:
+
+```ts
+import mongofire, { MongoFire, SyncConfig, SyncResult, ConflictData } from 'mongofire';
+
+const config: SyncConfig = {
+  collections: ['users'],
+  atlasUri: process.env.ATLAS_URI,
+  realtime: true,
+};
+
+await mongofire.start(config);
+
+mongofire.on('sync', (result: SyncResult) => {
+  console.log(`Uploaded: ${result.uploaded}, Downloaded: ${result.downloaded}`);
+});
+
+mongofire.on('conflict', (data: ConflictData) => {
+  console.warn(`Conflict on ${data.collection}/${data.docId}`);
+});
+```
+
+---
+
+## How it Works
+
+1. **Change Tracking** — Every `save`/`update`/`delete` via Mongoose hooks is recorded locally before syncing to Atlas
+2. **Upload** — Pending local changes are uploaded to Atlas inside MongoDB transactions, with automatic retry and idempotency
+3. **Download (Bootstrap)** — First sync streams all remote docs in batches. Resumable — a crash mid-bootstrap picks up from where it left off
+4. **Download (Delta)** — Subsequent syncs fetch only changes newer than the last seen position, with no gaps even at millisecond boundaries
+5. **Conflict Resolution** — Version number → timestamp → deviceId tiebreaker (deterministic, no coin flip)
+6. **Offline** — All reads/writes work locally. Changes queue up and upload automatically when Atlas reconnects
+
+---
+
+## Collection Name Rules
+
+Collection names passed to `mongofire.plugin()` and `config.collections` must:
+- Start with a letter or digit
+- Contain only letters, digits, underscores (`_`), hyphens (`-`), or dots (`.`)
+- **Not** contain a colon (`:`)
+- **Not** start with `_mf_` (reserved for MongoFire internal use)
+
+Invalid names throw a clear error at startup.
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `MONGOFIRE_VERIFY_REMOTE` | `0` | Set to `1` to verify each uploaded doc with a checksum round-trip |
+| `MONGOFIRE_COLLECTION_CONCURRENCY` | `4` | Number of collections synced in parallel |
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE)