nx-mongo 3.3.0

package/IMPROVEMENT_PLAN.md

@@ -0,0 +1,223 @@
+# Improvement Plan for SimpleMongoHelper
+
+Based on research of popular MongoDB helper packages and best practices, here's a comprehensive plan for future improvements.
+
+## Completed Features (v2.0.0)
+
+✅ Delete operations (deleteOne, deleteMany)
+✅ FindOne operation for single document queries
+✅ Count operations (countDocuments, estimatedDocumentCount)
+✅ Pagination support with limit, skip, and sort options
+✅ Aggregation pipeline support
+✅ Transaction support for multi-operation consistency
+✅ Connection retry logic with exponential backoff
+✅ Index management (createIndex, dropIndex, listIndexes)
+
+## Planned Improvements
+
+### Phase 1: Enhanced Query Features
+
+#### 1.1 Query Builder (Inspired by mquery)
+- **Priority:** Medium
+- **Description:** Fluent query builder API for constructing complex queries
+- **Benefits:** More readable and maintainable query code
+- **Example:**
+  ```typescript
+  await helper.query('users')
+    .where('age').gt(18)
+    .where('active').equals(true)
+    .sort({ createdAt: -1 })
+    .limit(10)
+    .exec();
+  ```
+
+#### 1.2 Advanced Filtering
+- **Priority:** Low
+- **Description:** Helper methods for common filter patterns
+- **Features:**
+  - Date range queries
+  - Text search helpers
+  - Array operations helpers
+
+#### 1.3 Bulk Operations
+- **Priority:** High
+- **Description:** Efficient bulk insert, update, and delete operations
+- **Benefits:** Better performance for large datasets
+- **Example:**
+  ```typescript
+  await helper.bulkInsert('users', usersArray);
+  await helper.bulkUpdate('users', updateOperations);
+  ```
+
+### Phase 2: Data Validation & Schema
+
+#### 2.1 Schema Validation (Inspired by Mongoose)
+- **Priority:** Medium
+- **Description:** Optional schema validation using Zod or Joi
+- **Benefits:** Type safety and data validation at runtime
+- **Implementation:** Use Zod (MIT licensed) for validation
+- **Example:**
+  ```typescript
+  const userSchema = z.object({
+    name: z.string(),
+    email: z.string().email(),
+    age: z.number().min(0).max(150)
+  });
+  
+  await helper.insert('users', userData, { schema: userSchema });
+  ```
+
+#### 2.2 Type-Safe Collections
+- **Priority:** Medium
+- **Description:** Collection-specific type definitions
+- **Benefits:** Better TypeScript inference and type safety
+
+### Phase 3: Performance & Caching
+
+#### 3.1 Query Result Caching
+- **Priority:** Medium
+- **Description:** Optional caching layer using node-cache (MIT licensed)
+- **Benefits:** Reduce database load for frequently accessed data
+- **Features:**
+  - TTL-based cache
+  - Cache invalidation
+  - Configurable cache keys
+
+#### 3.2 Connection Pooling Configuration
+- **Priority:** Low
+- **Description:** Expose MongoDB connection pool options
+- **Benefits:** Better control over connection management
+
+#### 3.3 Query Performance Monitoring
+- **Priority:** Low
+- **Description:** Optional query timing and performance metrics
+- **Benefits:** Identify slow queries
+
+### Phase 4: Developer Experience
+
+#### 4.1 Migration Support (Inspired by migrate-mongoose)
+- **Priority:** Medium
+- **Description:** Database migration framework
+- **Benefits:** Version-controlled schema changes
+- **Features:**
+  - Migration files
+  - Up/down migrations
+  - Migration history tracking
+
+#### 4.2 Seeding Support (Inspired by mongo-seeding)
+- **Priority:** Low
+- **Description:** Database seeding utilities
+- **Benefits:** Easy test data setup
+
+#### 4.3 Logging & Debugging
+- **Priority:** Low
+- **Description:** Optional logging for operations
+- **Features:**
+  - Query logging
+  - Error logging
+  - Performance logging
+- **Implementation:** Use winston or pino (MIT licensed)
+
+#### 4.4 Middleware/Hooks Support
+- **Priority:** Low
+- **Description:** Pre/post operation hooks
+- **Example:**
+  ```typescript
+  helper.beforeInsert((data) => {
+    data.createdAt = new Date();
+  });
+  ```
+
+### Phase 5: Advanced Features
+
+#### 5.1 Change Streams
+- **Priority:** Low
+- **Description:** Support for MongoDB change streams
+- **Benefits:** Real-time data monitoring
+
+#### 5.2 GridFS Support
+- **Priority:** Low
+- **Description:** File storage operations
+- **Benefits:** Store large files in MongoDB
+
+#### 5.3 Full-Text Search
+- **Priority:** Low
+- **Description:** Text search index and query helpers
+- **Benefits:** Better search capabilities
+
+### Phase 6: Testing & Documentation
+
+#### 6.1 Comprehensive Test Suite
+- **Priority:** High
+- **Description:** Unit tests, integration tests, and E2E tests
+- **Tools:** Jest or Mocha (MIT licensed)
+
+#### 6.2 API Documentation
+- **Priority:** High
+- **Description:** Auto-generated API docs using TypeDoc
+- **Benefits:** Better developer experience
+
+#### 6.3 Examples & Tutorials
+- **Priority:** Medium
+- **Description:** More examples and use cases in README
+- **Topics:**
+  - Common patterns
+  - Best practices
+  - Performance tips
+
+### Phase 7: Package Improvements
+
+#### 7.1 ESM Support
+- **Priority:** Medium
+- **Description:** Support for ES modules alongside CommonJS
+- **Benefits:** Modern JavaScript support
+
+#### 7.2 Tree Shaking
+- **Priority:** Low
+- **Description:** Optimize bundle size
+- **Benefits:** Smaller package size for users
+
+#### 7.3 Type Definitions
+- **Priority:** High
+- **Description:** Ensure all types are properly exported
+- **Status:** Mostly complete, needs review
+
+## Recommended Next Steps
+
+1. **Immediate (v2.1.0):**
+   - Add bulk operations (high impact, relatively simple)
+   - Improve test coverage
+   - Add more examples to README
+
+2. **Short-term (v2.2.0):**
+   - Add query builder (if users request it)
+   - Add optional Zod validation
+   - Add connection pooling configuration
+
+3. **Medium-term (v3.0.0):**
+   - Add migration support
+   - Add caching layer
+   - Add comprehensive test suite
+
+4. **Long-term:**
+   - Add advanced features based on user feedback
+   - Consider breaking into smaller packages if needed
+
+## Dependencies to Consider
+
+All packages below are MIT or Apache licensed:
+
+- **Zod** (MIT) - Schema validation
+- **node-cache** (MIT) - Caching layer
+- **winston** (MIT) - Logging
+- **Jest** (MIT) - Testing framework
+- **TypeDoc** (Apache 2.0) - Documentation generation
+
+## Notes
+
+- Keep the package lightweight - only add dependencies when necessary
+- Maintain backward compatibility when possible
+- Follow semantic versioning
+- Gather user feedback before implementing major features
+- Consider performance implications of new features
+

package/PROVIDER_INSTRUCTIONS.md

@@ -0,0 +1,460 @@
+# Generic Instructions for Config-Driven Ref Mapping and Signature-Based Deduplication
+
+## Abstract
+
+This document provides generic instructions for implementing a **config-driven collection mapping** and **signature-based deduplication** system for database operations. The solution enables applications to work with logical references (refs) instead of physical collection names, while automatically handling duplicate prevention through deterministic document signatures.
+
+## Core Concepts
+
+### 1. Reference-Based Abstraction
+
+Instead of hardcoding collection names throughout an application, define a mapping configuration that associates logical reference names with physical database collections. This provides:
+
+- **Flexibility**: Change collection names without modifying application code
+- **Environment-specific mappings**: Different collections per environment (dev, staging, prod)
+- **Query encapsulation**: Associate default queries with references
+- **Separation of concerns**: Business logic uses refs, infrastructure uses collections
+
+### 2. Signature-Based Deduplication
+
+Generate deterministic signatures for documents based on configurable key paths. Use these signatures to:
+
+- **Prevent duplicates**: Automatically detect and handle duplicate documents
+- **Upsert behavior**: Update existing documents or insert new ones based on signature matching
+- **Business logic independence**: Duplicate detection based on business rules, not database IDs
+
+## Configuration Schema
+
+### Input Configuration
+
+Define how to read data from collections:
+
+```typescript
+interface InputConfig {
+  ref: string;              // Logical reference name (e.g., "topology", "vulnerabilities")
+  collection: string;       // Physical collection name (e.g., "topology-definition-neo-data")
+  query?: Filter<any>;      // Optional default query filter
+}
+```
+
+**Purpose**: Map logical references to physical collections with optional query filters.
+
+**Example**:
+```json
+{
+  "ref": "highSeverityVulns",
+  "collection": "vulnerabilities-data",
+  "query": { "severity": { "$in": ["high", "critical"] } }
+}
+```
+
+### Output Configuration
+
+Define how to write data to collections:
+
+```typescript
+interface OutputConfig {
+  ref: string;              // Logical reference name
+  collection: string;       // Physical collection name
+  keys?: string[];         // Dot-notation paths for signature generation
+  mode?: "append" | "replace"; // Write mode
+}
+```
+
+**Purpose**: Map logical references to physical collections with signature keys and write behavior.
+
+**Example**:
+```json
+{
+  "ref": "paths",
+  "collection": "paths-neo-data",
+  "keys": ["segments[]", "edges[].from", "edges[].to", "target_role"],
+  "mode": "append"
+}
+```
+
+### Global Configuration
+
+```typescript
+interface Config {
+  inputs: InputConfig[];
+  outputs: OutputConfig[];
+  output?: {
+    mode?: "append" | "replace"; // Global default mode
+  };
+}
+```
+
+## Implementation Components
+
+### 1. Dot-Path Value Extraction
+
+**Purpose**: Extract values from nested objects using dot-notation paths with array wildcard support.
+
+**Requirements**:
+- Support simple paths: `"meta.id"` → extracts `meta.id`
+- Support array wildcards: `"segments[]"` → extracts all elements from `segments` array
+- Support nested array access: `"edges[].from"` → extracts `from` from all `edges` elements
+- Recursively flatten nested arrays
+- Handle null/undefined gracefully (return empty array)
+
+**Algorithm**:
+1. Parse path for array wildcards (`[]`)
+2. If wildcard at end: extract array, flatten recursively
+3. If wildcard in middle: extract from each array element, then continue path
+4. If simple path: traverse object properties
+5. Return array of extracted values
+
+**Example Implementation Pattern**:
+```typescript
+function extractByPath(obj: any, path: string): any[] {
+  // Handle array wildcards
+  if (path.endsWith('[]')) {
+    const basePath = path.slice(0, -2);
+    const array = extractByPath(obj, basePath);
+    return flatten(array);
+  }
+  
+  // Handle nested paths
+  // ... recursive extraction logic
+}
+```
+
+### 2. Value Normalization
+
+**Purpose**: Convert values to consistent string representations for signature computation.
+
+**Normalization Rules**:
+
+| Type | Normalization |
+|------|---------------|
+| String | As-is |
+| Number | `String(value)` |
+| Boolean | `"true"` or `"false"` |
+| Date | `value.toISOString()` (UTC) |
+| Null/Undefined | `"null"` |
+| Object | `JSON.stringify(value, Object.keys(value).sort())` (sorted keys) |
+| Array | Flatten → Normalize each → Deduplicate → Sort → `JSON.stringify` |
+
+**Critical**: Objects must have sorted keys for consistent stringification.
+
+### 3. Signature Computation
+
+**Purpose**: Generate deterministic hash signatures for documents.
+
+**Algorithm**:
+1. **Extract**: For each key path, extract values using dot-path extraction
+2. **Normalize**: Normalize all extracted values per normalization rules
+3. **Deduplicate**: Remove duplicate normalized values per key
+4. **Sort**: Sort normalized values lexicographically per key
+5. **Canonical Map**: Create `{ key1: [values], key2: [values], ... }`
+6. **Sort Keys**: Sort map keys alphabetically
+7. **Stringify**: `JSON.stringify(canonicalMap)`
+8. **Hash**: Apply hash algorithm (SHA-256 recommended)
+9. **Return**: Hex string
+
+**Properties**:
+- **Deterministic**: Same document + same keys → same signature
+- **Collision-resistant**: Different documents → different signatures (with high probability)
+- **Order-independent**: Array element order doesn't affect signature (due to sorting)
+
+### 4. Index Management
+
+**Purpose**: Ensure unique index exists on signature field for efficient lookups.
+
+**Requirements**:
+- **Idempotent**: Safe to call multiple times
+- **Conflict handling**: If index exists with wrong options, drop and recreate
+- **Field name**: Configurable (default: `_sig`)
+- **Uniqueness**: Always unique (prevents duplicates)
+
+**Algorithm**:
+1. List existing indexes on collection
+2. Find index on signature field
+3. If exists with correct options → return (no-op)
+4. If exists with wrong options → drop index
+5. Create index with correct options
+6. Return creation status and index name
+
+### 5. Load by Reference
+
+**Purpose**: Load data using logical reference name instead of collection name.
+
+**Algorithm**:
+1. Look up reference in `config.inputs[]`
+2. If not found → error
+3. Get `collection` and `query` from config
+4. Execute query on collection (with optional pagination)
+5. Return results
+
+**Features**:
+- Automatic query application
+- Pagination support
+- Transaction session support
+
+### 6. Write by Reference
+
+**Purpose**: Write data using logical reference with automatic deduplication.
+
+**Algorithm**:
+
+**If `mode === "replace"`**:
+1. Clear collection (within transaction if session provided)
+
+**If `keys` specified** (signature-based):
+1. Ensure signature index exists
+2. For each document:
+   - Compute signature using `keys`
+   - Add `_sig` field to document
+   - Create upsert operation: `{ filter: { _sig: sig }, update: { $set: doc }, upsert: true }`
+3. Execute bulk upsert operations (batch size: 1000, ordered: false)
+4. Aggregate results: count inserted vs updated
+
+**If `keys` not specified** (regular insert):
+1. Execute `insertMany` (ordered: false)
+2. Aggregate errors
+
+**Error Handling**:
+- **Never throw on first error**: Aggregate all errors
+- Continue processing after errors
+- Return structured result: `{ inserted, updated, errors[], indexCreated }`
+
+## Write Modes
+
+### Append Mode (Default)
+
+- **Behavior**: Add documents to collection
+- **Deduplication**: If keys specified, upsert by signature (update if exists, insert if new)
+- **Use case**: Incremental data updates, accumulating data over time
+
+### Replace Mode
+
+- **Behavior**: Clear collection, then insert documents
+- **Deduplication**: Still applies if keys specified (for the new batch)
+- **Use case**: Full data refresh, replacing entire dataset
+
+## Transaction Support
+
+All operations should support optional transaction sessions:
+
+- **Load operations**: Use session in query execution
+- **Write operations**: Use session in bulk operations
+- **Replace mode**: Wrap clear + write in transaction
+- **Error handling**: Rollback on transaction errors
+
+## Error Aggregation
+
+**Principle**: Never fail fast on first error. Continue processing and aggregate all errors.
+
+**Error Structure**:
+```typescript
+interface ErrorInfo {
+  index: number;      // Document index in input array
+  error: Error;     // Error object
+  doc?: any;        // Optional: problematic document
+}
+```
+
+**Benefits**:
+- Partial success: Some documents succeed even if others fail
+- Better observability: See all failures at once
+- Easier debugging: Know which documents failed
+
+## Performance Considerations
+
+### Batch Processing
+
+- Process documents in batches (recommended: 1000 per batch)
+- Use unordered bulk operations (`ordered: false`) for better performance
+- Continue processing even if a batch has errors
+
+### Index Management
+
+- Create signature index once (idempotent operation)
+- Index should be unique for duplicate prevention
+- Index name typically auto-generated (e.g., `_sig_1`)
+
+### Large Arrays in Signatures
+
+- **No automatic capping**: Arrays are processed fully
+- **Performance impact**: Large arrays (1000+ elements) may slow signature computation
+- **Future enhancement**: Consider optional `signatureLimit` parameter
+
+## Security Considerations
+
+### Index Creation Permissions
+
+- Requires `createIndex` privilege
+- Provide clear error messages if permission denied
+- Document permission requirements
+
+### Signature Field
+
+- Signature field (`_sig`) is added to documents automatically
+- Applications should not manually set this field
+- Field name is configurable but defaults to `_sig`
+
+## Testing Requirements
+
+### Unit Tests
+
+1. **Dot-path extraction**:
+   - Simple paths
+   - Array wildcards
+   - Nested arrays
+   - Null/undefined handling
+
+2. **Value normalization**:
+   - All data types
+   - Object key sorting
+   - Array flattening and sorting
+
+3. **Signature computation**:
+   - Deterministic (same input → same output)
+   - Uniqueness (different inputs → different outputs)
+   - Order independence
+
+### Integration Tests
+
+1. **Load by reference**:
+   - Valid ref lookup
+   - Invalid ref error
+   - Query application
+   - Pagination
+
+2. **Write by reference**:
+   - Append mode with deduplication
+   - Replace mode
+   - Error aggregation
+   - Transaction support
+
+3. **Index management**:
+   - Idempotent creation
+   - Conflict handling
+   - Permission errors
+
+### Performance Tests
+
+1. Large document arrays (1000+ documents)
+2. Large arrays in signatures (1000+ elements per key)
+3. Batch processing efficiency
+
+## Migration Guide
+
+### For Applications Using Direct Collection Names
+
+**Before**:
+```typescript
+const docs = await helper.loadCollection('topology-definition-neo-data', { type: 'active' });
+await helper.insert('paths-neo-data', pathDocs);
+```
+
+**After**:
+```typescript
+// Define config once
+const config = {
+  inputs: [
+    { ref: 'topology', collection: 'topology-definition-neo-data', query: { type: 'active' } }
+  ],
+  outputs: [
+    { ref: 'paths', collection: 'paths-neo-data', keys: ['segments[]', 'target_role'] }
+  ]
+};
+
+// Use refs in code
+const docs = await helper.loadByRef('topology');
+await helper.writeByRef('paths', pathDocs);
+```
+
+### Benefits
+
+- **Flexibility**: Change collection names in config only
+- **Automatic deduplication**: No manual duplicate checking
+- **Query encapsulation**: Queries defined once in config
+- **Type safety**: Ref names can be validated
+
+## Best Practices
+
+### 1. Choosing Signature Keys
+
+- **Select business-critical fields**: Fields that define document uniqueness
+- **Avoid volatile fields**: Don't include timestamps, auto-incrementing IDs
+- **Include all relevant fields**: Missing fields may cause false duplicates
+- **Test signature uniqueness**: Verify different documents produce different signatures
+
+### 2. Write Mode Selection
+
+- **Append mode**: For incremental updates, accumulating data
+- **Replace mode**: For full refreshes, replacing entire datasets
+- **Consider data size**: Replace mode may be slow for large collections
+
+### 3. Error Handling
+
+- **Always check error array**: Even if `inserted > 0`, check for errors
+- **Log errors**: Log failed documents for debugging
+- **Retry logic**: Consider retrying failed documents
+
+### 4. Configuration Management
+
+- **Environment-specific configs**: Different configs per environment
+- **Version control**: Keep configs in version control
+- **Validation**: Validate config structure at startup
+
+## API Design Principles
+
+### 1. Generic and Reusable
+
+- All logic built into the helper/library
+- Applications only pass refs and documents
+- No application-specific code in helper
+
+### 2. Backward Compatible
+
+- Existing APIs remain unchanged
+- New features are additive
+- Config is optional
+
+### 3. Explicit Configuration
+
+- Configuration is explicit (not inferred)
+- Clear separation between inputs and outputs
+- Per-output overrides for global defaults
+
+### 4. Observable Results
+
+- Return detailed results (inserted, updated, errors)
+- Include metadata (index created, etc.)
+- Never hide errors
+
+## Future Enhancements
+
+### Optional Features
+
+1. **Signature limit**: Cap array element counts for performance
+2. **Custom normalization**: Allow custom normalization functions
+3. **Signature validation**: Utilities to validate signature consistency
+4. **Config validation**: Schema validation for configuration
+5. **Metrics**: Performance metrics for operations
+
+### Considerations
+
+- **Concurrency**: Document concurrent write behavior
+- **Large datasets**: Consider streaming for very large datasets
+- **Custom hash algorithms**: Support for additional algorithms beyond SHA-256
+
+## Conclusion
+
+This solution provides a generic, reusable approach to:
+
+1. **Abstraction**: Work with logical references instead of physical collections
+2. **Deduplication**: Automatic duplicate prevention based on business logic
+3. **Flexibility**: Change collection names and queries without code changes
+4. **Observability**: Detailed results and error aggregation
+5. **Performance**: Efficient bulk operations with batching
+
+The implementation is database-agnostic in concept, though specific examples use MongoDB syntax. The principles apply to any database system that supports:
+- Indexed queries
+- Bulk write operations
+- Unique constraints
+- Transaction support
+