mdb-engine 0.1.6__py3-none-any.whl → 0.2.0__py3-none-any.whl

mdb_engine/database/README.md

@@ -180,9 +180,9 @@ Async-native interface for managing Atlas Search and Vector indexes.
 ```python
 from mdb_engine.database import AsyncAtlasIndexManager
 
-# Get index manager from collection
+# Get index manager from collection (automatically uses unscoped collection)
 collection = db.my_collection
-index_manager = AsyncAtlasIndexManager(collection._collection)  # Use unscoped collection
+index_manager = collection.index_manager  # Secure way to access index manager
 
 # Create vector search index
 await index_manager.create_vector_search_index(
@@ -277,25 +277,121 @@ await index_manager.create_search_index(
 )
 ```
 
-## Connection Pooling
+## Security Features
+
+The database module includes comprehensive security controls to prevent unauthorized access, NoSQL injection, resource exhaustion, and ensure data isolation:
+
+### Query Security
+
+All queries are automatically validated for security:
+- **Dangerous operator blocking**: Blocks `$where`, `$eval`, `$function`, and `$accumulator` operators that allow JavaScript execution
+- **Query depth limits**: Prevents deeply nested queries that could cause performance issues
+- **Regex complexity limits**: Prevents ReDoS (Regular Expression Denial of Service) attacks
+- **Pipeline validation**: Validates aggregation pipelines for safety and complexity
+
+```python
+# These queries will be blocked:
+db.collection.find({"$where": "this.status === 'active'"})  # ❌ Dangerous operator
+db.collection.aggregate([{"$match": {}}] * 100)  # ❌ Too many pipeline stages
 
-The database module provides shared MongoDB connection pooling for efficient resource usage.
+# These queries are safe:
+db.collection.find({"status": "active"})  # ✅ Safe
+db.collection.find({"age": {"$gt": 18}})  # ✅ Safe
+```
+
+### Resource Limits
 
-### Get Shared Client
+All operations have automatic resource limits to prevent resource exhaustion:
+- **Query timeouts**: All queries automatically have `maxTimeMS` set (default: 30 seconds, max: 5 minutes)
+- **Result size limits**: Maximum 10,000 documents per query (configurable)
+- **Batch size limits**: Maximum 1,000 documents per cursor batch
+- **Document size validation**: Documents are validated before insert (16MB MongoDB limit)
 
 ```python
-from mdb_engine.database import get_shared_mongo_client
+# Timeouts are automatically enforced:
+db.collection.find({"status": "active"})  # Automatically has maxTimeMS=30000
+
+# Result limits are enforced:
+db.collection.find({}, limit=20000)  # Automatically capped to 10,000
+
+# Document sizes are validated:
+large_doc = {"data": "x" * (20 * 1024 * 1024)}  # ❌ Exceeds 16MB limit
+await db.collection.insert_one(large_doc)  # Raises ResourceLimitExceeded
+```
+
+### Collection Name Validation
 
-# Get or create shared MongoDB client
-client = get_shared_mongo_client(
-    mongo_uri="mongodb://localhost:27017",
-    max_pool_size=10,
-    min_pool_size=1
+All collection names are validated for security:
+
+### Collection Name Validation
+
+All collection names are validated for security:
+- **Format validation**: Must match MongoDB naming rules (alphanumeric, underscore, dot, hyphen)
+- **Length limits**: 1-255 characters
+- **Reserved names**: System collections (`apps_config`) are blocked
+- **Reserved prefixes**: Collections starting with `system`, `admin`, `config`, or `local` are blocked
+- **Path traversal protection**: Blocks attempts to use `..`, `/`, or `\` in collection names
+
+```python
+# These will raise ValueError:
+db.system_users  # Reserved prefix
+db.apps_config    # Reserved name
+db["../other"]    # Path traversal attempt
+db["123invalid"] # Invalid format (starts with number)
+```
+
+### Cross-App Access Control
+
+Cross-app collection access is strictly controlled:
+- Apps can only read from collections of apps listed in their `read_scopes`
+- Unauthorized cross-app access attempts are logged and blocked
+- All cross-app access is logged for audit purposes
+
+```python
+# App can only read from authorized apps
+db = engine.get_scoped_db(
+    "my_app",
+    read_scopes=["my_app", "shared_app"]  # Can read from these apps
 )
 
-db = client["my_database"]
+# This works (authorized):
+collection = db.get_collection("shared_app_data")
+
+# This fails (unauthorized):
+collection = db.get_collection("other_app_data")  # Raises ValueError
 ```
 
+### Scope Validation
+
+The `get_scoped_db()` method validates all scopes:
+- `read_scopes` must be a non-empty list of valid app slugs
+- `write_scope` must be a non-empty string
+- Invalid scopes raise `ValueError` with clear error messages
+
+### Audit Logging
+
+All security-relevant events are logged:
+- Invalid collection name attempts
+- Unauthorized cross-app access attempts
+- Reserved name/prefix access attempts
+- Collection name validation failures
+
+Logs include app context (app_slug, collection_name, action) for security monitoring.
+
+### Best Practices
+
+1. **Always use scoped databases**: Never access raw MongoDB clients or databases
+2. **Validate collection names**: Use descriptive, valid collection names
+3. **Limit cross-app access**: Only grant `read_scopes` to apps that need cross-app data
+4. **Monitor audit logs**: Review security logs regularly for suspicious patterns
+5. **Follow naming conventions**: Use lowercase, underscore-separated names (e.g., `user_profiles`)
+
+## Connection Pooling
+
+The database module provides shared MongoDB connection pooling for efficient resource usage. Connection pooling is handled automatically by the engine - users should always use `engine.get_scoped_db()` for database access.
+
+**Security Note:** Direct MongoDB client creation functions are internal and not part of the public API. Always use scoped databases to ensure proper app isolation.
+
 ### Pool Metrics
 
 Monitor connection pool usage:
@@ -474,13 +570,40 @@ except OperationFailure as e:
     print(f"MongoDB operation failed: {e.details}")
 except AutoReconnect as e:
     print(f"MongoDB reconnection: {e}")
-except Exception as e:
-    print(f"Unexpected error: {e}")
+except (ConnectionFailure, ServerSelectionTimeoutError) as e:
+    print(f"Connection error: {e}")
 ```
 
 ## Integration Examples
 
-### FastAPI Integration
+### FastAPI Integration (Recommended)
+
+Use the request-scoped `get_scoped_db` dependency from `mdb_engine.dependencies`:
+
+```python
+from fastapi import Depends
+from mdb_engine import MongoDBEngine
+from mdb_engine.dependencies import get_scoped_db
+
+engine = MongoDBEngine(mongo_uri="...", db_name="...")
+app = engine.create_app(slug="my_app", manifest=Path("manifest.json"))
+
+@app.get("/data")
+async def get_data(db=Depends(get_scoped_db)):
+    # db is automatically scoped to "my_app"
+    docs = await db.my_collection.find({}).to_list(length=10)
+    return {"data": docs}
+
+@app.post("/data")
+async def create_data(db=Depends(get_scoped_db)):
+    # Writes are automatically scoped to "my_app"
+    result = await db.my_collection.insert_one({"name": "New Document"})
+    return {"inserted_id": str(result.inserted_id)}
+```
+
+### Legacy FastAPI Integration
+
+For apps not using `engine.create_app()`:
 
 ```python
 from fastapi import FastAPI, Depends
@@ -508,8 +631,8 @@ db1 = engine.get_scoped_db("app1")
 db2 = engine.get_scoped_db("app2")
 
 # Cross-app read (read from app1, write to app2)
-shared_db = ScopedMongoWrapper(
-    real_db=engine.mongo_db,
+shared_db = engine.get_scoped_db(
+    app_slug="shared",
     read_scopes=["app1", "app2"],
     write_scope="shared"
 )

mdb_engine/database/__init__.py

@@ -6,11 +6,20 @@ and MongoDB-style API for familiarity.
 """
 
 from .abstraction import AppDB, Collection, get_app_db
-from .connection import (close_shared_client, get_pool_metrics,
-                         get_shared_mongo_client, register_client_for_metrics,
-                         verify_shared_client)
-from .scoped_wrapper import (AsyncAtlasIndexManager, AutoIndexManager,
-                             ScopedCollectionWrapper, ScopedMongoWrapper)
+from .connection import (
+    close_shared_client,
+    get_pool_metrics,
+    register_client_for_metrics,
+    verify_shared_client,
+)
+from .query_validator import QueryValidator
+from .resource_limiter import ResourceLimiter
+from .scoped_wrapper import (
+    AsyncAtlasIndexManager,
+    AutoIndexManager,
+    ScopedCollectionWrapper,
+    ScopedMongoWrapper,
+)
 
 __all__ = [
     # Scoped wrappers
@@ -18,12 +27,14 @@ __all__ = [
     "ScopedCollectionWrapper",
     "AsyncAtlasIndexManager",
     "AutoIndexManager",
+    # Query security
+    "QueryValidator",
+    "ResourceLimiter",
     # Database abstraction
     "AppDB",
     "Collection",
     "get_app_db",
     # Connection pooling
-    "get_shared_mongo_client",
     "verify_shared_client",
     "get_pool_metrics",
     "register_client_for_metrics",

mdb_engine/database/abstraction.py

@@ -30,9 +30,13 @@ from ..exceptions import MongoDBEngineError
 from .scoped_wrapper import ScopedMongoWrapper
 
 try:
-    from pymongo.errors import (AutoReconnect, ConnectionFailure,
-                                InvalidOperation, OperationFailure,
-                                ServerSelectionTimeoutError)
+    from pymongo.errors import (
+        AutoReconnect,
+        ConnectionFailure,
+        InvalidOperation,
+        OperationFailure,
+        ServerSelectionTimeoutError,
+    )
 except ImportError:
     OperationFailure = Exception
     AutoReconnect = Exception
@@ -41,8 +45,12 @@ except ImportError:
 
 try:
     from motor.motor_asyncio import AsyncIOMotorCursor
-    from pymongo.results import (DeleteResult, InsertManyResult,
-                                 InsertOneResult, UpdateResult)
+    from pymongo.results import (
+        DeleteResult,
+        InsertManyResult,
+        InsertOneResult,
+        UpdateResult,
+    )
 except ImportError:
     AsyncIOMotorCursor = None
     InsertOneResult = None
@@ -112,9 +120,7 @@ class Collection:
                 context={"operation": "find_one"},
             ) from e
 
-    def find(
-        self, filter: Optional[Dict[str, Any]] = None, *args, **kwargs
-    ) -> AsyncIOMotorCursor:
+    def find(self, filter: Optional[Dict[str, Any]] = None, *args, **kwargs) -> AsyncIOMotorCursor:
         """
         Find documents matching the filter.
 
@@ -140,9 +146,7 @@ class Collection:
         """
         return self._collection.find(filter or {}, *args, **kwargs)
 
-    async def insert_one(
-        self, document: Dict[str, Any], *args, **kwargs
-    ) -> InsertOneResult:
+    async def insert_one(self, document: Dict[str, Any], *args, **kwargs) -> InsertOneResult:
         """
         Insert a single document.
 
@@ -460,9 +464,7 @@ class Collection:
                 context={"operation": "count_documents"},
             ) from e
 
-    def aggregate(
-        self, pipeline: List[Dict[str, Any]], *args, **kwargs
-    ) -> AsyncIOMotorCursor:
+    def aggregate(self, pipeline: List[Dict[str, Any]], *args, **kwargs) -> AsyncIOMotorCursor:
         """
         Perform aggregation pipeline.
 
@@ -554,11 +556,17 @@ class AppDB:
         Example:
             db.users.get("user_123")  # Instead of db.collection("users").get("user_123")
         """
-        # Only proxy collection names, not internal attributes
-        if name.startswith("_"):
+        # Explicitly block access to 'database' property (removed for security)
+        if name == "database":
             raise AttributeError(
-                f"'{type(self).__name__}' object has no attribute '{name}'"
+                "'database' property has been removed for security. "
+                "Use collection.index_manager for index operations. "
+                "All data access must go through scoped collections."
             )
+
+        # Only proxy collection names, not internal attributes
+        if name.startswith("_"):
+            raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
         return self.collection(name)
 
     @property
@@ -576,26 +584,6 @@ class AppDB:
         """
         return self._wrapper
 
-    @property
-    def database(self):
-        """
-        Access the underlying AsyncIOMotorDatabase (unscoped).
-
-        This is useful for advanced operations that need direct access to the
-        real database without scoping, such as index management or administrative
-        operations.
-
-        Returns:
-            The underlying AsyncIOMotorDatabase instance
-
-        Example:
-            # Access underlying database for index management
-            real_db = db.database
-            collection = real_db["my_collection"]
-            index_manager = AsyncAtlasIndexManager(collection)
-        """
-        return self._wrapper.database
-
 
 # FastAPI dependency helper
 async def get_app_db(request, get_scoped_db_func: Callable) -> AppDB:

mdb_engine/database/connection.py

@@ -26,8 +26,12 @@ import threading
 from typing import Any, Dict, Optional
 
 from motor.motor_asyncio import AsyncIOMotorClient
-from pymongo.errors import (ConnectionFailure, InvalidOperation,
-                            OperationFailure, ServerSelectionTimeoutError)
+from pymongo.errors import (
+    ConnectionFailure,
+    InvalidOperation,
+    OperationFailure,
+    ServerSelectionTimeoutError,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -87,10 +91,7 @@ def get_shared_mongo_client(
         # Verify client is still connected
         try:
             # Non-blocking check - if client was closed, it will be None or invalid
-            if (
-                hasattr(_shared_client, "_topology")
-                and _shared_client._topology is not None
-            ):
+            if hasattr(_shared_client, "_topology") and _shared_client._topology is not None:
                 return _shared_client
         except (AttributeError, RuntimeError):
             # Client was closed or invalid, reset and recreate
@@ -103,10 +104,7 @@ def get_shared_mongo_client(
         # Double-check pattern: another thread may have initialized while we waited
         if _shared_client is not None:
             try:
-                if (
-                    hasattr(_shared_client, "_topology")
-                    and _shared_client._topology is not None
-                ):
+                if hasattr(_shared_client, "_topology") and _shared_client._topology is not None:
                     return _shared_client
             except (AttributeError, RuntimeError):
                 # Client was closed or invalid, reset and recreate
@@ -180,7 +178,7 @@ async def verify_shared_client() -> bool:
         OperationFailure,
         InvalidOperation,
     ) as e:
-        logger.error(f"Shared MongoDB client verification failed: {e}")
+        logger.exception(f"Shared MongoDB client verification failed: {e}")
         return False
 
 
@@ -236,10 +234,7 @@ async def get_pool_metrics(
     for registered_client in _registered_clients:
         try:
             # Verify client is still valid
-            if (
-                hasattr(registered_client, "_topology")
-                and registered_client._topology is not None
-            ):
+            if hasattr(registered_client, "_topology") and registered_client._topology is not None:
                 return await _get_client_pool_metrics(registered_client)
         except (AttributeError, RuntimeError):
             # Type 2: Recoverable - if this client is invalid, try next one
@@ -338,9 +333,7 @@ async def _get_client_pool_metrics(client: AsyncIOMotorClient) -> Dict[str, Any]
         if max_pool_size and current_connections is not None:
             usage_percent = (current_connections / max_pool_size) * 100
             metrics["pool_usage_percent"] = round(usage_percent, 2)
-            metrics["active_connections"] = (
-                current_connections  # Alias for compatibility
-            )
+            metrics["active_connections"] = current_connections  # Alias for compatibility
 
             # Warn if pool usage is high
             if usage_percent > 80: