devdox-ai-locust 0.1.3.post1__py3-none-any.whl → 0.1.4__py3-none-any.whl

devdox_ai_locust/templates/mongo/db_integration.j2

@@ -0,0 +1,352 @@
+# MongoDB Integration Guide for Locust Load Testing
+
+## 🎯 Overview
+
+This guide explains how to integrate MongoDB with your Locust load tests to use realistic, production-like data instead of randomly generated data.
+
+## 📋 Table of Contents
+
+1. [Why Use MongoDB?](#why-use-mongodb)
+2. [Architecture](#architecture)
+3. [Quick Start](#quick-start)
+4. [Configuration](#configuration)
+5. [Seeding Data](#seeding-data)
+6. [Using MongoDB Data in Tests](#using-mongodb-data-in-tests)
+7. [Performance Tuning](#performance-tuning)
+8. [Troubleshooting](#troubleshooting)
+
+---
+
+## Why Use MongoDB?
+
+### Benefits
+
+✅ **Realistic Load Testing**: Use actual production-like data patterns  
+✅ **Data Consistency**: Same data across test runs for reproducibility  
+✅ **Complex Scenarios**: Test with real data relationships and edge cases  
+✅ **Performance Baseline**: Measure with production-scale data volumes  
+✅ **Debugging**: Easier to trace specific test scenarios  
+
+### When NOT to Use MongoDB
+
+❌ High-concurrency tests (1000+ users) - may bottleneck on DB  
+❌ Simple smoke tests - generated data is faster  
+❌ No MongoDB infrastructure available  
+
+---
+
+## Architecture
+
+```
+┌─────────────────┐
+│  Locust Users   │
+│   (50-500)      │
+└────────┬────────┘
+         │
+         ├──────────────┐
+         │              │
+    ┌────▼────┐    ┌───▼──────┐
+    │  Cache  │    │ MongoDB  │
+    │ (Fast)  │◄───┤ (Source) │
+    └─────────┘    └──────────┘
+         │
+    ┌────▼─────────────┐
+    │  Test Execution  │
+    └──────────────────┘
+```
+
+### Components
+
+1. **db_config.py**: MongoDB connection pool management
+2. **mongo_data_provider.py**: Smart caching and data retrieval
+3. **seed_mongodb.py**: Data seeding utility
+4. **test_data.py**: Enhanced with MongoDB fallback
+
+---
+
+## Quick Start
+
+### 1. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 2. Start MongoDB
+
+```bash
+# Using Docker (recommended)
+docker run -d -p 27017:27017 --name mongodb mongo:latest
+
+# Or using local installation
+mongod --dbpath /data/db
+```
+
+### 3. Configure Environment
+
+Create a `.env` file from the example:
+
+```bash
+cp .env.example .env
+```
+
+Enable MongoDB in `.env`:
+
+```bash
+# Enable MongoDB
+ENABLE_MONGODB=true
+USE_MONGODB_FOR_TEST_DATA=true
+
+# Connection string
+MONGODB_URI=mongodb://localhost:27017/
+MONGODB_DATABASE=locust_test_data
+
+# Pool size should be >= LOCUST_USERS
+MONGODB_MAX_POOL_SIZE=100
+LOCUST_USERS=50
+```
+
+### 4. Test Connection
+
+```bash
+python db_config.py
+```
+
+Expected output:
+```
+============================================================
+MongoDB Connection Test
+============================================================
+
+Connection URI: mongodb://localhost:27017/
+Database: locust_test_data
+Pool Size: 10 - 100
+...
+✅ Connection successful!
+```
+
+### 5. Seed Test Data
+
+```bash
+# Seed all collections with 5000 documents each
+python seed_mongodb.py --all 5000
+
+# Or seed specific collections
+python seed_mongodb.py --pets 1000 --users 500 --orders 2000
+```
+
+### 6. Run Load Tests
+
+```bash
+locust -f locustfile.py
+```
+
+---
+
+## Configuration
+
+### Critical Settings
+
+#### Connection Pool Size
+
+```bash
+# RULE: max_pool_size >= LOCUST_USERS
+MONGODB_MAX_POOL_SIZE=100  # For 50-100 concurrent users
+MONGODB_MIN_POOL_SIZE=10
+```
+
+**⚠️ WARNING**: If `max_pool_size < LOCUST_USERS`, you WILL experience:
+- Connection exhaustion
+- Test failures
+- Timeout errors
+- Cascading failures
+
+**Formula**: `MONGODB_MAX_POOL_SIZE = LOCUST_USERS × 2` (for safety margin)
+
+#### Timeout Settings
+
+```bash
+# Connection timeouts
+MONGODB_CONNECT_TIMEOUT_MS=5000      # Initial connection
+MONGODB_SOCKET_TIMEOUT_MS=10000      # Read/write operations
+MONGODB_WAIT_QUEUE_TIMEOUT_MS=10000  # Waiting for connection from pool
+```
+
+**Best Practices**:
+- `CONNECT_TIMEOUT` < `SOCKET_TIMEOUT` < `WAIT_QUEUE_TIMEOUT`
+- Increase timeouts for slow networks
+- Decrease for fast local testing
+
+---
+
+## Seeding Data
+
+### Basic Seeding
+
+```bash
+# Seed 10,000 documents per collection
+python seed_mongodb.py --all 10000
+
+# Seed specific amounts
+python seed_mongodb.py --pets 5000 --users 1000 --orders 10000
+
+# Custom batch size for faster insertion
+python seed_mongodb.py --all 5000 --batch-size 500
+```
+
+### Verify Seeded Data
+
+```bash
+# Check document counts
+python seed_mongodb.py --verify
+
+# View sample documents
+python seed_mongodb.py --sample
+```
+
+### Clear Data
+
+```bash
+# Clear all collections
+python seed_mongodb.py --clear
+
+# Clear specific collection
+python seed_mongodb.py --clear-pets
+```
+
+---
+
+## Using MongoDB Data in Tests
+
+### Automatic Integration
+
+The `TestDataGenerator` automatically uses MongoDB when available:
+
+```python
+# In your workflow tasks
+data_generator = TestDataGenerator()
+
+# This will try MongoDB first, then fallback to generation
+pet_data = data_generator.get_from_mongodb_or_generate(
+    "pets",
+    lambda: data_generator.generate_product_data()
+)
+```
+
+### Manual MongoDB Queries
+
+```python
+from mongo_data_provider import mongo_data_provider
+
+# Get random document
+pet = mongo_data_provider.get_random_document("pets")
+
+# Get with query filter
+available_pet = mongo_data_provider.get_random_document(
+    "pets",
+    query={"status": "available"}
+)
+
+# Get multiple documents
+pets = mongo_data_provider.get_multiple_documents("pets", count=10)
+```
+
+---
+
+## Performance Tuning
+
+### 1. Connection Pool Optimization
+
+```bash
+# For 50 users
+MONGODB_MAX_POOL_SIZE=100
+
+# For 200 users
+MONGODB_MAX_POOL_SIZE=400
+
+# For 500 users
+MONGODB_MAX_POOL_SIZE=1000
+```
+
+### 2. Preload Cache Before Tests
+
+```python
+from locust import events
+from mongo_data_provider import mongo_data_provider
+
+@events.test_start.add_listener
+def on_test_start(environment, **kwargs):
+    logger.info("Preloading MongoDB cache...")
+    mongo_data_provider.preload_cache("pets")
+    mongo_data_provider.preload_cache("users")
+    mongo_data_provider.preload_cache("orders")
+    logger.info("✅ Cache preloaded")
+```
+
+---
+
+## Troubleshooting
+
+### Connection Errors
+
+**Error**: `ServerSelectionTimeoutError`
+
+**Solution**:
+```bash
+# Check if MongoDB is running
+docker ps | grep mongo
+
+# Start MongoDB
+docker start mongodb
+
+# Verify connection
+python db_config.py
+```
+
+### Connection Pool Exhaustion
+
+**Error**: `WaitQueueTimeoutError`
+
+**Solution**:
+```bash
+# In .env, increase pool size
+MONGODB_MAX_POOL_SIZE=200  # Double your LOCUST_USERS
+```
+
+### Slow Performance
+
+**Solutions**:
+
+1. Check cache hit rate (should be >90%)
+2. Create indexes on frequently queried fields
+3. Increase cache size
+4. Preload cache before tests
+
+---
+
+## Best Practices
+
+### ✅ DO
+
+- Set `MONGODB_MAX_POOL_SIZE` >= `LOCUST_USERS`
+- Preload cache before tests
+- Monitor cache hit rates
+- Create indexes on frequently queried fields
+- Use connection pooling (singleton pattern)
+
+### ❌ DON'T
+
+- Set pool size smaller than user count
+- Create new connections per request
+- Query MongoDB directly in tight loops
+- Ignore timeout errors
+- Skip data seeding
+
+---
+
+## Support
+
+For issues or questions:
+1. Check logs: `python db_config.py`
+2. Verify configuration: review warnings from `validate_config()`
+3. Monitor stats: `mongo_data_provider.get_stats()`

devdox_ai_locust/templates/readme.md.j2

@@ -29,6 +29,8 @@ Professional-grade performance testing suite for **{{ api_info.title }} {{ api_i
 - **Network access** to the target API
 - **Minimum 2GB RAM** for load testing
 
+{{ db_using }}
+
 ### Installation
 ```bash
 # 1. Clone or download the test suite
@@ -43,4 +45,4 @@ cp .env.example .env
 # Edit .env with your API configuration
 
 # 4. Run your first test
-locust -f locust.py
+locust -f locustfile.py

devdox_ai_locust/templates/requirement.txt.j2

@@ -25,7 +25,10 @@ seaborn>=0.12.0
 structlog>=23.0.0
 colorama>=0.4.6
 Faker==37.6.0
+jsonschema==4.25.1
 
 # Optional: Database connectivity
-# psycopg2-binary>=2.9.0  # PostgreSQL
-# pymongo>=4.3.0  # MongoDB
+#psycopg2-binary>=2.9.0  # PostgreSQL
+#psycopg2==2.9.11
+pymongo>=4.3.0  # MongoDB
+motor>=3.3.0  # Async MongoDB driver (optional for advanced use)

devdox_ai_locust/templates/test_data.py.j2

@@ -10,7 +10,11 @@ import uuid
 from datetime import datetime, timedelta
 from typing import Any, Dict, List, Optional, Union
 from faker import Faker
-import json
+{% if data_provider_content %}
+from data_provider import mongo_data_provider
+{% endif %}
+
+
 
 fake = Faker()
 

{devdox_ai_locust-0.1.3.post1.dist-info → devdox_ai_locust-0.1.4.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devdox_ai_locust
-Version: 0.1.3.post1
+Version: 0.1.4
 Summary: AI-powered Locust load test generator from API documentation
 Author-email: Hayat Bourji <hayat.bourgi@montyholding.com>
 Maintainer-email: Hayat Bourji <hayat.bourgi@montyholding.com>
@@ -73,19 +73,27 @@ Dynamic: license-file
 DevDox AI Locust automatically generates comprehensive Locust load testing scripts from your API documentation (OpenAPI/Swagger specs). Using advanced AI capabilities, it creates realistic test scenarios, handles complex authentication flows, and generates production-ready performance tests.
 
 
-## 🆕 What's New in 0.1.3.post1
+## 🆕 What's New in 0.1.4
 
-### Performance & Reliability Improvements
+### MongoDB Integration
 
-- **🚀 Asynchronous API Calls**: Migrated from `Together` to `AsyncTogether` for non-blocking API interactions, significantly improving generation speed and responsiveness
-- **⚡ Enhanced Timeout Handling**: Implemented robust timeout logic with configurable retry mechanisms for better error resilience
-- **🔧 Improved Code Extraction**: Enhanced `<code>` block parsing with detailed validation, multiple fallback scenarios, and better error messages
-- **🛡️ Better Error Management**: Comprehensive error handling throughout the AI generation pipeline with graceful degradation
+- Added a new data provider class: MongoDataProvider
+- Connects Locust test data generation directly to MongoDB
+- Enables realistic test data retrieval for entities like users, products, orders, affiliates, etc.
+- Supports **real data** from the database and **synthetic fallback generation** when MongoDB is disabled or unavailable
 
-### Bug Fixes & Stability
+#### 2. **New MongoDataProvider Methods**
+| Method                                                   | Description |
+|----------------------------------------------------------|--------------|
+| `get_document(collection_name)`                      | Retrieves a single realistic document from MongoDB or fallback generator |
+| `get_multiple_documents(collection_name, count=10, query=None)` | Retrieves multiple documents or generates them in batches |
+| `clear_cache()`                                          | Clears in-memory cached data for all collections |
+| `get_stats()`                                            | Returns usage and cache statistics for debugging and optimization |
+
+#### 3. **Smart Fallbacks**
+If MongoDB is disabled (`enable_mongodb = false` in `db_config.py`),  
+the system automatically switches to **synthetic data generation** using the LLM-based `TestDataGenerator`.
 
-- Fixed edge cases in code block extraction where malformed responses could cause generation failures
-- Improved retry logic to handle transient API errors without interrupting the generation process
 
 ## ✨ Features
 
@@ -139,14 +147,21 @@ devdox_ai_locust generate --openapi-url https://api.example.com/openapi.json --o
 
 # Generate with custom configuration
 devdox_ai_locust generate \
-   https://petstore.swagger.io/v3/swagger.json \
+   https://petstore3.swagger.io/api/v3/openapi.json \
   --output ./petstore-tests \
   --together-api-key your_api_key \
   
+ # Generate with db integration 
+devdox_ai_locust generate \
+  https://petstore3.swagger.io/api/v3/openapi.json \
+  --output ./petstore-tests \
+  --db-type mongo \
 ```
 
 
 
+
+
 ## 📖 Documentation
 
 ### Command Line Interface

{devdox_ai_locust-0.1.3.post1.dist-info → devdox_ai_locust-0.1.4.dist-info}/RECORD

@@ -1,14 +1,14 @@
 devdox_ai_locust/__init__.py,sha256=LhG8nXZxLkyvWwJxB_OCe9t4TLa4udLtzAVfHpD3CkU,276
-devdox_ai_locust/cli.py,sha256=DDSAbT37-Rtj9XPFMtfr9nF6JIqz3md12u2DmCTD4MU,13844
-devdox_ai_locust/config.py,sha256=UfyqyI63QYbylxp5PQdAa-MoIGWTxcxr86LQ9cBr12s,453
-devdox_ai_locust/hybrid_loctus_generator.py,sha256=npnNw8OleKjXfhZMuULL1qFwSEjuUzGVGVOUDlIvXpM,36199
-devdox_ai_locust/locust_generator.py,sha256=dt455ONBs91W4JxRipNX395PlDmHojigM79aKWHHHxs,27017
+devdox_ai_locust/cli.py,sha256=80CPYwJhqym0HYfplZcAP1PygTEPobevLObNzsmK92A,14257
+devdox_ai_locust/config.py,sha256=2rCcwtCS8BE3RcM2M2WQwnpeTJgyDhXP2p5OHxXNx98,447
+devdox_ai_locust/hybrid_loctus_generator.py,sha256=alTwpdT-4FLLoT-i4vMDTGoGkfSAoOe-JwdXA1aFjwY,37187
+devdox_ai_locust/locust_generator.py,sha256=5wOk3jE6ka8RBalcHLevBAQEliLWuaO2WDH8Ao-Q8_s,29449
 devdox_ai_locust/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 devdox_ai_locust/prompt/domain.j2,sha256=ThK1mZK8bmDF6v1YsOB9wjvijljPRoseLy6lWEUG-vM,1570
 devdox_ai_locust/prompt/locust.j2,sha256=1OM-g5X1Y8fqerfSHjSBPWGRVgeLm9h22aBsA7Bt5fc,852
-devdox_ai_locust/prompt/test_data.j2,sha256=8tGmPUGkhoc2Oxe5I-2Nr1LGND5KH0LgUjDR_S3h0zA,2659
+devdox_ai_locust/prompt/test_data.j2,sha256=4QxjezD9EOGbYN7MNspPSKwQfrg_-46veTAREWbKNTQ,8796
 devdox_ai_locust/prompt/validation.j2,sha256=4r-T8oVpkIjv3ADabD9d5klKqYfHICy85xpL0TVZKr4,487
-devdox_ai_locust/prompt/workflow.j2,sha256=5Ea0y0_zyAyiFMXQGNvG_bDD-cJHhdWi10IstS3NaRk,5330
+devdox_ai_locust/prompt/workflow.j2,sha256=6g6DjgAIk3wrvpt_nOl-rYIDSM8zGHtmvEZN3ZYYjew,14864
 devdox_ai_locust/schemas/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 devdox_ai_locust/schemas/processing_result.py,sha256=jja6oJh19FLrcnV7zHAQYXQ7g0_g_xt4JEQz3i9Sa9A,789
 devdox_ai_locust/templates/base_workflow.py.j2,sha256=KQNamuRuTFAPsmY3MhH2WkYNiNP_Vu2iBUze8ZyENJk,6346
@@ -18,17 +18,20 @@ devdox_ai_locust/templates/endpoint_template.py.j2,sha256=edZ7_VETOCjCluUE0Gx4un
 devdox_ai_locust/templates/env.example.j2,sha256=MYCZu8Y2OZILB7XUnNapwebhvHC6ASD4YN7VKFOzaOY,84
 devdox_ai_locust/templates/fallback_locust.py.j2,sha256=gUEBMsKWmwq05aoxAxhWDLgujVQzP05lgfMW92ORWAM,691
 devdox_ai_locust/templates/locust.py.j2,sha256=jIc4tbJkHzBIxOfpeqrUXr_vEZWYGV5AbgtKQyIe7dQ,1843
-devdox_ai_locust/templates/readme.md.j2,sha256=WyTiq7CdNSL9YjuHs3ziciMA4s5moV_wkbGufPkgybo,1281
-devdox_ai_locust/templates/requirement.txt.j2,sha256=kfyd1GRmGU7mZxRBQ5uEDbdr8RIN46CKtV2Pcra0_os,530
-devdox_ai_locust/templates/test_data.py.j2,sha256=Xga10fOn-BghnIZCSB3lAZ4jZhH1pnVv9GYdbzURrwk,10088
+devdox_ai_locust/templates/readme.md.j2,sha256=eG085_ivXcK1Ik1xX09tgkVed2F31WaxNXiJXbrrJ7U,1301
+devdox_ai_locust/templates/requirement.txt.j2,sha256=FQqK0BN-0IqUBCYR73wZ-LoggPeeOrmhtC2s4j7oA8s,629
+devdox_ai_locust/templates/test_data.py.j2,sha256=HED9eQ9YK_RpvII52Rh1LKidxkl5Q_vdInLkkqLFZ8k,10167
 devdox_ai_locust/templates/utils.py.j2,sha256=2lcPIHypGNlfNRMv4nyC67WnvOFl07nxgJuz40NhFIg,13920
+devdox_ai_locust/templates/mongo/data_provider.py.j2,sha256=y8iqhwKmzrNgpslt-Fzr_Yco-tGUTiRc0DJqYQjjMpk,10822
+devdox_ai_locust/templates/mongo/db_config.py.j2,sha256=aMy82lSFf5V-caKb8Y7FbKvuMiJXjtNOgOl6R4aQKvY,10186
+devdox_ai_locust/templates/mongo/db_integration.j2,sha256=rK6VjSpof0A2IUQmCm4qsWr55DsY5q5CK6cLJ52jHQU,7475
 devdox_ai_locust/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 devdox_ai_locust/utils/file_creation.py,sha256=sN6rW07VBKfzwUEAG298tCRvmnzx0886w_phdEnsQZg,3957
 devdox_ai_locust/utils/open_ai_parser.py,sha256=EJsPpPSM9RiTZ0iySYIJ66knREegS324Q7mSk_4CxGM,13158
 devdox_ai_locust/utils/swagger_utils.py,sha256=L2CV_5J4krCYyIcl-KYW_SAkBxzIKOsn2kcHhZ2CI7k,3191
-devdox_ai_locust-0.1.3.post1.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
-devdox_ai_locust-0.1.3.post1.dist-info/METADATA,sha256=5z4bSGjdnkY3AznoJfsUFvMn6s3WJWgJ6JQBNh5RJ10,14239
-devdox_ai_locust-0.1.3.post1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-devdox_ai_locust-0.1.3.post1.dist-info/entry_points.txt,sha256=Eoq-gJd4WxkwwQ8pUMsqeSrfZG3yW-NmJ82iVxOc9JA,95
-devdox_ai_locust-0.1.3.post1.dist-info/top_level.txt,sha256=ZIpK9RS5xc9RXgG8mw9xPs0kwln8Kggi_7VURxtERQE,17
-devdox_ai_locust-0.1.3.post1.dist-info/RECORD,,
+devdox_ai_locust-0.1.4.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
+devdox_ai_locust-0.1.4.dist-info/METADATA,sha256=pKhgIHlGGDt6kGLoG8aPq6sr-zqwtaZhUKGXkt5plyQ,14799
+devdox_ai_locust-0.1.4.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+devdox_ai_locust-0.1.4.dist-info/entry_points.txt,sha256=Eoq-gJd4WxkwwQ8pUMsqeSrfZG3yW-NmJ82iVxOc9JA,95
+devdox_ai_locust-0.1.4.dist-info/top_level.txt,sha256=ZIpK9RS5xc9RXgG8mw9xPs0kwln8Kggi_7VURxtERQE,17
+devdox_ai_locust-0.1.4.dist-info/RECORD,,