genlayer-test 0.4.1__tar.gz → 0.5.0__tar.gz

{genlayer_test-0.4.1 → genlayer_test-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: genlayer-test
-Version: 0.4.1
+Version: 0.5.0
 Summary: GenLayer Testing Suite
 Author: GenLayer
 License-Expression: MIT
@@ -15,7 +15,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: pytest
 Requires-Dist: setuptools>=77.0
-Requires-Dist: genlayer-py==0.6.1
+Requires-Dist: genlayer-py==0.7.2
 Requires-Dist: colorama>=0.4.6
 Requires-Dist: pyyaml
 Requires-Dist: python-dotenv
@@ -59,8 +59,8 @@ contract = factory.deploy(account=other_account)
 assert contract.account == other_account
 
 # Interact with the contract
-result = contract.get_value()  # Read method
-tx_receipt = contract.set_value(args=["new_value"])  # Write method
+result = contract.get_value().call()  # Read method
+tx_receipt = contract.set_value(args=["new_value"]).transact()  # Write method
 
 assert tx_execution_succeeded(tx_receipt)
 ```
@@ -73,6 +73,15 @@ assert tx_execution_succeeded(tx_receipt)
 - [Installation and Usage](#installation-and-usage)
 - [Key Features](#-key-features)
 - [Examples](#-examples)
+  - [Project Structure](#project-structure)
+  - [Storage Contract Example](#storage-contract-example)
+  - [Contract Deployment](#contract-deployment)
+  - [Read Methods](#read-methods)
+  - [Write Methods](#write-methods)
+  - [Assertions](#assertions)
+  - [Test Fixtures](#test-fixtures)
+  - [Statistical Analysis with `.analyze()`](#statistical-analysis-with-analyze)
+  - [Mock LLM Responses](#mock-llm-responses)
 - [Best Practices](#-best-practices)
 - [Troubleshooting](#-troubleshooting)
 - [Contributing](#-contributing)
@@ -112,19 +121,33 @@ The GenLayer Testing Suite can be configured using an optional but recommended `
 networks:
   default: localnet  # Default network to use
 
-  localnet:  # Local development network configuration
+  localnet:  # Local development network configuration (pre-configured)
     url: "http://127.0.0.1:4000/api"
+    leader_only: false  # Set to true to run all contracts in leader-only mode by default
 
-  testnet_asimov:  # Test network configuration
-    id: 4221
-    url: "http://34.32.169.58:9151"
+  studionet:  # Studio network configuration (pre-configured)
+    # Pre-configured network - accounts are automatically generated
+    # You can override any settings if needed
+
+  testnet_asimov:  # Test network configuration (pre-configured)
+    # Pre-configured network - requires accounts to be specified
     accounts:
       - "${ACCOUNT_PRIVATE_KEY_1}"
       - "${ACCOUNT_PRIVATE_KEY_2}"
       - "${ACCOUNT_PRIVATE_KEY_3}"
+    from: "${ACCOUNT_PRIVATE_KEY_2}"  # Optional: specify default account
+
+  custom_network:  # Custom network configuration
+    id: 1234
+    url: "http://custom.network:8545"
+    accounts:
+      - "${CUSTOM_ACCOUNT_1}"
+      - "${CUSTOM_ACCOUNT_2}"
+    from: "${CUSTOM_ACCOUNT_1}"  # Optional: specify default account
 
 paths:
   contracts: "contracts"  # Path to your contracts directory
+  artifacts: "artifacts" # Path to your artifacts directory
 
 environment: .env  # Path to your environment file containing private keys and other secrets
 ```
@@ -133,16 +156,35 @@ Key configuration sections:
 
 1. **Networks**: Define different network environments
    - `default`: Specifies which network to use by default
+   - **Pre-configured Networks**:
+     - `localnet`: Local development network with auto-generated test accounts
+     - `studionet`: GenLayer Studio network with auto-generated test accounts
+     - `testnet_asimov`: Public testnet (requires account configuration)
    - Network configurations can include:
-     - `url`: The RPC endpoint for the network
-     - `id`: Chain ID
+     - `url`: The RPC endpoint for the network (optional for pre-configured networks)
+     - `id`: Chain ID (optional for pre-configured networks)
      - `accounts`: List of account private keys (using environment variables)
-   - Special case for `localnet`:
-     - If a network is named `localnet`, missing fields will be filled with default values
-     - For all other network names, `id`, `url`, and `accounts` are required fields
+     - `from`: Specify which account to use as the default for transactions (optional, defaults to first account)
+     - `leader_only`: Leader only mode
+   - For custom networks (non-pre-configured), `id`, `url`, and `accounts` are required fields
+
+**Note on Environment Variables**: When using environment variables in your configuration (e.g., `${ACCOUNT_PRIVATE_KEY_1}`), ensure they are properly set in your `environment` file. If an environment variable is not found, the system will raise a clear error message indicating which variable is missing.
+
+**Default Account Selection**: The `from` field allows you to specify which account from the `accounts` list should be used as the default for deployments and transactions. If not specified, the first account in the list is used by default. This is useful when you want a specific account to be the primary account for your tests without having to specify it in every transaction.
+
+Example:
+```yaml
+testnet_asimov:
+  accounts:
+    - "${DEPLOYER_KEY}"      # accounts[0]
+    - "${USER_KEY}"          # accounts[1] 
+    - "${ADMIN_KEY}"         # accounts[2]
+  from: "${ADMIN_KEY}"       # Use ADMIN_KEY as default instead of DEPLOYER_KEY
+```
 
 2. **Paths**: Define important directory paths
    - `contracts`: Location of your contract files
+   - `artifacts`: Location of your artifacts files (analysis results will be stored here)
 
 3. **Environment**: Path to your `.env` file containing sensitive information like private keys
 
@@ -187,11 +229,23 @@ $ gltest --contracts-dir <path_to_contracts>
 # Run tests on localnet (default)
 $ gltest --network localnet
 
-# Run tests on testnet
+# Run tests on studionet
+$ gltest --network studionet
+
+# Run tests on testnet (requires account configuration)
 $ gltest --network testnet_asimov
+
+# Run tests on a custom network
+$ gltest --network custom_network
 ```
 The `--network` flag allows you to specify which network configuration to use from your `gltest.config.yaml`. If not specified, it will use the `default` network defined in your config file.
 
+**Pre-configured Networks**:
+- `localnet` and `studionet`: Work out of the box with auto-generated test accounts
+- `testnet_asimov`: Requires account configuration in `gltest.config.yaml`
+
+When using `testnet_asimov` without proper account configuration, you'll receive a clear error message directing you to configure accounts in your config file.
+
 7. Run tests with a custom RPC url
 ```bash
 $ gltest --rpc-url <custom_rpc_url>
@@ -231,6 +285,20 @@ def test_with_mocked_llm(setup_validators):
 
 Note: This feature is only available when running tests on localnet.
 
+11. Run tests with leader-only mode enabled
+```bash
+$ gltest --leader-only
+```
+The `--leader-only` flag configures all contract deployments and write operations to run only on the leader node. This is useful for:
+- Faster test execution by avoiding consensus
+- Testing specific leader-only scenarios
+- Development and debugging purposes
+- Reducing computational overhead in test environments
+
+When this flag is enabled, all contracts deployed and all write transactions will automatically use leader-only mode, regardless of individual method parameters.
+
+**Note:** Leader-only mode is only available for studio-based networks (localhost, 127.0.0.1, *.genlayer.com, *.genlayerlabs.com). When enabled on other networks, it will have no effect and a warning will be logged.
+
 ## 🚀 Key Features
 
 - **Pytest Integration** – Extends pytest to support intelligent contract testing, making it familiar and easy to adopt.
@@ -251,8 +319,9 @@ Before diving into the examples, let's understand the basic project structure:
 genlayer-example/
 ├── contracts/              # Contract definitions
 │   └── storage.py          # Example storage contract
-└── test/                   # Test files
-    └── test_contract.py    # Contract test cases
+├── test/                   # Test files
+│   └── test_contract.py    # Contract test cases
+└── gltest.config.yaml      # Configuration file
 ```
 
 ### Storage Contract Example
@@ -311,7 +380,6 @@ def test_deployment():
         args=["initial_value"],  # Constructor arguments
         account=get_default_account(),  # Account to deploy from
         consensus_max_rotations=3,  # Optional: max consensus rotations
-        leader_only=False,  # Optional: whether to run only on leader
     )
     
     # Contract is now deployed and ready to use
@@ -320,7 +388,7 @@ def test_deployment():
 
 ### Read Methods
 
-Reading from the contract is straightforward:
+Reading from the contract requires calling `.call()` on the method:
 
 ```python
 from gltest import get_contract_factory
@@ -332,7 +400,7 @@ def test_read_methods():
     contract = factory.deploy()
 
     # Call a read-only method
-    result = contract.get_value(args=[])
+    result = contract.get_storage(args=[]).call()
     
     # Assert the result matches the initial value
     assert result == "initial_value"
@@ -340,7 +408,7 @@ def test_read_methods():
 
 ### Write Methods
 
-Writing to the contract requires transaction handling:
+Writing to the contract requires calling `.transact()` on the method. Method arguments are passed to the write method, while transaction parameters are passed to `.transact()`:
 
 ```python
 from gltest import get_contract_factory
@@ -354,9 +422,9 @@ def test_write_methods():
     # Call a write method with arguments
     tx_receipt = contract.update_storage(
         args=["new_value"],  # Method arguments
+    ).transact(
         value=0,  # Optional: amount of native currency to send
         consensus_max_rotations=3,  # Optional: max consensus rotations
-        leader_only=False,  # Optional: whether to run only on leader
         wait_interval=1,  # Optional: seconds between status checks
         wait_retries=10,  # Optional: max number of retries
     )
@@ -365,7 +433,7 @@ def test_write_methods():
     assert tx_execution_succeeded(tx_receipt)
     
     # Verify the value was updated
-    assert contract.get_storage() == "new_value"
+    assert contract.get_storage().call() == "new_value"
 ```
 
 ### Assertions
@@ -503,6 +571,172 @@ Fixtures help maintain clean, DRY test code by:
 - Ensuring consistent test environments
 - Managing resource cleanup automatically
 - Providing appropriate scoping for performance
+### Statistical Analysis with `.analyze()`
+
+The GenLayer Testing Suite provides a powerful `.analyze()` method for write operations that performs statistical analysis through multiple simulation runs. This is particularly useful for testing LLM-based contracts where outputs may vary:
+
+```python
+from gltest import get_contract_factory
+
+def test_analyze_method():
+    factory = get_contract_factory("LlmContract")
+    contract = factory.deploy()
+    
+    # Analyze a write method's behavior across multiple runs
+    analysis = contract.process_with_llm(args=["input_data"]).analyze(
+        provider="openai",           # LLM provider
+        model="gpt-4o",             # Model to use
+        runs=100,                   # Number of simulation runs (default: 100)
+        config=None,                # Optional: provider-specific config
+        plugin=None,                # Optional: plugin name
+        plugin_config=None,         # Optional: plugin configuration
+    )
+    
+    # Access analysis results
+    print(f"Method: {analysis.method}")
+    print(f"Success rate: {analysis.success_rate:.2f}%")
+    print(f"Reliability score: {analysis.reliability_score:.2f}%")
+    print(f"Unique states: {analysis.unique_states}")
+    print(f"Execution time: {analysis.execution_time:.1f}s")
+    
+    # The analysis returns a MethodStatsSummary object with:
+    # - method: The contract method name
+    # - args: Arguments passed to the method
+    # - total_runs: Total number of simulation runs
+    # - successful_runs: Number of successful executions
+    # - failed_runs: Number of failed executions
+    # - unique_states: Number of unique contract states observed
+    # - reliability_score: Percentage of runs with the most common state
+    # - execution_time: Total time for all simulations
+```
+
+The `.analyze()` method helps you:
+- Test non-deterministic contract methods
+- Measure consistency of LLM-based operations
+- Identify edge cases and failure patterns
+- Benchmark performance across multiple runs
+
+### Mock LLM Responses
+
+The Mock LLM system allows you to simulate Large Language Model responses in GenLayer tests. This is essential for creating deterministic tests by providing predefined responses instead of relying on actual LLM calls.
+
+#### Basic Structure
+
+The mock system consists of a response dictionary that maps GenLayer methods to their mocked responses:
+
+```python
+mock_response = {
+    "response": {},                               # Optional: mocks gl.nondet.exec_prompt
+    "eq_principle_prompt_comparative": {},        # Optional: mocks gl.eq_principle.prompt_comparative
+    "eq_principle_prompt_non_comparative": {}     # Optional: mocks gl.eq_principle.prompt_non_comparative
+}
+
+setup_validators(mock_response)
+```
+
+#### Method Mappings
+
+| Mock Key | GenLayer Method |
+|----------|----------------|
+| `"response"` | `gl.nondet.exec_prompt` |
+| `"eq_principle_prompt_comparative"` | `gl.eq_principle.prompt_comparative` |
+| `"eq_principle_prompt_non_comparative"` | `gl.eq_principle.prompt_non_comparative` |
+
+#### How It Works
+
+The mock system works by pattern matching against the user message that gets built internally. When a GenLayer method is called:
+
+1. A user message is constructed internally (`<user_message>`)
+2. The mock system searches for strings within that message
+3. If a matching string is found in the mock dictionary, the associated response is returned
+
+##### String Matching Rules
+
+The system performs **substring matching** on the user message. The key in your mock dictionary must be contained within the actual user message.
+
+#### Examples
+
+##### Basic Example
+
+```python
+# Mock setup
+mock_response = {
+    "eq_principle_prompt_comparative": {
+        "The value of give_coin has to match": True
+    }
+}
+setup_validators(mock_response)
+
+# In your contract
+result = gl.eq_principle.prompt_comparative(
+    get_wizard_answer, 
+    "The value of give_coin has to match"  # This string will be matched
+)
+# result will be True
+```
+
+##### Substring Matching Examples
+
+✅ **Will work** - Partial match:
+```python
+"eq_principle_prompt_comparative": {
+    "The value of give_coin": True  # Substring of the full message
+}
+```
+
+❌ **Won't work** - Extra words break the match:
+```python
+"eq_principle_prompt_comparative": {
+    "The good value of give_coin": True  # "good" is not in the actual message
+}
+```
+
+##### Complete Example
+
+```python
+from gltest import get_contract_factory
+from gltest.fixtures import setup_validators
+
+def test_with_mocked_llm(setup_validators):
+    # Define mock responses
+    mock_response = {
+        "response": {
+            "What is the weather?": "It's sunny today",
+            "Calculate 2+2": "4"
+        },
+        "eq_principle_prompt_comparative": {
+            "values must be equal": True,
+            "amounts should match": False
+        },
+        "eq_principle_prompt_non_comparative": {
+            "Is this valid?": True
+        }
+    }
+    
+    # Initialize the mock system
+    setup_validators(mock_response)
+    
+    # Deploy and test your contract
+    factory = get_contract_factory("MyLLMContract")
+    contract = factory.deploy()
+    
+    # Your LLM methods will use the mocked responses
+    result = contract.check_weather()  # Uses mocked response
+```
+
+#### Best Practices
+
+1. **Be specific with match strings**: Use unique substrings that won't accidentally match other prompts
+2. **Test your matches**: Verify that your mock strings actually appear in the generated user messages
+3. **Keep mocks simple**: Mock responses should be minimal and focused on the test case
+4. **Document your mocks**: Comment why specific responses are mocked for future reference
+5. **Use with `--test-with-mocks` flag**: Enable mocking when running tests: `gltest --test-with-mocks`
+
+#### Notes
+
+- Mock responses are only available when running tests on localnet
+- The `setup_validators` fixture handles the mock setup when provided with a mock_response
+- Mocking is particularly useful for CI/CD pipelines where deterministic results are required
 
 ## 📝 Best Practices
 
@@ -546,6 +780,7 @@ Fixtures help maintain clean, DRY test code by:
    ```python
    tx_receipt = contract.set_value(
        args=["new_value"],
+   ).transact(
        wait_interval=2,  # Increase wait interval between status checks
        wait_retries=20,  # Increase number of retry attempts
    )
@@ -558,13 +793,11 @@ Fixtures help maintain clean, DRY test code by:
    # Try with increased consensus parameters
    contract = factory.deploy(
        consensus_max_rotations=5,  # Increase number of consensus rotations
-       leader_only=True,  # Try leader-only mode for faster execution
    )
    
    # For critical operations, use more conservative settings
    contract = factory.deploy(
        consensus_max_rotations=10,  # More rotations for better reliability
-       leader_only=False,  # Full consensus for better security
        wait_interval=3,  # Longer wait between checks
        wait_retries=30  # More retries for consensus
    )