pdd-cli 0.0.90__py3-none-any.whl → 0.0.121__py3-none-any.whl

pdd/prompts/generate_test_LLM.prompt

@@ -9,6 +9,16 @@
  - The example file will be saved at: <test_file_path>{test_file_path}</test_file_path>
  - The module name (without extension) is: <module_name>{module_name}</module_name>
 
+% EXISTING TESTS (if provided - your output will be APPENDED to this file):
+<existing_tests>{existing_tests}</existing_tests>
+
+% If existing tests are provided above:
+    - Generate ONLY NEW test functions (your output will be appended to the existing file)
+    - Do NOT include import statements (they already exist in the file)
+    - Do NOT duplicate any existing test function names
+    - Maintain consistent style with existing tests (fixtures, naming conventions)
+    - Focus on testing functionality NOT already covered by existing tests
+
 % Follow these rules:
     - CRITICAL: You MUST analyze the actual code provided in code_under_test and generate tests for the EXACT functions defined in that code
     - CRITICAL: Import statements must use the ACTUAL module name from the code file path, not generic names. Include the necessary code/path info to import the code under test.
@@ -22,7 +32,15 @@
     - Setup and teardown methods should only use public APIs and environment variables, never reset internal module state directly.
     - Design tests to be independent of implementation details that might change when code is regenerated.
     - For test isolation, use fixtures and mocking of external dependencies rather than manipulating internal module state. In general minimize the amount of mocking needed so that the tests are more robust to changes in the code under test and more code is tested.
-<include>./context/test.prompt</include>
+
+% TEST ISOLATION PRINCIPLE:
+% CRITICAL: Each test MUST be isolated and not pollute state for other tests.
+    - Tests must clean up any state they modify (environment variables, global state, files, mocks)
+    - Use the testing framework's built-in isolation mechanisms (fixtures, setup/teardown, context managers)
+    - Avoid modifying global or module-level state directly; if unavoidable, always restore original state
+    - Prefer function-scoped test resources over shared/module-scoped ones to ensure isolation
+
+<include>context/test.prompt</include>
 
 <instructions>
     1. FIRST: Carefully analyze the ACTUAL code provided in code_under_test:

pdd/prompts/generate_test_from_example_LLM.prompt

@@ -0,0 +1,366 @@
+% You are an expert Software Test Engineer. Your goal is to generate tests based on the intended behavior described in a prompt and demonstrated in an example file.
+
+% Here a description of what the code is supposed to do and was the prompt that generated the code: <prompt_that_generated_code>{prompt_that_generated_code}</prompt_that_generated_code>
+
+% Here is an example showing how the module should be used: <example_usage>{example}</example_usage>
+
+% File path information:
+ - The example file is located at: <example_file_path>{source_file_path}</example_file_path>
+ - The test file will be saved at: <test_file_path>{test_file_path}</test_file_path>
+ - The module name (without extension) is: <module_name>{module_name}</module_name>
+
+% EXISTING TESTS (if provided - your output will be APPENDED to this file):
+<existing_tests>{existing_tests}</existing_tests>
+
+% If existing tests are provided above:
+    - Generate ONLY NEW test functions (your output will be appended to the existing file)
+    - Do NOT include import statements (they already exist in the file)
+    - Do NOT duplicate any existing test function names
+    - Maintain consistent style with existing tests (fixtures, naming conventions)
+    - Focus on testing functionality NOT already covered by existing tests
+
+% Follow these rules:
+    - CRITICAL: Analyze the EXAMPLE to understand the API (function names, parameters, return values)
+    - CRITICAL: Import statements must match the module structure shown in the example
+    - CRITICAL: Test the intended function names and behavior based on the prompt
+    - The module name for the code under test will have the same name as the function name
+    - The unit test should be in {language}. If Python, use pytest.
+    - Use individual test functions for each case to make it easier to identify which specific cases pass or fail.
+    - Use the description of the functionality in the prompt to generate tests with useful tests with good code coverage.
+    - Focus on testing the INTENDED FUNCTIONALITY, not implementation details.
+    - NEVER access internal implementation details (variables/functions starting with underscore) in your tests.
+    - Setup and teardown methods should only use public APIs and environment variables, never reset internal module state directly.
+    - Design tests to be independent of implementation details.
+    - For test isolation, use fixtures and mocking of external dependencies rather than manipulating internal module state. In general minimize the amount of mocking needed so that the tests are more robust to changes in the code under test and more code is tested.
+    - Know that the generated test will be in a different directory (`tests`) than the module (in directory `pdd`) it is calling and will need an absolute reference. The module file name will be same as the function name.
+    - Created files should be in the `output` directory.
+    - Data files (language_format.csv and llm_model.csv) already exist in the PDD_PATH/`data` directory. Do not write over them. It already contains data for popular languages and LLM models and can be used for tests.
+    - The PDD_PATH environment variable is already set.
+
+% PYTEST TEST ISOLATION AND ANTI-POLLUTION RULES:
+% CRITICAL: Generated tests MUST be isolated and not pollute state for other tests. Follow these rules strictly:
+
+% 1. ENVIRONMENT VARIABLES:
+    - ALWAYS use monkeypatch.setenv() or monkeypatch.delenv() instead of os.environ["VAR"] = "value"
+    - NEVER use direct os.environ manipulation - it persists beyond the test and pollutes other tests
+    - BAD:  os.environ["API_KEY"] = "test_key"  # POLLUTION: persists after test ends
+    - GOOD: monkeypatch.setenv("API_KEY", "test_key")  # Auto-cleaned by pytest
+
+% 2. MOCKING EXTERNAL DEPENDENCIES:
+    - Use context managers or monkeypatch for mocks - they auto-cleanup after the test
+    - Prefer monkeypatch.setattr() over unittest.mock.patch() decorators at module level
+    - BAD:  @patch('module.func') at module/class level  # Can leak if exception occurs
+    - GOOD: monkeypatch.setattr('module.func', mock_func)  # Always cleaned up
+    - GOOD: with patch('module.func') as mock:  # Context manager ensures cleanup
+
+% 3. FIXTURE CLEANUP WITH YIELD:
+    - Use yield-based fixtures with cleanup code after yield for any resources
+    - Prefer function-scoped fixtures over module or session scope to ensure isolation
+    - BAD:  @pytest.fixture(scope="module") without cleanup  # State leaks between tests
+    - GOOD: @pytest.fixture with yield and cleanup after yield  # Always cleans up
+    - Example of proper fixture:
+        @pytest.fixture
+        def temp_resource():
+            resource = setup_resource()
+            yield resource
+            resource.cleanup()  # Always runs after test, even on failure
+
+% 4. SYS.MODULES MANIPULATION:
+    - AVOID manipulating sys.modules directly whenever possible
+    - If unavoidable, ALWAYS save and restore in try/finally or fixture with yield
+    - BAD:  sys.modules["module"] = mock_module  # Pollutes all subsequent tests
+    - GOOD: Use a fixture that saves, mocks, and restores:
+        @pytest.fixture
+        def mock_module():
+            saved = sys.modules.get("module")
+            sys.modules["module"] = MagicMock()
+            yield
+            if saved is not None:
+                sys.modules["module"] = saved
+            elif "module" in sys.modules:
+                del sys.modules["module"]
+
+% 5. FILE SYSTEM OPERATIONS:
+    - ALWAYS use the tmp_path fixture for creating temporary files and directories
+    - NEVER create files in the working directory or fixed paths
+    - BAD:  with open("test_output.txt", "w") as f: ...  # Leaves file behind
+    - GOOD: def test_file(tmp_path): (tmp_path / "test_output.txt").write_text(...)
+
+% 6. GLOBAL/MODULE STATE:
+    - Never modify global variables or module-level state directly in tests
+    - Use monkeypatch.setattr() for any module-level variables that need changing
+    - Reset any singleton instances using fixtures with proper teardown
+
+% SUMMARY OF GOOD PATTERNS:
+    - Use tmp_path fixture for file operations
+    - Use monkeypatch fixture for environment variables and attributes
+    - Use pytest.raises() as context manager for exception testing
+    - Prefer function-scoped fixtures over module or session scope
+    - Use yield in fixtures to ensure cleanup runs even on test failure
+
+% 7. MODULE-LEVEL SYS.MODULES FOR IMPORT-TIME DEPENDENCIES:
+    - Sometimes you must mock modules BEFORE importing the code under test
+      (e.g., when decorators or top-level imports need mocking)
+    - ALWAYS save original values, apply mocks, load module, then RESTORE immediately
+    - BAD:  sys.modules.update(mocks); exec_module(...)  # No cleanup - pollutes all tests!
+    - GOOD: See PATTERN 7 in pytest_isolation_example.py for the full save/restore pattern
+
+<isolation_example>
+"""
+Example code patterns demonstrating proper test isolation to prevent test pollution.
+
+This file provides reference implementations of CORRECT patterns that should be used
+in generated tests. These patterns prevent test pollution and ensure tests are independent.
+
+IMPORTANT: This is a context file for the LLM, not a runnable test file.
+"""
+
+import os
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+
+# =============================================================================
+# PATTERN 1: Environment Variable Handling with monkeypatch
+# =============================================================================
+
+def test_set_env_var_with_monkeypatch(monkeypatch):
+    """GOOD: Use monkeypatch.setenv() for setting env vars.
+
+    monkeypatch automatically restores the original value after the test,
+    preventing pollution of subsequent tests.
+    """
+    monkeypatch.setenv("TEST_API_KEY", "test_key_123")
+    assert os.environ["TEST_API_KEY"] == "test_key_123"
+    # Automatically cleaned up after test
+
+
+def test_delete_env_var_with_monkeypatch(monkeypatch):
+    """GOOD: Use monkeypatch.delenv() for removing env vars."""
+    monkeypatch.setenv("TEMP_VAR_TO_DELETE", "value")
+    monkeypatch.delenv("TEMP_VAR_TO_DELETE")
+    assert "TEMP_VAR_TO_DELETE" not in os.environ
+
+
+def test_multiple_env_vars(monkeypatch):
+    """GOOD: Set multiple env vars safely with monkeypatch."""
+    monkeypatch.setenv("VAR_ONE", "value1")
+    monkeypatch.setenv("VAR_TWO", "value2")
+    monkeypatch.setenv("VAR_THREE", "value3")
+    # All automatically cleaned up
+
+
+# =============================================================================
+# PATTERN 2: Mocking with monkeypatch and context managers
+# =============================================================================
+
+def test_mock_function_with_monkeypatch(monkeypatch):
+    """GOOD: Use monkeypatch.setattr() for mocking functions."""
+    def mock_getcwd():
+        return "/mock/path"
+
+    monkeypatch.setattr(os, "getcwd", mock_getcwd)
+    assert os.getcwd() == "/mock/path"
+    # Original function automatically restored after test
+
+
+def test_mock_with_context_manager():
+    """GOOD: Use patch as context manager for scoped mocking."""
+    with patch("os.path.exists") as mock_exists:
+        mock_exists.return_value = True
+        assert os.path.exists("/fake/nonexistent/path") is True
+    # Mock is automatically removed when context exits
+
+
+# =============================================================================
+# PATTERN 3: File System Operations with tmp_path
+# =============================================================================
+
+def test_create_temp_file(tmp_path):
+    """GOOD: Use tmp_path fixture for temporary files."""
+    test_file = tmp_path / "test_output.txt"
+    test_file.write_text("test content")
+    assert test_file.exists()
+    assert test_file.read_text() == "test content"
+    # tmp_path is automatically cleaned up by pytest
+
+
+def test_create_temp_directory_structure(tmp_path):
+    """GOOD: Create directory structures in tmp_path."""
+    subdir = tmp_path / "subdir" / "nested"
+    subdir.mkdir(parents=True)
+    config_file = subdir / "config.json"
+    config_file.write_text('{{"key": "value"}}')
+    assert config_file.exists()
+
+
+# =============================================================================
+# PATTERN 4: Fixtures with Proper Cleanup
+# =============================================================================
+
+@pytest.fixture
+def resource_with_cleanup():
+    """GOOD: Fixture with proper cleanup using yield.
+
+    The cleanup code after yield always runs, even if the test fails.
+    """
+    # Setup
+    resource = {{"initialized": True, "data": []}}
+    yield resource
+    # Cleanup - always runs
+    resource["initialized"] = False
+    resource["data"].clear()
+
+
+@pytest.fixture
+def mock_module_with_cleanup():
+    """GOOD: Fixture for sys.modules with save/restore.
+
+    This pattern ensures sys.modules is always restored to its original
+    state after the test, preventing pollution.
+    """
+    module_name = "test_mock_module"
+    saved = sys.modules.get(module_name)
+
+    mock_module = MagicMock()
+    sys.modules[module_name] = mock_module
+
+    yield mock_module
+
+    # Cleanup - restore original state
+    if saved is not None:
+        sys.modules[module_name] = saved
+    elif module_name in sys.modules:
+        del sys.modules[module_name]
+
+
+def test_with_resource_cleanup(resource_with_cleanup):
+    """Test using fixture with automatic cleanup."""
+    assert resource_with_cleanup["initialized"] is True
+    resource_with_cleanup["data"].append("test_item")
+
+
+def test_with_mock_module_cleanup(mock_module_with_cleanup):
+    """Test using sys.modules fixture with cleanup."""
+    assert "test_mock_module" in sys.modules
+
+
+# =============================================================================
+# PATTERN 5: Exception Testing with Context Manager
+# =============================================================================
+
+def test_exception_with_context_manager():
+    """GOOD: Use pytest.raises() as context manager."""
+    with pytest.raises(ValueError) as exc_info:
+        raise ValueError("expected error message")
+    assert "expected error message" in str(exc_info.value)
+
+
+def test_exception_with_match():
+    """GOOD: Use match parameter for regex matching."""
+    with pytest.raises(ValueError, match=r"invalid.*value"):
+        raise ValueError("invalid input value provided")
+
+
+# =============================================================================
+# PATTERN 6: Combining Multiple Isolation Techniques
+# =============================================================================
+
+def test_combined_env_and_file(monkeypatch, tmp_path):
+    """GOOD: Combine monkeypatch and tmp_path for full isolation."""
+    config_path = tmp_path / "config"
+    config_path.mkdir()
+    monkeypatch.setenv("CONFIG_DIR", str(config_path))
+
+    config_file = config_path / "settings.json"
+    config_file.write_text('{{"debug": true}}')
+
+    assert os.environ["CONFIG_DIR"] == str(config_path)
+    assert config_file.exists()
+    # Both automatically cleaned up
+
+
+def test_combined_mock_and_env(monkeypatch):
+    """GOOD: Combine function mocking with environment variables."""
+    monkeypatch.setattr(os.path, "isfile", lambda x: True)
+    monkeypatch.setenv("TEST_MODE", "true")
+
+    assert os.path.isfile("/any/path") is True
+    assert os.environ["TEST_MODE"] == "true"
+    # Both automatically cleaned up
+
+
+# =============================================================================
+# PATTERN 7: Module-Level sys.modules for Import-Time Dependencies
+# =============================================================================
+#
+# When you need to mock modules BEFORE importing code under test
+# (e.g., for decorators or top-level imports), use this pattern.
+#
+# This is necessary when the code under test has decorators or module-level
+# imports that you need to mock. Unlike fixture-based mocking, this happens
+# at test file load time, before any test functions run.
+#
+# CRITICAL: You MUST restore original modules after loading, or you will
+# pollute sys.modules for all other test files during collection!
+#
+# Example usage (place at module level, outside any test function):
+#
+# import importlib.util
+# from unittest.mock import MagicMock
+#
+# # Step 1: Define mocks for dependencies that need mocking at import time
+# _mock_decorator = lambda f: f  # Pass-through decorator
+# _mock_dependency = MagicMock(some_decorator=_mock_decorator)
+# _module_mocks = {{
+#     "some.dependency": _mock_dependency,
+# }}
+#
+# # Step 2: Save originals BEFORE patching
+# _original_modules = {{key: sys.modules.get(key) for key in _module_mocks}}
+#
+# # Step 3: Apply mocks to sys.modules
+# sys.modules.update(_module_mocks)
+#
+# # Step 4: Load the module under test using importlib
+# _module_path = os.path.join(os.path.dirname(__file__), "..", "src", "module.py")
+# _module_path = os.path.abspath(_module_path)
+# _spec = importlib.util.spec_from_file_location("my_module", _module_path)
+# _module = importlib.util.module_from_spec(_spec)
+# sys.modules["my_module"] = _module
+# _spec.loader.exec_module(_module)
+# function_to_test = _module.function_to_test
+#
+# # Step 5: RESTORE originals immediately after load
+# # This is CRITICAL to avoid polluting other test files!
+# for key, original in _original_modules.items():
+#     if original is None:
+#         sys.modules.pop(key, None)
+#     else:
+#         sys.modules[key] = original
+#
+# # Now you can write normal test functions using function_to_test
+# def test_something():
+#     result = function_to_test()
+#     assert result == expected
+
+</isolation_example>
+
+
+<instructions>
+    1. FIRST: Carefully analyze the EXAMPLE to understand:
+        - How to import the module (exact import statements)
+        - What functions/classes are exposed
+        - How they are called (parameters, return values)
+    2. SECOND: Analyze the prompt that describes the intended functionality and edge cases.
+    3. THIRD: For each edge case explain whether it is better to do the test using Z3 formal verification or unit tests.
+    4. FOURTH: Develop a detailed test plan that will ensure the intended functionality is correct. This should involve both Z3 formal verification and unit tests.
+    5. FIFTH: Write the test file with:
+        a) The first part of the test file should be the detailed test plan from step 4 above in comments.
+        b) Import statements matching the module structure from the example
+        c) Tests for the intended function names and behavior from the prompt
+        d) Z3 formal verification tests that are runnable as unit tests.
+</instructions>