testrium 0.1.0__tar.gz → 0.2.0__tar.gz

testrium-0.2.0/PKG-INFO

@@ -0,0 +1,293 @@
+Metadata-Version: 2.1
+Name: testrium
+Version: 0.2.0
+Summary: A modern, advanced testing framework.
+Home-page: https://github.com/letalboy/testrium
+License: MIT
+Author: Cristian Camargo
+Author-email: ccf@cdone.com.br
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: colorama (>=0.4.6,<0.5.0)
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Requires-Dist: toml (>=0.10.2,<0.11.0)
+Project-URL: Documentation, https://github.com/letalboy/testrium#readme
+Project-URL: Repository, https://github.com/letalboy/testrium
+Description-Content-Type: text/markdown
+
+# Testrium
+
+Testrium is a circuit-level end-to-end testing framework for Python systems.
+
+It is built for tests where the important question is not only "did this function return the right value?", but:
+
+```text
+Did this event, command, request, or message travel through the whole system correctly?
+```
+
+Testrium helps coordinate multiple Python processes, record probes from each unit, verify that the expected endpoints were activated, and measure latency and consistency across the full circuit.
+
+## What Testrium Is For
+
+Use Testrium when a test needs to prove behavior across multiple runtime units:
+
+- a host and one or more clients
+- two services exchanging messages
+- an API that triggers a worker
+- a producer writing to a queue and a consumer processing it
+- a scheduler activating downstream logic
+- a local microservice flow
+- a module-to-module circuit where the final effect happens away from the source
+
+Testrium is designed for tests where success means:
+
+```text
+processes started
+units synchronized
+events happened
+messages crossed boundaries
+expected probes completed
+the target endpoint activated
+latency and consistency were measured
+the whole circuit passed
+```
+
+## What Testrium Is Not
+
+Testrium is not a replacement for `pytest`, `unittest`, or direct API test tools.
+
+Those tools are excellent for direct tests:
+
+- function input/output
+- class behavior
+- isolated modules
+- single endpoint responses
+- fixtures and local assertions
+
+Testrium sits beside them. It exists for the cases where direct assertions are not enough because the behavior crosses process, service, module, queue, socket, or callback boundaries.
+
+## Why This Library Exists
+
+In many systems, the actual behavior is a circuit:
+
+```text
+source unit -> event/message/request -> intermediate unit -> endpoint activation
+```
+
+Traditional test tools can test each piece, but the full circuit often becomes custom scripts, fixed sleeps, process joins, shared status flags, log scraping, and repeated manual assertions.
+
+Testrium turns that repeated pattern into a framework:
+
+```text
+declare units
+start coordinated processes
+emit probes
+collect events
+verify the circuit
+measure latency
+report pass/fail
+store history
+```
+
+## Core Concepts
+
+### Unit
+
+A unit is one actor in the scenario, such as `host`, `client`, `api`, `worker`, `scheduler`, or `consumer`.
+
+### Probe
+
+A probe is a completion tag emitted by a unit when something important happens.
+
+Examples:
+
+```text
+host-ready
+client-contacted
+command-sent
+command-received
+worker-activated
+response-received
+```
+
+### Event
+
+An event is the stored record of a probe. Events include the unit, completed step, event type, event key, and timestamp.
+
+### Circuit
+
+A circuit is the full path from cause to effect.
+
+Examples:
+
+```text
+client command -> host receives -> host callback runs -> client receives response
+API request -> event emitted -> worker consumes -> database update completes
+service A sends -> service B receives -> service B responds -> service A completes
+```
+
+## What Testrium Measures
+
+Testrium should make these checks first-class:
+
+| Circuit concern | What Testrium verifies |
+| --- | --- |
+| Start point | Which unit emitted the event or command |
+| Route | Which units should receive or process it |
+| Endpoint activation | Which callback, handler, worker, or endpoint ran |
+| Completion tags | Which probes completed in each unit |
+| Consistency | Whether `Send` and `Receive` pairs match by `EventKey` |
+| Latency | Time between send/receive events and total scenario duration |
+| Failure evidence | Missing probes, exception events, orphan sends, orphan receives, duplicate keys |
+
+## Comparison
+
+| Tool or category | Best for | Where Testrium fits |
+| --- | --- | --- |
+| `pytest` | Direct function, class, fixture, and module tests | Use Testrium when the assertion depends on events crossing runtime boundaries |
+| `unittest` | Standard-library direct tests | Use Testrium for coordinated multi-unit scenarios |
+| API test tools | One endpoint request/response | Use Testrium when the endpoint triggers async or downstream behavior |
+| Contract testing | Schema/provider compatibility | Use Testrium to prove the live circuit actually executed |
+| Browser E2E tools | UI flows | Use Testrium for backend/service/module circuits |
+| Load tools | Throughput and stress | Use Testrium for per-circuit correctness and latency |
+| Custom scripts | Project-specific orchestration | Use Testrium to avoid rebuilding process coordination and probe verification |
+
+## Current Features
+
+- Config-driven test groups
+- Unit definitions through TOML files
+- Config-first unit entrypoints using `module:function`
+- Coordinated Python unit processes
+- Unit lifecycle states and readiness probes
+- Isolated SQLite runtime state per test group run
+- Process-based setup support
+- Event/probe logging with SQLite
+- `Default`, `Exception`, `Send`, and `Receive` event types
+- Required probe verification
+- Send/receive correlation with latency metrics
+- Callback hooks for extra validation and final result handling
+- Basic result summaries
+- Template generation for starter config files
+
+The remaining near-term work is focused on richer fixtures, public history APIs, and broader reporting polish.
+
+## Installation
+
+Using Poetry:
+
+```bash
+poetry add testrium
+```
+
+Using pip:
+
+```bash
+pip install testrium
+```
+
+## Quick Start
+
+Generate a starter config:
+
+```bash
+testrium gen config-template .
+```
+
+Run Testrium from a folder that contains test groups:
+
+```bash
+testrium run
+```
+
+Run with verbose logs:
+
+```bash
+testrium --verbose run
+```
+
+Exclude specific test groups:
+
+```bash
+testrium --less test_redirect run
+```
+
+## Example Shape
+
+```text
+tests/
+  test_connection/
+    config.toml
+    setup.py
+    test_case.py
+    units/
+      target.toml
+      source.toml
+```
+
+Example unit config:
+
+```toml
+["source"]
+init = 1
+enabled = true
+entrypoint = "test_case:run_source"
+ready_event = "source-ready"
+use_setup = false
+in-except = "Resume"
+unit_dependencies = ["target"]
+events = [
+  "source-ready",
+  "command-sent",
+  "response-received"
+]
+```
+
+Example probe:
+
+```python
+from testrium.modules.events import Events_Manager
+
+events = Events_Manager(Unit="source", path=".")
+
+events.Set_Event("source-ready")
+events.Set_Event("command-sent", event_type="Send", event_key="request-001")
+```
+
+The receiving unit can emit:
+
+```python
+from testrium.modules.events import Events_Manager
+
+events = Events_Manager(Unit="target", path=".")
+
+events.Set_Event("command-received", event_type="Receive", event_key="request-001")
+```
+
+Testrium can then verify that the expected probes happened and that the circuit completed.
+
+## Documentation
+
+- [Developer guide](docs/developer-guide.md): how to structure scenarios, units, probes, and circuit-level tests.
+- [Design decisions](docs/design-decisions.md): architecture, module responsibilities, flow diagrams, and implementation direction.
+- [Implementation roadmap](docs/implementation-roadmap.md): issue-aligned gaps, acceptance gates, and MVP follow-up work.
+
+## Project Direction
+
+The near-term implementation priority is:
+
+1. Make the CLI runnable and dependency-complete.
+2. Add strict config and unit validation.
+3. Implement unit orchestration by order, dependency, and setup policy.
+4. Implement event/probe verification.
+5. Add event correlation and history metrics.
+
+Testrium's core promise is simple:
+
+```text
+test the whole circuit, not only one component inside it
+```
+

testrium-0.2.0/README.md

@@ -0,0 +1,271 @@
+# Testrium
+
+Testrium is a circuit-level end-to-end testing framework for Python systems.
+
+It is built for tests where the important question is not only "did this function return the right value?", but:
+
+```text
+Did this event, command, request, or message travel through the whole system correctly?
+```
+
+Testrium helps coordinate multiple Python processes, record probes from each unit, verify that the expected endpoints were activated, and measure latency and consistency across the full circuit.
+
+## What Testrium Is For
+
+Use Testrium when a test needs to prove behavior across multiple runtime units:
+
+- a host and one or more clients
+- two services exchanging messages
+- an API that triggers a worker
+- a producer writing to a queue and a consumer processing it
+- a scheduler activating downstream logic
+- a local microservice flow
+- a module-to-module circuit where the final effect happens away from the source
+
+Testrium is designed for tests where success means:
+
+```text
+processes started
+units synchronized
+events happened
+messages crossed boundaries
+expected probes completed
+the target endpoint activated
+latency and consistency were measured
+the whole circuit passed
+```
+
+## What Testrium Is Not
+
+Testrium is not a replacement for `pytest`, `unittest`, or direct API test tools.
+
+Those tools are excellent for direct tests:
+
+- function input/output
+- class behavior
+- isolated modules
+- single endpoint responses
+- fixtures and local assertions
+
+Testrium sits beside them. It exists for the cases where direct assertions are not enough because the behavior crosses process, service, module, queue, socket, or callback boundaries.
+
+## Why This Library Exists
+
+In many systems, the actual behavior is a circuit:
+
+```text
+source unit -> event/message/request -> intermediate unit -> endpoint activation
+```
+
+Traditional test tools can test each piece, but the full circuit often becomes custom scripts, fixed sleeps, process joins, shared status flags, log scraping, and repeated manual assertions.
+
+Testrium turns that repeated pattern into a framework:
+
+```text
+declare units
+start coordinated processes
+emit probes
+collect events
+verify the circuit
+measure latency
+report pass/fail
+store history
+```
+
+## Core Concepts
+
+### Unit
+
+A unit is one actor in the scenario, such as `host`, `client`, `api`, `worker`, `scheduler`, or `consumer`.
+
+### Probe
+
+A probe is a completion tag emitted by a unit when something important happens.
+
+Examples:
+
+```text
+host-ready
+client-contacted
+command-sent
+command-received
+worker-activated
+response-received
+```
+
+### Event
+
+An event is the stored record of a probe. Events include the unit, completed step, event type, event key, and timestamp.
+
+### Circuit
+
+A circuit is the full path from cause to effect.
+
+Examples:
+
+```text
+client command -> host receives -> host callback runs -> client receives response
+API request -> event emitted -> worker consumes -> database update completes
+service A sends -> service B receives -> service B responds -> service A completes
+```
+
+## What Testrium Measures
+
+Testrium should make these checks first-class:
+
+| Circuit concern | What Testrium verifies |
+| --- | --- |
+| Start point | Which unit emitted the event or command |
+| Route | Which units should receive or process it |
+| Endpoint activation | Which callback, handler, worker, or endpoint ran |
+| Completion tags | Which probes completed in each unit |
+| Consistency | Whether `Send` and `Receive` pairs match by `EventKey` |
+| Latency | Time between send/receive events and total scenario duration |
+| Failure evidence | Missing probes, exception events, orphan sends, orphan receives, duplicate keys |
+
+## Comparison
+
+| Tool or category | Best for | Where Testrium fits |
+| --- | --- | --- |
+| `pytest` | Direct function, class, fixture, and module tests | Use Testrium when the assertion depends on events crossing runtime boundaries |
+| `unittest` | Standard-library direct tests | Use Testrium for coordinated multi-unit scenarios |
+| API test tools | One endpoint request/response | Use Testrium when the endpoint triggers async or downstream behavior |
+| Contract testing | Schema/provider compatibility | Use Testrium to prove the live circuit actually executed |
+| Browser E2E tools | UI flows | Use Testrium for backend/service/module circuits |
+| Load tools | Throughput and stress | Use Testrium for per-circuit correctness and latency |
+| Custom scripts | Project-specific orchestration | Use Testrium to avoid rebuilding process coordination and probe verification |
+
+## Current Features
+
+- Config-driven test groups
+- Unit definitions through TOML files
+- Config-first unit entrypoints using `module:function`
+- Coordinated Python unit processes
+- Unit lifecycle states and readiness probes
+- Isolated SQLite runtime state per test group run
+- Process-based setup support
+- Event/probe logging with SQLite
+- `Default`, `Exception`, `Send`, and `Receive` event types
+- Required probe verification
+- Send/receive correlation with latency metrics
+- Callback hooks for extra validation and final result handling
+- Basic result summaries
+- Template generation for starter config files
+
+The remaining near-term work is focused on richer fixtures, public history APIs, and broader reporting polish.
+
+## Installation
+
+Using Poetry:
+
+```bash
+poetry add testrium
+```
+
+Using pip:
+
+```bash
+pip install testrium
+```
+
+## Quick Start
+
+Generate a starter config:
+
+```bash
+testrium gen config-template .
+```
+
+Run Testrium from a folder that contains test groups:
+
+```bash
+testrium run
+```
+
+Run with verbose logs:
+
+```bash
+testrium --verbose run
+```
+
+Exclude specific test groups:
+
+```bash
+testrium --less test_redirect run
+```
+
+## Example Shape
+
+```text
+tests/
+  test_connection/
+    config.toml
+    setup.py
+    test_case.py
+    units/
+      target.toml
+      source.toml
+```
+
+Example unit config:
+
+```toml
+["source"]
+init = 1
+enabled = true
+entrypoint = "test_case:run_source"
+ready_event = "source-ready"
+use_setup = false
+in-except = "Resume"
+unit_dependencies = ["target"]
+events = [
+  "source-ready",
+  "command-sent",
+  "response-received"
+]
+```
+
+Example probe:
+
+```python
+from testrium.modules.events import Events_Manager
+
+events = Events_Manager(Unit="source", path=".")
+
+events.Set_Event("source-ready")
+events.Set_Event("command-sent", event_type="Send", event_key="request-001")
+```
+
+The receiving unit can emit:
+
+```python
+from testrium.modules.events import Events_Manager
+
+events = Events_Manager(Unit="target", path=".")
+
+events.Set_Event("command-received", event_type="Receive", event_key="request-001")
+```
+
+Testrium can then verify that the expected probes happened and that the circuit completed.
+
+## Documentation
+
+- [Developer guide](docs/developer-guide.md): how to structure scenarios, units, probes, and circuit-level tests.
+- [Design decisions](docs/design-decisions.md): architecture, module responsibilities, flow diagrams, and implementation direction.
+- [Implementation roadmap](docs/implementation-roadmap.md): issue-aligned gaps, acceptance gates, and MVP follow-up work.
+
+## Project Direction
+
+The near-term implementation priority is:
+
+1. Make the CLI runnable and dependency-complete.
+2. Add strict config and unit validation.
+3. Implement unit orchestration by order, dependency, and setup policy.
+4. Implement event/probe verification.
+5. Add event correlation and history metrics.
+
+Testrium's core promise is simple:
+
+```text
+test the whole circuit, not only one component inside it
+```

{testrium-0.1.0 → testrium-0.2.0}/pyproject.toml

@@ -1,27 +1,28 @@
-[tool.poetry]
-name = "testrium"
-version = "0.1.0"
-description = "A modern, advanced testing framework."
-authors = ["Cristian Camargo <ccf@cdone.com.br>"]
-readme = "README.md"
-homepage = "https://github.com/letalboy/testrium"
-repository = "https://github.com/letalboy/testrium"
-license = "MIT"
-
-[tool.poetry.dependencies]
-python = "^3.10"
-toml = "^0.10.2"
-pandas = "^2.2.2"
-
-[tool.poetry.group.dev.dependencies]
-pytest = "^8.2.2"
-
-[build-system]
-requires = ["poetry-core"]
-build-backend = "poetry.core.masonry.api"
-
-[tool.poetry.scripts]
-testrium = "testrium.core:main"
-
-[tool.poetry.urls]
-Documentation = "https://github.com/letalboy/testrium#readme"
+[tool.poetry]
+name = "testrium"
+version = "0.2.0"
+description = "A modern, advanced testing framework."
+authors = ["Cristian Camargo <ccf@cdone.com.br>"]
+readme = "README.md"
+homepage = "https://github.com/letalboy/testrium"
+repository = "https://github.com/letalboy/testrium"
+license = "MIT"
+
+[tool.poetry.dependencies]
+python = "^3.10"
+toml = "^0.10.2"
+pandas = "^2.2.2"
+colorama = "^0.4.6"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^9.0.3"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry.scripts]
+testrium = "testrium.core:main"
+
+[tool.poetry.urls]
+Documentation = "https://github.com/letalboy/testrium#readme"

testrium-0.2.0/testrium/common/generator/config.toml

@@ -0,0 +1,15 @@
+# This is the overall config for a circuit-level test group
+["Configs"]
+units = ["target", "source"]
+debug-modes = ["DEBUG", "INFO", "WARNING", "EXCEPT"]
+test-modes = ["DEBUG"]                               # Can be "<MODE>", ["<MODE1>", <"MODE2">, ...], "All"
+enabled = true
+timeout = 10
+use-ai = false
+diagnostics = true
+detect-bad-behavior = true
+save-resume = true
+repeat = 0                                           # can be any between 0 - 10
+gen-metrics = true
+save-metrics = true
+save-scores = true

testrium-0.2.0/testrium/common/generator/main.py

@@ -0,0 +1,38 @@
+import os
+import shutil
+from colorama import Fore
+
+
+def resolve_generation_path(path: str | None) -> str:
+    target_path = path or "."
+    return os.path.abspath(os.path.expanduser(target_path))
+
+
+def _copy_template_file(source: str, destination: str) -> None:
+    destination_file = os.path.join(destination, os.path.basename(source))
+    if os.path.isfile(destination_file):
+        print(f"{Fore.BLUE} {os.path.basename(source)} already exists")
+        return
+    shutil.copy2(source, destination)
+
+
+def resolve_template(type: str, path: str | None = "."):
+    if type == "config-template":
+        target_path = resolve_generation_path(path)
+        if not os.path.isdir(target_path):
+            print(f"{Fore.RED} path argument is invalid")
+            return
+
+        template_dir = os.path.dirname(__file__)
+        _copy_template_file(os.path.join(template_dir, "config.toml"), target_path)
+        _copy_template_file(os.path.join(template_dir, "test_case.py"), target_path)
+
+        units_source = os.path.join(template_dir, "units")
+        units_target = os.path.join(target_path, "units")
+        os.makedirs(units_target, exist_ok=True)
+        for filename in os.listdir(units_source):
+            if filename.endswith(".toml"):
+                _copy_template_file(os.path.join(units_source, filename), units_target)
+
+        print(f"{Fore.GREEN} config template generated in {target_path}")
+

testrium-0.2.0/testrium/common/generator/test_case.py

@@ -0,0 +1,27 @@
+from testrium.modules.events import Events_Manager
+
+
+def run_target():
+    events = Events_Manager(Unit="target", path=".")
+    all_events = Events_Manager(Unit="*", path=".")
+
+    events.Set_Event("target-ready")
+    if not all_events.wait_for_event("command-sent", unit="source", timeout=5):
+        events.Set_Event("source-command-timeout", event_type="Exception")
+        return
+
+    events.Set_Event("command-received", event_type="Receive", event_key="request-001")
+    events.Set_Event("endpoint-activated")
+
+
+def run_source():
+    events = Events_Manager(Unit="source", path=".")
+    all_events = Events_Manager(Unit="*", path=".")
+
+    events.Set_Event("source-ready")
+    events.Set_Event("command-sent", event_type="Send", event_key="request-001")
+
+    if all_events.wait_for_event("endpoint-activated", unit="target", timeout=5):
+        events.Set_Event("response-received")
+    else:
+        events.Set_Event("response-timeout", event_type="Exception")