lightassay 0.3.0__tar.gz

lightassay-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vadim Larin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lightassay-0.3.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: lightassay
+Version: 0.3.0
+Summary: File-based orchestrator for structured evaluation of applied LLM workflows: humans declare intent, LLMs reason about quality, code runs and records raw facts
+Author-email: Vadim Larin <vadimlarintech@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/vadimlarintech/lightassay
+Project-URL: Repository, https://github.com/vadimlarintech/lightassay
+Project-URL: Issues, https://github.com/vadimlarintech/lightassay/issues
+Project-URL: Changelog, https://github.com/vadimlarintech/lightassay/blob/main/CHANGELOG.md
+Keywords: llm,eval,evaluation,testing,workflow,ai,ai-agents
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# lightassay
+
+`lightassay` is a simple first way to test an LLM workflow.
+
+- You describe what worries you in plain language.
+- Your agent, using the LLM access you already have, helps turn that into directions, test cases, and analysis.
+- You do not need to build a formal eval system first.
+- The code runs the workflow and records raw facts.
+- The results are analyzed in terms that make sense to you.
+
+---
+
+## How it works
+
+```
+target → sources → intention → directions → cases → run → analysis → compare
+```
+
+Use it to check and compare:
+
+- different LLM models or providers for the same workflow
+- different workflow architectures around the same model
+- structured and free-text LLM responses
+- any applied LLM workflow where behavior quality matters
+
+It keeps the process visible in normal files:
+
+- workbook: Markdown
+- run artifact: JSON
+- analysis and compare: Markdown
+
+---
+
+## Quick start
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+### One-shot `quickstart` (main self-serve entrypoint)
+
+Start from one plain-language message.  No hand-authored target,
+preparation, semantic, or workflow config is required — use
+`--backend` to pick the adapter bundle.
+
+```bash
+lightassay quickstart my-eval \
+  --message "Check myapp.pipeline.run. I care about obvious mistakes, over-correction, and preserving names and numbers." \
+  --target "myapp.pipeline.run" \
+  --backend claude-cli
+```
+
+Quickstart performs the full first pass end-to-end:
+
+- resolves the target and execution shape,
+- generates a small, high-signal suite,
+- runs the workflow,
+- writes the analysis artifact with structured next-step
+  recommendations (each answering "to ensure what?"),
+- leaves a canonical workbook ready for further iterations.
+
+List the built-in backends:
+
+```bash
+lightassay --list-backends
+```
+
+### Follow-up `continue`
+
+```bash
+# Add instructions in the workbook's "## Continue Next Run" block,
+# or pass --message, or both.
+lightassay continue --backend claude-cli --compare-previous
+```
+
+`continue` extends/refines directions + cases, runs again, analyzes
+again, and — when `--compare-previous` is set — compares with the
+prior run.  The active workbook pointer (`.lightassay/active_workbook.json`)
+is updated after each successful `quickstart` or `continue`, so
+`continue` picks up the right workbook automatically.
+
+### Library path
+
+```python
+from lightassay import quickstart, continue_workbook
+
+result = quickstart(
+    "my-eval",
+    message="Check myapp.pipeline.run for preservation of names and numbers.",
+    target_hint="myapp.pipeline.run",
+    backend="claude-cli",
+)
+print(result.workbook_path, result.analysis_artifact_path, result.conclusion)
+
+next_result = continue_workbook(
+    message="Also check edge cases around very short inputs.",
+    backend="claude-cli",
+    compare_previous=True,
+)
+```
+
+For the earlier, explicit flow (init + prepare + run + analyze), see
+[`docs/quickstart.md`](docs/quickstart.md).
+
+For a runnable end-to-end example, see
+[`examples/quickstart/`](examples/quickstart/).
+
+The first-party Claude / Codex / stub adapters are packaged inside
+`lightassay.builtin_adapters` and selected via `--backend <name>`.  No
+separate reference scripts are needed.
+
+---
+
+## Documentation
+
+- [`quickstart.md`](docs/quickstart.md) — normal start path
+- [`workbook_spec.md`](docs/workbook_spec.md) — workbook structure
+- [`workflow_config_spec.md`](docs/workflow_config_spec.md) — workflow execution config
+- [`semantic_adapter_spec.md`](docs/semantic_adapter_spec.md) — analysis and compare config
+- [`code_architecture.md`](docs/code_architecture.md) — code structure
+
+---
+
+## Status
+
+`0.3.0`
+
+---
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

lightassay-0.3.0/README.md

@@ -0,0 +1,134 @@
+# lightassay
+
+`lightassay` is a simple first way to test an LLM workflow.
+
+- You describe what worries you in plain language.
+- Your agent, using the LLM access you already have, helps turn that into directions, test cases, and analysis.
+- You do not need to build a formal eval system first.
+- The code runs the workflow and records raw facts.
+- The results are analyzed in terms that make sense to you.
+
+---
+
+## How it works
+
+```
+target → sources → intention → directions → cases → run → analysis → compare
+```
+
+Use it to check and compare:
+
+- different LLM models or providers for the same workflow
+- different workflow architectures around the same model
+- structured and free-text LLM responses
+- any applied LLM workflow where behavior quality matters
+
+It keeps the process visible in normal files:
+
+- workbook: Markdown
+- run artifact: JSON
+- analysis and compare: Markdown
+
+---
+
+## Quick start
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+### One-shot `quickstart` (main self-serve entrypoint)
+
+Start from one plain-language message.  No hand-authored target,
+preparation, semantic, or workflow config is required — use
+`--backend` to pick the adapter bundle.
+
+```bash
+lightassay quickstart my-eval \
+  --message "Check myapp.pipeline.run. I care about obvious mistakes, over-correction, and preserving names and numbers." \
+  --target "myapp.pipeline.run" \
+  --backend claude-cli
+```
+
+Quickstart performs the full first pass end-to-end:
+
+- resolves the target and execution shape,
+- generates a small, high-signal suite,
+- runs the workflow,
+- writes the analysis artifact with structured next-step
+  recommendations (each answering "to ensure what?"),
+- leaves a canonical workbook ready for further iterations.
+
+List the built-in backends:
+
+```bash
+lightassay --list-backends
+```
+
+### Follow-up `continue`
+
+```bash
+# Add instructions in the workbook's "## Continue Next Run" block,
+# or pass --message, or both.
+lightassay continue --backend claude-cli --compare-previous
+```
+
+`continue` extends/refines directions + cases, runs again, analyzes
+again, and — when `--compare-previous` is set — compares with the
+prior run.  The active workbook pointer (`.lightassay/active_workbook.json`)
+is updated after each successful `quickstart` or `continue`, so
+`continue` picks up the right workbook automatically.
+
+### Library path
+
+```python
+from lightassay import quickstart, continue_workbook
+
+result = quickstart(
+    "my-eval",
+    message="Check myapp.pipeline.run for preservation of names and numbers.",
+    target_hint="myapp.pipeline.run",
+    backend="claude-cli",
+)
+print(result.workbook_path, result.analysis_artifact_path, result.conclusion)
+
+next_result = continue_workbook(
+    message="Also check edge cases around very short inputs.",
+    backend="claude-cli",
+    compare_previous=True,
+)
+```
+
+For the earlier, explicit flow (init + prepare + run + analyze), see
+[`docs/quickstart.md`](docs/quickstart.md).
+
+For a runnable end-to-end example, see
+[`examples/quickstart/`](examples/quickstart/).
+
+The first-party Claude / Codex / stub adapters are packaged inside
+`lightassay.builtin_adapters` and selected via `--backend <name>`.  No
+separate reference scripts are needed.
+
+---
+
+## Documentation
+
+- [`quickstart.md`](docs/quickstart.md) — normal start path
+- [`workbook_spec.md`](docs/workbook_spec.md) — workbook structure
+- [`workflow_config_spec.md`](docs/workflow_config_spec.md) — workflow execution config
+- [`semantic_adapter_spec.md`](docs/semantic_adapter_spec.md) — analysis and compare config
+- [`code_architecture.md`](docs/code_architecture.md) — code structure
+
+---
+
+## Status
+
+`0.3.0`
+
+---
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

lightassay-0.3.0/pyproject.toml

@@ -0,0 +1,54 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "lightassay"
+version = "0.3.0"
+description = "File-based orchestrator for structured evaluation of applied LLM workflows: humans declare intent, LLMs reason about quality, code runs and records raw facts"
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Vadim Larin", email = "vadimlarintech@gmail.com" },
+]
+keywords = ["llm", "eval", "evaluation", "testing", "workflow", "ai", "ai-agents"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Software Development :: Quality Assurance",
+    "Topic :: Software Development :: Testing",
+]
+
+[project.urls]
+Homepage = "https://github.com/vadimlarintech/lightassay"
+Repository = "https://github.com/vadimlarintech/lightassay"
+Issues = "https://github.com/vadimlarintech/lightassay/issues"
+Changelog = "https://github.com/vadimlarintech/lightassay/blob/main/CHANGELOG.md"
+
+[project.scripts]
+lightassay = "lightassay.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py39"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["E501", "E402"]
+"tests/fixtures/*" = ["E501", "UP"]

lightassay-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lightassay-0.3.0/src/lightassay/__init__.py

@@ -0,0 +1,134 @@
+"""lightassay: file-based orchestrator for structured evaluation of applied LLM workflows.
+
+One rule runs through the whole design: humans declare intent, LLMs do the
+semantic reasoning, code orchestrates execution and measures raw facts — and
+never judges output quality.  The workbook (markdown), run artifact (JSON),
+and analysis/compare artifacts (markdown) are the source of truth; the
+library is an orchestrator around them.
+
+The ordinary public entrypoint is the L1 library surface.  Start here::
+
+    from lightassay import (
+        open_session,
+        init_workbook,
+        quick_try,
+        quick_try_workbook,
+        refine_workbook,
+        explore_workbook,
+        compare_runs,
+    )
+
+    # Create a workbook (or use an existing one).
+    wb_path = init_workbook("my-eval", output_dir=".")
+
+    # Or run a one-shot quick try to see the full workbook shape.
+    quick = quick_try(
+        "my-quick-try",
+        target=EvalTarget(
+            kind="workflow",
+            name="summarize",
+            locator="myapp.pipeline.run",
+            boundary="high-level pipeline boundary",
+            sources=["myapp/pipeline.py", "myapp/prompts/summarize.py"],
+        ),
+        user_request="Check how the pipeline handles obvious failures without over-correcting.",
+        preparation_config="prep.json",
+        output_dir=".",
+    )
+
+    # Open a session.
+    session = open_session(
+        wb_path,
+        preparation_config="prep.json",
+        workflow_config="wf.json",
+        semantic_config="sem.json",
+    )
+
+    # Inspect state, prepare, run, analyze.
+    state = session.state()
+    result = session.prepare()
+    ...
+
+    # Compare runs (no session/workbook required).
+    compare_result = compare_runs(
+        ["run_a.json", "run_b.json"],
+        semantic_config="sem.json",
+    )
+
+Deeper engine internals are not part of the ordinary L1 surface.
+Use ``open_diagnostics()`` on a session to enter the L2
+diagnostics/recovery layer with structured reports, evidence, and
+bounded recovery actions.  The ``DiagnosticsHandle`` type returned
+by ``open_diagnostics()`` lives in ``lightassay.types`` but
+is not part of the ordinary top-level export set.  L2 detail types
+live in ``lightassay.diagnostics``.
+
+For deep inspection and bounded low-level control, escalate from
+L2 to L3 via ``diag.open_expert()``.  L3 types live in
+``lightassay.expert``.
+"""
+
+__version__ = "0.3.0"
+
+# L1 public surface ──────────────────────────────────────────────────────────
+
+from .errors import EvalError
+from .surface import (
+    EvalSession,
+    compare_runs,
+    continue_workbook,
+    explore_workbook,
+    init_workbook,
+    list_backends,
+    open_session,
+    quick_try,
+    quick_try_workbook,
+    quickstart,
+    refine_workbook,
+)
+from .types import (
+    AnalyzeResult,
+    CompareResult,
+    ContinueResult,
+    EvalState,
+    EvalTarget,
+    ExploreResult,
+    PreparationStage,
+    PrepareResult,
+    QuickstartResult,
+    QuickTryResult,
+    RefineResult,
+    RunResult,
+)
+
+__all__ = [
+    # Version
+    "__version__",
+    # L1 control
+    "open_session",
+    "init_workbook",
+    "quick_try",
+    "quick_try_workbook",
+    "refine_workbook",
+    "explore_workbook",
+    "compare_runs",
+    "quickstart",
+    "continue_workbook",
+    "list_backends",
+    "EvalSession",
+    # L1 types
+    "EvalTarget",
+    "EvalState",
+    "ExploreResult",
+    "PreparationStage",
+    "PrepareResult",
+    "QuickstartResult",
+    "QuickTryResult",
+    "ContinueResult",
+    "RefineResult",
+    "RunResult",
+    "AnalyzeResult",
+    "CompareResult",
+    # L1 error boundary
+    "EvalError",
+]