value-stream 0.1.1__py3-none-any.whl

value_stream/__init__.py

@@ -0,0 +1 @@
+__all__ = []

value_stream/app/SPEC.md

@@ -0,0 +1,79 @@
+# Simulation web application specification
+
+The implementation design and validation milestones are to be documented in `.agent/SIMULATION_APP_EXECPLAN.md`, maintained according to `.agent/PLANS.md`. This specification records the agreed behavior; the ExecPlan resolves version 1 implementation choices.
+
+## Goals
+- Create a standalone web application that provides a user interface to the API offered by `value_stream.service`.
+- Provide insight on how model properties affect the loss of value for a delivered set of tasks.
+- Identify the source(s) of loss within a simulation.
+- Provide suggestions for reducing loss.
+
+## Version 1 requirements
+
+- Application source files belong under `src/value_stream/app`.
+
+### Collect User Input
+
+Create a set of tasks, given: task count, story points, initial value, and depreciation rate. Story points and initial value may be specified as constants or bounded uniform ranges. Initial value defaults to 1, and depreciation rate defaults to 0.005 (0.5% per simulation time unit). Use explicit time-unit labels. There is no separate complexity property; task effort is represented by `Task.story_points`.
+
+Create one or more models.  Models may differ:
+  - In the size and properties of developer teams, QA Tester pools, toolchain pools
+  - In the properties of deployment cadence, support interval, support task story points
+  - Developers within a team may differ in efficiency. Collect team size, minimum efficiency, maximum efficiency, and a linear or bounded normal distribution. Linear means evenly spaced values between the bounds; a one-person linear team uses the midpoint. Developer teams may differ between models.
+  - For V1, QA testers are specified as a pooled resource.  Pool size and properties may differ between models.
+  - For V1, the toolchain will be specified as a pooled resource with user-specified deployment duration and failure rate.
+
+A job is a request to execute a simulation for every model, applying each to the same task set.  A task set may be reused across jobs and must remain consistent within a job.
+
+Support both individually configured models and parameter sweeps across multiple properties. A sweep generates all combinations of selected values. Sweeps are the primary model-creation workflow; show the model count and validate limits before execution. Support duplicating and editing model definitions without repetitive entry.
+
+Give every scenario a stable identity and a human-readable name. Freeze generated task sets and developer teams, and retain generation and execution seeds so reordering, repeating, or selectively rerunning scenarios does not change their numerical outcomes. Provide an explicit task-regeneration action.
+
+Utilize best practices in UI design to guide the user and avoid repetitive data entry.
+
+### Execute Simulation
+- A set of tasks and one or more models are submitted to the web service for execution as a job.
+- Provide progress updates and update plots as individual models finish. Streaming intermediate results from inside a running model is not required.
+- Cancellation stops the whole job and preserves already completed results. Individual model failures do not prevent other models from completing.
+
+### Plot Results
+
+- Render results using plots similar to those implemented in `src/value_stream/client/views/metadata_viewer.py` and `result_viewer.py`, including:
+  - Loss vs. Cadence
+  - Loss vs. Team Size
+  - Mean Stage Loss
+  - Resource Utilization
+  - Resource Backlog (the existing Resource Capacity view)
+
+- Plots are updated in realtime as results are returned by the simulation while it is executed.
+- Support switching between plots via labeled, keyboard-accessible tabs.
+- Support pan, zoom, and reset zoom, with consistent scenario colors and accessible alternatives to plot-only information.
+- Display value lost as a positive percentage, with lower values better. Overall value lost is `100 * (initial value - delivered value) / initial value`; show N/A when total initial value is zero. Explain the separate meaning of mean stage loss instead of presenting it as an additive breakdown of overall loss. Resource metric labels must accurately describe the telemetry available.
+- For V1, preserve the resource metrics used by the current viewers. Explain that resource utilization shows recorded activity shares, not a precise measure of total capacity utilization, and that backlog counts resource requests, which can contain task batches. Expanded simulation telemetry is deferred.
+
+- Support an interactive mode after a job is executed. Retain the baseline and add the latest interactive comparison, with a Pin comparison action to retain additional comparisons. For example, a team-size sweep produces one series at a fixed cadence; changing cadence produces another series using the same task set and the same teams for each size. This provides an alternative to sweeping cadence in advance. Commit slider changes on release, replace unfinished interactive work when superseded, reuse unchanged results, and keep prior plots visible while updating. Baseline and pinned comparisons remain immutable.
+
+- Errors are handled gracefully.  When possible, continue if there is an error while executing the simulation, presenting the user with relevant details.
+
+- Support separate CSV exports for model summaries, stage events, and resource history, with scenario identities and relevant configuration/seed provenance.
+- Provide explainable, rule-based observations and suggestions for reducing loss, with an explicit action to test a suggested change. Distinguish observed evidence from an untested hypothesis. Automatic optimization and machine learning are deferred.
+
+### Other Requirements
+
+- Put storage behind a replaceable interface, using an in-memory implementation for V1. Application life means the life of the server: browser refreshes retain task sets, model definitions, and retained comparisons. Document configurable retention limits; state need not survive server shutdown.
+- Include unit and integration tests
+- Handle invalid input, execution failures, and connection failures with a documented error contract.
+- Provide an option to package and run the application as a Docker image, and support local standalone startup from the command line.
+- Add configurable protections against excessive resource use, including admission and execution limits. Select initial defaults during design and document their errors in the API contract.
+- Provide single startup of web application and simulation service, with UI and application API on one browser origin. Also permit an explicitly configured simulation service endpoint.
+- User interface design must be clean, modern, and self-describing.
+
+
+
+## Intended use and future scope
+
+Version 1 is for individuals demonstrating the value-stream project and runs as a single server instance. Horizontal scaling, shared multi-user deployment, authentication, policy extensibility, service discovery, persistent results, and machine-learning optimization are later phases. Keep storage, job coordination, and simulation execution separable, with stable identifiers, to allow future shared storage and distributed workers. V1 does not implement distributed infrastructure or claim multi-instance support.
+
+## Implementation design prerequisites
+
+- Write an ExecPlan for this significant feature before coding, following `.agent/PLANS.md` as required by `AGENTS.md`.

value_stream/app/__init__.py

@@ -0,0 +1 @@
+"""Interactive, temporary workspaces for value-stream simulation."""

value_stream/app/__main__.py

@@ -0,0 +1,28 @@
+"""Run the packaged browser application and its local simulation service."""
+
+import argparse
+from pathlib import Path
+import uvicorn
+from .server import create_app
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=8081)
+    parser.add_argument("--service-url", default=None)
+    args = parser.parse_args()
+    if not (Path(__file__).parent / "static" / "index.html").is_file():
+        parser.error(
+            "Build the UI first: npm ci --prefix src/value_stream/app/frontend && npm run build --prefix src/value_stream/app/frontend"
+        )
+    uvicorn.run(
+        create_app(service_url=args.service_url),
+        host=args.host,
+        port=args.port,
+        workers=1,
+    )
+
+
+if __name__ == "__main__":
+    main()

value_stream/app/contracts.py

@@ -0,0 +1,18 @@
+"""Regenerate both checked-in OpenAPI contracts without starting servers."""
+
+import json
+from pathlib import Path
+from .server import create_app
+from value_stream.service.app import create_app as create_service
+
+
+def main():
+    root = Path(__file__).resolve().parents[1]
+    for name, factory in [("app", create_app), ("service", create_service)]:
+        (root / name / "openapi.json").write_text(
+            json.dumps(factory().openapi(), indent=2, sort_keys=True) + "\n"
+        )
+
+
+if __name__ == "__main__":
+    main()

value_stream/app/coordinator.py

@@ -0,0 +1,359 @@
+"""Application run coordination, separate from routes and worker execution."""
+
+from typing import Optional
+
+import asyncio
+import logging
+import time
+from uuid import uuid5, UUID
+from value_stream.app.schemas import RunStatus
+from value_stream.service.schemas import JobRequest, ErrorEnvelope, ResultData
+from .errors import AppError
+from .generation import concrete_model, validate_model_limits
+from .metrics import loss_percent
+from .schemas import TERMINAL, ModelSettings, RunRequest, Status
+from .storage import WorkspaceStore, RunRecord
+from .gateway import SimulationGateway
+
+logger = logging.getLogger(__name__)
+
+
+class RunCoordinator:
+    """Coordinates workspace runs and communicates with the simulation service."""
+
+    def __init__(self, store: WorkspaceStore, gateway: SimulationGateway) -> None:
+        """Configure run coordination with its workspace store and simulation gateway.
+
+        Args:
+            store (WorkspaceStore): Store used to retrieve or update data.
+            gateway (SimulationGateway): Simulation service gateway.
+        """
+        self.store, self.gateway = store, gateway
+        self.tasks = {}
+        self.closed = False
+
+    async def start_run(self, workspace_id: UUID, request: RunRequest) -> RunStatus:
+        """Start a run for the requested workspace scenarios.
+
+        Args:
+            workspace_id (UUID): Identifier of the workspace.
+            request (RunRequest): Request data to process.
+
+        Raises:
+            AppError: If the service is unavailable, the request is invalid, or an active run conflicts with it.
+
+        Returns:
+            RunStatus: The resulting value.
+        """
+
+        if self.closed:
+            raise AppError("SERVICE_UNAVAILABLE", "Application is shutting down", 503)
+        w = self.store.get(workspace_id)
+        previous = self.store.previous(w, request)
+        if previous:
+            return previous.status
+        if request.intent == "interactive":
+            if (
+                request.source_run_id is None
+                or request.property is None
+                or request.property not in ModelSettings.model_fields.keys()
+            ):
+                raise AppError("INVALID_INPUT", "Choose a source comparison and a model property")
+            source = self.store.run(workspace_id, request.source_run_id)
+            if source.status.state not in TERMINAL or not source.results:
+                raise AppError("INVALID_INPUT", "Interactive mode requires a completed comparison")
+            if w.current_task_set != source.task_set.id:
+                raise AppError(
+                    "REVISION_CONFLICT",
+                    "Interactive comparisons must use the current task set",
+                    409,
+                )
+            scenarios = [
+                o.scenario.model_copy(deep=True)
+                for o in source.status.outcomes
+                if (
+                    request.definition_id is None
+                    or o.scenario.definition_id == request.definition_id
+                )
+                and getattr(o.scenario.settings, request.property) == request.family_value
+            ]
+            if not scenarios:
+                raise AppError(
+                    "INVALID_INPUT",
+                    "Select an existing value/family before changing it",
+                )
+            for scenario in scenarios:
+                scenario.settings = ModelSettings.model_validate(
+                    {**scenario.settings.model_dump(), request.property: request.value}
+                )
+                validate_model_limits(scenario.settings, self.store.settings)
+                scenario.model = concrete_model(scenario.settings, scenario.team_seed)
+                scenario.revision += 1
+                scenario.name = (
+                    f"{scenario.name.split(' → ')[0]} → {request.property}={request.value}"
+                )
+            task_set = source.task_set
+            # Supersession is explicit and confirmed before replacement admission.
+            active = next((r for r in w.runs if r.state not in TERMINAL), None)
+            if active:
+                if active.intent != "interactive":
+                    raise AppError("RUN_ACTIVE", "Finish or cancel the manual run first", 409)
+                active.superseded = True
+                await self.cancel_run(workspace_id, active.id)
+                raise AppError(
+                    "RUN_ACTIVE",
+                    "Waiting for the previous interactive job to stop",
+                    409,
+                )
+        else:
+            preview = self.store.preview(workspace_id)
+            if preview.digest != request.preview_digest:
+                raise AppError(
+                    "REVISION_CONFLICT",
+                    "Preview is stale. Preview the current inputs again.",
+                    409,
+                )
+            scenarios = preview.scenarios
+            task_set = next(t for t in w.task_sets if t.id == preview.task_set_id)
+        record, created = self.store.register(workspace_id, request, scenarios, task_set)
+        if created:
+            self.launch(workspace_id, record)
+        return record.status
+
+    def launch(self, workspace_id: UUID, record: RunRecord) -> None:
+        """Launch execution of a simulation run.
+
+        Args:
+            workspace_id (UUID): Identifier of the workspace.
+            record (RunRecord): Run record to update.
+        """
+        record.status.retry_paused = False
+        task = asyncio.create_task(self.execute(workspace_id, record))
+        self.tasks[record.status.id] = task
+        task.add_done_callback(
+            lambda done: (
+                self.tasks.pop(record.status.id, None)
+                if self.tasks.get(record.status.id) is done
+                else None
+            )
+        )
+
+    def complete_outcome(
+        self,
+        record: RunRecord,
+        index: int,
+        status: Status,
+        result: Optional[ResultData] = None,
+        error: ErrorEnvelope | None = None,
+        cached: bool = False,
+    ) -> None:
+        """Record the completed scenario outcome.
+
+        Args:
+            record (RunRecord): Run record to update.
+            index (int): Index of the scenario or model.
+            status (Status): Current outcome status.
+            result (Optional[ResultData]): Simulation result to retain.
+            error (ErrorEnvelope | None): Optional error returned for the scenario.
+            cached (bool): Whether the result came from the cache.
+        """
+        outcome = record.status.outcomes[index]
+        if outcome.cursor:
+            return
+        if result is not None:
+            self.store.retain_result(record, index, result)
+            outcome.loss_percent = loss_percent(
+                sum(t.initial_value for t in record.task_set.tasks),
+                result.summary_result.total_delivered_value,
+            )
+            outcome.completion_time = result.summary_result.completion_time
+            outcome.delivered_value = result.summary_result.total_delivered_value
+        outcome.status, outcome.error, outcome.cached = status, error, cached
+        record.status.last_cursor += 1
+        outcome.cursor = record.status.last_cursor
+
+    async def execute(self, workspace_id: UUID, record: RunRecord) -> None:
+        """Execute the requested simulation operation.
+
+        Args:
+            workspace_id (UUID): Identifier of the workspace.
+            record (RunRecord): Run record to update.
+
+        Raises:
+            AppError: If the simulation service rejects the job or cannot be reached.
+        """
+        status = record.status
+        retry_start = None
+        delay = self.store.settings.poll_seconds
+        try:
+            for i, key in enumerate(record.cache_keys):
+                if i not in record.service_indices and not status.outcomes[i].cursor:
+                    self.complete_outcome(
+                        record,
+                        i,
+                        Status.SUCCEEDED,
+                        self.store.cached_result(key),
+                        cached=True,
+                    )
+            if not record.service_indices:
+                status.state = "completed"
+                self.store.finish(workspace_id, record)
+                return
+            request = JobRequest(
+                tasks=record.task_set.tasks,
+                models=[status.outcomes[i].scenario.model for i in record.service_indices],
+                model_seeds=[
+                    status.outcomes[i].scenario.execution_seed for i in record.service_indices
+                ],
+                submission_id=uuid5(status.id, "simulation"),
+            )
+            if len(request.model_dump_json().encode()) > self.store.settings.max_body_bytes:
+                raise AppError(
+                    "BODY_TOO_LARGE",
+                    "Expanded simulation request exceeds the body limit",
+                    413,
+                )
+            while status.state not in TERMINAL:
+                try:
+                    if status.job_id is None:
+                        await self.gateway.ready()
+                        accepted = await self.gateway.submit(request)
+                        status.job_id = accepted.job_id
+                    if status.cancel_requested:
+                        status.state = "cancelling"
+                        await self.gateway.cancel(status.job_id)
+                    upstream = await self.gateway.status(status.job_id)
+                    page = await self.gateway.outcomes(status.job_id, record.service_cursor)
+                    for outcome in page.outcomes:
+                        if outcome.cursor <= record.service_cursor:
+                            continue
+                        if outcome.model_index >= len(record.service_indices):
+                            raise AppError(
+                                "SERVICE_INVALID_RESPONSE",
+                                "Service returned an invalid model index",
+                                502,
+                            )
+                        i = record.service_indices[outcome.model_index]
+                        error = (
+                            ErrorEnvelope(code=outcome.error.code, message=outcome.error.message)
+                            if outcome.error
+                            else None
+                        )
+                        self.complete_outcome(record, i, outcome.status, outcome.result, error)
+                    record.service_cursor = page.next_cursor
+                    status.error = None
+                    retry_start = None
+                    delay = self.store.settings.poll_seconds
+                    if upstream.status in {
+                        "completed",
+                        "completed_with_errors",
+                        "cancelled",
+                    }:
+                        if any(not o.cursor for o in status.outcomes):
+                            # Status and outcomes are separate reads; a terminal state must
+                            # have a complete outcome set by the time the page is fetched.
+                            raise AppError(
+                                "SERVICE_INVALID_RESPONSE",
+                                "Terminal job is missing outcomes",
+                                502,
+                            )
+                        status.state = upstream.status
+                        self.store.finish(workspace_id, record)
+                        return
+                    status.state = "cancelling" if status.cancel_requested else upstream.status
+                except AppError as exc:
+                    if exc.code != "SERVICE_UNAVAILABLE":
+                        raise
+                    status.state = "reconnecting"
+                    status.error = ErrorEnvelope(**exc.body())
+                    retry_start = retry_start or time.monotonic()
+                    if time.monotonic() - retry_start >= self.store.settings.retry_seconds:
+                        status.retry_paused = True
+                        return
+                    delay = min(5, delay * 2)
+                await asyncio.sleep(delay)
+        except asyncio.CancelledError:
+            raise
+        except Exception as exc:
+            logger.exception("Application run %s failed", status.id)
+            error = (
+                exc
+                if isinstance(exc, AppError)
+                else AppError("INTERNAL_ERROR", "Unexpected application execution error", 500)
+            )
+            if error.code == "JOB_NOT_FOUND":
+                error = AppError(
+                    "SERVICE_JOB_LOST",
+                    "The simulation job expired or the service restarted. Completed results were retained.",
+                    404,
+                )
+            status.error = ErrorEnvelope(**error.body())
+            if status.job_id:
+                try:
+                    await self.gateway.cancel(status.job_id)
+                except Exception:
+                    logger.warning("Unable to confirm cancellation for job %s", status.job_id)
+            for i, o in enumerate(status.outcomes):
+                if not o.cursor:
+                    self.complete_outcome(record, i, Status.FAILED, error=status.error)
+            status.state = "failed"
+            self.store.finish(workspace_id, record)
+
+    async def cancel_run(self, workspace_id: UUID, run_id: UUID) -> RunStatus:
+        """Cancel a run that has not reached a terminal state.
+
+        Args:
+            workspace_id (UUID): Identifier of the workspace.
+            run_id (UUID): Identifier of the simulation run.
+
+        Returns:
+            RunStatus: Updated status of the cancelled run.
+        """
+        record = self.store.run(workspace_id, run_id)
+        if record.status.state not in TERMINAL:
+            record.status.cancel_requested = True
+            record.status.state = "cancelling"
+            if run_id not in self.tasks:
+                self.launch(workspace_id, record)
+        return record.status
+
+    def resume(self, workspace_id: UUID, run_id: UUID) -> RunStatus:
+        """Resume a paused simulation run.
+
+        Args:
+            workspace_id (UUID): Identifier of the workspace.
+            run_id (UUID): Identifier of the simulation run.
+
+        Returns:
+            RunStatus: Current status of the resumed run.
+        """
+        record = self.store.run(workspace_id, run_id)
+        if record.status.state not in TERMINAL and run_id not in self.tasks:
+            self.launch(workspace_id, record)
+        return record.status
+
+    async def close(self) -> None:
+        """Close the client and release its resources."""
+        self.closed = True
+
+        async def cancel(record: RunRecord):
+            if record.status.job_id:
+                try:
+                    await self.gateway.cancel(record.status.job_id)
+                except Exception:
+                    logger.warning(
+                        "Shutdown could not confirm cancellation for %s",
+                        record.status.job_id,
+                    )
+
+        try:
+            await asyncio.wait_for(
+                asyncio.gather(*(cancel(r) for r in self.store.active_records())),
+                timeout=5,
+            )
+        except asyncio.TimeoutError:
+            logger.warning("Shutdown cancellation timed out")
+        tasks = list(self.tasks.values())
+        for task in tasks:
+            task.cancel()
+        await asyncio.gather(*tasks, return_exceptions=True)

value_stream/app/errors.py

@@ -0,0 +1,17 @@
+"""Application errors shared by routes, storage, and the service gateway."""
+
+
+class AppError(Exception):
+    """Represents a structured error returned by the application."""
+
+    def __init__(self, code: str, message: str, status: int = 422, details=None):
+        super().__init__(message)
+        self.code, self.message, self.status, self.details = (
+            code,
+            message,
+            status,
+            details,
+        )
+
+    def body(self):
+        return {"code": self.code, "message": self.message, "details": self.details}

value_stream/app/exports.py

@@ -0,0 +1,142 @@
+"""Streaming CSV exports from immutable scenario snapshots."""
+
+from typing import Generator, Literal
+
+import csv
+import io
+from .generation import canonical
+from .schemas import ENGINE_VERSION, GENERATOR_VERSION
+from .storage import RunRecord, WorkspaceStore
+
+
+def safe_text(value):
+    if isinstance(value, str) and value.lstrip().startswith(("=", "+", "-", "@", "\t", "\r")):
+        return "'" + value
+    return value
+
+
+def export_csv(
+    store: WorkspaceStore,
+    record: RunRecord,
+    kind: Literal["summary", "events", "resources"],
+) -> Generator[str, None, None]:
+    """Export a simulation result as CSV.
+
+    Args:
+        store (WorkspaceStore): Store containing the run results.
+        record (RunRecord): Run record to export.
+        kind (Literal["summary", "events", "resources"]): Data section to export.
+
+    Yields:
+        str: One CSV row at a time.
+    """
+    common = [
+        "scenario_id",
+        "scenario_name",
+        "scenario_revision",
+        "task_set_id",
+        "task_set_hash",
+        "run_id",
+        "comparison_name",
+        "execution_seed",
+        "task_generation_seed",
+        "team_generation_seed",
+        "generator_version",
+        "engine_version",
+        "status",
+        "settings",
+        "team_efficiencies",
+    ]
+    fields = {
+        "summary": [
+            "error_code",
+            "error_message",
+            "completion_time",
+            "initial_value",
+            "delivered_value",
+            "value_lost_percent",
+        ],
+        "events": [
+            "event",
+            "event_type",
+            "time",
+            "status",
+            "loss",
+            "task_type",
+            "is_rework",
+            "duration",
+            "resource_id",
+            "stage_loss_percent",
+        ],
+        "resources": [
+            "time",
+            "state",
+            "waiting",
+            "allocated",
+            "active",
+            "success_t",
+            "failure_t",
+            "interruption_t",
+            "waiting_t",
+            "idle_t",
+        ],
+    }[kind]
+    # Event status is distinct from the scenario's terminal status.
+    headers = ["scenario_status" if x == "status" else x for x in common] + fields
+    output = io.StringIO(newline="")
+    writer = csv.writer(output)
+    writer.writerow(headers)
+    yield output.getvalue()
+    for outcome in record.status.outcomes:
+        s = outcome.scenario
+        base = [
+            str(s.id),
+            safe_text(s.name),
+            s.revision,
+            str(record.task_set.id),
+            record.task_set.content_hash,
+            str(record.status.id),
+            safe_text(record.status.name),
+            s.execution_seed,
+            record.task_set.spec.seed,
+            s.team_seed,
+            GENERATOR_VERSION,
+            ENGINE_VERSION,
+            outcome.status,
+            canonical(s.settings.model_dump()),
+            canonical([d.efficiency for d in s.model.developer_team]),
+        ]
+        if kind == "summary":
+            rows = [
+                [
+                    outcome.error.code if outcome.error else "",
+                    safe_text(outcome.error.message) if outcome.error else "",
+                    outcome.completion_time,
+                    sum(t.initial_value for t in record.task_set.tasks),
+                    outcome.delivered_value,
+                    outcome.loss_percent,
+                ]
+            ]
+        elif outcome.status == "succeeded":
+            result = store.result(record, s.id)
+            data = (
+                result.metadata.event_metadata
+                if kind == "events"
+                else result.metadata.resource_metadata
+            )
+
+            def event_rows():
+                for item in data:
+                    values = item.model_dump(mode="json")
+                    if kind == "events":
+                        values["stage_loss_percent"] = -100 * item.loss
+                    yield [values.get(f) for f in fields]
+
+            rows = event_rows()
+        else:
+            rows = []
+        for row in rows:
+            output.seek(0)
+            output.truncate(0)
+            writer.writerow(base + row)
+            yield output.getvalue()