microvm-ctl 0.3.0__tar.gz → 0.4.0__tar.gz

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: microvm-ctl
-Version: 0.3.0
+Version: 0.4.0
 Summary: Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live.
 Author: Vivek Raja P S
 License: Apache-2.0
@@ -196,10 +196,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/README.md

@@ -160,10 +160,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/__init__.py

@@ -12,7 +12,7 @@ from microvm.images import ImageBuilder, ImageBuildError
 from microvm.lease import FanoutLimit, Lease, LeasePlan, LeasePlanRejected, LeasePolicy, plan_fanout
 from microvm.monitor import CostModel, FleetMonitor
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 __all__ = [
     "microvm_client",

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/cli.py

@@ -9,6 +9,7 @@
     mvm scale IMAGE N                      converge fleet to N
     mvm suspend|resume|terminate ID...     lifecycle control
     mvm drain IMAGE                        terminate the whole fleet
+    mvm dispatch IMAGE /path -d '{}' -n 64 spread requests over the warm fleet, --per-vm in flight each
     mvm call ID /path [-X POST -d '{}']    authenticated request into the VM
     mvm status ID | watch ID               job telemetry from the hook runtime (/status, /events)
     mvm watch --image NAME                 one live table for every leased VM of an image
@@ -226,6 +227,37 @@ def cmd_drain(args):
     console.print(f"[bold green]✓[/] terminated {n} microVMs of [cyan]{args.image}[/]")
 
 
+def cmd_dispatch(args):
+    if args.bodies:
+        with open(args.bodies) as f:
+            bodies = [json.loads(line) for line in f if line.strip()]
+    else:
+        bodies = [json.loads(args.data or "{}")] * args.n
+    fleet = Fleet(FleetManager(_cfg(args)), args.image)
+    started = time.time()
+    with console.status(f"[bold]dispatching {len(bodies)} requests over [cyan]{args.image}[/]…"):
+        results = fleet.dispatch(args.path, bodies, per_vm=args.per_vm, method=args.method, port=args.port,
+                                 timeout=args.timeout)
+    wall = time.time() - started
+    if args.out:
+        with open(args.out, "w") as f:
+            for r in results:
+                f.write(json.dumps(r, default=str) + "\n")
+    by_vm: dict = {}
+    for r in results:
+        by_vm.setdefault(r["microvm_id"], []).append(r)
+    t = Table(title=f"{len(results)} requests in {wall:.2f} s ({len(results) / wall:.1f}/s), {args.per_vm} in flight per VM")
+    for col in ("microVM", "requests", "ok", "failed", "p50 ms", "max ms"):
+        t.add_column(col)
+    for vid, rs in sorted(by_vm.items()):
+        lat = sorted(r["ms"] for r in rs)
+        ok = sum(1 for r in rs if r["status"] and r["status"] < 400)
+        t.add_row(vid, str(len(rs)), str(ok), str(len(rs) - ok), f"{lat[len(lat) // 2]:.0f}", f"{lat[-1]:.0f}")
+    console.print(t)
+    if any(not r["status"] or r["status"] >= 400 for r in results):
+        sys.exit(1)
+
+
 # ---------------------------------------------------------------- execution plane
 def cmd_call(args):
     cfg = _cfg(args)
@@ -531,11 +563,11 @@ def cmd_lease_run(args):
         task = json.loads(args.task) if args.task else {}
     except ValueError as e:
         sys.exit(f"mvm lease run: --task is not valid JSON: {e}")
+    policy = _lease_policy(args)
     lease = Lease(kind=args.kind, token=args.token or "", region=cfg.region, target=args.target,
-                  heartbeat_s=args.heartbeat_every, id=args.id)
+                  heartbeat_s=policy.heartbeat_every(args.heartbeat_every), id=args.id)
     fm = FleetManager(cfg)
-    vm = fm.lease(args.image, lease, task, _lease_policy(args), version=args.version,
-                  execution_role=args.execution_role)
+    vm = fm.lease(args.image, lease, task, policy, version=args.version, execution_role=args.execution_role)
     console.print(
         f"[bold green]✓[/] {vm.microvm_id}  {_state(vm.state)}  "
         f"[link=https://{vm.endpoint}]{vm.endpoint}[/link]  lease {lease.kind}"
@@ -562,14 +594,14 @@ def cmd_lease_run_many(args, cfg):
         sys.exit(f"mvm lease run: --task-template is not valid JSON: {e}")
     if not isinstance(template, dict):
         sys.exit("mvm lease run: --task-template must be a JSON object")
+    policy = _lease_policy(args)
     leases, tasks = [], []
     for i in range(args.shards):
         token = args.token_template.replace("{i}", str(i)) if args.token_template else ""
         lease_id = args.id.replace("{i}", str(i)) if args.id else None
         leases.append(Lease(kind=kind, token=token, region=cfg.region, target=args.target,
-                            heartbeat_s=args.heartbeat_every, id=lease_id))
+                            heartbeat_s=policy.heartbeat_every(args.heartbeat_every), id=lease_id))
         tasks.append(_fill_template(template, i))
-    policy = _lease_policy(args)
     fm = FleetManager(cfg)
     baseline = _baseline_mib(args, cfg)
     plan = fm.plan(args.shards, baseline, policy)
@@ -724,6 +756,19 @@ def main(argv: list[str] | None = None):
     d.add_argument("image")
     d.set_defaults(fn=cmd_drain)
 
+    dp = sub.add_parser("dispatch", help="spread requests over the RUNNING members of an image")
+    dp.add_argument("image")
+    dp.add_argument("path")
+    dp.add_argument("-X", "--method", default="POST")
+    dp.add_argument("-d", "--data", help="one JSON body, sent -n times")
+    dp.add_argument("-n", type=int, default=1, help="how many times to send --data")
+    dp.add_argument("--bodies", help="a JSON Lines file, one request body per line")
+    dp.add_argument("--per-vm", type=int, default=4, help="requests in flight per VM")
+    dp.add_argument("--port", type=int, default=8080)
+    dp.add_argument("--timeout", type=float, default=300)
+    dp.add_argument("--out", help="write every result as JSON Lines")
+    dp.set_defaults(fn=cmd_dispatch)
+
     c = sub.add_parser("call", help="authenticated request into a microVM")
     c.add_argument("id")
     c.add_argument("path")

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/fleet.py

@@ -12,9 +12,11 @@ retried with jittered backoff, so `scale_to(50)` is safe to call in one shot.
 from __future__ import annotations
 
 import concurrent.futures as futures
+import queue
+import threading
 import time
 from dataclasses import dataclass, field
-from typing import Callable
+from typing import Any, Callable
 
 from microvm.client import image_arn, microvm_client
 from microvm.config import TPS, PlaneConfig
@@ -311,15 +313,23 @@ class Fleet:
     _pool: futures.ThreadPoolExecutor = field(
         default_factory=lambda: futures.ThreadPoolExecutor(max_workers=Fleet.MAX_WORKERS), repr=False
     )
+    #: VMs this fleet has asked to terminate. ListMicrovms keeps reporting a terminated VM in its old
+    #: state for about a second, so without this a size() right after a scale-down counts the victim
+    #: and a drain terminates it twice.
+    _terminated: set = field(default_factory=set, repr=False)
 
     # -- observation -------------------------------------------------------------
     def members(self) -> list[Microvm]:
         return [
             vm
             for vm in self.manager.list(self.image, self.version)
-            if vm.state in ACTIVE_STATES
+            if vm.state in ACTIVE_STATES and vm.microvm_id not in self._terminated
         ]
 
+    def _terminate(self, microvm_id: str) -> None:
+        self.manager.terminate(microvm_id)
+        self._terminated.add(microvm_id)
+
     def size(self) -> int:
         return len(self.members())
 
@@ -345,7 +355,7 @@ class Fleet:
             return launched
         if delta < 0:
             for vm in self.scale_down_victims(current, -delta):
-                self.manager.terminate(vm.microvm_id)
+                self._terminate(vm.microvm_id)
         return []
 
     @staticmethod
@@ -384,7 +394,7 @@ class Fleet:
     def drain(self) -> int:
         """Terminate every member of the fleet."""
         vms = self.members()
-        list(self._pool.map(lambda v: self.manager.terminate(v.microvm_id), vms))
+        list(self._pool.map(lambda v: self._terminate(v.microvm_id), vms))
         return len(vms)
 
     # -- reaper ------------------------------------------------------------------
@@ -396,6 +406,81 @@ class Fleet:
         for vm in self.members():
             started = vm.started_epoch
             if started and now - started > max_age_seconds:
-                self.manager.terminate(vm.microvm_id)
+                self._terminate(vm.microvm_id)
                 reaped.append(vm.microvm_id)
         return reaped
+
+    # -- work distribution -------------------------------------------------------
+    def dispatch(
+        self,
+        path: str,
+        bodies: list,
+        *,
+        per_vm: int = 4,
+        method: str = "POST",
+        port: int = 8080,
+        timeout: float = 300,
+        client_factory: Callable[[str], Any] | None = None,
+        on_result: Callable[[dict], None] | None = None,
+    ) -> list[dict]:
+        """Send one request per body to the fleet's RUNNING members and return the results in order.
+
+        Each RUNNING member gets `per_vm` workers that pull from one shared queue, so a slow VM takes
+        fewer items and no VM ever holds more than `per_vm` requests in flight. This is the warm-fleet
+        path: hundreds of short tasks over VMs that are already up, where a lease per task would pay a
+        launch per task. A request that raises is recorded with its error and does not stop the others.
+
+        Each result is {"index", "microvm_id", "status", "ms", "body"} or, on an exception,
+        {"index", "microvm_id", "status": None, "ms", "error"}. `on_result` is called with each result
+        as it lands, from a worker thread.
+        """
+        if per_vm < 1:
+            raise ValueError("per_vm must be at least 1")
+        running = [vm for vm in self.members() if vm.state == "RUNNING"]
+        if not running:
+            raise RuntimeError(f"fleet {self.image} has no RUNNING members to dispatch to")
+        if client_factory is None:
+            from microvm.endpoint import EndpointClient
+
+            def client_factory(microvm_id: str):
+                return EndpointClient(self.manager.cfg, microvm_id, ports=[port])
+
+        work: queue.Queue = queue.Queue()
+        for i, body in enumerate(bodies):
+            work.put((i, body))
+        results: list = [None] * len(bodies)
+        lock = threading.Lock()
+
+        def worker(microvm_id: str, client) -> None:
+            while True:
+                try:
+                    i, body = work.get_nowait()
+                except queue.Empty:
+                    return
+                started = time.time()
+                try:
+                    resp = client.request(method, path, json=body, timeout=timeout, port=port)
+                    try:
+                        payload = resp.json()
+                    except ValueError:
+                        payload = resp.text
+                    out = {"index": i, "microvm_id": microvm_id, "status": resp.status_code,
+                           "ms": round((time.time() - started) * 1000, 1), "body": payload}
+                except Exception as e:  # one failed request must not stop the batch
+                    out = {"index": i, "microvm_id": microvm_id, "status": None,
+                           "ms": round((time.time() - started) * 1000, 1), "error": f"{type(e).__name__}: {e}"}
+                with lock:
+                    results[i] = out
+                if on_result:
+                    on_result(out)
+
+        threads = []
+        for vm in running:
+            client = client_factory(vm.microvm_id)
+            for _ in range(per_vm):
+                t = threading.Thread(target=worker, args=(vm.microvm_id, client), daemon=True)
+                t.start()
+                threads.append(t)
+        for t in threads:
+            t.join()
+        return results

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/hooks/server.py

@@ -514,7 +514,11 @@ class HookApp:
             self._deliver(ctx, completer, "failure", ctx.completion(error=err))
 
     def _heartbeat_loop(self, ctx: LeaseContext, completer, interval: float) -> None:
-        while not ctx._stop.wait(interval):
+        # The first heartbeat goes out at once: the orchestrator's heartbeat clock started before
+        # RunMicrovm returned, so waiting a full interval first can lose a tight lease.
+        first = True
+        while first or not ctx._stop.wait(interval):
+            first = False
             try:
                 with ctx._lock:  # serialized with _deliver: no heartbeat lands after the completion
                     if ctx.done:

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/integrations/durable.py

@@ -187,7 +187,7 @@ def lease_microvm(
         ),
     )
     lease = Lease(kind="durable", token=callback.callback_id, region=region,
-                  heartbeat_s=heartbeat_s, id=execution_name(context))
+                  heartbeat_s=policy.heartbeat_every(heartbeat_s), id=execution_name(context))
     policy_dict = {"budget_s": policy.budget_s, "heartbeat_timeout_s": policy.heartbeat_timeout_s,
                    "slack_s": policy.slack_s}
     launch_once = StepConfig(step_semantics=StepSemantics.AT_MOST_ONCE_PER_RETRY,

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/integrations/stepfunctions.py

@@ -103,8 +103,10 @@ def _lease_states(
     task_expr: str, heartbeat_s: int, lease_id_expr: str, client_token_expr: str,
     done_name: str = "Done",
 ) -> dict:
-    """The states of one lease: `name` -> Terminate -> Done, failures -> Reap ->
-    TerminateStale -> Failed. Used as the whole machine and as a Map item processor."""
+    """The states of one lease: `name` -> Terminate -> Done; a typed failure ->
+    OnLeaseError -> TerminateFailed (the VM the cause names) -> Reap -> TerminateStale ->
+    Failed; a timeout -> OnLeaseError -> Reap -> TerminateStale -> Failed. Used as the whole
+    machine and as a Map item processor."""
     max_duration = policy.max_duration()
     stale_ms = max_duration * 1000
     items_expr = (
@@ -118,7 +120,8 @@ def _lease_states(
             "Arguments": {
                 "ImageIdentifier": image_arn,
                 "ExecutionRoleArn": execution_role_arn,
-                "RunHookPayload": run_hook_payload_expr(region, heartbeat_s, task_expr, lease_id_expr),
+                "RunHookPayload": run_hook_payload_expr(
+                    region, policy.heartbeat_every(heartbeat_s), task_expr, lease_id_expr),
                 "IdlePolicy": _idle_policy_args(policy),
                 "MaximumDurationInSeconds": max_duration,
                 "ClientToken": _jsonata(f"$substring({client_token_expr}, 0, 128)"),
@@ -129,11 +132,30 @@ def _lease_states(
             "Catch": [{
                 "ErrorEquals": ["States.Timeout", "States.HeartbeatTimeout", "States.TaskFailed"],
                 "Assign": {"lease_error": _jsonata("$states.errorOutput")},
-                "Next": "Reap",
+                "Next": "OnLeaseError",
             }],
             "Assign": {"vm": _jsonata("$states.result.microvm_id")},
             "Next": "Terminate",
         },
+        # A typed failure's cause is the VM's own payload and names it: terminate that VM at
+        # once. A timeout carries no id; that VM is bounded by MaximumDurationInSeconds
+        # (budget + slack) and Reap terminates anything older than that.
+        "OnLeaseError": {
+            "Type": "Choice",
+            "Choices": [{
+                "Condition": _jsonata("$contains($string($lease_error.Cause), '\"microvm_id\"')"),
+                "Next": "TerminateFailed",
+            }],
+            "Default": "Reap",
+        },
+        "TerminateFailed": {
+            "Type": "Task",
+            "Resource": TERMINATE,
+            "Arguments": {"MicrovmIdentifier": _jsonata("$parse($lease_error.Cause).microvm_id")},
+            "Retry": _retry_throttling(),
+            "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Reap"}],
+            "Next": "Reap",
+        },
         "Terminate": {
             "Type": "Task",
             "Resource": TERMINATE,

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm/lease.py

@@ -119,6 +119,15 @@ class LeasePolicy:
     def to_dict(self) -> dict:
         return asdict(self)
 
+    def heartbeat_every(self, requested: int | float | None = None) -> int:
+        """The interval the VM should heartbeat at: the requested seconds (default 30), never
+        more than a third of `heartbeat_timeout_s` and never under 5 s. A heartbeat interval at
+        or above the timeout loses the lease before the first heartbeat lands, because the
+        orchestrator's clock starts before RunMicrovm returns."""
+        want = 30 if requested is None else float(requested)
+        ceiling = max(5.0, self.heartbeat_timeout_s / 3.0)
+        return int(max(5.0, min(want, ceiling)))
+
     def idle_policy(self):
         """No auto-resume: a leased VM that goes idle is finished, not dormant."""
         from microvm.fleet import IdlePolicy

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/microvm_ctl.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: microvm-ctl
-Version: 0.3.0
+Version: 0.4.0
 Summary: Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live.
 Author: Vivek Raja P S
 License: Apache-2.0
@@ -196,10 +196,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "microvm-ctl"
-version = "0.3.0"
+version = "0.4.0"
 description = "Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live."
 readme = "README.md"
 requires-python = ">=3.9"

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_fleet.py

@@ -84,3 +84,67 @@ def test_run_params_carries_client_token(monkeypatch):
     assert params["runHookPayload"] == '{"callback_id": "abc"}'
     assert params["maximumDurationInSeconds"] == 900
     assert "clientToken" not in fm.run_params("img")
+
+
+def test_scale_down_victim_is_not_counted_while_list_still_shows_it():
+    # ListMicrovms reports a terminated VM in its old state for a moment; the fleet must not count it.
+    members = [_vm(1, "RUNNING", 600), _vm(2, "SUSPENDED", 300), _vm(3, "RUNNING", 10)]
+    fm = FakeManager(members)
+    fleet = Fleet(fm, "img")
+    fleet.scale_to(2)
+    assert fm.terminated == ["vm-2"]
+    assert fleet.size() == 2
+    assert fleet.drain() == 2
+    assert sorted(fm.terminated) == ["vm-1", "vm-2", "vm-3"]
+
+
+class FakeResponse:
+    def __init__(self, status, payload):
+        self.status_code, self._payload = status, payload
+
+    def json(self):
+        return self._payload
+
+
+class FakeClient:
+    def __init__(self, microvm_id, log, fail_on=None, delay=0.0):
+        self.microvm_id, self.log, self.fail_on, self.delay = microvm_id, log, fail_on, delay
+
+    def request(self, method, path, json=None, timeout=None, port=None):
+        import time as _t
+        _t.sleep(self.delay)
+        self.log.append((self.microvm_id, method, path, json))
+        if self.fail_on is not None and json == self.fail_on:
+            raise ConnectionError("boom")
+        return FakeResponse(200, {"echo": json, "vm": self.microvm_id})
+
+
+def test_dispatch_spreads_bodies_over_running_members_and_keeps_order():
+    fm = FakeManager([_vm(1, "RUNNING", 60), _vm(2, "RUNNING", 60), _vm(3, "SUSPENDED", 60)])
+    log = []
+    fleet = Fleet(fm, "img")
+    seen = []
+    results = fleet.dispatch("/task", [{"n": i} for i in range(10)], per_vm=2,
+                             client_factory=lambda vm_id: FakeClient(vm_id, log, delay=0.01),
+                             on_result=seen.append)
+    assert [r["index"] for r in results] == list(range(10))
+    assert [r["body"]["echo"] for r in results] == [{"n": i} for i in range(10)]
+    assert {r["microvm_id"] for r in results} == {"vm-1", "vm-2"}  # the SUSPENDED member gets nothing
+    assert all(m == "POST" and p == "/task" for _, m, p, _ in log)
+    assert len(seen) == 10
+
+
+def test_dispatch_records_a_failed_request_and_finishes_the_rest():
+    fm = FakeManager([_vm(1, "RUNNING", 60)])
+    fleet = Fleet(fm, "img")
+    results = fleet.dispatch("/task", [{"n": 0}, {"n": 1}, {"n": 2}], per_vm=1,
+                             client_factory=lambda vm_id: FakeClient(vm_id, [], fail_on={"n": 1}))
+    assert results[0]["status"] == 200 and results[2]["status"] == 200
+    assert results[1]["status"] is None and "ConnectionError" in results[1]["error"]
+
+
+def test_dispatch_needs_a_running_member():
+    import pytest
+    fleet = Fleet(FakeManager([_vm(1, "SUSPENDED", 60)]), "img")
+    with pytest.raises(RuntimeError, match="no RUNNING members"):
+        fleet.dispatch("/task", [{}], client_factory=lambda vm_id: None)

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_integrations.py

@@ -23,7 +23,8 @@ def test_asl_is_jsonata_and_json_serialisable():
     json.dumps(asl)  # no sets, no dataclasses
     assert asl["QueryLanguage"] == "JSONata"
     assert asl["StartAt"] == "Lease"
-    assert set(asl["States"]) == {"Lease", "Terminate", "Reap", "TerminateStale", "Failed", "Done"}
+    assert set(asl["States"]) == {"Lease", "Terminate", "OnLeaseError", "TerminateFailed", "Reap",
+                                  "TerminateStale", "Failed", "Done"}
     assert asl["States"]["Done"] == {"Type": "Succeed"}
     assert asl["States"]["Failed"]["Type"] == "Fail"
     assert asl["States"]["Failed"]["Error"] == "LeaseFailed"
@@ -51,7 +52,16 @@ def test_lease_state_resources_and_arguments():
     assert lease["Retry"][0]["BackoffRate"] == 2
     catch = lease["Catch"][0]
     assert catch["ErrorEquals"] == ["States.Timeout", "States.HeartbeatTimeout", "States.TaskFailed"]
-    assert catch["Next"] == "Reap"
+    assert catch["Next"] == "OnLeaseError"
+    asl = _asl()
+    choice = asl["States"]["OnLeaseError"]
+    assert choice["Type"] == "Choice" and choice["Default"] == "Reap"
+    assert choice["Choices"][0]["Next"] == "TerminateFailed"
+    assert "microvm_id" in choice["Choices"][0]["Condition"]
+    tf = asl["States"]["TerminateFailed"]
+    assert tf["Resource"] == asl["States"]["Terminate"]["Resource"]
+    assert "$parse($lease_error.Cause).microvm_id" in tf["Arguments"]["MicrovmIdentifier"]
+    assert tf["Next"] == "Reap" and tf["Catch"][0]["Next"] == "Reap"
     assert catch["Assign"] == {"lease_error": "{% $states.errorOutput %}"}
 
 

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_integrations_durable.py

@@ -91,7 +91,7 @@ def test_success_carries_the_lease_and_terminates_once(fm):
         launch = fm.leases[0]
         lease = launch["lease"]
         assert lease.kind == "durable" and lease.token == cb and lease.region == "us-east-1"
-        assert lease.heartbeat_s == 30 and lease.id
+        assert lease.heartbeat_s == 5 and lease.id  # clamped: timeout 2 s -> floor of 5 s
         assert launch["image"] == "handoff-agent" and launch["task"] == TASK
         assert launch["policy"].budget_s == 4 and launch["policy"].max_duration() == 124
         assert launch["version"] == "3" and launch["execution_role"] == "arn:aws:iam::1:role/agent"

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_integrations_durable_scale.py

@@ -118,7 +118,8 @@ def test_two_shards_happy_path(fm):
         cb1 = runner.wait_for_callback(arn, name="job-1-0-callback", timeout=30)
         assert cb0 != cb1
         assert fm.plans == [(2, 2048, POLICY)]  # the plan step ran once, with the reconstructed policy
-        assert [launch["task"] for launch in fm.leases] == SHARDS
+        # map items launch concurrently, so compare as a set of shards, not in launch order
+        assert sorted((launch["task"] for launch in fm.leases), key=lambda t: t["i"]) == SHARDS
         assert all(launch["version"] == "3" and launch["policy"] == POLICY for launch in fm.leases)
         _succeed(runner, cb0, 1)
         _succeed(runner, cb1, 2)

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_integrations_scale.py

@@ -11,7 +11,8 @@ from microvm.lease import LeasePolicy
 IMAGE = "arn:aws:lambda:us-east-1:123456789012:microvm-image:handoff-agent"
 ROLE = "arn:aws:iam::123456789012:role/agent"
 TOPIC = "arn:aws:sns:us-east-1:1:t"
-LEASE_STATES = {"Lease", "Terminate", "Reap", "TerminateStale", "Failed", "Done"}
+LEASE_STATES = {"Lease", "Terminate", "OnLeaseError", "TerminateFailed", "Reap", "TerminateStale", "Failed",
+                "Done"}
 
 def _indexed(expr: str) -> str:
     return "{% $map(" + expr + ", function($v, $i) { {'index': $i, 'task': $v} }) %}"

{microvm_ctl-0.3.0 → microvm_ctl-0.4.0}/tests/test_scale.py

@@ -329,3 +329,24 @@ def test_package_exports():
 
     assert microvm.LeasePlan is LeasePlan and microvm.LeasePlanRejected is LeasePlanRejected
     assert microvm.FanoutLimit is FanoutLimit and microvm.plan_fanout is plan_fanout
+
+
+def test_heartbeat_every_clamps_to_a_third_of_the_timeout():
+    from microvm.lease import LeasePolicy
+
+    assert LeasePolicy().heartbeat_every() == 30            # 120 s timeout: the default 30 s stands
+    assert LeasePolicy(heartbeat_timeout_s=30).heartbeat_every() == 10
+    assert LeasePolicy(heartbeat_timeout_s=30).heartbeat_every(30) == 10
+    assert LeasePolicy(heartbeat_timeout_s=30).heartbeat_every(4) == 5
+    assert LeasePolicy(heartbeat_timeout_s=9).heartbeat_every() == 5
+    assert LeasePolicy(heartbeat_timeout_s=600).heartbeat_every(45) == 45
+
+
+def test_state_machine_and_durable_lease_use_the_clamped_interval():
+    from microvm.integrations.stepfunctions import lease_state_machine
+    from microvm.lease import LeasePolicy
+
+    asl = lease_state_machine(image_arn="arn:aws:lambda:us-east-1:1:microvm-image:x",
+                              execution_role_arn="arn:aws:iam::1:role/r",
+                              policy=LeasePolicy(heartbeat_timeout_s=30), region="us-east-1", heartbeat_s=30)
+    assert "'heartbeat_s': 10" in asl["States"]["Lease"]["Arguments"]["RunHookPayload"]