microvm-ctl 0.3.1__tar.gz → 0.4.0__tar.gz

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: microvm-ctl
-Version: 0.3.1
+Version: 0.4.0
 Summary: Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live.
 Author: Vivek Raja P S
 License: Apache-2.0
@@ -196,10 +196,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/README.md

@@ -160,10 +160,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/microvm/__init__.py

@@ -12,7 +12,7 @@ from microvm.images import ImageBuilder, ImageBuildError
 from microvm.lease import FanoutLimit, Lease, LeasePlan, LeasePlanRejected, LeasePolicy, plan_fanout
 from microvm.monitor import CostModel, FleetMonitor
 
-__version__ = "0.3.1"
+__version__ = "0.4.0"
 
 __all__ = [
     "microvm_client",

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/microvm/cli.py

@@ -9,6 +9,7 @@
     mvm scale IMAGE N                      converge fleet to N
     mvm suspend|resume|terminate ID...     lifecycle control
     mvm drain IMAGE                        terminate the whole fleet
+    mvm dispatch IMAGE /path -d '{}' -n 64 spread requests over the warm fleet, --per-vm in flight each
     mvm call ID /path [-X POST -d '{}']    authenticated request into the VM
     mvm status ID | watch ID               job telemetry from the hook runtime (/status, /events)
     mvm watch --image NAME                 one live table for every leased VM of an image
@@ -226,6 +227,37 @@ def cmd_drain(args):
     console.print(f"[bold green]✓[/] terminated {n} microVMs of [cyan]{args.image}[/]")
 
 
+def cmd_dispatch(args):
+    if args.bodies:
+        with open(args.bodies) as f:
+            bodies = [json.loads(line) for line in f if line.strip()]
+    else:
+        bodies = [json.loads(args.data or "{}")] * args.n
+    fleet = Fleet(FleetManager(_cfg(args)), args.image)
+    started = time.time()
+    with console.status(f"[bold]dispatching {len(bodies)} requests over [cyan]{args.image}[/]…"):
+        results = fleet.dispatch(args.path, bodies, per_vm=args.per_vm, method=args.method, port=args.port,
+                                 timeout=args.timeout)
+    wall = time.time() - started
+    if args.out:
+        with open(args.out, "w") as f:
+            for r in results:
+                f.write(json.dumps(r, default=str) + "\n")
+    by_vm: dict = {}
+    for r in results:
+        by_vm.setdefault(r["microvm_id"], []).append(r)
+    t = Table(title=f"{len(results)} requests in {wall:.2f} s ({len(results) / wall:.1f}/s), {args.per_vm} in flight per VM")
+    for col in ("microVM", "requests", "ok", "failed", "p50 ms", "max ms"):
+        t.add_column(col)
+    for vid, rs in sorted(by_vm.items()):
+        lat = sorted(r["ms"] for r in rs)
+        ok = sum(1 for r in rs if r["status"] and r["status"] < 400)
+        t.add_row(vid, str(len(rs)), str(ok), str(len(rs) - ok), f"{lat[len(lat) // 2]:.0f}", f"{lat[-1]:.0f}")
+    console.print(t)
+    if any(not r["status"] or r["status"] >= 400 for r in results):
+        sys.exit(1)
+
+
 # ---------------------------------------------------------------- execution plane
 def cmd_call(args):
     cfg = _cfg(args)
@@ -724,6 +756,19 @@ def main(argv: list[str] | None = None):
     d.add_argument("image")
     d.set_defaults(fn=cmd_drain)
 
+    dp = sub.add_parser("dispatch", help="spread requests over the RUNNING members of an image")
+    dp.add_argument("image")
+    dp.add_argument("path")
+    dp.add_argument("-X", "--method", default="POST")
+    dp.add_argument("-d", "--data", help="one JSON body, sent -n times")
+    dp.add_argument("-n", type=int, default=1, help="how many times to send --data")
+    dp.add_argument("--bodies", help="a JSON Lines file, one request body per line")
+    dp.add_argument("--per-vm", type=int, default=4, help="requests in flight per VM")
+    dp.add_argument("--port", type=int, default=8080)
+    dp.add_argument("--timeout", type=float, default=300)
+    dp.add_argument("--out", help="write every result as JSON Lines")
+    dp.set_defaults(fn=cmd_dispatch)
+
     c = sub.add_parser("call", help="authenticated request into a microVM")
     c.add_argument("id")
     c.add_argument("path")

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/microvm/fleet.py

@@ -12,9 +12,11 @@ retried with jittered backoff, so `scale_to(50)` is safe to call in one shot.
 from __future__ import annotations
 
 import concurrent.futures as futures
+import queue
+import threading
 import time
 from dataclasses import dataclass, field
-from typing import Callable
+from typing import Any, Callable
 
 from microvm.client import image_arn, microvm_client
 from microvm.config import TPS, PlaneConfig
@@ -311,15 +313,23 @@ class Fleet:
     _pool: futures.ThreadPoolExecutor = field(
         default_factory=lambda: futures.ThreadPoolExecutor(max_workers=Fleet.MAX_WORKERS), repr=False
     )
+    #: VMs this fleet has asked to terminate. ListMicrovms keeps reporting a terminated VM in its old
+    #: state for about a second, so without this a size() right after a scale-down counts the victim
+    #: and a drain terminates it twice.
+    _terminated: set = field(default_factory=set, repr=False)
 
     # -- observation -------------------------------------------------------------
     def members(self) -> list[Microvm]:
         return [
             vm
             for vm in self.manager.list(self.image, self.version)
-            if vm.state in ACTIVE_STATES
+            if vm.state in ACTIVE_STATES and vm.microvm_id not in self._terminated
         ]
 
+    def _terminate(self, microvm_id: str) -> None:
+        self.manager.terminate(microvm_id)
+        self._terminated.add(microvm_id)
+
     def size(self) -> int:
         return len(self.members())
 
@@ -345,7 +355,7 @@ class Fleet:
             return launched
         if delta < 0:
             for vm in self.scale_down_victims(current, -delta):
-                self.manager.terminate(vm.microvm_id)
+                self._terminate(vm.microvm_id)
         return []
 
     @staticmethod
@@ -384,7 +394,7 @@ class Fleet:
     def drain(self) -> int:
         """Terminate every member of the fleet."""
         vms = self.members()
-        list(self._pool.map(lambda v: self.manager.terminate(v.microvm_id), vms))
+        list(self._pool.map(lambda v: self._terminate(v.microvm_id), vms))
         return len(vms)
 
     # -- reaper ------------------------------------------------------------------
@@ -396,6 +406,81 @@ class Fleet:
         for vm in self.members():
             started = vm.started_epoch
             if started and now - started > max_age_seconds:
-                self.manager.terminate(vm.microvm_id)
+                self._terminate(vm.microvm_id)
                 reaped.append(vm.microvm_id)
         return reaped
+
+    # -- work distribution -------------------------------------------------------
+    def dispatch(
+        self,
+        path: str,
+        bodies: list,
+        *,
+        per_vm: int = 4,
+        method: str = "POST",
+        port: int = 8080,
+        timeout: float = 300,
+        client_factory: Callable[[str], Any] | None = None,
+        on_result: Callable[[dict], None] | None = None,
+    ) -> list[dict]:
+        """Send one request per body to the fleet's RUNNING members and return the results in order.
+
+        Each RUNNING member gets `per_vm` workers that pull from one shared queue, so a slow VM takes
+        fewer items and no VM ever holds more than `per_vm` requests in flight. This is the warm-fleet
+        path: hundreds of short tasks over VMs that are already up, where a lease per task would pay a
+        launch per task. A request that raises is recorded with its error and does not stop the others.
+
+        Each result is {"index", "microvm_id", "status", "ms", "body"} or, on an exception,
+        {"index", "microvm_id", "status": None, "ms", "error"}. `on_result` is called with each result
+        as it lands, from a worker thread.
+        """
+        if per_vm < 1:
+            raise ValueError("per_vm must be at least 1")
+        running = [vm for vm in self.members() if vm.state == "RUNNING"]
+        if not running:
+            raise RuntimeError(f"fleet {self.image} has no RUNNING members to dispatch to")
+        if client_factory is None:
+            from microvm.endpoint import EndpointClient
+
+            def client_factory(microvm_id: str):
+                return EndpointClient(self.manager.cfg, microvm_id, ports=[port])
+
+        work: queue.Queue = queue.Queue()
+        for i, body in enumerate(bodies):
+            work.put((i, body))
+        results: list = [None] * len(bodies)
+        lock = threading.Lock()
+
+        def worker(microvm_id: str, client) -> None:
+            while True:
+                try:
+                    i, body = work.get_nowait()
+                except queue.Empty:
+                    return
+                started = time.time()
+                try:
+                    resp = client.request(method, path, json=body, timeout=timeout, port=port)
+                    try:
+                        payload = resp.json()
+                    except ValueError:
+                        payload = resp.text
+                    out = {"index": i, "microvm_id": microvm_id, "status": resp.status_code,
+                           "ms": round((time.time() - started) * 1000, 1), "body": payload}
+                except Exception as e:  # one failed request must not stop the batch
+                    out = {"index": i, "microvm_id": microvm_id, "status": None,
+                           "ms": round((time.time() - started) * 1000, 1), "error": f"{type(e).__name__}: {e}"}
+                with lock:
+                    results[i] = out
+                if on_result:
+                    on_result(out)
+
+        threads = []
+        for vm in running:
+            client = client_factory(vm.microvm_id)
+            for _ in range(per_vm):
+                t = threading.Thread(target=worker, args=(vm.microvm_id, client), daemon=True)
+                t.start()
+                threads.append(t)
+        for t in threads:
+            t.join()
+        return results

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/microvm_ctl.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: microvm-ctl
-Version: 0.3.1
+Version: 0.4.0
 Summary: Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live.
 Author: Vivek Raja P S
 License: Apache-2.0
@@ -196,10 +196,12 @@ The four-part article series **Building on AWS Lambda MicroVMs** on the AWS Buil
 1. [Control and scale AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JIDTpz0ZgatSBv24drra3gEod9/control-and-scale-aws-lambda-microvms-with-microvm-ctl): this plane and its measurements.
 2. [Seven workloads Lambda could never run, until MicroVMs](https://builder.aws.com/content/3JJ2oNWY9EsZzivMMx044cSlrFQ/seven-workloads-lambda-could-never-run-until-microvms): the first seven examples through build, run, cost, and gotchas.
 3. [A kernel for every customer: scaling AI agents to 1,000 tenants on AWS Lambda MicroVMs with microvm-ctl](https://builder.aws.com/content/3JJ7tASPSSUUTrcpWnWtxOuu8g3/a-kernel-for-every-customer-scaling-ai-agents-to-1000-tenants-on-aws-lambda-microvms-with-microvm-ctl): the multi-tenant finale with the decision guide.
-4. Hand a task to a MicroVM from anywhere (part 4, [source in awesome-microvm](https://github.com/Vivek0712/awesome-microvm/blob/main/blog/03-handoff.md), Builder Center link to follow): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
+4. [Hand a task to a MicroVM from anywhere: one lease, Step Functions, durable functions, or your own controller](https://builder.aws.com/content/3Jcx0VcEOq9QQBtqr59y4S8dVFC/hand-a-task-to-a-microvm-from-anywhere-one-lease-step-functions-durable-functions-or-your-own-controller): the lease contract, Step Functions, durable functions, your own controller, and leases at scale.
 
 The article sources and a long-form deep dive per example live under [awesome-microvm/blog](https://github.com/Vivek0712/awesome-microvm/tree/main/blog).
 
+Every lease scenario also runs live, on Step Functions and on a Lambda durable function, in [microvm-handoff-demo](https://github.com/Vivek0712/microvm-handoff-demo): nine scenarios on both orchestrators plus one lease from the CLI, with the execution histories, VM and orchestrator logs, benchmarks, and console screenshots checked in. Its first pass ran on 0.3.0 and found the two things that became 0.3.1: a typed failure whose VM Step Functions never terminated, and a 30 s heartbeat interval against a 30 s heartbeat timeout that lost a durable callback before the first heartbeat landed.
+
 ## Requirements and regions
 
 Python 3.9 or newer on the machine running the plane. The VMs themselves are ARM64 (Graviton) only, so audit binary wheels before you build an image. The service is available in `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`.

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "microvm-ctl"
-version = "0.3.1"
+version = "0.4.0"
 description = "Control and execution plane for AWS Lambda MicroVMs: build images, launch Firecracker microVMs, scale fleets inside your quotas, call into them securely, and watch it all live."
 readme = "README.md"
 requires-python = ">=3.9"

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/tests/test_fleet.py

@@ -84,3 +84,67 @@ def test_run_params_carries_client_token(monkeypatch):
     assert params["runHookPayload"] == '{"callback_id": "abc"}'
     assert params["maximumDurationInSeconds"] == 900
     assert "clientToken" not in fm.run_params("img")
+
+
+def test_scale_down_victim_is_not_counted_while_list_still_shows_it():
+    # ListMicrovms reports a terminated VM in its old state for a moment; the fleet must not count it.
+    members = [_vm(1, "RUNNING", 600), _vm(2, "SUSPENDED", 300), _vm(3, "RUNNING", 10)]
+    fm = FakeManager(members)
+    fleet = Fleet(fm, "img")
+    fleet.scale_to(2)
+    assert fm.terminated == ["vm-2"]
+    assert fleet.size() == 2
+    assert fleet.drain() == 2
+    assert sorted(fm.terminated) == ["vm-1", "vm-2", "vm-3"]
+
+
+class FakeResponse:
+    def __init__(self, status, payload):
+        self.status_code, self._payload = status, payload
+
+    def json(self):
+        return self._payload
+
+
+class FakeClient:
+    def __init__(self, microvm_id, log, fail_on=None, delay=0.0):
+        self.microvm_id, self.log, self.fail_on, self.delay = microvm_id, log, fail_on, delay
+
+    def request(self, method, path, json=None, timeout=None, port=None):
+        import time as _t
+        _t.sleep(self.delay)
+        self.log.append((self.microvm_id, method, path, json))
+        if self.fail_on is not None and json == self.fail_on:
+            raise ConnectionError("boom")
+        return FakeResponse(200, {"echo": json, "vm": self.microvm_id})
+
+
+def test_dispatch_spreads_bodies_over_running_members_and_keeps_order():
+    fm = FakeManager([_vm(1, "RUNNING", 60), _vm(2, "RUNNING", 60), _vm(3, "SUSPENDED", 60)])
+    log = []
+    fleet = Fleet(fm, "img")
+    seen = []
+    results = fleet.dispatch("/task", [{"n": i} for i in range(10)], per_vm=2,
+                             client_factory=lambda vm_id: FakeClient(vm_id, log, delay=0.01),
+                             on_result=seen.append)
+    assert [r["index"] for r in results] == list(range(10))
+    assert [r["body"]["echo"] for r in results] == [{"n": i} for i in range(10)]
+    assert {r["microvm_id"] for r in results} == {"vm-1", "vm-2"}  # the SUSPENDED member gets nothing
+    assert all(m == "POST" and p == "/task" for _, m, p, _ in log)
+    assert len(seen) == 10
+
+
+def test_dispatch_records_a_failed_request_and_finishes_the_rest():
+    fm = FakeManager([_vm(1, "RUNNING", 60)])
+    fleet = Fleet(fm, "img")
+    results = fleet.dispatch("/task", [{"n": 0}, {"n": 1}, {"n": 2}], per_vm=1,
+                             client_factory=lambda vm_id: FakeClient(vm_id, [], fail_on={"n": 1}))
+    assert results[0]["status"] == 200 and results[2]["status"] == 200
+    assert results[1]["status"] is None and "ConnectionError" in results[1]["error"]
+
+
+def test_dispatch_needs_a_running_member():
+    import pytest
+    fleet = Fleet(FakeManager([_vm(1, "SUSPENDED", 60)]), "img")
+    with pytest.raises(RuntimeError, match="no RUNNING members"):
+        fleet.dispatch("/task", [{}], client_factory=lambda vm_id: None)

{microvm_ctl-0.3.1 → microvm_ctl-0.4.0}/tests/test_integrations_durable_scale.py

@@ -118,7 +118,8 @@ def test_two_shards_happy_path(fm):
         cb1 = runner.wait_for_callback(arn, name="job-1-0-callback", timeout=30)
         assert cb0 != cb1
         assert fm.plans == [(2, 2048, POLICY)]  # the plan step ran once, with the reconstructed policy
-        assert [launch["task"] for launch in fm.leases] == SHARDS
+        # map items launch concurrently, so compare as a set of shards, not in launch order
+        assert sorted((launch["task"] for launch in fm.leases), key=lambda t: t["i"]) == SHARDS
         assert all(launch["version"] == "3" and launch["policy"] == POLICY for launch in fm.leases)
         _succeed(runner, cb0, 1)
         _succeed(runner, cb1, 2)