langgraph-api 0.16.0.dev7__tar.gz → 0.16.0.dev9__tar.gz

{langgraph_api-0.16.0.dev7 → langgraph_api-0.16.0.dev9}/PKG-INFO

@@ -1,10 +1,11 @@
 Metadata-Version: 2.4
 Name: langgraph-api
-Version: 0.16.0.dev7
+Version: 0.16.0.dev9
 Author-email: Will Fu-Hinthorn <will@langchain.dev>, Josh Rogers <josh@langchain.dev>, Parker Rule <parker@langchain.dev>
 License: Elastic-2.0
 License-File: LICENSE
 Requires-Python: >=3.11
+Requires-Dist: anyio>=4.14.2
 Requires-Dist: cloudpickle>=3.0.0
 Requires-Dist: cryptography>=50.0.0
 Requires-Dist: grpcio-health-checking<1.82.0,>=1.81.0

{langgraph_api-0.16.0.dev7 → langgraph_api-0.16.0.dev9}/benchmark/README.md

@@ -129,7 +129,11 @@ by replaying them. That moves cost from write to read, and the read cost grows w
 thread depth. `delta_turns` measures it: one thread per VU, one turn per iteration,
 `turn_duration` tagged by depth bucket.
 
-Run the cells one at a time and compare `turn_duration` by depth. Both sides get slower
+`benchmark-delta.yml` runs four cells nightly, 20 users for 25 minutes each: plain channel, then
+delta snapshotting every 50 (`benchmark_delta_freq`, the deepagents setting), 500
+(`benchmark_delta_500`) and 1000 (`benchmark_delta`, the library default). `delta-report.js` posts
+a headline table (non-delta vs delta at 50: read, write, storage by depth) and one table per metric
+across the three frequencies. The messages and migration cells stay manual. Run the cells one at a time and compare `turn_duration` by depth. Both sides get slower
 with depth for different reasons, the control by re-writing a growing value and delta by
 replaying more writes, so the deliverable is which curve is worse and where they cross.
 
@@ -138,6 +142,7 @@ replaying more writes, so the deliverable is which curve is worse and where they
 | control | `GRAPH_ID=benchmark EXPECT_DELTA=false` |
 | delta, snapshot every 1000 | `GRAPH_ID=benchmark_delta` |
 | delta, snapshot every 50 | `GRAPH_ID=benchmark_delta_freq` |
+| delta, snapshot every 500 | `GRAPH_ID=benchmark_delta_500` |
 | delta on a message list | `GRAPH_ID=benchmark_delta_messages PAYLOAD=messages` |
 | migration | `GRAPH_ID=benchmark_delta MIGRATE_FROM_GRAPH_ID=benchmark MIGRATE_AFTER=200` |
 
@@ -154,7 +159,7 @@ Knobs beyond the common ones:
 - `MIGRATE_FROM_GRAPH_ID` / `MIGRATE_AFTER` - run the first N turns on another graph, then
   switch on the same thread.
 - `VERIFY_EVERY` - cold re-read cadence in turns. Default 50, `0` disables.
-- `DEPTH_BUCKET` - width of a depth bucket. Default 100.
+- `DEPTH_BUCKET` - width of a depth bucket. Defaults to `VERIFY_EVERY`, so each bucket holds one cold read per thread.
 
 Two things to know before reading any result:
 
@@ -166,7 +171,9 @@ Two things to know before reading any result:
 
 The runner fails a turn rather than reporting a false pass when delta is not actually on:
 the `DeltaChannel` import is version-guarded, and its stub degrades to a plain channel.
-At every `VERIFY_EVERY` turn it also reads `GET /threads/{thread_id}/storage` and records
+At every `VERIFY_EVERY` turn it also times the cold `GET /threads/{thread_id}/state` as
+`state_read_duration` (the read path alone; a turn is one save plus the read before it), and
+reads `GET /threads/{thread_id}/storage` and records
 `thread_storage_bytes` tagged by depth bucket and graph, so the control and delta cells can
 be compared on the bytes they occupy, which is what the feature exists to reduce.
 
@@ -185,6 +192,12 @@ be compared on the bytes they occupy, which is what the feature exists to reduce
 
 - Daily ramp benchmark: `.github/workflows/benchmark.yml`
 - Daily capacity matrix: `.github/workflows/benchmark-capacity-matrix.yml`
+- Daily delta-channel cells: `.github/workflows/benchmark-delta.yml`, reported by `delta-report.js`
+
+The staging deployments build their graphs from
+[langgraph-server-benchmarks](https://github.com/langchain-ai/langgraph-server-benchmarks), not
+from this repo. Its `src/agent/graph.py` is a copy of `api/tests/graphs/benchmark.py`; a graph
+change here reaches the nightly only once that copy is updated.
 
 ### Capacity matrix setup
 
@@ -192,9 +205,18 @@ be compared on the bytes they occupy, which is what the feature exists to reduce
 2. Fill in `deploymentId` and `baseUrl` in `CAPACITY_DEPLOYMENTS` at the top of [`capacity-ci.js`](capacity-ci.js).
 3. Scheduled run: two parallel CI jobs (one per deployment), aggregate job posts Slack summary.
 
-Each cell also records the deployment's own view of its window (queue and API pod counts, CPU peaks) from the control plane's deployment metrics route, and the posted table lists it under each deployment. Every tier starts at one queue pod and autoscales during a cell, so a capacity number is partly a pod-count number; pinning pods at the tier maximum would need the benchmark deployments recreated as `external_docker` sources, the only kind whose `resource_spec` the API accepts.
+Each cell also records the deployment's own view of its window (queue and API pod counts, CPU peaks) from the control plane's deployment metrics route. The posted message names any component at or over 95% of its CPU request as that tier's bottleneck, and the full per-cell table goes to the message's thread. Every tier starts at one queue pod and autoscales during a cell, so a capacity number is partly a pod-count number; pinning pods at the tier maximum would need the benchmark deployments recreated as `external_docker` sources, the only kind whose `resource_spec` the API accepts.
+
+### Slack reports
+
+Every report follows one shape so the channel reads the same way each morning, on a phone as well as a desktop:
+
+- The first line is the verdict: a dot (green quiet, yellow something regressed or a bottleneck or errors, red something failed), the report name and date, `manual run` when dispatched by hand, the API version, and a clause saying what needs attention. The second line says what the benchmark is, for a reader outside the team.
+- Then `Failed`, `Regressed`, `Improved` and `Bottlenecks` or `Errors` lines, worst first. `Regressed` prints `none` rather than going quiet.
+- One table per deployment, about 40 characters wide so it fits a phone, with `▲ better` / `▼ worse` markers on cells that left their usual range.
+- One legend line (pass rule, arrows, dot colors), then a link to the thread, which holds the per-cell or per-run numbers and the longer *How to read*, and to the run's artifacts.
 
-Both the daily and the capacity reports compare tonight with the previous scheduled nights (`history.js`, artifacts fetched with `gh`), flagging a figure only when it leaves that runner's own observed range.
+Both reports compare tonight with the previous scheduled nights (`history.js`, artifacts fetched with `gh`). A figure is flagged only when it leaves that cell's own observed range, and only once five same-kind nights exist; before that the table says how many nights it has. The daily benchmark posts one digest per workflow once its last runner finishes (`daily-report.js`, fed by each runner's artifact), not one message per runner. The delta-channel report (`delta-report.js`) keeps the plain-vs-delta table in the channel and puts the snapshot-frequency sweep in the thread.
 
 Local commands:
 

{langgraph_api-0.16.0.dev7 → langgraph_api-0.16.0.dev9}/benchmark/benchmark-runners/delta_turns.ts

@@ -30,6 +30,8 @@ const PAYLOAD_CHANNELS = [
 
 /** Tagged by depth: VUs ramp in, so clock time is not thread length. */
 const turnDuration = new Trend('turn_duration', true);
+/** The read path alone: a cold GET /state replays the channel without writing. */
+const stateReadDuration = new Trend('state_read_duration', true);
 const threadStorageBytes = new Trend('thread_storage_bytes', false);
 
 const TURN_CONFIG: Omit<DeltaTurnsOptions, 'graphId'> = {
@@ -37,7 +39,8 @@ const TURN_CONFIG: Omit<DeltaTurnsOptions, 'graphId'> = {
   migrateAfter: parseInt(__ENV.MIGRATE_AFTER || '0', 10),
   expectDelta: (__ENV.EXPECT_DELTA || 'true') === 'true',
   verifyEvery: parseInt(__ENV.VERIFY_EVERY || '50', 10),
-  bucketSize: parseInt(__ENV.DEPTH_BUCKET || '100', 10),
+  // One bucket per cold read, so every row of the report is one sample per thread.
+  bucketSize: parseInt(__ENV.DEPTH_BUCKET || __ENV.VERIFY_EVERY || '50', 10),
 };
 
 // Each k6 VU runs this module in its own context, so these are per VU.
@@ -96,7 +99,13 @@ export class DeltaTurns extends BenchmarkRunner {
     // A failed body is not state, and comparing against it would report drift
     // on top of the failure that caused it.
     if (response.status === 200 && (plan.guard || plan.verify)) {
+      const readStartedAt = Date.now();
       const state = http.get(`${baseUrl}/threads/${threadId}/state`, requestParams);
+      // The turn-2 guard read is at depth 2, not the bucket's end; timing it
+      // would pull the first row's read median down.
+      if (state.status === 200 && plan.verify) {
+        stateReadDuration.add(Date.now() - readStartedAt, { depth: plan.depthBucket });
+      }
       addResponse(responses, 'state', state);
       const body = state.status === 200 ? safeJson(state.body) : null;
       if (body === null) {
@@ -111,7 +120,13 @@ export class DeltaTurns extends BenchmarkRunner {
         }
       }
       if (plan.verify) {
-        const storage = http.get(`${baseUrl}/threads/${threadId}/storage`, requestParams);
+        // Servers before 0.16 have no storage endpoint. A 404 there is a missing
+        // sample, not a failed request: without this, one 404 per 50 turns is
+        // about 2% and trips the 1% error budget on every cell.
+        const storage = http.get(`${baseUrl}/threads/${threadId}/storage`, {
+          ...requestParams,
+          responseCallback: http.expectedStatuses(200, 404),
+        });
         addResponse(responses, 'storage', storage);
         const total = storage.status === 200 ? safeJson(storage.body)?.total_bytes : null;
         if (typeof total === 'number') {