@rnd-news/newsletter 1.32.0 → 1.34.0

package/README.md

@@ -4,7 +4,7 @@ The research newsletter of mcp-tool-shop: what we measured, and what it means. E
 in plain language, with the numbers, the charts, what held up and what didn't.
 
 This package mirrors the published issues and is updated automatically after each issue goes live on the site.
-It holds 30 issues. The newest is issue 32, 2026-10-27: [27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/).
+It holds 32 issues. The newest is issue 34, 2026-10-29: [29 October: repaired test sets certify 44 of 50, long tool lists cost a model its "no", and Cosmos 3 on one rented card](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-34/).
 
 - **Read it on the web:** https://mcp-tool-shop-org.github.io/rnd/
 - **RSS:** https://mcp-tool-shop-org.github.io/rnd/newsletter/rss.xml

package/data/issues.jsonl

@@ -28,3 +28,5 @@
 {"issue": 30, "date": "2026-10-25", "title": "25 October: a crash test, a lab for tests with proven answers, and our Colab cards measured", "description": "The first daily issue. We killed a rented machine mid-write to test Attestia's event store, and the log survived, but the kill didn't test the fix, so we tore the same file by hand. We started a lab that builds tests whose answers are proved before any wording is written. And we turned 21 rented-GPU windows of receipts into a profile of each Colab card, with what's still unmeasured.", "highlights": ["<strong>A crash test that didn't test the fix.</strong> A machine killed mid-write left a log that a fresh machine opened whole, but no line was torn, so the fix never ran. We tore the same file by hand: the old code lost an event and refused to open, the fixed code didn't.", "<strong>A lab for tests with proven answers.</strong> reasoning-dataset-lab maps a dataset before any item exists and proves each answer in code before a model writes a word. Its first pilot map has 24 cells, 12 of them buildable today.", "<strong>Each Colab card, measured.</strong> From 21 windows of receipts: what each card has, what it costs per hour, how often a session was lost, and what no receipt has measured yet."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/", "body_markdown": "This is the first of the daily issues: one email a day, with an article for each piece of the day's work worth\nreporting. Today has three.\n\n**In this issue:**\n\n1. [We killed a machine mid-write. The log survived, but the kill didn't test the fix](#we-killed-a-machine-mid-write-the-log-survived-but-the-kill-didnt-test-the-fix)\n2. [A lab for tests whose answers are proved first](#a-lab-for-tests-whose-answers-are-proved-first)\n3. [What each Colab card is, from the receipts](#what-each-colab-card-is-from-the-receipts)\n\n## We killed a machine mid-write. The log survived, but the kill didn't test the fix\n\nAttestia is one of the studio's three proof products. Its **event store** keeps a log as a plain file, one event per\nline. Each event carries the hash of the one before it, so the events form a chain. If the chain is broken anywhere,\nthe store refuses to open. That's on purpose: a log you can't trust is worse than no log.\n\nWe want to use that store as the run log for experiments on rented machines, and rented machines vanish. So before\ntrusting it, we tried to break it.\n\n### The bug\n\nWe found the problem on our own machine first, before spending anything. A writer that's killed partway through\nwriting a line leaves a **torn** last line: half an event, with no line ending. Then:\n\n1. A new writer opens the store. It skips the torn line, checks the chain up to the last whole event, and opens.\n2. Its first new event is written straight after the torn fragment, on the same line, because nothing put a line\n   ending in between.\n3. The next time anything opens the store, it skips that glued line too, so the new event is gone. The event after\n   it now points at a predecessor that isn't there, and the chain is broken.\n4. With checking on, which is the default, the store refuses to open at all.\n\nSo one crash followed by one honest restart turned a good log into one that wouldn't open.\n\n### The fix\n\nThe fix is small. When the store opens a file that doesn't end in a line ending, it writes one before its first new\nevent. The torn fragment stays in the file on a line of its own and is skipped every time. Nothing is cut off or\nrewritten, so the store keeps its rule that the file only ever grows.\n\n### A real kill\n\nThen we tested it for real, on a rented Colab machine running on CPU:\n\n- a writer added 10 events a second for an hour;\n- a copy of the store was taken off the machine every 60 seconds;\n- at 3,654 seconds we killed the machine outright, with no warning to the writer;\n- 56 seconds later, a fresh machine opened the last copy, with checking on.\n\nThe copy held **36,498 events, all valid**. The new machine carried on writing and added 27,000 more. The final log\nheld **63,498 events: a valid chain, no gaps and no duplicates**. Its first 36,498 events are byte for byte the copy\nit started from, and its first new event links to the copy's last.\n\n![Line chart of events across the hard kill at 3,654 seconds and the resume 56 seconds later, with a step detail beside it. Machine 1 rises at 10 events a second to its last report of 36,516 events, one second before the kill. The last 60-second copy held 36,498 valid events, so at least 18 were lost with the machine. Machine 2 opened that copy 56 seconds later and added 27,000 events to finish at 63,498 valid events, with 0 gaps and 0 duplicate ids](https://mcp-tool-shop-org.github.io/rnd/charts/issue-30-crash-kill-resume.svg)\n\n### What the kill didn't test\n\nTwo things were true that a headline would skip.\n\n**The fix never ran.** The copy we resumed from ended on a whole line. There was no torn fragment for the fix to deal\nwith. The run proved that a hard kill and a resume on a new machine work, but it didn't prove the fix.\n\n**Some events died with the machine.** Its last progress report, one second before the kill, counted at least\n36,516 events. The copy held 36,498. So at least 18 events, about two seconds of writing, were on a machine that no\nlonger exists. That isn't a store bug: a resume from a copy can only have what the copy had. Anyone running a log\non a machine that can disappear has to decide how much of it they can afford to lose, and copy at least that often.\n\n### The fix, tested on the same file\n\nSo we tore the file ourselves. We took the copy from the real kill and added the first 328 bytes of a real 657-byte\nevent, with no line ending: exactly what a writer killed mid-line leaves. Then we resumed it twice, once with the\nold code and once with the fixed code, on our own machine.\n\n- **The old code** skipped the fragment and opened. It added 60 events, and the first was glued onto the torn line\n  and lost. The chain broke at position 36,500, and the store then refused to open.\n- **The fixed code** skipped the fragment, added the same 60 events, and lost none. The chain was valid with no gaps,\n  and the store opened again cleanly.\n\n![Two lanes from the same start: a store of 36,498 valid events plus the first 328 bytes of a 657-byte line, with no line ending. The old code skips the tear, adds 60 events, and glues the first one onto the torn line; the next open skips that line too, so 1 event is lost, 36,557 load, the chain breaks at position 36,500, and the store refuses to open. The fixed code writes a line ending before its first event, so only the fragment is skipped, 36,558 load, the chain is valid with 0 gaps, and the store opens](https://mcp-tool-shop-org.github.io/rnd/charts/issue-30-crash-torn-file.svg)\n\n### What it cost\n\n| Step | Where | Cost |\n|---|---|---|\n| A short trial run of the whole procedure | Colab, CPU | 0.01 units |\n| The real kill and resume | Colab, CPU | 0.14 units |\n| The torn-file test, old code and fixed | our own machine | nothing |\n\n### Where it stands\n\n- **Proven:** a hard kill, and a resume on a fresh machine, with every event in the copy kept and the chain intact.\n- **Proven separately:** the fix, on that same file with a torn line added by hand.\n- **Not proven:** that a real kill tears a line, or how often. Ours didn't.\n- **By design:** events written after the last copy are lost with the machine.\n\nThe fix shipped today in **Attestia 2.1.2**, on npm and as a GitHub release. The run logs, both stores and the\ntorn-file test are in the studio's records, and every figure here was recounted from the store files themselves.\n\n## A lab for tests whose answers are proved first\n\n[Issue 29](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-29/) ended with a decision: we stopped deciding anything about our critic from that\ntest panel until we have a test whose answers are known for certain. This article is the start of that test.\n\n**reasoning-dataset-lab** is a new public repository at\n[mcp-tool-shop-org/reasoning-dataset-lab](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab). It builds\npuzzles, quizzes and other tests. Its rules are the reverse of how a test is often made, where a model writes the\nitems and someone checks them afterwards:\n\n- **Map first.** Before any item exists, a `map.yaml` says what the dataset is: its axes (the skill tested, the kind of\n  item, the difficulty, the subject, and the wrong answer each item is built to catch), every cell those axes make,\n  how many items each cell gets, and the rules the finished set must pass. The map is hashed, so it can't drift\n  quietly.\n- **Prove, then word.** Code builds each item's underlying problem and proves its answer before a language model\n  writes any prose around it. An item kind whose answer can't be proved that way is marked `judged`, and gets in only\n  if judges from several model families all agree.\n- **Show the gaps.** A cell the generator can't build yet stays in the map with a capacity of 0 and a reason. The\n  map doesn't shrink to make coverage look complete.\n- **Provenance on every item,** so any item can be traced back to the map, the source material and the code that\n  made it.\n\n### The first pilot map\n\nThe first map, in the repository's `examples/`, is for a single skill: reading a stated quantity correctly. Its wrong\nanswers are four kinds of misreading: an exact amount read as a minimum, an exact amount read as a maximum, and a\nminimum or a maximum read as exact. The subjects are lamps and cups, at three difficulties.\n\nThat makes **24 cells**. **12 are buildable today**, with 5 items each: 60 items, 48 for training and 12 held out for\nevaluation. The other **12 are in the map at capacity 0**, because the generator can't build those misreadings yet.\n\nThe rules for the finished set were fixed before anything is generated:\n\n- every answer key proved (a proof rate of 1.0);\n- at least half the items passing in each class;\n- a sample of 20 items read by a person, with at most 1 defect.\n\n![Grid of the 24 cells in the first pilot map: 4 misreading families by 2 subjects (lamps, cups) by 3 difficulties. The 12 cells for an exact amount read as a minimum or as a maximum are buildable, at 5 items each, 60 items in all (48 train, 12 eval). The 12 cells for a minimum or a maximum read as exact are in the map at capacity 0, because the generator can't build them yet. Fixed rules: a proof rate of 1.0, at least half passing per class, at most 1 defect in a sample of 20](https://mcp-tool-shop-org.github.io/rnd/charts/issue-30-lab-pilot-map.svg)\n\n### Where the lab stands\n\nThe repository has a map checker, the project files and their fingerprint, and, as of today, the first part of the\npilot: the interface every generator follows, and an adapter that drives the studio's existing puzzle generator\nunchanged. No pilot items have been built yet. The pilot's first batch, with its numbers and a sample of items, is\nthe next step.\n\n## What each Colab card is, from the receipts\n\nWe rent GPU time on Google Colab in **windows**: a session on one card, booked and closed by a script that keeps a\nreceipt of everything it did. Before planning the next set of measurements, we read every receipt we have, 21\nwindows and the probes around them, and wrote down what each card actually is. Every value below comes from a\nreceipt. Where no receipt measures something, we say so.\n\n### The cards\n\n| Card | Memory | Rate (units an hour) | Sessions | What went wrong |\n|---|---|---|---|---|\n| A100, 40 GB | 40,960 MiB | 5.3 | 62 or more | 3 sessions lost mid-run on 23 October, plus one machine dropped mid-download and one session pruned earlier; 3 full four-hour windows with no loss |\n| A100, 80 GB | 81,920 MiB | 6.77 | about 8 | 1 session ended at 132 seconds when we stopped it (from 2 windows; the other ran 10,732 seconds clean) |\n| RTX PRO 6000 (G4) | 97,887 MiB | 8.9 | about 9 | none (from 2 windows); one ran 8,680 seconds clean |\n| L4 | 23,034 MiB | 1.54 | about 8 | only one window used it |\n| T4 | not measured | about 1.07, worked out, not read | assigned 6 times | never used |\n| H100 | — | — | never assigned | refused 5 times, \"Service Unavailable\" |\n| CPU (standard) | 12.3 GB of RAM free | 0.08 | 41 or more | the download check rejected 5 of 52 tries as too slow |\n\nThe plan allows up to 20 sessions at once: 3 each of the five GPU cards we can be assigned (the two A100s, the RTX PRO 6000, the L4 and\nthe T4) and 5 CPU machines. The next request is refused.\n\n### Faster isn't always cheaper\n\nOn the same job, a critic model checking items:\n\n- the **RTX PRO 6000** took a median 46.7 seconds a call against the 40 GB A100's 66.5: about 1.4 times as fast, at\n  1.68 times the price;\n- the **80 GB A100** ran the model 1.11 times as fast as the 40 GB one, at 1.28 times the price, with identical\n  output on all 178 calls compared.\n\nSo for that job, the cheaper card cost less per result. That holds for that job only. Another model or another size\ncan change it.\n\n### What no receipt measures yet\n\n- the time to the first token, on every card;\n- how fast a model downloads through Ollama (no receipt has an Ollama log);\n- how long a card takes from the request to ready;\n- almost everything about the T4, which has never run a job, and the L4, which ran one window.\n\nThose gaps are the reason for the next sweep, **C2**, which measures the cards the same way: the largest model\ncontext each holds, the speed at several prompt lengths and loads, and the throughput of small embedding models.\n\n### Licences first\n\nBefore a model's first window, it needs an entry in our licence registry, read at its source. Six of the models in\nthe C2 sweep had none, so we read them: **phi3.5** (MIT), **nomic-embed-text-v1.5** (Apache-2.0),\n**bge-base-en-v1.5** (MIT), **bge-reranker-v2-m3** (Apache-2.0), **nli-deberta-v3-base** (Apache-2.0) and\n**Qwen3-Embedding-0.6B** (Apache-2.0). No card or linked page restricts our use. Only phi3.5 ships a licence file;\nthe other five rest on the licence their model card names. Each entry keeps a hashed snapshot of the page and an\narchive.org copy.\n"}
 {"issue": 31, "date": "2026-10-26", "title": "26 October: our speed table was timing a cache, plus three releases and a newsletter that travels", "description": "Three small test runs on a rented T4 showed that interlock's generate ladder timed a prompt cache instead of the prompt, and counted queued requests as parallel ones. We fixed it in three releases and corrected the docs. Also: Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, and this newsletter now publishes itself to Hugging Face, npm, PyPI and GitHub.", "highlights": ["<strong>A correction.</strong> In interlock 1.0.5 and earlier, the generate ladder's time to first token measured a prompt cache, and its speed with several requests at once was close to the number of requests times too high. Neither is used any more.", "<strong>Found cheaply.</strong> Three short runs on a rented T4 card found it, one defect at a time: a prompt that looped, prompts cut in half, a cache, a queue, and cells mislabelled when the card ran out of room.", "<strong>Three releases.</strong> Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, Cognate with signed requests.", "<strong>The newsletter, everywhere.</strong> Every issue now lands on Hugging Face, npm and PyPI, and gets its own GitHub release."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/", "body_markdown": "Today's lead is a correction to numbers our own tool produced. It's followed by three releases, and a change to how\nthis newsletter reaches you.\n\n**In this issue:**\n\n1. [Our speed table was timing a cache](#our-speed-table-was-timing-a-cache)\n2. [Three proof products ship](#three-proof-products-ship)\n3. [The newsletter learns to travel](#the-newsletter-learns-to-travel)\n\n## Our speed table was timing a cache\n\n**interlock** picks a model, an engine and a device for a job from speeds it has measured. To measure rented\ncards, it runs a **generate ladder**: the same model at prompt lengths from 512 to 32,768 tokens, with one, four\nand eight requests at a time, recording how long the first token takes and how fast the rest come.\n\nBefore spending real money on a full sweep, we ran short trial **rungs** on Colab's cheapest card, a T4. Three of\nthem found five problems, and every one was in our measurement, not in the card.\n\n### Rung 1: a prompt that looped\n\nThe ladder built its prompt by repeating one word. One model, phi3.5, kept repeating it until Ollama stopped the\nrun, so some cells timed a loop instead of an answer. The errors were also filed under the wrong label. **interlock\n1.0.5** builds a varied prompt, calibrated to the length asked for, and gives an unmeasured cell its own label so it\ncan never read as a speed.\n\n### Rung 2: cut in half, cached, and queued\n\nThe second rung found three problems at once, all confirmed in Ollama's own source:\n\n- **Long prompts were cut in half.** The prompt landed about 1% over its target, the window left no room for that, and\n  Ollama halved the prompt at 16k and 32k tokens. Those cells came back unmeasured.\n- **The first token came from a cache.** Ollama keeps the start of the last prompt, and the ladder sent the same\n  prompt for its check call, its warm-up and every timed run. So each timed run read a remembered prompt: **15 ms at\n  512 tokens and 33 ms at 8,192**, which is too fast to be real.\n- **Requests queued instead of running together.** Ollama serves one request at a time unless told otherwise. The\n  ladder sent four or eight at once, they waited in line, and the way it added up their speeds left out the wait. So\n  \"four at once\" read close to four times one stream.\n\n**interlock 1.0.6** fixes all three. Each timed call begins with a short prefix of its own,\nso nothing is cached. The server gets as\nmany slots as the cell has requests. And combined speed is total tokens over the pass's decode window, from the first decode start to the last\nfinish. A prompt-read time that doesn't grow with the prompt now counts as unmeasured.\n\nWe check every change on our own machine first, where a run costs nothing. Measured that way there (on an RTX\n5090), the time to read the prompt grows as it should: **23 ms at 512 tokens, 100 ms\nat 2k, 463 ms at 8k, 1,244 ms at 16k and 3,791 ms at 32k.**\n\n![Log-scale chart of the time to read the prompt against prompt length, for phi3.5 with one request. On the fixed ladder on an RTX 5090, with no cache, it grows from 23 ms at 512 tokens to 100 ms at 2k, 463 ms at 8k, 1,244 ms at 16k and 3,791 ms at 32k. The old ladder on the Colab T4, with the prompt cached, read 15 ms at 512 tokens and 33 ms at 8k. Different cards, so only the shapes compare](https://mcp-tool-shop-org.github.io/rnd/charts/issue-31-ladder-prefill.svg)\n\n### Rung 3: out of room, mislabelled\n\nOn the fixed ladder, some long, busy cells ran out of room. Part of the model didn't fit on the card and ran on\nthe processor until the request timed out, or the machine's memory filled and the run was killed. Those are honest\n\"too big for this card\" results, but they were recorded as a timeout or as unmeasured. **interlock 1.0.7** reads the\nload from Ollama's log before any timed call, records those cells as out of memory, and scales each cell's timeout\nfrom a calibration call of its own.\n\n### What's still valid, and the correction\n\nThe speed of one stream of tokens and the largest context each card holds were not affected; the embedding\nthroughput runs on a different path and wasn't part of these defects. Rows from 1.0.4 and earlier also overstate\ntheir prompt length about twice over, so their one-stream speeds belong to a prompt about half as long as labelled. The first-token times and the speeds with more than one request at a time, from 1.0.5 and earlier, were\nnot valid measures. interlock no longer uses them, and its README and handbook carry a correction notice.\n\nThe trial runs so far, three on a CPU and four on the T4, cost about 1.3 of the 800 units set aside for this sweep. With the real timings, the whole sweep is\nnow priced at about 328 to 347 units.\n\n## Three proof products ship\n\nOn 25 October, each of the studio's three proof products had a release:\n\n- **Attestia 2.1.3** cleared every open advisory in its production dependencies by upgrading them, and its chain\n  observer now stops with a clear error when a chain returns an integer too large to hold exactly. (2.1.2, earlier\n  the same day, carried [issue 30](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/)'s crash fix.)\n- **RepoMesh 2.5.0** adds a required test gate on every change, and its anchor posting stays on the test network: it\n  won't connect unless the network is testnet and the server is a known testnet server.\n- **Cognate 1.0.0** reaches 1.0. Every write to its server is signed with the caller's own key, checked for its time\n  and for replay, and recorded under the signer's name; every route acts only in the signer's tenant; and decisions\n  cite the evaluations they rest on. From 1.0, a breaking change to its API or formats means a new major version.\n\n## The newsletter learns to travel\n\nFrom issue 30 on, every issue publishes itself to four more places as soon as it's live on the site:\n\n| Where | What you get |\n|---|---|\n| Hugging Face: [mcp-tool-shop/rnd-newsletter](https://huggingface.co/datasets/mcp-tool-shop/rnd-newsletter) | every issue as a dataset row and a Markdown page |\n| npm: [`@rnd-news/newsletter`](https://www.npmjs.com/package/@rnd-news/newsletter) | the issues as JSON and Markdown |\n| PyPI: [`mcptoolshop-rnd-newsletter`](https://pypi.org/project/mcptoolshop-rnd-newsletter/) | `rnd_newsletter.latest()`, `issue(n)` and `issues()` |\n| GitHub releases | one per issue, with the issue attached |\n\nAll four are built from the same published files as the site. A run with nothing new publishes nothing, and a\ncorrection to an issue already out becomes a new revision, not a silent change.\n"}
 {"issue": 32, "date": "2026-10-27", "title": "27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp", "description": "A spotlight issue on three tools the studio ships in the open: AI Jam Sessions, which teaches an AI to play, sing and listen; interlock, which went from 1.0.8 to 1.0.10 while it measured its first rented card; and XRPL Camp, which teaches the XRP Ledger in one sitting. Plus one short note from today's runs.", "highlights": ["<strong>AI Jam Sessions.</strong> An MCP server that teaches an AI to play piano and guitar, sing and listen: 109 annotated songs, 56 tools, and three hymns sung whole by a synthetic voice.", "<strong>interlock 1.0.8 to 1.0.10.</strong> Three releases made from what the first rented-card runs showed: a time limit on every cell, a restart when one hangs, and noise recorded instead of thrown away.", "<strong>XRPL Camp 1.5.</strong> Learn the XRP Ledger in about ten minutes on the test network, and keep a proof of what you did that anyone can check."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/", "body_markdown": "Today's Colab work produced little to report, so this issue is a spotlight: three of the studio's tools that are\npublic and shipping, what each one does, and what changed in it lately.\n\n**In this issue:**\n\n1. [AI Jam Sessions: an AI that plays, sings and listens](#ai-jam-sessions-an-ai-that-plays-sings-and-listens)\n2. [interlock 1.0.8 to 1.0.10](#interlock-108-to-1010)\n3. [XRPL Camp: the XRP Ledger in one sitting](#xrpl-camp-the-xrp-ledger-in-one-sitting)\n4. [A short note from today's runs](#a-short-note-from-todays-runs)\n\n## AI Jam Sessions: an AI that plays, sings and listens\n\n[AI Jam Sessions](https://github.com/mcp-tool-shop-org/ai-jam-sessions) is an MCP server that teaches a language model\nto play piano and guitar, and to sing. A model can read text, but it has no ears and no fingers, so the server gives\nit senses it can use:\n\n- **Reading:** 109 songs across 12 genres, every one annotated with its history, its structure bar by bar, and what\n  it teaches.\n- **Hearing:** six sound engines, from a sampled piano to a modelled guitar and a vocal tract, playing through real\n  speakers.\n- **Seeing:** a piano roll the model can read back, a guitar tablature editor, and a browser cockpit.\n- **Remembering:** a practice journal that carries over from one session to the next.\n- **Listening:** point it at a recording and it measures what is there, with the same kind of tools it uses on the\n  score.\n\nIt has 56 MCP tools in all, and installs from npm as `@mcptoolshop/ai-jam-sessions`; the current release is 2.8.0.\n\n### Three hymns, sung whole\n\nVersion 2.8.0 added whole songs sung by a synthetic voice on the piano's clock: Amazing Grace and America the\nBeautiful, every verse. The Battle Hymn of the Republic followed on the [landing\npage](https://mcp-tool-shop-org.github.io/ai-jam-sessions/#listen). The performances were rendered offline on a rented GPU,\nnot by the server's own sound engines. Each is built from sixteen takes of a singer conditioned on the score, with one take chosen for each phrase and checked for timing and pitch by two\ninstruments each, then passed by ear.\n\n| | Amazing Grace | America the Beautiful | Battle Hymn |\n|---|---|---|---|\n| syllables within 40 ms of the score | 107 of 112 | 217 of 224 | 285 of 414 |\n| notes within 50 cents | 135 of 140 | 218 of 224 | 368 of 424 |\n\nThe Battle Hymn's quick dotted notes are left on the singer's own timing, which is where most of its timing misses\ncome from. The voice is [SoulX-Singer](https://github.com/Soul-AILab/SoulX-Singer) (Apache-2.0) with its own example\ntimbre, imitating no real singer, and all three hymns are in the public domain.\n\n### Data others can use\n\nTwo datasets from the same work are public on Hugging Face:\n\n- [jam-actions-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-v0): 57 multi-turn MCP tool-use traces\n  over four classical piano works, for teaching a model to answer questions about music by calling tools rather than\n  guessing.\n- [jam-actions-acoustic-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-acoustic-v0): 72 records over\n  audio rather than notation, each label checked against what the analysis tools actually return. It ships no audio:\n  each record carries a recipe and the hash of the waveform it produces.\n\nBoth carry their licences and sources in their cards. Each was corrected on 25 September when an audit found files\nwhose arrangement licence couldn't be established, and the records built from them were withdrawn.\n\n## interlock 1.0.8 to 1.0.10\n\n[interlock](https://github.com/mcp-tool-shop-org/interlock) picks a model, an engine and a device for a job from\nspeeds it has measured. [Issue 31](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/) told how its first runs on a rented T4 card exposed\ndefects in how it measured, fixed in 1.0.5 to 1.0.7. The next three releases came from the same runs:\n\n- **1.0.8: nothing waits for ever.** Every cell has a wall-time cap, and a call that reads the card's memory now has\n  a deadline: one run had gone silent for 56 minutes waiting on it. The harness prints a line at each step, so the\n  next hang names its own call. Rig checks get their own identity instead of borrowing a Colab card's name. And\n  `interlock route --concurrency N` ranks results for the number of requests you'll actually send.\n- **1.0.9: a hung cell is cleaned up.** When a cell's deadline fires, the server is restarted and any late answer\n  from the abandoned cell is refused. The fallback reading of the card's memory is checked strictly, and an answer it\n  can't trust is recorded as unmeasured.\n- **1.0.10: noise is recorded, not used to discard.** A check meant to catch a cached prompt was also discarding real\n  cells, because the time to read the prompt varies from pass to pass. Each measured row now records that spread\n  instead, and the cells are kept.\n\ninterlock 1.0.10 is on PyPI as `mcptoolshop-interlock` and needs Python 3.13 or newer.\n\n## XRPL Camp: the XRP Ledger in one sitting\n\n[XRPL Camp](https://github.com/mcp-tool-shop-org/xrpl-camp) teaches the XRP Ledger by doing it: real transactions on\nthe ledger's test network, in about ten minutes, ending with a proof of what you did that anyone can check against\nthe ledger. It's on PyPI as `xrpl-camp` and npm as `@mcptoolshop/xrpl-camp`, both at 1.5.2.\n\n- **1.5.0** wired in the last parts of the course: a walkthrough of the reserve an account owes for the things it\n  owns, a plain-text certificate you can keep and share, and dated receipts each time a proof is verified.\n- **1.5.1 and 1.5.2** tightened the edges. A submission whose result comes back missing is no longer taken as a\n  success: the camp looks the transaction up on the ledger and reports what it finds. The reserve step says plainly\n  when a change couldn't be measured yet. And the standalone downloads no longer bundle a leftover install record from\n  their build.\n\n## A short note from today's runs\n\nOn today's T4 runs, the Colab machine's memory limit stopped some of the largest cells: when a model's working cache\nspilled out of the card into the machine's memory, the limit killed the model server at about 11.5 GB of its own\nmemory. In three runs the process running our harness was killed a moment later as well (two seen directly, one\ninferred from its logs); the rows written up to that point were kept, but the rest of that step was lost. The run\nlauncher now notices a dead run and stops it within about two minutes, instead of waiting out its full time. A check\nthat refuses such cells before they load is being built for interlock 1.0.11.\n"}
+{"issue": 33, "date": "2026-10-28", "title": "28 October: the T4 sweep is done, a ready-made critic fails its test, and Cognate learns to choose between plans", "description": "Yesterday the T4 part of the C2 sweep finished. Replayed over the earlier windows, interlock 1.0.11's new check would have refused every load that had been killed and none that had completed; on its live windows, nothing crashed. We also tested whether an off-the-shelf language-inference model could check our verifier's quotes. It couldn't: on real supporting quotes it agreed 21% and 7% of the time, against a bar of 85%. And Cognate took in a plan chooser, with three fixes on the way in. In reasoning-dataset-lab, two models agreeing certified 26 test sets, and a blind second key found only 3 of them sound. A first paid run on batching found that it rewrites most of a model's replies, though few of its verdicts. Overnight, a CPU-only test run turned out to be on the GPU, and CPU runs now hide every route to it. The issue ends with the first daily watch.", "highlights": ["<strong>The T4 sweep is done.</strong> Three repetitions on Colab's T4 card closed on 27 October. By the end of the T4 part the sweep had spent 19.572 of the 800 units set aside for it; the L4 and A100 cards started running overnight.", "<strong>A check that held on a live run.</strong> interlock 1.0.11 refuses a cell whose memory would overflow the machine, before loading it. On its first live windows, for phi3.5 it refused the same five cells in each of four quants, and nothing crashed.", "<strong>A ready-made critic, failed.</strong> An off-the-shelf language-inference model read real supporting quotes as support 21% and 7% of the time, against a pre-set bar of 85%. It won't be used.", "<strong>Cognate chooses between plans.</strong> civility-kernel joins Cognate: it drops plans that break a hard rule, scores the rest, and executes the best or asks a person. Its decisions can now be hashed and recorded.", "<strong>Two readers agreeing wasn't enough.</strong> Two models certified 26 of 50 test sets, but a blind second key found only 3 that hold a contradiction on every pair. The sets are being rewritten before any rerun.", "<strong>Batch 1 for verdicts that must repeat.</strong> Served one request at a time, a 30B model repeated all 60 replies. Batched eight at a time, only 1 of 60 matched. Decisions moved far less, on 1 to 4 of 60.", "<strong>CPU runs now hide every GPU.</strong> A CPU-only test server that hid only CUDA put a 24B model on the RTX 5090 through Vulkan. CPU runs now hide every route, ask for no GPU layers, and stop if the server takes any of the card's memory."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-33/", "body_markdown": "Two results from 27 October: a sweep that finished, with a fix that held on its first live windows, and a test that\nfailed, as its pre-registration said it might. Then a change that landed overnight in Cognate, a certification that two\nmodels agreed on and a second key didn't, a test of what batching does to a model's replies, a CPU run that turned\nout to be on the GPU, and, from this issue on, a short daily watch at the end.\n\n**In this issue:**\n\n1. [The T4 sweep is done, and the host-memory check held](#the-t4-sweep-is-done-and-the-host-memory-check-held)\n2. [A ready-made critic fails its test](#a-ready-made-critic-fails-its-test)\n3. [Cognate learns to choose between plans](#cognate-learns-to-choose-between-plans)\n4. [Two readers agreed, and it wasn't enough](#two-readers-agreed-and-it-wasnt-enough)\n5. [Batching rewrites a model's replies](#batching-rewrites-a-models-replies)\n6. [A CPU run that wasn't](#a-cpu-run-that-wasnt)\n7. [The daily watch](#the-daily-watch)\n\n## The T4 sweep is done, and the host-memory check held\n\n**C2** measures how fast and how large the studio's models run on Colab's rented cards, with interlock.\n[Issue 31](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/) told how its first trial runs on a T4 card exposed problems in our\nmeasurement, and [issue 32](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/) noted one more: some cells filled the machine's memory, and\nits limit killed the model server, three times taking our harness down with it.\n\n### The fix, checked against the past first\n\n**interlock 1.0.11**, released on 27 October, works out before each cell how much of the machine's memory the\ncell's working cache would need, from the cell before it. If that, with a 5% margin added, is over the machine's\nlimit (12,954 MiB on a T4), the cell is recorded as out of memory and never loaded.\n\nBefore it ran live, it was replayed over the seven T4 windows already on record. Of 234 loads, 12 were the first of\ntheir run and can't be projected. Of the other 222, it refused **all 34 loads that had been killed** and **none of\nthe 188 that had completed**.\n\n### Then live\n\nThe first window on 1.0.11, s2c, ran phi3.5 at Q4_K_M: 21 cells, 13 measured and 8 out of memory. Five of those\neight were refused by the new check before loading. Two were the exact cells that had killed the harness in windows\ns2a and s2a2; with the margin, the check projected them at **15,544 MiB** and **42,349 MiB**. The machine's memory log showed\n**no model-server kills and no harness kills.**\n\nThe last window, s3a2, ran all four phi3.5 quants on 1.0.11 and refused the same five cells in each, again with no\nkills.\n\n### Its limits, stated with it\n\n- The first cell of each run is never projected, because there's nothing earlier to scale from. No first cell on\n  record was killed.\n- It can refuse a cell that would have fitted. A cell projected at **12,337 to 12,954 MiB** is under the T4's\n  limit, but the margin lifts it over, so it is refused; and the projection overstates a partly offloaded load. The\n  closest completed load on record was projected at **12,021 MiB** (12,622 with the margin), 316 MiB short of\n  that band.\n\n### What the sweep cost\n\nThe T4 part of C2 ran three repetitions, finishing with window s3a2 on the evening of 27 October. Not every window\ncame back whole: one stopped at its spending cap, one was lost to the memory limit before the fix, one was stopped by the launcher after the same kill,\nand one was lost twice when our own machine restarted, then re-run in full. By the end of the T4 part, counting the\ntrial runs before it, C2 had spent **19.572 of the 800 units** set aside for it. The other cards began overnight. The\nT4 bills **1.07 units an hour**.\n\n## A ready-made critic fails its test\n\n**aspire-si** trains models to judge the way a teacher does. Its current plan trains a **verifier**: a model that\nchecks a claim about code and quotes the line that settles it. A cheap way to check those quotes would be a ready-made **natural-language\ninference** model, one trained to say whether one sentence supports another. Smoke test **S1** asked whether one\nsuch model, `cross-encoder/nli-deberta-v3-base`, can tell a real supporting quote from one that fooled a verifier.\n\nThe pass line was fixed before the run. Real supporting quotes had to be read as support at least **85%** of the\ntime, and quotes that had fooled a verifier on an unsupported claim no more than **15%** of the time (the upper end\nof a 95% interval).\n\n### The result\n\n| check type | real supporting quotes read as support | fooling quotes read as support (upper bound) | pass |\n|---|---|---|---|\n| grounded | **0.210** (of 267) | 0.073 (of 110; upper 0.137) | no |\n| reasoning | **0.071** (of 210) | 0.029 (of 68; upper 0.101) | no |\n\nThe model rarely calls a fooling quote support, so that half holds. But most of what it reads, it calls\n\"neutral\": on the NPU, 498 of the 677 pairs it scored came back neutral. It was trained on prose, and these are lines of code. The pre-registration named\nthat risk, and pairs with a prose comment in the code didn't rescue it: on grounded claims, 0.208 with a comment and\n0.153 without.\n\n(The run's own note first gave these two counts as 473 of 637 and 636 of 637. Both had been counted with each quote\ncut to 80 characters, which made the 677 pairs look like 637; the note was corrected on 28 October.)\n\nIt ran on both of the rig's Intel devices. They agreed on 676 of 677 labels (99.9%), and the integrated GPU was about three\ntimes faster than the NPU (0.089 s a pair against 0.271 s). That's moot for this model, which fails on both.\n\n### What it means\n\nThe model won't be used. That was the pre-registered consequence of a fail, and nothing else changes: it was never\ngoing to enter the training reward, and S1 says nothing about the verifier's answer keys, which are provisional. The\nnext smoke tests, S2 and S3, are separate pre-registrations and haven't run.\n\n## Cognate learns to choose between plans\n\n[Cognate](https://github.com/mcp-tool-shop-org/cognate) records how AI is governed: which model version was used,\nwhich policy applied, which agent acted, with each record provable afterwards. Its policy engine checks one event at a\ntime, such as a prompt, an output or a spend. Until now it had nothing for the step before: deciding which of several\nplans an agent should carry out.\n\nOn 28 October Cognate took in **civility-kernel**, imported from its earlier standalone package, as a new package of\nits own. Given an agent's candidate plans, it drops any that break a hard rule (an irreversible change, a spend over a\nlimit), scores the rest by the policy's weights, and then either carries out the best one or, when that plan is too\nuncertain, asks a person. If no plan survives, it asks the person to relax a rule or offer another option.\n\n### Three fixes on the way in\n\n- **A cache keyed on too little.** From the policy, the kernel's compile cache read only the version, so two\n  policies with the same version but different weights or constraints got the first one's rules. The cache is gone.\n- **Decisions that can be reproduced.** A caller can now supply the decision's id and time; then the same inputs give\n  a byte-identical record.\n- **A number JSON can't hold.** A plan that broke a rule was scored minus infinity, which JSON writes as `null`, so a\n  record read back from storage was no longer the record that was made. Such plans now score `null` from the start.\n\n### What it adds\n\nA decision can now be hashed in the form Cognate's run pre-registration uses, and recorded on its event log. The record\ncarries the outcome and which rules each rejected plan broke, and never the plans' steps; the chosen plan's digest\nstill covers them, so a step changed afterwards no longer matches.\n\nThe new package has **164 tests**: 143 carried over and 21 new. Its tests cover **91.89%** of its lines (578 of 629),\nagainst a floor of 90%. It is on Cognate's main branch and not yet in a release.\n\n## Two readers agreed, and it wasn't enough\n\n[reasoning-dataset-lab](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab) builds test sets with a known\nanswer. One kind is a set of five values for one attribute of a machine, chosen so that only one can be true of a\nunit: a transformer's vector group, a vessel's berth status, a reactor's coolant. Any two of them then make a\ncontradiction, and a test item can be built from each pair. Before a set is used, it is **certified**: two language\nmodels, each reading blind, must call every one of its 20 ordered pairs a contradiction.\n\nOn 28 October the lab ran certification on 50 sets with two readers, Mistral Small 3.2 (24B, thinking off) and\nNemotron 3.5 Lightning (30B). Both first passed a fixed test on 20 known-compatible pairs (19 and 20 of 20), so neither\nwas simply answering \"contradiction\" to everything.\n\n### The run\n\n- **26 of 50 sets certified**, and 24 went to a review queue. There were 2,000 calls and none failed.\n- 121 pairs kept those 24 sets out. On 55, one reader called a contradiction and the other called the pair compatible. Both readers called\n  27 compatible. Another 25 were Nemotron runs whose thinking hit the length limit, which count as failures, not\n  verdicts. On the other 14, a reader answered \"unsure\".\n\n### A second key\n\nAgreement between readers is only worth something if the pairs really are contradictions, so we had every pair of the\n26 certified sets keyed a second time, by a model blind to the readers' verdicts. It was allowed three answers:\ncontradiction, arguable, or compatible.\n\n| Pairs | Contradiction | Arguable | Compatible |\n|---|---|---|---|\n| 520 pairs in the 26 certified sets | 272 | 216 | 32 |\n| 27 pairs both readers called compatible | 2 | 19 | 6 |\n\nOnly **3 of the 26 certified sets** came back as contradictions on all 20 pairs. The faults are in the sets, and both\nreaders missed them the same way:\n\n- Under the collision regulations a vessel that is drifting is still \"underway\", so *underway* and *adrift* can both\n  hold.\n- A large generator is often cooled by hydrogen and by water at once.\n- A three-winding transformer carries two vector groups.\n\nOur own read of a sample, made before the second key and without sight of it, had found the first two. The second key\nagreed with every fault that read named, and found others, such as the three-winding transformer, in sets the read\nhad called clean.\n\n### What it means\n\nTwo readers agreeing is a weak filter for this task: it rules out clear failures, but it doesn't prove a pair is a\ncontradiction. That's a finding about the test sets, not a measurement of the readers. \"Arguable\" is a judgement, and\nsome of the second key's calls are strict ones (most people would read \"the transformer\" as a two-winding unit). The\nsets are being rewritten before any rerun, under four rules: one axis per attribute, no relative terms (\"higher\"), no\nlabel that contains another value, and no values that can occur together on one unit.\n\n## Batching rewrites a model's replies\n\nA model served one request at a time gives the same reply to the same request, if the seed and settings are fixed.\nServers are faster when they batch several requests together, but batching changes the arithmetic. We wanted to know\nhow much it changes the answers.\n\nOn 28 October we ran Muse Glimmer 30B on a Colab A100 (40 GB) under llama.cpp's server, and sent it the same 60\ncritic requests (52 asking for a verdict, 8 for the flaw in a check) at batch widths 1, 2, 4 and 8 (the server's slot\ncount). Widths 1 and 4\nwere run twice. The seed, the settings and the server build were fixed. (The study's code is public, in\n[role-set](https://github.com/mcp-tool-shop-org/role-set), under `scripts/batchdrift/`.)\n\n| Compared with batch 1 | Identical replies | Decision changed |\n|---|---|---|\n| Batch 1, run again | 60 of 60 | 0 |\n| Batch 2 | 30 of 60 | 3 |\n| Batch 4 | 12 of 60 | 1 |\n| Batch 4, second run | 12 of 60 | 3 |\n| Batch 8 | 1 of 60 | 1 |\n\n- **Batch 1 is deterministic** on this card and build. It matched itself on all 60, and matched an earlier reference\n  run on all 60.\n- **Batching rewrites most replies**: half of them at width 2, all but one at width 8.\n- **And a batched run doesn't repeat itself.** Batch 4 run twice gave the same reply on only 37 of 60.\n- **Decisions move much less than the text.** A reply's decision is its verdict (or, for a flaw request, the flaw\n  it names) plus the fields that say what it covered. It changed on 1 to 4 of the 60 requests in each comparison that\n  involved batching. The verdict itself changed on 0 to 4, no flaw label changed, and batch 8's one change was in a\n  coverage field. With 60 requests that is still loose: the 95% intervals\n  reach about 16%.\n\n### What we take from it\n\nAny verdict that has to be reproducible runs at batch 1. Batched runs are their own tier, and their replies can't be\ncompared one by one with batch 1's. Two limits: we recorded the server's slot count but not how many requests the\nclient kept in flight, so we can't yet say every slot was full. And this is one model on one card under one build;\nother engines and cards may behave differently.\n\n## A CPU run that wasn't\n\nOn 28 October we ran a dry run of a 24B model, the Mistral reader above, on the rig's processor only, to check a code\npath before asking for time on the RTX 5090. The second Ollama server it started hid CUDA with `CUDA_VISIBLE_DEVICES=-1`, and that was all\nit hid. Ollama 0.40.9 also ships a Vulkan backend, and through it the \"CPU\" server loaded the model onto the 5090\nanyway: its process held 15,114 MiB of the card's memory for about 29 minutes before we found it and stopped it.\nNothing broke, but the card was meant to stay free, and the run wasn't measuring what it said.\n\n### The guard\n\nCPU-only runs now do three things:\n\n- **Hide every GPU route**, not just one. The server starts with CUDA, Vulkan and HIP hidden.\n- **Ask for no GPU layers on every request.** A small wrapper adds `num_gpu: 0` to each call, so a backend we didn't\n  think of still gets no layers.\n- **Check, and stop on a rise.** Every 10 seconds a guard reads the GPU memory held by the server's own processes and\n  the card's total, and stops the job unless the server holds nothing and the total hasn't risen by more than 512 MiB.\n\nWe tested the stop path three ways, including a forced baseline and a live run that failed closed on a counter it\ncouldn't read. Then the guard passed on a 4B model and on the same 24B model: both runs reported `library=cpu`, the\n24B server started with no GPU layers (`-ngl 0`), the server held 0 MiB of the card, and the card's total went down\nslightly over each run rather than up.\n\n### A two-second name\n\nThe same night turned up a smaller cost. An embedding job that talked to Ollama at `localhost` hit its 15-minute stop\n(902.4 s) without finishing. Each call was taking about 2 seconds against hundredths of a second at `127.0.0.1`,\nmost likely because the rig tries IPv6 first and falls back. With `127.0.0.1` the whole job took 30.8 seconds.\n\n### A study whose retrieval can't miss\n\nThat job was the retrieval arm of a study we're preparing: how a model's tool choice holds up as the list of tools\ngrows. (Its code is public, in [role-set](https://github.com/mcp-tool-shop-org/role-set), under `scripts/toolcount/`.)\nWith real embeddings, the arm found the right tool in its top five for all 170 answerable items, at every list size\nfrom 10 to 100. That isn't a result: the study's paid run hasn't happened. It says something about the items, which\nname their tool almost word for word. The study will report that alongside whatever it finds.\n\n## The daily watch\n\n*New since the [weekly watch](https://mcp-tool-shop-org.github.io/rnd/watch/2026-10-26/) of 26 October, checked against each item's own source.*\n\n**NeMo Relay 0.11.0** (NVIDIA, released 2026-10-28;\n[release notes](https://github.com/NVIDIA/NeMo-Relay/releases/tag/0.11.0)). This is the agent-tracing tool we tried in\na sandbox on 0.10.1. The new version changes how its coding-agent hooks behave: tool checks now fail closed by\ndefault, and a hook that can't deliver returns exit code 2. It also adds opt-in diagnostics and continues traces\nthrough Claude Code hooks. For us, a hook that fails closed can stop an agent mid-task, so the 0.10.1 trial doesn't\ncarry over. *What we'll do:* run the sandbox trial again on 0.11.0, hooks included, before we use it in our own\nharnesses.\n\n**ComfyUI v0.43.0** (Comfy-Org, released 2026-10-27;\n[release notes](https://github.com/Comfy-Org/ComfyUI/releases/tag/v0.43.0)). Among many partner-node additions, it\nfixes the top-p sampling threshold and Gemma 4 31B's global attention, starts dynamic VRAM KV caches, and sets a\nminimum Pillow version. For us, our image pipeline pins one ComfyUI commit per run, so this is the candidate for the\nnext pin. *What we'll do:* check it on a CPU run before anything paid.\n\n**Ollama v0.40.12** (released 2026-10-27; [release notes](https://github.com/ollama/ollama/releases/tag/v0.40.12)).\nIts listed changes are on the Apple-silicon side (MLX, faster prompt processing), an updated llama.cpp, and Liquid AI's\nnew LFM3 models, which run on Apple hardware only for now. For us, the C2 sweep ran on 0.40.11. *What we'll do:*\nnothing for now; C2 stays on 0.40.11, so its windows stay comparable.\n\n**Serving World Models: Latency vs Throughput with NVIDIA Cosmos 3 Super on Nebius** (NVIDIA Developer livestream,\ntoday at 12:00 PM ET; [watch on YouTube](https://www.youtube.com/watch?v=i9FZyG9-Gx0)). Two speakers from Nebius's\nPhysical AI team will cover serving world models for video generation, which trades latency against throughput\ndifferently from serving language models. They'll show Cosmos 3 Super benchmarks through vLLM-Omni on HGX B200 and\nH200 systems, across four layouts from one eight-GPU replica to eight single-GPU replicas, with which settings to hold\nfixed and how to check the generated clips are whole and actually move. Its measure is, in the description's words,\n\"validated video-seconds per node-hour\", and the description links a public\n[benchmark repository](https://github.com/nebius/nebius-physical-ai) for reproducing it. For us, the trade-off is the\none the batching study above ran into from the other side: serving several requests at once changed most of a model's\nreplies. We don't serve video models on datacentre cards, but the method is the part we can use: what to hold\nconstant between runs, counting only output that passes a check, and when more replicas or more concurrency stop\nhelping. Cosmos 3 also sits in our plan for robotics in simulation, which hasn't started, and its licence is read\nbefore anything is run. *What we'll do:* read the benchmark repository and the talk's written version (next item),\nand compare their list of settings to hold constant and their checks on output with the protocols of our batching\nstudy. We won't run anything from it, and nothing changes in the robotics plan before the Cosmos licence has been read.\n\n**Serving world models: Request latency and node throughput tradeoffs** (Nebius, published 2026-10-26;\n[the post](https://nebius.com/blog/posts/serving-world-models-latency-node-throughput)). The written version of the\ntalk, by the same two speakers, and the post the talk links as its technical blog. It serves Cosmos 3 Super through\nvLLM-Omni on eight-GPU B200 and H200 nodes split four ways. By the authors' figures, on B200 eight one-GPU replicas\ngave 42.5% more validated video per node-hour than one eight-GPU replica, at 5.5 times the mean request latency\n(378.1 s against 68.5 s); on H200 the gain was 25.7%, at 6.3 times. Only clips that pass a validation check count,\nwhile failed attempts still use the node's time. The post lists what it held constant, from the model revision and\ncontainer to the seeds, resolution, frame count and denoising steps, and it ran one request at a time per replica.\nFor us, it is the batching study's trade-off put in node terms: the layout with the most output per node is the one\nwith the slowest single request, and its method is close to ours. *What we'll do:* compare its list of what's held\nconstant, and its validation check, with our batching protocol. We run nothing from it.\n\n**NVIDIA Cosmos Opens New Tools to Train, Evaluate and Deploy World Models** (NVIDIA, published 2026-10-21;\n[the post](https://blogs.nvidia.com/blog/cosmos-open-train-evaluate-deploy-world-models/)). It announces Cosmos Data\nFactory (tools and blueprints to label, curate, augment and evaluate training data), the Cosmos Framework (training,\noptimisation and deployment tools offered as agent skills), Cosmos 3 Nano checkpoints for testing autonomous policies\nin simulation, updated Cosmos 3 NIM microservices that NVIDIA says give \"up to 2x faster inference with no quality loss in\nvalidated configurations\", and version 3.3 of its Metropolis blueprint for video search and summarization (VSS). For us, the Nano simulation checkpoints are the part closest to our plan for\nrobotics in simulation, and the post states no licence terms for any of it. *What we'll do:* read the licences on the\ncheckpoints' own model cards and repositories, as part of the planned licence read, before anything is run.\n"}
+{"issue": 34, "date": "2026-10-29", "title": "29 October: repaired test sets certify 44 of 50, long tool lists cost a model its \"no\", and Cosmos 3 on one rented card", "description": "Rewritten so that every pair is a clear contradiction, and keyed blind before any reader saw them, the reasoning-dataset-lab sets certified 44 of 50, against 26 before; the thinking setting we suspected turned out not to be what cut the length stops. A test of tool lists from 5 to 100 found that two local models kept picking the right tool, but one of them declined less often when the prompt grew long, and retrieving five tools kept its \"no\" intact. NVIDIA's Cosmos 3 made its first clips on rented cards: one Nano clip on one card, a 30-second rollout from the two-armed robot checkpoint, and five shrunk versions of the bigger Super model, of which Intel's int4 export is our working choice for now. Then the raw files behind our figures get a public home, a read of GitHub's Universe announcements, notes in brief, and the daily watch.", "highlights": ["<strong>44 of 50 sets certified.</strong> Every pair rewritten to be a clear contradiction and keyed blind first, the sets certified 44 of 50, against 26 before. The cleaner sets, not the thinking setting, cut the length stops.", "<strong>Long tool lists held; the \"no\" slipped.</strong> With 5 to 100 tools, Granite 4.2 8B and Qwen3.8-27B kept picking the right tool. Granite declined 19 of 30 no-tool requests at 40 tools and 18 on a padded 10-tool prompt; retrieving five tools kept it at 27 to 30.", "<strong>A world model on one rented card.</strong> Cosmos 3 Nano made an 8-second clip on one RTX PRO 6000 card in a 628-second window, about 1.55 compute units, with generation within 2% of our estimate.", "<strong>Super on one card, for now in int4.</strong> Five shrunk versions of Cosmos 3 Super ran on one G4. Intel's int4 export wasn't separable from FP8 on fidelity, and ran faster in less memory, so it's our working choice for now.", "<strong>rnd-data is public.</strong> The raw files behind our figures, 111 runs so far, now have a repository of their own.", "<strong>GitHub Universe 2026.</strong> We read the agent-centred announcements on GitHub's own pages, and checked an AI search summary of them against those pages."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-34/", "body_markdown": "Results from 28 October: a repaired test set and its rerun, a test of long tool lists, and NVIDIA's Cosmos 3 on\nrented cards, from one clip to five shrunk versions of its biggest model. Then the raw files behind our figures get a\npublic home, a read of GitHub's Universe announcements, notes in brief, and the daily watch.\n\n**In this issue:**\n\n1. [The repaired sets: 44 of 50 certified](#the-repaired-sets-44-of-50-certified-and-the-thinking-setting-wasnt-the-fix)\n2. [How many tools can a local model take?](#how-many-tools-can-a-local-model-take-the-list-held-the-no-slipped)\n3. [A world model on one rented card](#a-world-model-on-one-rented-card)\n4. [Cosmos 3, continued: the robot-arm checkpoint, and Super on one card](#cosmos-3-continued-the-robot-arm-checkpoint-and-super-on-one-card)\n5. [rnd-data: the raw files behind the research](#rnd-data-the-raw-files-behind-the-research)\n6. [GitHub Universe 2026: the platform turns toward agents](#github-universe-2026-the-platform-turns-toward-agents)\n7. [In brief](#in-brief)\n8. [The daily watch](#the-daily-watch)\n\n## The repaired sets: 44 of 50 certified, and the thinking setting wasn't the fix\n\nIssue 33 showed that two readers agreeing on a set of values is a weak filter: most of the sets they \"certified\" held\npairs that were arguable, not contradictions. This is the repair and the rerun.\n\n**The repair.** All 50 five-value sets in reasoning-dataset-lab were rewritten so that every ordered pair is a\ncontradiction no one could argue with. New tests catch the faults we found before: an attribute must name one property,\nno relative terms, no value whose label contains another, and no two values a single machine can hold at once. Then an\noutside model keyed all 1,000 pairs blind, before any reader saw them. It took three rounds: 120 of the 1,000 pairs came\nback \"arguable\" in the first, 16 of the 560 re-keyed in the second, and none of the 60 in the third. On the old certified\nsets, only 3 of 26 had been clean on all 20 pairs.\n\n**What \"certified\" means now:** a set passed the blind key first, and then both readers called every one of its 20 pairs\na contradiction.\n\n**The rerun.** 44 of 50 sets certified, against 26 before. 2,000 reader calls, none failed. Eleven pairs kept the other\nsix sets out, and we read each one:\n\n- 6 are faults in the pairs that the blind key passed. Four are \"accelerating\" pairs in two operating-mode sets (a\n  variable-speed compressor can speed up while loaded; a turbine \"accelerating\" can read as a load ramp while online). Two\n  are in a metering-device set, where an electronic valve can run a constant-pressure mode. The readers were right to\n  refuse them, so those three sets belong in the queue.\n- 5 are reader mistakes: 3 by one reader, 2 by the other.\n\nA read of 10 of the 44 certified sets found 9 clean and 1 with a mild wording note. The same read of the old sets found 5\nof 10 clean.\n\n![Three blind-key rounds, then the rerun. Round 1 found 120 of 1,000 pairs arguable across all 50 sets, round 2 16 of 560 pairs across the 28 sets re-keyed, and round 3 none of 60 across the last 3. In 2,000 reader calls with none failed, certified sets rose from 26 of 50 to 44 of 50. Six sets were queued on 11 refused pairs, 6 pair faults and 5 reader mistakes. Length stops for one reader: 25 at full and 22 at medium on the old sets, 2 at medium on the repaired sets.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-certification-repair.svg)\n\n**The thinking setting.** One reader (Nemotron 3.5 Lightning) now runs at \"medium\", an effort hint its template\nunderstands, not a cap on thinking. In the rerun it hit the token limit only twice in 1,000 pairs, against 25 before. That\nlooked like the setting at work, so we ran it at medium on the old pairs too: 22 limit stops against 25 at full thinking,\nand mostly on different pairs. The drop came from the cleaner sets, not the setting. Medium did give the same verdict as\nfull thinking on 95.5% of the pairs that got a verdict at both, in about 8% less time, so it stays; it just isn't the fix\nfor length stops.\n\n**Limits.** Our read of the 11 pairs and of the sample is one reader's judgement, and the blind keys come from one outside\nmodel. Where the 35 flipped verdicts between settings sit hasn't been read yet.\n\n**Next:** reword or replace the three faulty sets, and add a generator rule for modes a variable-speed machine can hold\ntogether.\n\n**The data is public.** Everything above is in reasoning-dataset-lab under\n[`certification/v3/`](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/tree/main/certification/v3): the\nmanifest of the 44 certified sets, the queue of 6, both reader controls, and the findings. `medium-arm/` holds the\nmedium run on the old pairs, filed as an addendum to the earlier run so the two settings are never pooled, and\n`second-keying/` holds all three blind-key rounds (1,000, 560 and 60 pairs). Before it merged, a separate model\nrecomputed every figure from the committed files. Every number comes from those JSON files except run times and peak\nmemory, which come from the run logs, so you can redo the arithmetic yourself.\n\nCode: reasoning-dataset-lab PRs 21 and 23; data: PR 24.\n\n## How many tools can a local model take? The list held; the \"no\" slipped\n\nAgent setups hand a model a list of tools and expect two things: it picks the right one when one fits, and it declines\nwhen none does. A common worry is that a long list breaks the first. We tested both on two local models, Granite 4.2 8B\nand Qwen3.8-27B, with lists of 5 to 100 tools.\n\n**The setup.** Each list size got 200 calls: 170 where a tool fits and 30 where none does. The same 30 no-tool requests\nran in every cell, so cells compare request by request. Two more arms separated the causes:\n\n- **A length control:** 10 tools, padded with neutral text to the length of the 100-tool prompt.\n- **A retriever:** it picks the 5 most relevant tools from the full list, so the prompt stays about 1,040 tokens.\n\nSettings: temperature 0, each model's own tool template, one A100 40 GB card.\n\n**Picking the right tool held.** Granite chose correctly on 164 to 170 of the 170 calls in every cell, its lowest at\n100 tools. Qwen managed 169 or 170 throughout. Up to 100 tools there was no cliff for either.\n\n**Declining is where Granite slipped,** and it followed the prompt's length, not the number of tools:\n\n| Tools in the prompt | Prompt length (tokens) | Granite declined | Qwen declined |\n|---|---|---|---|\n| 5 | 1,004 | 28 of 30 | 30 of 30 |\n| 10 | 1,737 | 29 of 30 | 30 of 30 |\n| 20 | 3,184 | 26 of 30 | 30 of 30 |\n| 40 | 6,108 | 19 of 30 | 29 of 30 |\n| 70 | 10,457 | 22 of 30 | 27 of 30 |\n| 100 | 14,804 | 27 of 30 | 28 of 30 |\n| 10, padded to 100-tool length | 13,270 | 18 of 30 | 29 of 30 |\n| retriever's top 5 | about 1,040 | 27–30 of 30 | 29–30 of 30 |\n\nThe prompt lengths are Granite's. Qwen3.8-27B ran as two halves of the design, split by seed, and its counts pool both.\n\n![Tool lists from 5 to 100. Picking the right tool held flat: Granite 4.2 8B 164 to 170 of 170, Qwen3.8-27B 169 to 170. Declining on 30 no-tool requests, plotted against Granite's prompt length with 95% intervals and no trend line: Granite 28 of 30 at 5 tools, 29 at 10, 26 at 20, 19 at 40, 22 at 70, 27 at 100, 18 on the 10-tool prompt padded to 100-tool length, and 27 to 30 with the retriever's top 5. Qwen, pooled from two halves, 27 to 30 of 30 everywhere. Only the 40-tool and padded drops are significant after correction (p 0.02 and 0.011); the 70-tool drop is not (p 0.35). Interim: Gemma 4 31B is still running.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-tool-count-abstention.svg)\n\n- **Two drops are real after correcting for the number of comparisons:** 40 tools, and the padded 10-tool prompt. The\n  padded prompt carries only 10 tools, yet does as badly as any list, so length alone can do it.\n- **It isn't a steady slide.** 100 tools came back to 27 of 30, and the 70-tool drop doesn't survive the correction.\n  \"The longer the list, the less it declines\" would overstate it.\n- **With the retriever,** Granite declined 27 to 30 of 30 at every list size.\n- **Qwen stayed flat,** with no change near significance.\n\n**Limits.** 30 no-tool requests per cell is small: one request is 3.3 points, and most intervals overlap. This is\ntwo model families, with one template each. The padding was neutral text, so it shows that length alone can do this,\nnot which part of a real tool list does. The retriever is one retriever. A third family, Gemma 4 31B, is running now,\nso these results are interim.\n\n**What we take from it:** here, retrieving a few tools kept the prompt short and protected Granite's \"no\". Whatever the\nsetup, test declining on its own. Accuracy alone didn't show this.\n\n## A world model on one rented card\n\nRobotics in simulation is our next research track, and NVIDIA's Cosmos 3 is where it starts: open world models that\npredict what a camera would see next, including checkpoints tuned to show how a scene changes after a robot acts. The\nfirst rung asked only one thing: does it run on the cloud GPUs we rent, and what does a clip actually cost?\n\n**Which model.** NVIDIA ships Cosmos 3 Super (64B) only in BF16, about 128 GB of weights, so it fits on no single Colab card.\nCosmos 3 Nano (16B, about 35 GB, under the OpenMDW-1.1 licence) fits.\n\n**The run.** One clip of the model card's own example prompt (1280×720, 189 frames, 35 steps, one seed) on one RTX PRO\n6000 Blackwell card:\n\n| Stage | Measured |\n|---|---|\n| Download, 35 GB | 97 s (362 MB/s) |\n| Setup (environment and pinned wheels) | 53 s |\n| Generate 35 steps and decode | 430 s (about 11.6 s a step) |\n| Whole window | 628 s, about 1.55 compute units |\n\nPeak GPU memory was about 42 GiB allocated and 51 GiB reserved, with roughly 45 GB of the card to spare; host memory\nbarely moved (2.9 GB). The output is an 8-second, 24 fps clip of a robot arm scrubbing a plate.\n\n**Estimate against measurement.** The design scaled NVIDIA's published figure (93 s a clip on a B300) to this card and\npredicted about 420 s of generation; it came in at 430, within 2%. Setup and download were much faster than guessed, so\nthe window cost under half the design's projection of about 3.6 units.\n\n**How it was run.** A free CPU stub ran every install step first, for about 0.005 units. A test suite against a fake\nColab command line checked that the launcher refuses a wrong card, too little memory or disk, or a changed script before\nanything downloads, and it caught a launcher defect before any spend. An outside model reviewed the harness and the\nlicence.\n\n**Limits.** One clip and one seed: this says the model runs and what it costs, not how good its output is. The next rung,\na simulation checkpoint on its own example, gets its own design.\n\n## Cosmos 3, continued: the robot-arm checkpoint, and Super on one card\n\nThe Nano clip above was the first rung. Three more runs followed the same day: a checkpoint built for a two-armed robot,\nfive shrunk versions of the bigger Super model on one Colab card, and Super at full precision on one rented\nB200 to serve as their reference.\n\n### The two-armed robot checkpoint\n\nCosmos3-Nano-Sim-Bimanual predicts what a camera sees as a simulated two-armed robot carries out a list of actions. We\nran one 30-second rollout (901 frames at 832×480, one seed) through its own video API on a G4 card (RTX PRO 6000\nBlackwell, 96 GB).\n\n- **Inference took 29.1 s,** which is 31.0 frames a second. The model card gives 20.6 for this card at 480p.\n- **Peak GPU memory** was 51,980 MB, and the whole window took 5 min 27 s and 0.81 compute units.\n- **Two earlier tries failed on the install, not the model.** First a dependency needed a Rust compiler, which a free\n  CPU stub caught. Then on the paid run, the serving stack's just-in-time CUDA compile found no compiler. A toolchain\n  step now fixes that, and every stub proves it.\n\n![The two-armed robot checkpoint's 327-second window on one G4: 110.8 s of setup, 91.2 s downloading 31.8 GB at 348.9 MB/s, 35.0 s starting the server, 29.1 s of inference for a 901-frame rollout at 31.0 frames a second against the model card's 20.6, and 60.9 s for opening, the VM check, stopping and fetching. Closed-loop control is planned, not yet measured.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-bimanual-window.svg)\n\nThis run plays back a fixed action file, with no controller in the loop, so it says the checkpoint runs and how fast.\nIt says nothing about whether the robot does its task well. Faster than the card is one run on one card type, not a\nbenchmark. Closed-loop control is the next rung, and it isn't designed yet.\n\n### Super, shrunk to fit one card\n\nSuper won't fit a single Colab card in BF16. Uploads on Hugging Face shrink it, so we asked which of them\nrun on one G4, how fast, and in how much memory. Each made the same three 61-frame clips at 480p (35 steps, the same\nprompts and seed), plus a 50-step still:\n\n| Version | Time a step | 61-frame clip | Peak memory allocated | Window cost |\n|---|---|---|---|---|\n| FP8 (made with NVIDIA ModelOpt) | 3.83 s | 137–139 s | 69,954 MiB | 3.09 units |\n| NVFP4 (ModelOpt) | 7.14 s | 253–254 s | 47,396 MiB | 3.07 units |\n| NF4 (bitsandbytes) | 3.37–3.38 s | 122 s | 39,750 MiB | 2.52 units |\n| int4 (AutoRound, a second export) | 3.33–3.35 s | 120–125 s | 41,066 MiB | 1.59 units |\n\n- **All four ran every clip they were given,** and none ran out of memory. The four windows cost 10.27 units together.\n- **At 720p** (FP8 and NF4 only), a clip took 339 s and 326 s, at a peak of 76,054 and 45,851 MiB.\n- **Cutting the steps from 35 to 20** cut the FP8 clip from 138.6 s to 80.0 s, 42% less, with no change in memory.\n- **A fifth version,** Intel's own int4 AutoRound export, needed changes to its config files to load and joined later.\n  It took 3.36 s a step, at a peak of 39,581 MiB at 480p and 45,680 MiB at 720p.\n\nThe table shows what runs and what it costs. How close each version looks to full precision is a separate measure.\nWe scored every clip against the B200 reference below in two ways: LPIPS, a learned measure of how different two\nframes look (0 means identical), and the cosine similarity of DINOv2 image features (1 means identical). A first round\nused one seed, three prompts and a still. A wider round then used 8 prompts and 3 seeds; these are the 21 clips that all\nfive versions share:\n\n| Version | LPIPS (lower is closer) | DINOv2 (higher is closer) |\n|---|---|---|\n| int4 (AutoRound, Intel's export) | 0.320 | 0.765 |\n| FP8 | 0.341 | 0.770 |\n| NVFP4 | 0.387 | 0.724 |\n| NF4 (bitsandbytes) | 0.401 | 0.716 |\n| int4 (AutoRound, a second export) | 0.406 | 0.723 |\n| Nano at full precision (a different model) | 0.578 | 0.558 |\n\nPaired clip by clip against FP8:\n\n- **Intel's int4 and FP8 can't be told apart.** FP8 was closer on 9 of 21 clips (sign test p = 0.66). FP8's mean LPIPS\n  was 0.021 higher, with a 95% bootstrap interval of −0.014 to +0.058, which takes in zero.\n- **FP8 is closer than the other three.** It was closer than NVFP4 on 16 of 21 clips (p = 0.027), and than NF4 and the\n  second int4 export on 15 of 21 each (p = 0.078). Its mean was 0.045 to 0.065 lower, and all three intervals stay below\n  zero.\n- **Nano is a baseline, not a shrunk Super.** Its distance shows how different the smaller model's clips are, not what\n  shrinking costs.\n\nThe two are not separable on fidelity, and on everything else measured Intel's int4 is ahead. It took 3.36 s a step against\nFP8's 3.83 in the wider round, and peaked at 39,581 MiB at 480p against 69,954. It ran 720p in 45,680 MiB against\n76,054, and had the narrowest spread across the three seeds of any Super version (0.311 to 0.328). So it's our working\nchoice, for now. It goes back to FP8 if any of three checks still to run prefers FP8 by more than that check's own\nrun-to-run spread: scores that need no reference, a vision-language model judging clips in pairs, and a blind\npreference test by people.\n\nThe scores have limits:\n\n- **Similarity, not preference.** They measure closeness to full precision, not which clip a person would pick.\n- **One card type,** and 8 prompts at 480p, 61 frames and 35 steps.\n- **Repeats.** Repeats of FP8, NF4 and the second int4 export matched the first run byte for byte, so on one card type the\n  differences come from the prompt and the seed, not from noise between runs. NVFP4 and Intel's int4 weren't repeated.\n- **The card counts too.** Part of each gap comes from the card, not from the shrinking. On the first round's clips,\n  FP8 on the B200 scored 0.345 against full precision on that same card, and FP8 on the G4 scored 0.193 against FP8 on\n  the B200.\n\nWhy FP8 is no faster than the 4-bit versions here hasn't been looked into.\n\n![Five shrunk versions of Cosmos 3 Super against the full-precision model over 21 shared clips (8 prompts and 3 seeds), plus the smaller 16B Nano model at full precision as a separate baseline. Mean LPIPS distance to full precision (0 is identical, lower is closer), with the range of the three seed means, time a step and peak memory at 480p: Intel's int4 AutoRound 0.320 (seeds 0.311–0.328; 3.36 s; 39,581 MiB), FP8 0.341 (0.311–0.363; 3.83 s; 69,954 MiB), NVFP4 0.387 (0.337–0.426; 7.12 s; 47,396 MiB), NF4 (bitsandbytes) 0.401 (0.372–0.441; 3.37 s; 39,753 MiB), a second int4 AutoRound export 0.406 (0.383–0.438; 3.39 s; 41,066 MiB), and Nano 0.578 (0.575–0.581; 0.81 s; 36,002 MiB). Paired against FP8, Intel's int4 is not separable on fidelity (FP8's mean 0.021 higher, 95% interval −0.014 to +0.058, FP8 closer on 9 of 21 clips, p = 0.66); the other three trail FP8, with intervals entirely below zero. Intel's int4 is the provisional working choice, and goes back to FP8 if reference-free scores, a vision-language judge or a blind preference test by people prefer FP8 by more than their own run-to-run spread.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-super-shrunk-comparison.svg)\n\n### The reference: full precision on one B200\n\nThe scoring needs Super's own clips at full precision, with the same prompts and seed. No Colab card holds it, so this\none run went to a rented B200 (about 180 GB) on RunPod.\n\n- **The 132.7 GB download** took 42.6 s (3,116 MB/s), and the model loaded in 29.1 s.\n- **A 61-frame 480p clip** took 37.9–44.0 s (1.03 s a step), the still 10.5 s, and the 720p clip 98.3 s.\n- **Peak memory allocated** was 129,206–129,207 MiB at 480p and 135,320 MiB at 720p.\n- **The run** took about 7.5 minutes and was booked at $1.09.\n\nThe B200's times aren't a speed comparison with the shrunk versions: it is a different card at a different precision.\n\n## rnd-data: the raw files behind the research\n\nEach issue quotes figures from runs: a window's log, a reader's verdicts, a cost ledger. Those files now have a public\nhome of their own, [rnd-data](https://github.com/mcp-tool-shop-org/rnd-data), so a reader can check a figure against\nthe run that produced it, not only against our summary of it.\n\n**What's in it.** 111 runs so far, from two ingests on October 19 and 20: 83 Colab runs (each retry of a window counts as its own run), the weekly watch's runs and\nreports, a handful of RunPod and tool receipts, and 11 xrpl-lab runs on the XRP Ledger's test network. Each run has a\nmanifest listing its files, and every public file in it carries its SHA-256, its size and its licence. There are 1,274\npublic files:\n- **1,096 small files** (256 KB or less, about 5.7 MB in all) are in git.\n- **178 larger ones** are stored once each, by hash, so they come to only 13 distinct files, about 430 MB. Most are one\n  llama.cpp server build that many Colab windows uploaded. These sit in a blob store on the studio's backup drive for\n  now, and are due to go public as a Hugging Face dataset in a later phase.\n\nThe data is CC BY 4.0; scripts are MIT, and a third-party file such as the llama.cpp build keeps its own licence\n(MIT), which its manifest names.\n\n**What stays private, and how that shows.** Open is the default, and a file is private only for a named reason. Of the\nfiles held back, 566 are private for their content (381 hold test-item text, 185 are plans not yet approved). They are\nleft out of the manifests and only counted by reason. Another 192 are private only because the identity scan flagged\nthem; these are listed by path and size, with no hash. No private file's hash is published, because the hash of a\nshort file can be matched by guessing its contents. The private copies are held in a separate private repository.\n\n**How to check a figure.** Download a file and compare its SHA-256 with its manifest; `sha256sum` is enough, and the\nREADME gives a one-line command that checks every file in git at once. CI does the same on every pull request: it\nre-hashes every file, and fails on a file no manifest names, a private entry with a hash, or a machine path in a\nmanifest. An experiment in the rnd library points to its runs by id: the stage-1 regeneration experiment names 74 of\nthem.\n\n**What's not done yet.** The runs in this issue (the certification rerun, the Cosmos clip, the current sweep) haven't\nbeen ingested yet, so for now their receipts are only in the experiment write-ups. The small read-only Python package\nthat lists and shows runs (`rnd-data`, version 0.1.0) has its release workflow in place, but it isn't on PyPI yet.\n\nRead more: [the landing page and handbook](https://mcp-tool-shop-org.github.io/rnd-data/).\n\n## GitHub Universe 2026: the platform turns toward agents\n\nGitHub's Universe announcements on 2026-10-28 have one theme: the unit of work is no longer a line of code a person\nwrites, but a task an agent carries from issue to merge. We read the announcements on GitHub's own pages, and we also\nchecked an AI search summary of them against those pages. The summary was mostly right, and the places it went wrong\nare worth a look.\n\n**What GitHub shipped**\n- **The Copilot app,** a separate app built around agent work. Inside it, **Copilot Projects** (preview) splits a goal\n  into parallel agent sessions across one or more repositories. **Computer use** (preview, macOS and Windows) lets\n  Copilot operate desktop apps. It asks before it takes control, and organizations can switch it off.\n- **Copilot Fusion** decides per task whether to use one model or combine several, at a Standard, High or Max effort\n  setting. GitHub's figure: on the Artificial Analysis Coding Agent Index, Fusion Max scored within 2 points of Opus 5.5\n  Max, at half the cost and 40% less time per task.\n- **Security.** AI Scan checks a whole repository on demand and hands what it finds to Copilot to fix. It also checks\n  pull requests in languages CodeQL doesn't cover. AI push protection blocks unstructured secrets, such as plain\n  passwords, at push time. A package firewall stops malicious or risky npm packages before they install, and\n  reachability analysis shows which vulnerable functions a JavaScript codebase actually calls.\n- **The platform.** Stacked pull requests split a large change into linked PRs that are reviewed apart and merged as a\n  stack. Copilot Automations run tasks on demand, on a schedule or from a trigger. Copilot Connectors give one sign-in\n  to MCP servers such as Vercel, Cloudflare and Sentry.\n- **Capacity.** GitHub reports adding 4 million CPU cores, 160 PB of fast storage and 50 Tb/s of network capacity this\n  year. Its Git architecture now separates storage from compute, and its move to a multi-region Azure deployment is due\n  to finish early next year.\n\n**The Octoverse numbers** (October 2025 to September 2026, public and private repositories)\n\n| Measure | Figure |\n|---|---|\n| Commits | 34.3 billion |\n| Pushes | 17.9 billion, more than 2024 and 2025 combined |\n| Developers | over 225 million (United States 38.3 million, India 32.0 million) |\n| Pull requests begun as an agent's first draft | over 40% in September, against 17% in March |\n\nThe agent share counts only agents GitHub can detect, so it is a floor. TypeScript is still the most used language by\ncontributors. Rust grew fastest of the top languages; the report's text says it doubled, while its chart puts the\ngrowth at 162%.\n\n**Where the summary drifted.** The AI summary had most of this right, but:\n- It put Fusion at \"40% faster\" and \"up to 77%\" cheaper. GitHub's own figure is half the cost and 40% less time, and\n  only for the Max setting on one benchmark against one model. None of GitHub's pages gives a 77% figure.\n- It dropped two qualifiers: that Projects and computer use are previews, and that the agent share is a floor.\n- It gave Rust's growth as a plain doubling, without the chart's 162%.\n\nNone of these is a big error, but each one makes the claim stronger than GitHub's own, which is why we check a\nsummary against the page it summarises.\n\n**For the studio.** We publish npm packages, so the package firewall and reachability analysis apply to our work. AI\npush protection would add a check alongside the identity scan we run before each push. Fusion's figures are GitHub's\nown benchmark; we haven't measured them. Relevance: watch.\n\nSources: [the Universe recap](https://github.com/events/universe/recap),\n[GitHub's announcement](https://github.blog/news-insights/company-news/github-universe-2026-evolving-every-layer-of-the-stack/),\n[the Copilot app changelog](https://github.blog/changelog/2026-10-28-github-copilot-app-universe-releases/),\n[Octoverse 2026](https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/).\n\n## In brief\n\n**CUDA Rust, live.** NVIDIA Developer streamed \"CUDA Live: Introduction to CUDA Rust\" on 2026-10-28 at 18:00 UTC\n([recording](https://www.youtube.com/live/6DKmniAOhok)): the team writes, builds and runs Rust GPU kernels and explains\nwhen to use each of its two tracks. The code is [NVIDIA/cuda-rust](https://github.com/NVIDIA/cuda-rust) (Apache-2.0),\nwhich its README calls an early-stage research project, with API breakage to be expected.\n- **cuda-oxide** is the SIMT track: a custom rustc codegen backend that compiles Rust kernels ahead of time to PTX, with\n  threads, blocks, warps and shared memory in your hands. It pins its own Rust toolchain, needs CUDA Toolkit 13.0+ and a\n  13.x driver, and is tested on Linux (Ubuntu 24.04).\n- **cutile-rs** is the Tile track: you work on tensor tiles, with no thread indexing or synchronization to write, and it\n  compiles at first launch to the CUDA Tile IR. It runs on stable Rust 1.89+ and lists Ampere, Ada, Hopper and Blackwell\n  GPUs (consumer Blackwell, sm_120, included). It is on crates.io as `cutile` 0.4.0. (The crates.io name `cuda-oxide` belongs to an unrelated\n  2021 crate.)\n- Both tracks carry Rust's borrow rules across the kernel launch, and kernels from the two can be used together.\n\nFor the studio: the rig's RTX 5090 is sm_120, so cutile-rs lists our card, but both tracks are tested on Linux only and\nthe rig runs Windows; trying it would mean WSL or a Linux box. Relevance: watch.\n\n**JAX on a Colab TPU: a fix to flax.nnx's import, then a first training smoke.** The single-chip TPU runtime we opened on 2026-10-29 (a v5e-1) comes with jax, jaxlib and flax all at 0.11.2, and `flax.nnx` fails to import against that jax:\n\n- flax 0.11.2 asks for `jax.array_ref` and `jax.ArrayRef`, which jax 0.11.2 now calls `new_ref` and `Ref`. Its fallback names aren't in jax 0.11.2 either.\n- It also calls `jax.core.get_opaque_trace_state`, which jax removed from `jax.core` in 0.11.0 (it now lives in `jax.extend.core`).\n\nWe worked around those two breaks with three one-line aliases, set only inside our own training process. An outside review read both libraries' sources and judged them the same objects under new names. That holds for these two versions only, and images change, so check yours. A newer flax pinned in your own environment, matched to the image's jax, is the cleaner fix.\n\nWith the aliases in place, a first training smoke passed on one v5e chip. It was a 10-million-parameter `flax.nnx` transformer, run for 50 steps after a CPU stub of the same path:\n\n- The first step took 4.71 s with compilation, and a steady step took 1.69 ms. A model this small is limited by dispatch, so that is not a chip-speed figure.\n- Peak TPU memory was 345 MB of 15.75 GiB.\n- JAX's persistent compile cache cut a second process's compile to 0.65 s, 7.2 times faster.\n- An Orbax checkpoint restored bit for bit, all 208 arrays, with the same next-step loss.\n- The session ran under two minutes, at 2.92 units an hour.\n\nIt trained on one fixed random batch, so the loss falling to 0.01 is memorisation: it shows the loop works and nothing more. Next is a 181-million-parameter nanoGPT on the same card, where throughput starts to mean something. Sources: [flax 0.11.2](https://github.com/google/flax/blob/v0.11.2/flax/nnx/variablelib.py) and [jax 0.11.2](https://github.com/jax-ml/jax/blob/jax-v0.11.2/jax/core.py).\n\n**RunPod's REST v1 is closing.** RunPod's public APIs have enforced a rate limit since 2026-10-27, and REST v1 is\nbeing wound down. Its limit is 180 requests a minute, 3,600 an hour and 43,200 a day now. Those halve on November 2\n(90, 1,800, 21,600) and halve again on November 9 (45, 900, 10,800). From November 16 every v1 request returns\n410 Gone. Clients should back off exponentially on a 429. REST v2 is the long-term replacement, and GraphQL for\nintegrators is due to be deprecated in early 2027. (Source: RunPod's service notice to account holders.)\n\nFor the studio: offrig already calls REST v2 for pods, volumes and the GPU catalog, and waits for `Retry-After` on a\n429 when it creates a pod. It still reads the account balance and the data-centre list over GraphQL, which has to move\nbefore early 2027.\n\n## The daily watch\n\n4 new since [issue 33](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-33/).\n\n- **PyTorch 2.15.0** (2026-10-28): the deep-learning framework's new release makes CUDA 13.2 the default wheel (13.4 alongside), adds `torch.compile` on Python 3.15, and says 4-bit NVFP4 decoding is faster on Blackwell GPUs. Breaking: `import torch` now runs once per process, and its distributed checkpoints won't load in 2.14 ([release notes](https://github.com/pytorch/pytorch/releases/tag/v2.15.0)).\n- **llama.cpp v0.8.0** (2026-10-28): a tagged stable release of the local inference engine; its numbered builds are pre-releases. It adds Liquid AI's LFM3 models (including speech in and out), `/v1/realtime` and `/tts` server endpoints, a selectable reasoning effort, and Nemotron 3.5 Embed support. Scripts beware: the CLI now requires `--model` or `-hf` ([release notes](https://github.com/ggml-org/llama.cpp/releases/tag/v0.8.0)).\n- **Ollama v0.40.13** (2026-10-28): Liquid AI's LFM3 models, including the text-to-speech `lfm3-audio` and `lfm3-omni`, now run on Linux and Windows, where 0.40.12 had them on Apple silicon only. That makes a small local speech model a one-line pull on most machines; check the model's licence before you build on it ([release notes](https://github.com/ollama/ollama/releases/tag/v0.40.13)).\n- **NVIDIA skills catalogue** (2026-10-28): NVIDIA's public catalogue of agent skills added 53 skills for its DOCA 3.5 networking bundle, six for the PVA SDK (vision accelerators), Omniverse Content Services 0.1.0, and Run:ai quota-audit, cluster-health and utilization-overview skills ([commit history](https://github.com/NVIDIA/skills/commits/main)).\n"}

package/index.json

@@ -419,5 +419,38 @@
     ],
     "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/",
     "file": "issues/issue-32.md"
+  },
+  {
+    "issue": 33,
+    "date": "2026-10-28",
+    "title": "28 October: the T4 sweep is done, a ready-made critic fails its test, and Cognate learns to choose between plans",
+    "description": "Yesterday the T4 part of the C2 sweep finished. Replayed over the earlier windows, interlock 1.0.11's new check would have refused every load that had been killed and none that had completed; on its live windows, nothing crashed. We also tested whether an off-the-shelf language-inference model could check our verifier's quotes. It couldn't: on real supporting quotes it agreed 21% and 7% of the time, against a bar of 85%. And Cognate took in a plan chooser, with three fixes on the way in. In reasoning-dataset-lab, two models agreeing certified 26 test sets, and a blind second key found only 3 of them sound. A first paid run on batching found that it rewrites most of a model's replies, though few of its verdicts. Overnight, a CPU-only test run turned out to be on the GPU, and CPU runs now hide every route to it. The issue ends with the first daily watch.",
+    "highlights": [
+      "<strong>The T4 sweep is done.</strong> Three repetitions on Colab's T4 card closed on 27 October. By the end of the T4 part the sweep had spent 19.572 of the 800 units set aside for it; the L4 and A100 cards started running overnight.",
+      "<strong>A check that held on a live run.</strong> interlock 1.0.11 refuses a cell whose memory would overflow the machine, before loading it. On its first live windows, for phi3.5 it refused the same five cells in each of four quants, and nothing crashed.",
+      "<strong>A ready-made critic, failed.</strong> An off-the-shelf language-inference model read real supporting quotes as support 21% and 7% of the time, against a pre-set bar of 85%. It won't be used.",
+      "<strong>Cognate chooses between plans.</strong> civility-kernel joins Cognate: it drops plans that break a hard rule, scores the rest, and executes the best or asks a person. Its decisions can now be hashed and recorded.",
+      "<strong>Two readers agreeing wasn't enough.</strong> Two models certified 26 of 50 test sets, but a blind second key found only 3 that hold a contradiction on every pair. The sets are being rewritten before any rerun.",
+      "<strong>Batch 1 for verdicts that must repeat.</strong> Served one request at a time, a 30B model repeated all 60 replies. Batched eight at a time, only 1 of 60 matched. Decisions moved far less, on 1 to 4 of 60.",
+      "<strong>CPU runs now hide every GPU.</strong> A CPU-only test server that hid only CUDA put a 24B model on the RTX 5090 through Vulkan. CPU runs now hide every route, ask for no GPU layers, and stop if the server takes any of the card's memory."
+    ],
+    "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-33/",
+    "file": "issues/issue-33.md"
+  },
+  {
+    "issue": 34,
+    "date": "2026-10-29",
+    "title": "29 October: repaired test sets certify 44 of 50, long tool lists cost a model its \"no\", and Cosmos 3 on one rented card",
+    "description": "Rewritten so that every pair is a clear contradiction, and keyed blind before any reader saw them, the reasoning-dataset-lab sets certified 44 of 50, against 26 before; the thinking setting we suspected turned out not to be what cut the length stops. A test of tool lists from 5 to 100 found that two local models kept picking the right tool, but one of them declined less often when the prompt grew long, and retrieving five tools kept its \"no\" intact. NVIDIA's Cosmos 3 made its first clips on rented cards: one Nano clip on one card, a 30-second rollout from the two-armed robot checkpoint, and five shrunk versions of the bigger Super model, of which Intel's int4 export is our working choice for now. Then the raw files behind our figures get a public home, a read of GitHub's Universe announcements, notes in brief, and the daily watch.",
+    "highlights": [
+      "<strong>44 of 50 sets certified.</strong> Every pair rewritten to be a clear contradiction and keyed blind first, the sets certified 44 of 50, against 26 before. The cleaner sets, not the thinking setting, cut the length stops.",
+      "<strong>Long tool lists held; the \"no\" slipped.</strong> With 5 to 100 tools, Granite 4.2 8B and Qwen3.8-27B kept picking the right tool. Granite declined 19 of 30 no-tool requests at 40 tools and 18 on a padded 10-tool prompt; retrieving five tools kept it at 27 to 30.",
+      "<strong>A world model on one rented card.</strong> Cosmos 3 Nano made an 8-second clip on one RTX PRO 6000 card in a 628-second window, about 1.55 compute units, with generation within 2% of our estimate.",
+      "<strong>Super on one card, for now in int4.</strong> Five shrunk versions of Cosmos 3 Super ran on one G4. Intel's int4 export wasn't separable from FP8 on fidelity, and ran faster in less memory, so it's our working choice for now.",
+      "<strong>rnd-data is public.</strong> The raw files behind our figures, 111 runs so far, now have a repository of their own.",
+      "<strong>GitHub Universe 2026.</strong> We read the agent-centred announcements on GitHub's own pages, and checked an AI search summary of them against those pages."
+    ],
+    "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-34/",
+    "file": "issues/issue-34.md"
   }
 ]

package/issues/issue-33.md

@@ -0,0 +1,323 @@
+---
+issue: 33
+date: "2026-10-28"
+title: "28 October: the T4 sweep is done, a ready-made critic fails its test, and Cognate learns to choose between plans"
+description: "Yesterday the T4 part of the C2 sweep finished. Replayed over the earlier windows, interlock 1.0.11's new check would have refused every load that had been killed and none that had completed; on its live windows, nothing crashed. We also tested whether an off-the-shelf language-inference model could check our verifier's quotes. It couldn't: on real supporting quotes it agreed 21% and 7% of the time, against a bar of 85%. And Cognate took in a plan chooser, with three fixes on the way in. In reasoning-dataset-lab, two models agreeing certified 26 test sets, and a blind second key found only 3 of them sound. A first paid run on batching found that it rewrites most of a model's replies, though few of its verdicts. Overnight, a CPU-only test run turned out to be on the GPU, and CPU runs now hide every route to it. The issue ends with the first daily watch."
+url: "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-33/"
+highlights:
+  - "<strong>The T4 sweep is done.</strong> Three repetitions on Colab's T4 card closed on 27 October. By the end of the T4 part the sweep had spent 19.572 of the 800 units set aside for it; the L4 and A100 cards started running overnight."
+  - "<strong>A check that held on a live run.</strong> interlock 1.0.11 refuses a cell whose memory would overflow the machine, before loading it. On its first live windows, for phi3.5 it refused the same five cells in each of four quants, and nothing crashed."
+  - "<strong>A ready-made critic, failed.</strong> An off-the-shelf language-inference model read real supporting quotes as support 21% and 7% of the time, against a pre-set bar of 85%. It won't be used."
+  - "<strong>Cognate chooses between plans.</strong> civility-kernel joins Cognate: it drops plans that break a hard rule, scores the rest, and executes the best or asks a person. Its decisions can now be hashed and recorded."
+  - "<strong>Two readers agreeing wasn't enough.</strong> Two models certified 26 of 50 test sets, but a blind second key found only 3 that hold a contradiction on every pair. The sets are being rewritten before any rerun."
+  - "<strong>Batch 1 for verdicts that must repeat.</strong> Served one request at a time, a 30B model repeated all 60 replies. Batched eight at a time, only 1 of 60 matched. Decisions moved far less, on 1 to 4 of 60."
+  - "<strong>CPU runs now hide every GPU.</strong> A CPU-only test server that hid only CUDA put a 24B model on the RTX 5090 through Vulkan. CPU runs now hide every route, ask for no GPU layers, and stop if the server takes any of the card's memory."
+---
+
+Two results from 27 October: a sweep that finished, with a fix that held on its first live windows, and a test that
+failed, as its pre-registration said it might. Then a change that landed overnight in Cognate, a certification that two
+models agreed on and a second key didn't, a test of what batching does to a model's replies, a CPU run that turned
+out to be on the GPU, and, from this issue on, a short daily watch at the end.
+
+**In this issue:**
+
+1. [The T4 sweep is done, and the host-memory check held](#the-t4-sweep-is-done-and-the-host-memory-check-held)
+2. [A ready-made critic fails its test](#a-ready-made-critic-fails-its-test)
+3. [Cognate learns to choose between plans](#cognate-learns-to-choose-between-plans)
+4. [Two readers agreed, and it wasn't enough](#two-readers-agreed-and-it-wasnt-enough)
+5. [Batching rewrites a model's replies](#batching-rewrites-a-models-replies)
+6. [A CPU run that wasn't](#a-cpu-run-that-wasnt)
+7. [The daily watch](#the-daily-watch)
+
+## The T4 sweep is done, and the host-memory check held
+
+**C2** measures how fast and how large the studio's models run on Colab's rented cards, with interlock.
+[Issue 31](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/) told how its first trial runs on a T4 card exposed problems in our
+measurement, and [issue 32](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/) noted one more: some cells filled the machine's memory, and
+its limit killed the model server, three times taking our harness down with it.
+
+### The fix, checked against the past first
+
+**interlock 1.0.11**, released on 27 October, works out before each cell how much of the machine's memory the
+cell's working cache would need, from the cell before it. If that, with a 5% margin added, is over the machine's
+limit (12,954 MiB on a T4), the cell is recorded as out of memory and never loaded.
+
+Before it ran live, it was replayed over the seven T4 windows already on record. Of 234 loads, 12 were the first of
+their run and can't be projected. Of the other 222, it refused **all 34 loads that had been killed** and **none of
+the 188 that had completed**.
+
+### Then live
+
+The first window on 1.0.11, s2c, ran phi3.5 at Q4_K_M: 21 cells, 13 measured and 8 out of memory. Five of those
+eight were refused by the new check before loading. Two were the exact cells that had killed the harness in windows
+s2a and s2a2; with the margin, the check projected them at **15,544 MiB** and **42,349 MiB**. The machine's memory log showed
+**no model-server kills and no harness kills.**
+
+The last window, s3a2, ran all four phi3.5 quants on 1.0.11 and refused the same five cells in each, again with no
+kills.
+
+### Its limits, stated with it
+
+- The first cell of each run is never projected, because there's nothing earlier to scale from. No first cell on
+  record was killed.
+- It can refuse a cell that would have fitted. A cell projected at **12,337 to 12,954 MiB** is under the T4's
+  limit, but the margin lifts it over, so it is refused; and the projection overstates a partly offloaded load. The
+  closest completed load on record was projected at **12,021 MiB** (12,622 with the margin), 316 MiB short of
+  that band.
+
+### What the sweep cost
+
+The T4 part of C2 ran three repetitions, finishing with window s3a2 on the evening of 27 October. Not every window
+came back whole: one stopped at its spending cap, one was lost to the memory limit before the fix, one was stopped by the launcher after the same kill,
+and one was lost twice when our own machine restarted, then re-run in full. By the end of the T4 part, counting the
+trial runs before it, C2 had spent **19.572 of the 800 units** set aside for it. The other cards began overnight. The
+T4 bills **1.07 units an hour**.
+
+## A ready-made critic fails its test
+
+**aspire-si** trains models to judge the way a teacher does. Its current plan trains a **verifier**: a model that
+checks a claim about code and quotes the line that settles it. A cheap way to check those quotes would be a ready-made **natural-language
+inference** model, one trained to say whether one sentence supports another. Smoke test **S1** asked whether one
+such model, `cross-encoder/nli-deberta-v3-base`, can tell a real supporting quote from one that fooled a verifier.
+
+The pass line was fixed before the run. Real supporting quotes had to be read as support at least **85%** of the
+time, and quotes that had fooled a verifier on an unsupported claim no more than **15%** of the time (the upper end
+of a 95% interval).
+
+### The result
+
+| check type | real supporting quotes read as support | fooling quotes read as support (upper bound) | pass |
+|---|---|---|---|
+| grounded | **0.210** (of 267) | 0.073 (of 110; upper 0.137) | no |
+| reasoning | **0.071** (of 210) | 0.029 (of 68; upper 0.101) | no |
+
+The model rarely calls a fooling quote support, so that half holds. But most of what it reads, it calls
+"neutral": on the NPU, 498 of the 677 pairs it scored came back neutral. It was trained on prose, and these are lines of code. The pre-registration named
+that risk, and pairs with a prose comment in the code didn't rescue it: on grounded claims, 0.208 with a comment and
+0.153 without.
+
+(The run's own note first gave these two counts as 473 of 637 and 636 of 637. Both had been counted with each quote
+cut to 80 characters, which made the 677 pairs look like 637; the note was corrected on 28 October.)
+
+It ran on both of the rig's Intel devices. They agreed on 676 of 677 labels (99.9%), and the integrated GPU was about three
+times faster than the NPU (0.089 s a pair against 0.271 s). That's moot for this model, which fails on both.
+
+### What it means
+
+The model won't be used. That was the pre-registered consequence of a fail, and nothing else changes: it was never
+going to enter the training reward, and S1 says nothing about the verifier's answer keys, which are provisional. The
+next smoke tests, S2 and S3, are separate pre-registrations and haven't run.
+
+## Cognate learns to choose between plans
+
+[Cognate](https://github.com/mcp-tool-shop-org/cognate) records how AI is governed: which model version was used,
+which policy applied, which agent acted, with each record provable afterwards. Its policy engine checks one event at a
+time, such as a prompt, an output or a spend. Until now it had nothing for the step before: deciding which of several
+plans an agent should carry out.
+
+On 28 October Cognate took in **civility-kernel**, imported from its earlier standalone package, as a new package of
+its own. Given an agent's candidate plans, it drops any that break a hard rule (an irreversible change, a spend over a
+limit), scores the rest by the policy's weights, and then either carries out the best one or, when that plan is too
+uncertain, asks a person. If no plan survives, it asks the person to relax a rule or offer another option.
+
+### Three fixes on the way in
+
+- **A cache keyed on too little.** From the policy, the kernel's compile cache read only the version, so two
+  policies with the same version but different weights or constraints got the first one's rules. The cache is gone.
+- **Decisions that can be reproduced.** A caller can now supply the decision's id and time; then the same inputs give
+  a byte-identical record.
+- **A number JSON can't hold.** A plan that broke a rule was scored minus infinity, which JSON writes as `null`, so a
+  record read back from storage was no longer the record that was made. Such plans now score `null` from the start.
+
+### What it adds
+
+A decision can now be hashed in the form Cognate's run pre-registration uses, and recorded on its event log. The record
+carries the outcome and which rules each rejected plan broke, and never the plans' steps; the chosen plan's digest
+still covers them, so a step changed afterwards no longer matches.
+
+The new package has **164 tests**: 143 carried over and 21 new. Its tests cover **91.89%** of its lines (578 of 629),
+against a floor of 90%. It is on Cognate's main branch and not yet in a release.
+
+## Two readers agreed, and it wasn't enough
+
+[reasoning-dataset-lab](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab) builds test sets with a known
+answer. One kind is a set of five values for one attribute of a machine, chosen so that only one can be true of a
+unit: a transformer's vector group, a vessel's berth status, a reactor's coolant. Any two of them then make a
+contradiction, and a test item can be built from each pair. Before a set is used, it is **certified**: two language
+models, each reading blind, must call every one of its 20 ordered pairs a contradiction.
+
+On 28 October the lab ran certification on 50 sets with two readers, Mistral Small 3.2 (24B, thinking off) and
+Nemotron 3.5 Lightning (30B). Both first passed a fixed test on 20 known-compatible pairs (19 and 20 of 20), so neither
+was simply answering "contradiction" to everything.
+
+### The run
+
+- **26 of 50 sets certified**, and 24 went to a review queue. There were 2,000 calls and none failed.
+- 121 pairs kept those 24 sets out. On 55, one reader called a contradiction and the other called the pair compatible. Both readers called
+  27 compatible. Another 25 were Nemotron runs whose thinking hit the length limit, which count as failures, not
+  verdicts. On the other 14, a reader answered "unsure".
+
+### A second key
+
+Agreement between readers is only worth something if the pairs really are contradictions, so we had every pair of the
+26 certified sets keyed a second time, by a model blind to the readers' verdicts. It was allowed three answers:
+contradiction, arguable, or compatible.
+
+| Pairs | Contradiction | Arguable | Compatible |
+|---|---|---|---|
+| 520 pairs in the 26 certified sets | 272 | 216 | 32 |
+| 27 pairs both readers called compatible | 2 | 19 | 6 |
+
+Only **3 of the 26 certified sets** came back as contradictions on all 20 pairs. The faults are in the sets, and both
+readers missed them the same way:
+
+- Under the collision regulations a vessel that is drifting is still "underway", so *underway* and *adrift* can both
+  hold.
+- A large generator is often cooled by hydrogen and by water at once.
+- A three-winding transformer carries two vector groups.
+
+Our own read of a sample, made before the second key and without sight of it, had found the first two. The second key
+agreed with every fault that read named, and found others, such as the three-winding transformer, in sets the read
+had called clean.
+
+### What it means
+
+Two readers agreeing is a weak filter for this task: it rules out clear failures, but it doesn't prove a pair is a
+contradiction. That's a finding about the test sets, not a measurement of the readers. "Arguable" is a judgement, and
+some of the second key's calls are strict ones (most people would read "the transformer" as a two-winding unit). The
+sets are being rewritten before any rerun, under four rules: one axis per attribute, no relative terms ("higher"), no
+label that contains another value, and no values that can occur together on one unit.
+
+## Batching rewrites a model's replies
+
+A model served one request at a time gives the same reply to the same request, if the seed and settings are fixed.
+Servers are faster when they batch several requests together, but batching changes the arithmetic. We wanted to know
+how much it changes the answers.
+
+On 28 October we ran Muse Glimmer 30B on a Colab A100 (40 GB) under llama.cpp's server, and sent it the same 60
+critic requests (52 asking for a verdict, 8 for the flaw in a check) at batch widths 1, 2, 4 and 8 (the server's slot
+count). Widths 1 and 4
+were run twice. The seed, the settings and the server build were fixed. (The study's code is public, in
+[role-set](https://github.com/mcp-tool-shop-org/role-set), under `scripts/batchdrift/`.)
+
+| Compared with batch 1 | Identical replies | Decision changed |
+|---|---|---|
+| Batch 1, run again | 60 of 60 | 0 |
+| Batch 2 | 30 of 60 | 3 |
+| Batch 4 | 12 of 60 | 1 |
+| Batch 4, second run | 12 of 60 | 3 |
+| Batch 8 | 1 of 60 | 1 |
+
+- **Batch 1 is deterministic** on this card and build. It matched itself on all 60, and matched an earlier reference
+  run on all 60.
+- **Batching rewrites most replies**: half of them at width 2, all but one at width 8.
+- **And a batched run doesn't repeat itself.** Batch 4 run twice gave the same reply on only 37 of 60.
+- **Decisions move much less than the text.** A reply's decision is its verdict (or, for a flaw request, the flaw
+  it names) plus the fields that say what it covered. It changed on 1 to 4 of the 60 requests in each comparison that
+  involved batching. The verdict itself changed on 0 to 4, no flaw label changed, and batch 8's one change was in a
+  coverage field. With 60 requests that is still loose: the 95% intervals
+  reach about 16%.
+
+### What we take from it
+
+Any verdict that has to be reproducible runs at batch 1. Batched runs are their own tier, and their replies can't be
+compared one by one with batch 1's. Two limits: we recorded the server's slot count but not how many requests the
+client kept in flight, so we can't yet say every slot was full. And this is one model on one card under one build;
+other engines and cards may behave differently.
+
+## A CPU run that wasn't
+
+On 28 October we ran a dry run of a 24B model, the Mistral reader above, on the rig's processor only, to check a code
+path before asking for time on the RTX 5090. The second Ollama server it started hid CUDA with `CUDA_VISIBLE_DEVICES=-1`, and that was all
+it hid. Ollama 0.40.9 also ships a Vulkan backend, and through it the "CPU" server loaded the model onto the 5090
+anyway: its process held 15,114 MiB of the card's memory for about 29 minutes before we found it and stopped it.
+Nothing broke, but the card was meant to stay free, and the run wasn't measuring what it said.
+
+### The guard
+
+CPU-only runs now do three things:
+
+- **Hide every GPU route**, not just one. The server starts with CUDA, Vulkan and HIP hidden.
+- **Ask for no GPU layers on every request.** A small wrapper adds `num_gpu: 0` to each call, so a backend we didn't
+  think of still gets no layers.
+- **Check, and stop on a rise.** Every 10 seconds a guard reads the GPU memory held by the server's own processes and
+  the card's total, and stops the job unless the server holds nothing and the total hasn't risen by more than 512 MiB.
+
+We tested the stop path three ways, including a forced baseline and a live run that failed closed on a counter it
+couldn't read. Then the guard passed on a 4B model and on the same 24B model: both runs reported `library=cpu`, the
+24B server started with no GPU layers (`-ngl 0`), the server held 0 MiB of the card, and the card's total went down
+slightly over each run rather than up.
+
+### A two-second name
+
+The same night turned up a smaller cost. An embedding job that talked to Ollama at `localhost` hit its 15-minute stop
+(902.4 s) without finishing. Each call was taking about 2 seconds against hundredths of a second at `127.0.0.1`,
+most likely because the rig tries IPv6 first and falls back. With `127.0.0.1` the whole job took 30.8 seconds.
+
+### A study whose retrieval can't miss
+
+That job was the retrieval arm of a study we're preparing: how a model's tool choice holds up as the list of tools
+grows. (Its code is public, in [role-set](https://github.com/mcp-tool-shop-org/role-set), under `scripts/toolcount/`.)
+With real embeddings, the arm found the right tool in its top five for all 170 answerable items, at every list size
+from 10 to 100. That isn't a result: the study's paid run hasn't happened. It says something about the items, which
+name their tool almost word for word. The study will report that alongside whatever it finds.
+
+## The daily watch
+
+*New since the [weekly watch](https://mcp-tool-shop-org.github.io/rnd/watch/2026-10-26/) of 26 October, checked against each item's own source.*
+
+**NeMo Relay 0.11.0** (NVIDIA, released 2026-10-28;
+[release notes](https://github.com/NVIDIA/NeMo-Relay/releases/tag/0.11.0)). This is the agent-tracing tool we tried in
+a sandbox on 0.10.1. The new version changes how its coding-agent hooks behave: tool checks now fail closed by
+default, and a hook that can't deliver returns exit code 2. It also adds opt-in diagnostics and continues traces
+through Claude Code hooks. For us, a hook that fails closed can stop an agent mid-task, so the 0.10.1 trial doesn't
+carry over. *What we'll do:* run the sandbox trial again on 0.11.0, hooks included, before we use it in our own
+harnesses.
+
+**ComfyUI v0.43.0** (Comfy-Org, released 2026-10-27;
+[release notes](https://github.com/Comfy-Org/ComfyUI/releases/tag/v0.43.0)). Among many partner-node additions, it
+fixes the top-p sampling threshold and Gemma 4 31B's global attention, starts dynamic VRAM KV caches, and sets a
+minimum Pillow version. For us, our image pipeline pins one ComfyUI commit per run, so this is the candidate for the
+next pin. *What we'll do:* check it on a CPU run before anything paid.
+
+**Ollama v0.40.12** (released 2026-10-27; [release notes](https://github.com/ollama/ollama/releases/tag/v0.40.12)).
+Its listed changes are on the Apple-silicon side (MLX, faster prompt processing), an updated llama.cpp, and Liquid AI's
+new LFM3 models, which run on Apple hardware only for now. For us, the C2 sweep ran on 0.40.11. *What we'll do:*
+nothing for now; C2 stays on 0.40.11, so its windows stay comparable.
+
+**Serving World Models: Latency vs Throughput with NVIDIA Cosmos 3 Super on Nebius** (NVIDIA Developer livestream,
+today at 12:00 PM ET; [watch on YouTube](https://www.youtube.com/watch?v=i9FZyG9-Gx0)). Two speakers from Nebius's
+Physical AI team will cover serving world models for video generation, which trades latency against throughput
+differently from serving language models. They'll show Cosmos 3 Super benchmarks through vLLM-Omni on HGX B200 and
+H200 systems, across four layouts from one eight-GPU replica to eight single-GPU replicas, with which settings to hold
+fixed and how to check the generated clips are whole and actually move. Its measure is, in the description's words,
+"validated video-seconds per node-hour", and the description links a public
+[benchmark repository](https://github.com/nebius/nebius-physical-ai) for reproducing it. For us, the trade-off is the
+one the batching study above ran into from the other side: serving several requests at once changed most of a model's
+replies. We don't serve video models on datacentre cards, but the method is the part we can use: what to hold
+constant between runs, counting only output that passes a check, and when more replicas or more concurrency stop
+helping. Cosmos 3 also sits in our plan for robotics in simulation, which hasn't started, and its licence is read
+before anything is run. *What we'll do:* read the benchmark repository and the talk's written version (next item),
+and compare their list of settings to hold constant and their checks on output with the protocols of our batching
+study. We won't run anything from it, and nothing changes in the robotics plan before the Cosmos licence has been read.
+
+**Serving world models: Request latency and node throughput tradeoffs** (Nebius, published 2026-10-26;
+[the post](https://nebius.com/blog/posts/serving-world-models-latency-node-throughput)). The written version of the
+talk, by the same two speakers, and the post the talk links as its technical blog. It serves Cosmos 3 Super through
+vLLM-Omni on eight-GPU B200 and H200 nodes split four ways. By the authors' figures, on B200 eight one-GPU replicas
+gave 42.5% more validated video per node-hour than one eight-GPU replica, at 5.5 times the mean request latency
+(378.1 s against 68.5 s); on H200 the gain was 25.7%, at 6.3 times. Only clips that pass a validation check count,
+while failed attempts still use the node's time. The post lists what it held constant, from the model revision and
+container to the seeds, resolution, frame count and denoising steps, and it ran one request at a time per replica.
+For us, it is the batching study's trade-off put in node terms: the layout with the most output per node is the one
+with the slowest single request, and its method is close to ours. *What we'll do:* compare its list of what's held
+constant, and its validation check, with our batching protocol. We run nothing from it.
+
+**NVIDIA Cosmos Opens New Tools to Train, Evaluate and Deploy World Models** (NVIDIA, published 2026-10-21;
+[the post](https://blogs.nvidia.com/blog/cosmos-open-train-evaluate-deploy-world-models/)). It announces Cosmos Data
+Factory (tools and blueprints to label, curate, augment and evaluate training data), the Cosmos Framework (training,
+optimisation and deployment tools offered as agent skills), Cosmos 3 Nano checkpoints for testing autonomous policies
+in simulation, updated Cosmos 3 NIM microservices that NVIDIA says give "up to 2x faster inference with no quality loss in
+validated configurations", and version 3.3 of its Metropolis blueprint for video search and summarization (VSS). For us, the Nano simulation checkpoints are the part closest to our plan for
+robotics in simulation, and the post states no licence terms for any of it. *What we'll do:* read the licences on the
+checkpoints' own model cards and repositories, as part of the planned licence read, before anything is run.

package/issues/issue-34.md

@@ -0,0 +1,411 @@
+---
+issue: 34
+date: "2026-10-29"
+title: "29 October: repaired test sets certify 44 of 50, long tool lists cost a model its \"no\", and Cosmos 3 on one rented card"
+description: "Rewritten so that every pair is a clear contradiction, and keyed blind before any reader saw them, the reasoning-dataset-lab sets certified 44 of 50, against 26 before; the thinking setting we suspected turned out not to be what cut the length stops. A test of tool lists from 5 to 100 found that two local models kept picking the right tool, but one of them declined less often when the prompt grew long, and retrieving five tools kept its \"no\" intact. NVIDIA's Cosmos 3 made its first clips on rented cards: one Nano clip on one card, a 30-second rollout from the two-armed robot checkpoint, and five shrunk versions of the bigger Super model, of which Intel's int4 export is our working choice for now. Then the raw files behind our figures get a public home, a read of GitHub's Universe announcements, notes in brief, and the daily watch."
+url: "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-34/"
+highlights:
+  - "<strong>44 of 50 sets certified.</strong> Every pair rewritten to be a clear contradiction and keyed blind first, the sets certified 44 of 50, against 26 before. The cleaner sets, not the thinking setting, cut the length stops."
+  - "<strong>Long tool lists held; the \"no\" slipped.</strong> With 5 to 100 tools, Granite 4.2 8B and Qwen3.8-27B kept picking the right tool. Granite declined 19 of 30 no-tool requests at 40 tools and 18 on a padded 10-tool prompt; retrieving five tools kept it at 27 to 30."
+  - "<strong>A world model on one rented card.</strong> Cosmos 3 Nano made an 8-second clip on one RTX PRO 6000 card in a 628-second window, about 1.55 compute units, with generation within 2% of our estimate."
+  - "<strong>Super on one card, for now in int4.</strong> Five shrunk versions of Cosmos 3 Super ran on one G4. Intel's int4 export wasn't separable from FP8 on fidelity, and ran faster in less memory, so it's our working choice for now."
+  - "<strong>rnd-data is public.</strong> The raw files behind our figures, 111 runs so far, now have a repository of their own."
+  - "<strong>GitHub Universe 2026.</strong> We read the agent-centred announcements on GitHub's own pages, and checked an AI search summary of them against those pages."
+---
+
+Results from 28 October: a repaired test set and its rerun, a test of long tool lists, and NVIDIA's Cosmos 3 on
+rented cards, from one clip to five shrunk versions of its biggest model. Then the raw files behind our figures get a
+public home, a read of GitHub's Universe announcements, notes in brief, and the daily watch.
+
+**In this issue:**
+
+1. [The repaired sets: 44 of 50 certified](#the-repaired-sets-44-of-50-certified-and-the-thinking-setting-wasnt-the-fix)
+2. [How many tools can a local model take?](#how-many-tools-can-a-local-model-take-the-list-held-the-no-slipped)
+3. [A world model on one rented card](#a-world-model-on-one-rented-card)
+4. [Cosmos 3, continued: the robot-arm checkpoint, and Super on one card](#cosmos-3-continued-the-robot-arm-checkpoint-and-super-on-one-card)
+5. [rnd-data: the raw files behind the research](#rnd-data-the-raw-files-behind-the-research)
+6. [GitHub Universe 2026: the platform turns toward agents](#github-universe-2026-the-platform-turns-toward-agents)
+7. [In brief](#in-brief)
+8. [The daily watch](#the-daily-watch)
+
+## The repaired sets: 44 of 50 certified, and the thinking setting wasn't the fix
+
+Issue 33 showed that two readers agreeing on a set of values is a weak filter: most of the sets they "certified" held
+pairs that were arguable, not contradictions. This is the repair and the rerun.
+
+**The repair.** All 50 five-value sets in reasoning-dataset-lab were rewritten so that every ordered pair is a
+contradiction no one could argue with. New tests catch the faults we found before: an attribute must name one property,
+no relative terms, no value whose label contains another, and no two values a single machine can hold at once. Then an
+outside model keyed all 1,000 pairs blind, before any reader saw them. It took three rounds: 120 of the 1,000 pairs came
+back "arguable" in the first, 16 of the 560 re-keyed in the second, and none of the 60 in the third. On the old certified
+sets, only 3 of 26 had been clean on all 20 pairs.
+
+**What "certified" means now:** a set passed the blind key first, and then both readers called every one of its 20 pairs
+a contradiction.
+
+**The rerun.** 44 of 50 sets certified, against 26 before. 2,000 reader calls, none failed. Eleven pairs kept the other
+six sets out, and we read each one:
+
+- 6 are faults in the pairs that the blind key passed. Four are "accelerating" pairs in two operating-mode sets (a
+  variable-speed compressor can speed up while loaded; a turbine "accelerating" can read as a load ramp while online). Two
+  are in a metering-device set, where an electronic valve can run a constant-pressure mode. The readers were right to
+  refuse them, so those three sets belong in the queue.
+- 5 are reader mistakes: 3 by one reader, 2 by the other.
+
+A read of 10 of the 44 certified sets found 9 clean and 1 with a mild wording note. The same read of the old sets found 5
+of 10 clean.
+
+![Three blind-key rounds, then the rerun. Round 1 found 120 of 1,000 pairs arguable across all 50 sets, round 2 16 of 560 pairs across the 28 sets re-keyed, and round 3 none of 60 across the last 3. In 2,000 reader calls with none failed, certified sets rose from 26 of 50 to 44 of 50. Six sets were queued on 11 refused pairs, 6 pair faults and 5 reader mistakes. Length stops for one reader: 25 at full and 22 at medium on the old sets, 2 at medium on the repaired sets.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-certification-repair.svg)
+
+**The thinking setting.** One reader (Nemotron 3.5 Lightning) now runs at "medium", an effort hint its template
+understands, not a cap on thinking. In the rerun it hit the token limit only twice in 1,000 pairs, against 25 before. That
+looked like the setting at work, so we ran it at medium on the old pairs too: 22 limit stops against 25 at full thinking,
+and mostly on different pairs. The drop came from the cleaner sets, not the setting. Medium did give the same verdict as
+full thinking on 95.5% of the pairs that got a verdict at both, in about 8% less time, so it stays; it just isn't the fix
+for length stops.
+
+**Limits.** Our read of the 11 pairs and of the sample is one reader's judgement, and the blind keys come from one outside
+model. Where the 35 flipped verdicts between settings sit hasn't been read yet.
+
+**Next:** reword or replace the three faulty sets, and add a generator rule for modes a variable-speed machine can hold
+together.
+
+**The data is public.** Everything above is in reasoning-dataset-lab under
+[`certification/v3/`](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/tree/main/certification/v3): the
+manifest of the 44 certified sets, the queue of 6, both reader controls, and the findings. `medium-arm/` holds the
+medium run on the old pairs, filed as an addendum to the earlier run so the two settings are never pooled, and
+`second-keying/` holds all three blind-key rounds (1,000, 560 and 60 pairs). Before it merged, a separate model
+recomputed every figure from the committed files. Every number comes from those JSON files except run times and peak
+memory, which come from the run logs, so you can redo the arithmetic yourself.
+
+Code: reasoning-dataset-lab PRs 21 and 23; data: PR 24.
+
+## How many tools can a local model take? The list held; the "no" slipped
+
+Agent setups hand a model a list of tools and expect two things: it picks the right one when one fits, and it declines
+when none does. A common worry is that a long list breaks the first. We tested both on two local models, Granite 4.2 8B
+and Qwen3.8-27B, with lists of 5 to 100 tools.
+
+**The setup.** Each list size got 200 calls: 170 where a tool fits and 30 where none does. The same 30 no-tool requests
+ran in every cell, so cells compare request by request. Two more arms separated the causes:
+
+- **A length control:** 10 tools, padded with neutral text to the length of the 100-tool prompt.
+- **A retriever:** it picks the 5 most relevant tools from the full list, so the prompt stays about 1,040 tokens.
+
+Settings: temperature 0, each model's own tool template, one A100 40 GB card.
+
+**Picking the right tool held.** Granite chose correctly on 164 to 170 of the 170 calls in every cell, its lowest at
+100 tools. Qwen managed 169 or 170 throughout. Up to 100 tools there was no cliff for either.
+
+**Declining is where Granite slipped,** and it followed the prompt's length, not the number of tools:
+
+| Tools in the prompt | Prompt length (tokens) | Granite declined | Qwen declined |
+|---|---|---|---|
+| 5 | 1,004 | 28 of 30 | 30 of 30 |
+| 10 | 1,737 | 29 of 30 | 30 of 30 |
+| 20 | 3,184 | 26 of 30 | 30 of 30 |
+| 40 | 6,108 | 19 of 30 | 29 of 30 |
+| 70 | 10,457 | 22 of 30 | 27 of 30 |
+| 100 | 14,804 | 27 of 30 | 28 of 30 |
+| 10, padded to 100-tool length | 13,270 | 18 of 30 | 29 of 30 |
+| retriever's top 5 | about 1,040 | 27–30 of 30 | 29–30 of 30 |
+
+The prompt lengths are Granite's. Qwen3.8-27B ran as two halves of the design, split by seed, and its counts pool both.
+
+![Tool lists from 5 to 100. Picking the right tool held flat: Granite 4.2 8B 164 to 170 of 170, Qwen3.8-27B 169 to 170. Declining on 30 no-tool requests, plotted against Granite's prompt length with 95% intervals and no trend line: Granite 28 of 30 at 5 tools, 29 at 10, 26 at 20, 19 at 40, 22 at 70, 27 at 100, 18 on the 10-tool prompt padded to 100-tool length, and 27 to 30 with the retriever's top 5. Qwen, pooled from two halves, 27 to 30 of 30 everywhere. Only the 40-tool and padded drops are significant after correction (p 0.02 and 0.011); the 70-tool drop is not (p 0.35). Interim: Gemma 4 31B is still running.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-tool-count-abstention.svg)
+
+- **Two drops are real after correcting for the number of comparisons:** 40 tools, and the padded 10-tool prompt. The
+  padded prompt carries only 10 tools, yet does as badly as any list, so length alone can do it.
+- **It isn't a steady slide.** 100 tools came back to 27 of 30, and the 70-tool drop doesn't survive the correction.
+  "The longer the list, the less it declines" would overstate it.
+- **With the retriever,** Granite declined 27 to 30 of 30 at every list size.
+- **Qwen stayed flat,** with no change near significance.
+
+**Limits.** 30 no-tool requests per cell is small: one request is 3.3 points, and most intervals overlap. This is
+two model families, with one template each. The padding was neutral text, so it shows that length alone can do this,
+not which part of a real tool list does. The retriever is one retriever. A third family, Gemma 4 31B, is running now,
+so these results are interim.
+
+**What we take from it:** here, retrieving a few tools kept the prompt short and protected Granite's "no". Whatever the
+setup, test declining on its own. Accuracy alone didn't show this.
+
+## A world model on one rented card
+
+Robotics in simulation is our next research track, and NVIDIA's Cosmos 3 is where it starts: open world models that
+predict what a camera would see next, including checkpoints tuned to show how a scene changes after a robot acts. The
+first rung asked only one thing: does it run on the cloud GPUs we rent, and what does a clip actually cost?
+
+**Which model.** NVIDIA ships Cosmos 3 Super (64B) only in BF16, about 128 GB of weights, so it fits on no single Colab card.
+Cosmos 3 Nano (16B, about 35 GB, under the OpenMDW-1.1 licence) fits.
+
+**The run.** One clip of the model card's own example prompt (1280×720, 189 frames, 35 steps, one seed) on one RTX PRO
+6000 Blackwell card:
+
+| Stage | Measured |
+|---|---|
+| Download, 35 GB | 97 s (362 MB/s) |
+| Setup (environment and pinned wheels) | 53 s |
+| Generate 35 steps and decode | 430 s (about 11.6 s a step) |
+| Whole window | 628 s, about 1.55 compute units |
+
+Peak GPU memory was about 42 GiB allocated and 51 GiB reserved, with roughly 45 GB of the card to spare; host memory
+barely moved (2.9 GB). The output is an 8-second, 24 fps clip of a robot arm scrubbing a plate.
+
+**Estimate against measurement.** The design scaled NVIDIA's published figure (93 s a clip on a B300) to this card and
+predicted about 420 s of generation; it came in at 430, within 2%. Setup and download were much faster than guessed, so
+the window cost under half the design's projection of about 3.6 units.
+
+**How it was run.** A free CPU stub ran every install step first, for about 0.005 units. A test suite against a fake
+Colab command line checked that the launcher refuses a wrong card, too little memory or disk, or a changed script before
+anything downloads, and it caught a launcher defect before any spend. An outside model reviewed the harness and the
+licence.
+
+**Limits.** One clip and one seed: this says the model runs and what it costs, not how good its output is. The next rung,
+a simulation checkpoint on its own example, gets its own design.
+
+## Cosmos 3, continued: the robot-arm checkpoint, and Super on one card
+
+The Nano clip above was the first rung. Three more runs followed the same day: a checkpoint built for a two-armed robot,
+five shrunk versions of the bigger Super model on one Colab card, and Super at full precision on one rented
+B200 to serve as their reference.
+
+### The two-armed robot checkpoint
+
+Cosmos3-Nano-Sim-Bimanual predicts what a camera sees as a simulated two-armed robot carries out a list of actions. We
+ran one 30-second rollout (901 frames at 832×480, one seed) through its own video API on a G4 card (RTX PRO 6000
+Blackwell, 96 GB).
+
+- **Inference took 29.1 s,** which is 31.0 frames a second. The model card gives 20.6 for this card at 480p.
+- **Peak GPU memory** was 51,980 MB, and the whole window took 5 min 27 s and 0.81 compute units.
+- **Two earlier tries failed on the install, not the model.** First a dependency needed a Rust compiler, which a free
+  CPU stub caught. Then on the paid run, the serving stack's just-in-time CUDA compile found no compiler. A toolchain
+  step now fixes that, and every stub proves it.
+
+![The two-armed robot checkpoint's 327-second window on one G4: 110.8 s of setup, 91.2 s downloading 31.8 GB at 348.9 MB/s, 35.0 s starting the server, 29.1 s of inference for a 901-frame rollout at 31.0 frames a second against the model card's 20.6, and 60.9 s for opening, the VM check, stopping and fetching. Closed-loop control is planned, not yet measured.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-bimanual-window.svg)
+
+This run plays back a fixed action file, with no controller in the loop, so it says the checkpoint runs and how fast.
+It says nothing about whether the robot does its task well. Faster than the card is one run on one card type, not a
+benchmark. Closed-loop control is the next rung, and it isn't designed yet.
+
+### Super, shrunk to fit one card
+
+Super won't fit a single Colab card in BF16. Uploads on Hugging Face shrink it, so we asked which of them
+run on one G4, how fast, and in how much memory. Each made the same three 61-frame clips at 480p (35 steps, the same
+prompts and seed), plus a 50-step still:
+
+| Version | Time a step | 61-frame clip | Peak memory allocated | Window cost |
+|---|---|---|---|---|
+| FP8 (made with NVIDIA ModelOpt) | 3.83 s | 137–139 s | 69,954 MiB | 3.09 units |
+| NVFP4 (ModelOpt) | 7.14 s | 253–254 s | 47,396 MiB | 3.07 units |
+| NF4 (bitsandbytes) | 3.37–3.38 s | 122 s | 39,750 MiB | 2.52 units |
+| int4 (AutoRound, a second export) | 3.33–3.35 s | 120–125 s | 41,066 MiB | 1.59 units |
+
+- **All four ran every clip they were given,** and none ran out of memory. The four windows cost 10.27 units together.
+- **At 720p** (FP8 and NF4 only), a clip took 339 s and 326 s, at a peak of 76,054 and 45,851 MiB.
+- **Cutting the steps from 35 to 20** cut the FP8 clip from 138.6 s to 80.0 s, 42% less, with no change in memory.
+- **A fifth version,** Intel's own int4 AutoRound export, needed changes to its config files to load and joined later.
+  It took 3.36 s a step, at a peak of 39,581 MiB at 480p and 45,680 MiB at 720p.
+
+The table shows what runs and what it costs. How close each version looks to full precision is a separate measure.
+We scored every clip against the B200 reference below in two ways: LPIPS, a learned measure of how different two
+frames look (0 means identical), and the cosine similarity of DINOv2 image features (1 means identical). A first round
+used one seed, three prompts and a still. A wider round then used 8 prompts and 3 seeds; these are the 21 clips that all
+five versions share:
+
+| Version | LPIPS (lower is closer) | DINOv2 (higher is closer) |
+|---|---|---|
+| int4 (AutoRound, Intel's export) | 0.320 | 0.765 |
+| FP8 | 0.341 | 0.770 |
+| NVFP4 | 0.387 | 0.724 |
+| NF4 (bitsandbytes) | 0.401 | 0.716 |
+| int4 (AutoRound, a second export) | 0.406 | 0.723 |
+| Nano at full precision (a different model) | 0.578 | 0.558 |
+
+Paired clip by clip against FP8:
+
+- **Intel's int4 and FP8 can't be told apart.** FP8 was closer on 9 of 21 clips (sign test p = 0.66). FP8's mean LPIPS
+  was 0.021 higher, with a 95% bootstrap interval of −0.014 to +0.058, which takes in zero.
+- **FP8 is closer than the other three.** It was closer than NVFP4 on 16 of 21 clips (p = 0.027), and than NF4 and the
+  second int4 export on 15 of 21 each (p = 0.078). Its mean was 0.045 to 0.065 lower, and all three intervals stay below
+  zero.
+- **Nano is a baseline, not a shrunk Super.** Its distance shows how different the smaller model's clips are, not what
+  shrinking costs.
+
+The two are not separable on fidelity, and on everything else measured Intel's int4 is ahead. It took 3.36 s a step against
+FP8's 3.83 in the wider round, and peaked at 39,581 MiB at 480p against 69,954. It ran 720p in 45,680 MiB against
+76,054, and had the narrowest spread across the three seeds of any Super version (0.311 to 0.328). So it's our working
+choice, for now. It goes back to FP8 if any of three checks still to run prefers FP8 by more than that check's own
+run-to-run spread: scores that need no reference, a vision-language model judging clips in pairs, and a blind
+preference test by people.
+
+The scores have limits:
+
+- **Similarity, not preference.** They measure closeness to full precision, not which clip a person would pick.
+- **One card type,** and 8 prompts at 480p, 61 frames and 35 steps.
+- **Repeats.** Repeats of FP8, NF4 and the second int4 export matched the first run byte for byte, so on one card type the
+  differences come from the prompt and the seed, not from noise between runs. NVFP4 and Intel's int4 weren't repeated.
+- **The card counts too.** Part of each gap comes from the card, not from the shrinking. On the first round's clips,
+  FP8 on the B200 scored 0.345 against full precision on that same card, and FP8 on the G4 scored 0.193 against FP8 on
+  the B200.
+
+Why FP8 is no faster than the 4-bit versions here hasn't been looked into.
+
+![Five shrunk versions of Cosmos 3 Super against the full-precision model over 21 shared clips (8 prompts and 3 seeds), plus the smaller 16B Nano model at full precision as a separate baseline. Mean LPIPS distance to full precision (0 is identical, lower is closer), with the range of the three seed means, time a step and peak memory at 480p: Intel's int4 AutoRound 0.320 (seeds 0.311–0.328; 3.36 s; 39,581 MiB), FP8 0.341 (0.311–0.363; 3.83 s; 69,954 MiB), NVFP4 0.387 (0.337–0.426; 7.12 s; 47,396 MiB), NF4 (bitsandbytes) 0.401 (0.372–0.441; 3.37 s; 39,753 MiB), a second int4 AutoRound export 0.406 (0.383–0.438; 3.39 s; 41,066 MiB), and Nano 0.578 (0.575–0.581; 0.81 s; 36,002 MiB). Paired against FP8, Intel's int4 is not separable on fidelity (FP8's mean 0.021 higher, 95% interval −0.014 to +0.058, FP8 closer on 9 of 21 clips, p = 0.66); the other three trail FP8, with intervals entirely below zero. Intel's int4 is the provisional working choice, and goes back to FP8 if reference-free scores, a vision-language judge or a blind preference test by people prefer FP8 by more than their own run-to-run spread.](https://mcp-tool-shop-org.github.io/rnd/charts/issue-34-super-shrunk-comparison.svg)
+
+### The reference: full precision on one B200
+
+The scoring needs Super's own clips at full precision, with the same prompts and seed. No Colab card holds it, so this
+one run went to a rented B200 (about 180 GB) on RunPod.
+
+- **The 132.7 GB download** took 42.6 s (3,116 MB/s), and the model loaded in 29.1 s.
+- **A 61-frame 480p clip** took 37.9–44.0 s (1.03 s a step), the still 10.5 s, and the 720p clip 98.3 s.
+- **Peak memory allocated** was 129,206–129,207 MiB at 480p and 135,320 MiB at 720p.
+- **The run** took about 7.5 minutes and was booked at $1.09.
+
+The B200's times aren't a speed comparison with the shrunk versions: it is a different card at a different precision.
+
+## rnd-data: the raw files behind the research
+
+Each issue quotes figures from runs: a window's log, a reader's verdicts, a cost ledger. Those files now have a public
+home of their own, [rnd-data](https://github.com/mcp-tool-shop-org/rnd-data), so a reader can check a figure against
+the run that produced it, not only against our summary of it.
+
+**What's in it.** 111 runs so far, from two ingests on October 19 and 20: 83 Colab runs (each retry of a window counts as its own run), the weekly watch's runs and
+reports, a handful of RunPod and tool receipts, and 11 xrpl-lab runs on the XRP Ledger's test network. Each run has a
+manifest listing its files, and every public file in it carries its SHA-256, its size and its licence. There are 1,274
+public files:
+- **1,096 small files** (256 KB or less, about 5.7 MB in all) are in git.
+- **178 larger ones** are stored once each, by hash, so they come to only 13 distinct files, about 430 MB. Most are one
+  llama.cpp server build that many Colab windows uploaded. These sit in a blob store on the studio's backup drive for
+  now, and are due to go public as a Hugging Face dataset in a later phase.
+
+The data is CC BY 4.0; scripts are MIT, and a third-party file such as the llama.cpp build keeps its own licence
+(MIT), which its manifest names.
+
+**What stays private, and how that shows.** Open is the default, and a file is private only for a named reason. Of the
+files held back, 566 are private for their content (381 hold test-item text, 185 are plans not yet approved). They are
+left out of the manifests and only counted by reason. Another 192 are private only because the identity scan flagged
+them; these are listed by path and size, with no hash. No private file's hash is published, because the hash of a
+short file can be matched by guessing its contents. The private copies are held in a separate private repository.
+
+**How to check a figure.** Download a file and compare its SHA-256 with its manifest; `sha256sum` is enough, and the
+README gives a one-line command that checks every file in git at once. CI does the same on every pull request: it
+re-hashes every file, and fails on a file no manifest names, a private entry with a hash, or a machine path in a
+manifest. An experiment in the rnd library points to its runs by id: the stage-1 regeneration experiment names 74 of
+them.
+
+**What's not done yet.** The runs in this issue (the certification rerun, the Cosmos clip, the current sweep) haven't
+been ingested yet, so for now their receipts are only in the experiment write-ups. The small read-only Python package
+that lists and shows runs (`rnd-data`, version 0.1.0) has its release workflow in place, but it isn't on PyPI yet.
+
+Read more: [the landing page and handbook](https://mcp-tool-shop-org.github.io/rnd-data/).
+
+## GitHub Universe 2026: the platform turns toward agents
+
+GitHub's Universe announcements on 2026-10-28 have one theme: the unit of work is no longer a line of code a person
+writes, but a task an agent carries from issue to merge. We read the announcements on GitHub's own pages, and we also
+checked an AI search summary of them against those pages. The summary was mostly right, and the places it went wrong
+are worth a look.
+
+**What GitHub shipped**
+- **The Copilot app,** a separate app built around agent work. Inside it, **Copilot Projects** (preview) splits a goal
+  into parallel agent sessions across one or more repositories. **Computer use** (preview, macOS and Windows) lets
+  Copilot operate desktop apps. It asks before it takes control, and organizations can switch it off.
+- **Copilot Fusion** decides per task whether to use one model or combine several, at a Standard, High or Max effort
+  setting. GitHub's figure: on the Artificial Analysis Coding Agent Index, Fusion Max scored within 2 points of Opus 5.5
+  Max, at half the cost and 40% less time per task.
+- **Security.** AI Scan checks a whole repository on demand and hands what it finds to Copilot to fix. It also checks
+  pull requests in languages CodeQL doesn't cover. AI push protection blocks unstructured secrets, such as plain
+  passwords, at push time. A package firewall stops malicious or risky npm packages before they install, and
+  reachability analysis shows which vulnerable functions a JavaScript codebase actually calls.
+- **The platform.** Stacked pull requests split a large change into linked PRs that are reviewed apart and merged as a
+  stack. Copilot Automations run tasks on demand, on a schedule or from a trigger. Copilot Connectors give one sign-in
+  to MCP servers such as Vercel, Cloudflare and Sentry.
+- **Capacity.** GitHub reports adding 4 million CPU cores, 160 PB of fast storage and 50 Tb/s of network capacity this
+  year. Its Git architecture now separates storage from compute, and its move to a multi-region Azure deployment is due
+  to finish early next year.
+
+**The Octoverse numbers** (October 2025 to September 2026, public and private repositories)
+
+| Measure | Figure |
+|---|---|
+| Commits | 34.3 billion |
+| Pushes | 17.9 billion, more than 2024 and 2025 combined |
+| Developers | over 225 million (United States 38.3 million, India 32.0 million) |
+| Pull requests begun as an agent's first draft | over 40% in September, against 17% in March |
+
+The agent share counts only agents GitHub can detect, so it is a floor. TypeScript is still the most used language by
+contributors. Rust grew fastest of the top languages; the report's text says it doubled, while its chart puts the
+growth at 162%.
+
+**Where the summary drifted.** The AI summary had most of this right, but:
+- It put Fusion at "40% faster" and "up to 77%" cheaper. GitHub's own figure is half the cost and 40% less time, and
+  only for the Max setting on one benchmark against one model. None of GitHub's pages gives a 77% figure.
+- It dropped two qualifiers: that Projects and computer use are previews, and that the agent share is a floor.
+- It gave Rust's growth as a plain doubling, without the chart's 162%.
+
+None of these is a big error, but each one makes the claim stronger than GitHub's own, which is why we check a
+summary against the page it summarises.
+
+**For the studio.** We publish npm packages, so the package firewall and reachability analysis apply to our work. AI
+push protection would add a check alongside the identity scan we run before each push. Fusion's figures are GitHub's
+own benchmark; we haven't measured them. Relevance: watch.
+
+Sources: [the Universe recap](https://github.com/events/universe/recap),
+[GitHub's announcement](https://github.blog/news-insights/company-news/github-universe-2026-evolving-every-layer-of-the-stack/),
+[the Copilot app changelog](https://github.blog/changelog/2026-10-28-github-copilot-app-universe-releases/),
+[Octoverse 2026](https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/).
+
+## In brief
+
+**CUDA Rust, live.** NVIDIA Developer streamed "CUDA Live: Introduction to CUDA Rust" on 2026-10-28 at 18:00 UTC
+([recording](https://www.youtube.com/live/6DKmniAOhok)): the team writes, builds and runs Rust GPU kernels and explains
+when to use each of its two tracks. The code is [NVIDIA/cuda-rust](https://github.com/NVIDIA/cuda-rust) (Apache-2.0),
+which its README calls an early-stage research project, with API breakage to be expected.
+- **cuda-oxide** is the SIMT track: a custom rustc codegen backend that compiles Rust kernels ahead of time to PTX, with
+  threads, blocks, warps and shared memory in your hands. It pins its own Rust toolchain, needs CUDA Toolkit 13.0+ and a
+  13.x driver, and is tested on Linux (Ubuntu 24.04).
+- **cutile-rs** is the Tile track: you work on tensor tiles, with no thread indexing or synchronization to write, and it
+  compiles at first launch to the CUDA Tile IR. It runs on stable Rust 1.89+ and lists Ampere, Ada, Hopper and Blackwell
+  GPUs (consumer Blackwell, sm_120, included). It is on crates.io as `cutile` 0.4.0. (The crates.io name `cuda-oxide` belongs to an unrelated
+  2021 crate.)
+- Both tracks carry Rust's borrow rules across the kernel launch, and kernels from the two can be used together.
+
+For the studio: the rig's RTX 5090 is sm_120, so cutile-rs lists our card, but both tracks are tested on Linux only and
+the rig runs Windows; trying it would mean WSL or a Linux box. Relevance: watch.
+
+**JAX on a Colab TPU: a fix to flax.nnx's import, then a first training smoke.** The single-chip TPU runtime we opened on 2026-10-29 (a v5e-1) comes with jax, jaxlib and flax all at 0.11.2, and `flax.nnx` fails to import against that jax:
+
+- flax 0.11.2 asks for `jax.array_ref` and `jax.ArrayRef`, which jax 0.11.2 now calls `new_ref` and `Ref`. Its fallback names aren't in jax 0.11.2 either.
+- It also calls `jax.core.get_opaque_trace_state`, which jax removed from `jax.core` in 0.11.0 (it now lives in `jax.extend.core`).
+
+We worked around those two breaks with three one-line aliases, set only inside our own training process. An outside review read both libraries' sources and judged them the same objects under new names. That holds for these two versions only, and images change, so check yours. A newer flax pinned in your own environment, matched to the image's jax, is the cleaner fix.
+
+With the aliases in place, a first training smoke passed on one v5e chip. It was a 10-million-parameter `flax.nnx` transformer, run for 50 steps after a CPU stub of the same path:
+
+- The first step took 4.71 s with compilation, and a steady step took 1.69 ms. A model this small is limited by dispatch, so that is not a chip-speed figure.
+- Peak TPU memory was 345 MB of 15.75 GiB.
+- JAX's persistent compile cache cut a second process's compile to 0.65 s, 7.2 times faster.
+- An Orbax checkpoint restored bit for bit, all 208 arrays, with the same next-step loss.
+- The session ran under two minutes, at 2.92 units an hour.
+
+It trained on one fixed random batch, so the loss falling to 0.01 is memorisation: it shows the loop works and nothing more. Next is a 181-million-parameter nanoGPT on the same card, where throughput starts to mean something. Sources: [flax 0.11.2](https://github.com/google/flax/blob/v0.11.2/flax/nnx/variablelib.py) and [jax 0.11.2](https://github.com/jax-ml/jax/blob/jax-v0.11.2/jax/core.py).
+
+**RunPod's REST v1 is closing.** RunPod's public APIs have enforced a rate limit since 2026-10-27, and REST v1 is
+being wound down. Its limit is 180 requests a minute, 3,600 an hour and 43,200 a day now. Those halve on November 2
+(90, 1,800, 21,600) and halve again on November 9 (45, 900, 10,800). From November 16 every v1 request returns
+410 Gone. Clients should back off exponentially on a 429. REST v2 is the long-term replacement, and GraphQL for
+integrators is due to be deprecated in early 2027. (Source: RunPod's service notice to account holders.)
+
+For the studio: offrig already calls REST v2 for pods, volumes and the GPU catalog, and waits for `Retry-After` on a
+429 when it creates a pod. It still reads the account balance and the data-centre list over GraphQL, which has to move
+before early 2027.
+
+## The daily watch
+
+4 new since [issue 33](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-33/).
+
+- **PyTorch 2.15.0** (2026-10-28): the deep-learning framework's new release makes CUDA 13.2 the default wheel (13.4 alongside), adds `torch.compile` on Python 3.15, and says 4-bit NVFP4 decoding is faster on Blackwell GPUs. Breaking: `import torch` now runs once per process, and its distributed checkpoints won't load in 2.14 ([release notes](https://github.com/pytorch/pytorch/releases/tag/v2.15.0)).
+- **llama.cpp v0.8.0** (2026-10-28): a tagged stable release of the local inference engine; its numbered builds are pre-releases. It adds Liquid AI's LFM3 models (including speech in and out), `/v1/realtime` and `/tts` server endpoints, a selectable reasoning effort, and Nemotron 3.5 Embed support. Scripts beware: the CLI now requires `--model` or `-hf` ([release notes](https://github.com/ggml-org/llama.cpp/releases/tag/v0.8.0)).
+- **Ollama v0.40.13** (2026-10-28): Liquid AI's LFM3 models, including the text-to-speech `lfm3-audio` and `lfm3-omni`, now run on Linux and Windows, where 0.40.12 had them on Apple silicon only. That makes a small local speech model a one-line pull on most machines; check the model's licence before you build on it ([release notes](https://github.com/ollama/ollama/releases/tag/v0.40.13)).
+- **NVIDIA skills catalogue** (2026-10-28): NVIDIA's public catalogue of agent skills added 53 skills for its DOCA 3.5 networking bundle, six for the PVA SDK (vision accelerators), Omniverse Content Services 0.1.0, and Run:ai quota-audit, cluster-health and utilization-overview skills ([commit history](https://github.com/NVIDIA/skills/commits/main)).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@rnd-news/newsletter",
-  "version": "1.32.0",
+  "version": "1.34.0",
   "description": "The research newsletter of mcp-tool-shop: every published issue as Markdown and JSON.",
   "author": "mcp-tool-shop",
   "license": "MIT",
@@ -29,5 +29,5 @@
     "data/issues.jsonl",
     "issues/"
   ],
-  "contentSha256": "bfacf7a5fe936d3ac5110b59c47fd3ae3a9c1fa2c35ba534c17865ea4b9a406d"
+  "contentSha256": "cc606610d8793984c447de4a42125303930a2cfb79ea38f808f8876ca70a9734"
 }