@rnd-news/newsletter 1.30.0 → 1.32.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/data/issues.jsonl +2 -0
- package/index.json +27 -0
- package/issues/issue-31.md +106 -0
- package/issues/issue-32.md +110 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@ The research newsletter of mcp-tool-shop: what we measured, and what it means. E
|
|
|
4
4
|
in plain language, with the numbers, the charts, what held up and what didn't.
|
|
5
5
|
|
|
6
6
|
This package mirrors the published issues and is updated automatically after each issue goes live on the site.
|
|
7
|
-
It holds
|
|
7
|
+
It holds 30 issues. The newest is issue 32, 2026-10-27: [27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/).
|
|
8
8
|
|
|
9
9
|
- **Read it on the web:** https://mcp-tool-shop-org.github.io/rnd/
|
|
10
10
|
- **RSS:** https://mcp-tool-shop-org.github.io/rnd/newsletter/rss.xml
|
package/data/issues.jsonl
CHANGED
|
@@ -26,3 +26,5 @@
|
|
|
26
26
|
{"issue": 28, "date": "2026-10-23", "title": "Why our critic refuses good drafts", "description": "Our training-data critic sometimes refuses drafts that are fine. In one day we found three separate causes. The critic is never told what its three verdict labels mean. The planner sometimes hands the writer a brief that contradicts itself. And some refusals are the critic being right. Here's how we found each one, and what we're measuring next.", "highlights": ["<strong>The writer gets definitions, the critic doesn't.</strong> The writer is told what each verdict label means. The critic is given only the three names. When two runs of the same critic disagreed, 6 of the 7 flips were between the same two labels.", "<strong>A brief that contradicts itself.</strong> The planner asked for 7 items whose material must be silent, while also asking that the material match the statement through an equivalent wording. On 6 of the 7 the critic said the item teaches a different lesson.", "<strong>Sometimes the critic is right.</strong> Graded blind, 7 of 30 test items were ambiguous. One refused draft named a product the material never mentions.", "<strong>Found in a dialogue.</strong> These came out of an open research conversation with an outside model, not a checklist review."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-28/", "body_markdown": ":::caution[Correction, 2026-10-23]\nThe first version of this issue said all 10 writers given the contradictory side were told to keep the material\nsilent. Only the 7 cannot_tell writers were. It also gave the guest-cap item as a clean example of an unmentioned\ndetail, though the blind grading below found that item ambiguous; it's replaced by one graded sound. The tea label's\nwording is also made exact. No figure changed. The same day, the issue gained six new charts and diagrams, and a\nredrawn slot chart, designed by Antigravity and checked against the data by the newsletter desk.\n:::\n\nThe studio builds training data with local language models. A **writer** model drafts each item, and a **critic**\nmodel checks it. Many items end in a verdict about a short piece of **material** (a notice, a label, a code snippet)\nand a **statement** about it. There are three labels:\n- **supported:** the material shows the statement is true;\n- **unsupported:** the material rules it out;\n- **cannot_tell:** the material says nothing either way.\n\nThe writer is told which label its item should carry. The critic reads the material and the statement and gives its\nown label. When they disagree, the draft is refused. Each refusal costs a rewrite, and a refusal of a good draft also\nthrows away a good item. In [issue 21](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-21/) the critic refused too many good items. This issue\nfollows the refusals, and finds three causes.\n\n\n\n## Cause 1: the critic is never told what the labels mean\n\nThe writer's instructions spell out what cannot_tell means. For a cannot_tell item, the writer reads: \"The material\nmust be silent on the statement: it neither shows it nor rules it out.\"\n\nThe critic's instructions name the three labels and nothing more. Its verdict request holds that instruction, the\nmaterial and the statement. So where \"silent\" ends and \"rules it out\" begins is left to the critic's own reading.\n\nThat's the line where its answers wobble. An earlier study, Track A, ran the critic twice over the same 102 items on\nthe same kind of GPU. As [issue 27](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-27/) found, the seed never reached the model in those runs,\nso each run sampled differently. The two runs disagreed on **7 of 102** verdicts, and **6 of the 7** were between\nunsupported and cannot_tell. The seventh went from unsupported to supported.\n\nTrack A also has 43 **intact items**, which the study's design left unchanged as controls. Over the two runs, the critic refused 3 of them at least once. Two of those refusals happened in one\nrun only, which is noise. The third was refused both times.\n\n\n\n## Cause 2: a brief that contradicts itself\n\nT10, the lesson in our pilot, teaches one thing: when a size, quantity or scope word in the statement differs from\nthe material (\"at least\", \"exactly\", \"every\"), check the exact size the statement claims. Each slot in the plan also\ngets a **side** of that lesson to teach. One is \"judges the exact size\". Another is \"an equivalent wording still\nmatches\": the statement says the same thing, phrased differently.\n\nThat second side can't go with cannot_tell. If an equivalent wording in the material matches, the answer is\nsupported. Yet in one writing wave of 60 slots, the planner gave that side to **7 cannot_tell slots and 3 unsupported\nslots**. Each of the 7 cannot_tell writers was told both to show a match and to keep the material silent. The 3\nunsupported writers weren't given the silence line, and the critic voted their items T10 every time.\n\n\n\nThe critic also votes three times on which lesson an item teaches. Here's what it said about the 12 cannot_tell slots:\n\n\n\n- **Planned with \"an equivalent wording still matches\":** on 6 of the 7, all three votes said T14, a different\n lesson: the material is silent on a detail, so don't fill it in. The writer had escaped the contradiction by asking\n about something the material never mentions:\n - how to store a cleaner;\n - a tea's caffeine, on a label whose only quantity is its weight;\n - which floor a service elevator is on;\n - more than ten utensils, on a box label that lists \"Utensils\" with no count.\n- **Planned with \"judges the exact size\":** 3 of the 5 were voted T10 three times out of three. They have one shape:\n the material gives a loose amount (\"a few items are redundant\", \"several team members raised concerns\"), and the\n statement claims an exact one (\"exactly three\"). The other 2 weren't clean. One swapped the North Wing for the South\n Wing; the other stretched a week's schedule over a month.\n\nThe fix in the planner is simple: never pair a verdict with a side it contradicts. A second fix is an option in the\nwriter's brief. It asks for that loose-versus-exact shape on T10's cannot_tell items, and forbids swapped names and\nunmentioned topics. Both have been agreed but not built. The option will be tested against today's brief before it\nbecomes the default.\n\n## Cause 3: sometimes the critic is right\n\nBefore measuring how often the critic wrongly refuses, we needed to know which items are sound. So we built a panel\nof 30 items, 10 for each label, and had them graded **blind**: the grader saw the material, the statement and the\nlabel definitions, with no stored label and no critic output.\n\nTwo graders worked separately: Antigravity, in a fresh conversation, and the R&D seat. An item counted as **sound**\nonly if both graders and its stored label agreed, and neither grader flagged it as unclear. Anything else counted as\n**ambiguous**, with no tiebreak.\n\n\n\nOne ambiguous item shows the critic doing its job. Its stored label was supported, and the critic refused it. Both\ngraders read cannot_tell. The statement named a payroll portal that the material never mentions. That draft's flaw\nwas the writer's, and the critic was right to refuse it.\n\nAnother needs a judgment the definitions don't settle. The material is a complete input check with no upper limit,\nand the statement says the code caps guests at 50. If that function is the whole check, the material rules the\nstatement out. If it might be one piece of a larger system, the material is silent. The two graders split. No prompt\nchange or vote can make an item like that stable, so the bench counts such items apart.\n\nR&D had looked at three of the 30 items before grading, so its grades on those three weren't blind. Two of them are\namong the 7 ambiguous ones.\n\n**A key file corrected before any run.** Four panel items come from Track A's traps. On 3 of the 4, both graders\ndisagreed with the stored label. The reason was in the study's design document, written four days earlier. Those items are deliberate traps: the deciding fact\nis removed, so the true label becomes cannot_tell while the stored label keeps the old answer on purpose. The panel\nbuilder had copied the stored label. The key file now holds the true one, and keeps the stored value alongside. The\ngraders' disagreement is what caught it.\n\n## A measure we tried: hesitation\n\nThe critic thinks before it answers. We counted how often its thinking switches between the three labels, and called\nit **hesitation**. On the two Track A runs:\n\n\n\nThe gap is real, but hesitation is a weak alarm. The 10 most hesitant items held only 1 of the 7 flips, and the top\n20 held 4. And it can\nonly find doubts the critic writes down. A confident wrong refusal leaves no trace. So it's useful as a hint, not as a\ngate.\n\n## How this was found\n\nNone of this came from a checklist. It came from an open research dialogue with Antigravity, the outside model we use\nfor reviews. It began as follow-up questions on issue 27's results. Antigravity lined the cannot_tell drafts up\nagainst the plan, and spotted both the contradictory pairing and the missing definitions. It also suggested the\nhesitation measure, and grading a sample blind. We checked each claim against the code and the run records. One list of item names it gave was wrong, though\nits count was right. The figures in this issue are ours, recounted from those records.\n\nAntigravity also designed this issue's charts and diagrams, over three rounds with the newsletter desk, which\nrecounted every number in them from the same records.\n\n## What's next\n\n\n\n- **A 30-item bench.** It runs on one rented A100, with each of the 30 items × 3 seeds × 2 prompts: today's critic\n prompt, and the same prompt plus one sentence giving the writer's definitions. The decision rule is fixed in\n advance:\n - the definitions are recommended only if they cut false refusals on sound items by at least half;\n - and they accept no more than one extra item whose sound label they contradict.\n\n Not yet measured.\n- **The planner fix and the writer-brief option.** Agreed, not built. The option's test measures two things: how\n often the critic votes T10, and how many drafts it accepts.\n\nThe raw notes, the bench design and the graded panel are in the studio's research library, and will move into rnd\nwith the bench's results.\n"}
|
|
27
27
|
{"issue": 29, "date": "2026-10-24", "title": "A win that didn't replicate, and a test that couldn't see", "description": "We tested a fix for our critic's false refusals against a rule written in advance. The first run passed by one call. The replication failed. Then we found that every refusal in both runs landed on six items, most of them picked because the old prompt had failed on them before. Here's what the bench measured, what it couldn't, and why we've stopped deciding from it.", "highlights": ["<strong>Passed, then failed.</strong> Giving the critic the label definitions cut false refusals from 9 to 4 of 69 calls, clearing our pre-set bar by one call. On three new seeds the cut was 9 to 7, and the bar wasn't met.", "<strong>A trade, not a fix.</strong> The definitions ended the old prompt's refusals on cannot_tell items, 12 to 0, and started new ones on unsupported items, 1 to 6.", "<strong>Six items carried everything.</strong> On 17 of the 23 sound items, neither prompt refused once in 102 calls. Every refusal was on six items tied to the old prompt's past failures.", "<strong>No decision from this test.</strong> We've stopped making critic decisions from this panel until we have a test whose answers are known for certain."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-29/", "body_markdown": "The studio builds training data with local language models. A **writer** drafts each item and a **critic** checks\nit: it reads the material and a statement, and gives one of three verdicts, **supported**, **unsupported** or\n**cannot_tell**. When its verdict differs from the item's label, the item is refused. A **false refusal** throws away\na good item.\n\n[Issue 28](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-28/) found that the writer is told what each label means and the critic isn't. It\nended with a plan: test the critic's prompt as it is (**A**) against the same prompt plus the writer's definitions\n(**B**), on 30 blind-graded items, with the decision rule fixed before any run. This issue reports that test.\n\n## The rule, written first\n\nOf the 30 items, 23 were graded **sound**: both blind graders and the item's label agreed, and nobody flagged it\nunclear. The other 7 were ambiguous and counted apart. Each prompt ran on every item with three seeds, so 69 calls on\nsound items per prompt.\n\nB would be recommended only if both held:\n- **(a)** B's false refusals on sound items are at most half of A's;\n- **(b)** B accepts at most one more of the items whose label the graders contradict.\n\n## Passed by one call\n\nThe first run, on seeds 0–2, gave A 9 false refusals and B 4. Half of 9 is 4.5, so (a) held, by one call. (b) held at\nits limit. The rule said B.\n\nOne call is a thin margin, so we ran it again on seeds 3, 4 and 5, with nothing else changed.\n\n## Failed on the rerun\n\nOn the new seeds A refused 9 again, and B refused 7. That's well over half, so (a) failed. Pooled over all six\nseeds, A refused 18 of 138 calls and B 11. Half of 18 is 9, and 11 is more, so the pooled result fails too. **The rule\nsays keep A.**\n\n\n\nThe first run wasn't wrong. It was a favourable draw: two items, P20 and P26, happened to go B's way more often on\nthose three seeds than on the next three.\n\n## A trade, not a fix\n\nSplitting the refusals by the item's true label shows what the definitions actually do:\n\n| Sound items, by label | Calls per prompt | A refused | B refused |\n|---|---|---|---|\n| cannot_tell (10 items) | 60 | 12 | 0 |\n| supported (7 items) | 42 | 5 | 5 |\n| unsupported (6 items) | 36 | 1 | 6 |\n\n\n\nA's main error is calling a silent item **unsupported**. The definitions fix that completely. But B then calls\nsome unsupported items **cannot_tell**, an error A almost never makes. B moves the line between the two labels; it\ndoesn't make the critic better at both.\n\n## Six items carried everything\n\nThen we looked at where the refusals were. **On 17 of the 23 sound items, neither prompt refused once: 0 of 102 calls\neach.** Every refusal, in both prompts and all six seeds, was on the other six.\n\n\n\nThose six weren't a random draw. Five came from an earlier study, Track A, and were picked for this panel because the\nold prompt had already failed on them there: three it refused, two it flipped on between runs. The sixth, T10-C-09,\ncame in under a rule that took every item of its kind from a later run, but its refusal in that run was one of the\nfailures that started this study.\n\nSo the test was built, deliberately, around the old prompt's known failures, and the new prompt was designed to fix\nthose failures. That's a fair way to find out whether a fix works on the cases it was made for. It can't tell you\nwhether the fix helps or hurts anywhere else, and the 17 other items gave no signal either way.\n\nMore seeds won't change that. Antigravity, the outside model we work with, worked through the numbers before the\nrerun came back: almost all the spread in this result comes from differences between items, not from run-to-run\nnoise, so rerunning the same 23 items barely narrows it. Its estimate was that doubling the seeds would shrink the\nuncertainty by only about 6%.\n\n## What we're doing about it\n\n**We're not deciding anything about the critic from this panel.** The old prompt stays, and no run that the critic\nscores is being started. That hold lasts until we have a test whose answers are known for certain, rather than graded\nafter the fact.\n\nThe lesson we're taking is simple: **a test built from your failures can only tell you about those failures.**\n\nThe raw results, the scoring scripts and the graded panel are in the studio's research library, and every figure\nhere was recounted from the critic's own output files.\n"}
|
|
28
28
|
{"issue": 30, "date": "2026-10-25", "title": "25 October: a crash test, a lab for tests with proven answers, and our Colab cards measured", "description": "The first daily issue. We killed a rented machine mid-write to test Attestia's event store, and the log survived, but the kill didn't test the fix, so we tore the same file by hand. We started a lab that builds tests whose answers are proved before any wording is written. And we turned 21 rented-GPU windows of receipts into a profile of each Colab card, with what's still unmeasured.", "highlights": ["<strong>A crash test that didn't test the fix.</strong> A machine killed mid-write left a log that a fresh machine opened whole, but no line was torn, so the fix never ran. We tore the same file by hand: the old code lost an event and refused to open, the fixed code didn't.", "<strong>A lab for tests with proven answers.</strong> reasoning-dataset-lab maps a dataset before any item exists and proves each answer in code before a model writes a word. Its first pilot map has 24 cells, 12 of them buildable today.", "<strong>Each Colab card, measured.</strong> From 21 windows of receipts: what each card has, what it costs per hour, how often a session was lost, and what no receipt has measured yet."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/", "body_markdown": "This is the first of the daily issues: one email a day, with an article for each piece of the day's work worth\nreporting. Today has three.\n\n**In this issue:**\n\n1. [We killed a machine mid-write. The log survived, but the kill didn't test the fix](#we-killed-a-machine-mid-write-the-log-survived-but-the-kill-didnt-test-the-fix)\n2. [A lab for tests whose answers are proved first](#a-lab-for-tests-whose-answers-are-proved-first)\n3. [What each Colab card is, from the receipts](#what-each-colab-card-is-from-the-receipts)\n\n## We killed a machine mid-write. The log survived, but the kill didn't test the fix\n\nAttestia is one of the studio's three proof products. Its **event store** keeps a log as a plain file, one event per\nline. Each event carries the hash of the one before it, so the events form a chain. If the chain is broken anywhere,\nthe store refuses to open. That's on purpose: a log you can't trust is worse than no log.\n\nWe want to use that store as the run log for experiments on rented machines, and rented machines vanish. So before\ntrusting it, we tried to break it.\n\n### The bug\n\nWe found the problem on our own machine first, before spending anything. A writer that's killed partway through\nwriting a line leaves a **torn** last line: half an event, with no line ending. Then:\n\n1. A new writer opens the store. It skips the torn line, checks the chain up to the last whole event, and opens.\n2. Its first new event is written straight after the torn fragment, on the same line, because nothing put a line\n ending in between.\n3. The next time anything opens the store, it skips that glued line too, so the new event is gone. The event after\n it now points at a predecessor that isn't there, and the chain is broken.\n4. With checking on, which is the default, the store refuses to open at all.\n\nSo one crash followed by one honest restart turned a good log into one that wouldn't open.\n\n### The fix\n\nThe fix is small. When the store opens a file that doesn't end in a line ending, it writes one before its first new\nevent. The torn fragment stays in the file on a line of its own and is skipped every time. Nothing is cut off or\nrewritten, so the store keeps its rule that the file only ever grows.\n\n### A real kill\n\nThen we tested it for real, on a rented Colab machine running on CPU:\n\n- a writer added 10 events a second for an hour;\n- a copy of the store was taken off the machine every 60 seconds;\n- at 3,654 seconds we killed the machine outright, with no warning to the writer;\n- 56 seconds later, a fresh machine opened the last copy, with checking on.\n\nThe copy held **36,498 events, all valid**. The new machine carried on writing and added 27,000 more. The final log\nheld **63,498 events: a valid chain, no gaps and no duplicates**. Its first 36,498 events are byte for byte the copy\nit started from, and its first new event links to the copy's last.\n\n\n\n### What the kill didn't test\n\nTwo things were true that a headline would skip.\n\n**The fix never ran.** The copy we resumed from ended on a whole line. There was no torn fragment for the fix to deal\nwith. The run proved that a hard kill and a resume on a new machine work, but it didn't prove the fix.\n\n**Some events died with the machine.** Its last progress report, one second before the kill, counted at least\n36,516 events. The copy held 36,498. So at least 18 events, about two seconds of writing, were on a machine that no\nlonger exists. That isn't a store bug: a resume from a copy can only have what the copy had. Anyone running a log\non a machine that can disappear has to decide how much of it they can afford to lose, and copy at least that often.\n\n### The fix, tested on the same file\n\nSo we tore the file ourselves. We took the copy from the real kill and added the first 328 bytes of a real 657-byte\nevent, with no line ending: exactly what a writer killed mid-line leaves. Then we resumed it twice, once with the\nold code and once with the fixed code, on our own machine.\n\n- **The old code** skipped the fragment and opened. It added 60 events, and the first was glued onto the torn line\n and lost. The chain broke at position 36,500, and the store then refused to open.\n- **The fixed code** skipped the fragment, added the same 60 events, and lost none. The chain was valid with no gaps,\n and the store opened again cleanly.\n\n\n\n### What it cost\n\n| Step | Where | Cost |\n|---|---|---|\n| A short trial run of the whole procedure | Colab, CPU | 0.01 units |\n| The real kill and resume | Colab, CPU | 0.14 units |\n| The torn-file test, old code and fixed | our own machine | nothing |\n\n### Where it stands\n\n- **Proven:** a hard kill, and a resume on a fresh machine, with every event in the copy kept and the chain intact.\n- **Proven separately:** the fix, on that same file with a torn line added by hand.\n- **Not proven:** that a real kill tears a line, or how often. Ours didn't.\n- **By design:** events written after the last copy are lost with the machine.\n\nThe fix shipped today in **Attestia 2.1.2**, on npm and as a GitHub release. The run logs, both stores and the\ntorn-file test are in the studio's records, and every figure here was recounted from the store files themselves.\n\n## A lab for tests whose answers are proved first\n\n[Issue 29](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-29/) ended with a decision: we stopped deciding anything about our critic from that\ntest panel until we have a test whose answers are known for certain. This article is the start of that test.\n\n**reasoning-dataset-lab** is a new public repository at\n[mcp-tool-shop-org/reasoning-dataset-lab](https://github.com/mcp-tool-shop-org/reasoning-dataset-lab). It builds\npuzzles, quizzes and other tests. Its rules are the reverse of how a test is often made, where a model writes the\nitems and someone checks them afterwards:\n\n- **Map first.** Before any item exists, a `map.yaml` says what the dataset is: its axes (the skill tested, the kind of\n item, the difficulty, the subject, and the wrong answer each item is built to catch), every cell those axes make,\n how many items each cell gets, and the rules the finished set must pass. The map is hashed, so it can't drift\n quietly.\n- **Prove, then word.** Code builds each item's underlying problem and proves its answer before a language model\n writes any prose around it. An item kind whose answer can't be proved that way is marked `judged`, and gets in only\n if judges from several model families all agree.\n- **Show the gaps.** A cell the generator can't build yet stays in the map with a capacity of 0 and a reason. The\n map doesn't shrink to make coverage look complete.\n- **Provenance on every item,** so any item can be traced back to the map, the source material and the code that\n made it.\n\n### The first pilot map\n\nThe first map, in the repository's `examples/`, is for a single skill: reading a stated quantity correctly. Its wrong\nanswers are four kinds of misreading: an exact amount read as a minimum, an exact amount read as a maximum, and a\nminimum or a maximum read as exact. The subjects are lamps and cups, at three difficulties.\n\nThat makes **24 cells**. **12 are buildable today**, with 5 items each: 60 items, 48 for training and 12 held out for\nevaluation. The other **12 are in the map at capacity 0**, because the generator can't build those misreadings yet.\n\nThe rules for the finished set were fixed before anything is generated:\n\n- every answer key proved (a proof rate of 1.0);\n- at least half the items passing in each class;\n- a sample of 20 items read by a person, with at most 1 defect.\n\n\n\n### Where the lab stands\n\nThe repository has a map checker, the project files and their fingerprint, and, as of today, the first part of the\npilot: the interface every generator follows, and an adapter that drives the studio's existing puzzle generator\nunchanged. No pilot items have been built yet. The pilot's first batch, with its numbers and a sample of items, is\nthe next step.\n\n## What each Colab card is, from the receipts\n\nWe rent GPU time on Google Colab in **windows**: a session on one card, booked and closed by a script that keeps a\nreceipt of everything it did. Before planning the next set of measurements, we read every receipt we have, 21\nwindows and the probes around them, and wrote down what each card actually is. Every value below comes from a\nreceipt. Where no receipt measures something, we say so.\n\n### The cards\n\n| Card | Memory | Rate (units an hour) | Sessions | What went wrong |\n|---|---|---|---|---|\n| A100, 40 GB | 40,960 MiB | 5.3 | 62 or more | 3 sessions lost mid-run on 23 October, plus one machine dropped mid-download and one session pruned earlier; 3 full four-hour windows with no loss |\n| A100, 80 GB | 81,920 MiB | 6.77 | about 8 | 1 session ended at 132 seconds when we stopped it (from 2 windows; the other ran 10,732 seconds clean) |\n| RTX PRO 6000 (G4) | 97,887 MiB | 8.9 | about 9 | none (from 2 windows); one ran 8,680 seconds clean |\n| L4 | 23,034 MiB | 1.54 | about 8 | only one window used it |\n| T4 | not measured | about 1.07, worked out, not read | assigned 6 times | never used |\n| H100 | — | — | never assigned | refused 5 times, \"Service Unavailable\" |\n| CPU (standard) | 12.3 GB of RAM free | 0.08 | 41 or more | the download check rejected 5 of 52 tries as too slow |\n\nThe plan allows up to 20 sessions at once: 3 each of the five GPU cards we can be assigned (the two A100s, the RTX PRO 6000, the L4 and\nthe T4) and 5 CPU machines. The next request is refused.\n\n### Faster isn't always cheaper\n\nOn the same job, a critic model checking items:\n\n- the **RTX PRO 6000** took a median 46.7 seconds a call against the 40 GB A100's 66.5: about 1.4 times as fast, at\n 1.68 times the price;\n- the **80 GB A100** ran the model 1.11 times as fast as the 40 GB one, at 1.28 times the price, with identical\n output on all 178 calls compared.\n\nSo for that job, the cheaper card cost less per result. That holds for that job only. Another model or another size\ncan change it.\n\n### What no receipt measures yet\n\n- the time to the first token, on every card;\n- how fast a model downloads through Ollama (no receipt has an Ollama log);\n- how long a card takes from the request to ready;\n- almost everything about the T4, which has never run a job, and the L4, which ran one window.\n\nThose gaps are the reason for the next sweep, **C2**, which measures the cards the same way: the largest model\ncontext each holds, the speed at several prompt lengths and loads, and the throughput of small embedding models.\n\n### Licences first\n\nBefore a model's first window, it needs an entry in our licence registry, read at its source. Six of the models in\nthe C2 sweep had none, so we read them: **phi3.5** (MIT), **nomic-embed-text-v1.5** (Apache-2.0),\n**bge-base-en-v1.5** (MIT), **bge-reranker-v2-m3** (Apache-2.0), **nli-deberta-v3-base** (Apache-2.0) and\n**Qwen3-Embedding-0.6B** (Apache-2.0). No card or linked page restricts our use. Only phi3.5 ships a licence file;\nthe other five rest on the licence their model card names. Each entry keeps a hashed snapshot of the page and an\narchive.org copy.\n"}
|
|
29
|
+
{"issue": 31, "date": "2026-10-26", "title": "26 October: our speed table was timing a cache, plus three releases and a newsletter that travels", "description": "Three small test runs on a rented T4 showed that interlock's generate ladder timed a prompt cache instead of the prompt, and counted queued requests as parallel ones. We fixed it in three releases and corrected the docs. Also: Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, and this newsletter now publishes itself to Hugging Face, npm, PyPI and GitHub.", "highlights": ["<strong>A correction.</strong> In interlock 1.0.5 and earlier, the generate ladder's time to first token measured a prompt cache, and its speed with several requests at once was close to the number of requests times too high. Neither is used any more.", "<strong>Found cheaply.</strong> Three short runs on a rented T4 card found it, one defect at a time: a prompt that looped, prompts cut in half, a cache, a queue, and cells mislabelled when the card ran out of room.", "<strong>Three releases.</strong> Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, Cognate with signed requests.", "<strong>The newsletter, everywhere.</strong> Every issue now lands on Hugging Face, npm and PyPI, and gets its own GitHub release."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/", "body_markdown": "Today's lead is a correction to numbers our own tool produced. It's followed by three releases, and a change to how\nthis newsletter reaches you.\n\n**In this issue:**\n\n1. [Our speed table was timing a cache](#our-speed-table-was-timing-a-cache)\n2. [Three proof products ship](#three-proof-products-ship)\n3. [The newsletter learns to travel](#the-newsletter-learns-to-travel)\n\n## Our speed table was timing a cache\n\n**interlock** picks a model, an engine and a device for a job from speeds it has measured. To measure rented\ncards, it runs a **generate ladder**: the same model at prompt lengths from 512 to 32,768 tokens, with one, four\nand eight requests at a time, recording how long the first token takes and how fast the rest come.\n\nBefore spending real money on a full sweep, we ran short trial **rungs** on Colab's cheapest card, a T4. Three of\nthem found five problems, and every one was in our measurement, not in the card.\n\n### Rung 1: a prompt that looped\n\nThe ladder built its prompt by repeating one word. One model, phi3.5, kept repeating it until Ollama stopped the\nrun, so some cells timed a loop instead of an answer. The errors were also filed under the wrong label. **interlock\n1.0.5** builds a varied prompt, calibrated to the length asked for, and gives an unmeasured cell its own label so it\ncan never read as a speed.\n\n### Rung 2: cut in half, cached, and queued\n\nThe second rung found three problems at once, all confirmed in Ollama's own source:\n\n- **Long prompts were cut in half.** The prompt landed about 1% over its target, the window left no room for that, and\n Ollama halved the prompt at 16k and 32k tokens. Those cells came back unmeasured.\n- **The first token came from a cache.** Ollama keeps the start of the last prompt, and the ladder sent the same\n prompt for its check call, its warm-up and every timed run. So each timed run read a remembered prompt: **15 ms at\n 512 tokens and 33 ms at 8,192**, which is too fast to be real.\n- **Requests queued instead of running together.** Ollama serves one request at a time unless told otherwise. The\n ladder sent four or eight at once, they waited in line, and the way it added up their speeds left out the wait. So\n \"four at once\" read close to four times one stream.\n\n**interlock 1.0.6** fixes all three. Each timed call begins with a short prefix of its own,\nso nothing is cached. The server gets as\nmany slots as the cell has requests. And combined speed is total tokens over the pass's decode window, from the first decode start to the last\nfinish. A prompt-read time that doesn't grow with the prompt now counts as unmeasured.\n\nWe check every change on our own machine first, where a run costs nothing. Measured that way there (on an RTX\n5090), the time to read the prompt grows as it should: **23 ms at 512 tokens, 100 ms\nat 2k, 463 ms at 8k, 1,244 ms at 16k and 3,791 ms at 32k.**\n\n\n\n### Rung 3: out of room, mislabelled\n\nOn the fixed ladder, some long, busy cells ran out of room. Part of the model didn't fit on the card and ran on\nthe processor until the request timed out, or the machine's memory filled and the run was killed. Those are honest\n\"too big for this card\" results, but they were recorded as a timeout or as unmeasured. **interlock 1.0.7** reads the\nload from Ollama's log before any timed call, records those cells as out of memory, and scales each cell's timeout\nfrom a calibration call of its own.\n\n### What's still valid, and the correction\n\nThe speed of one stream of tokens and the largest context each card holds were not affected; the embedding\nthroughput runs on a different path and wasn't part of these defects. Rows from 1.0.4 and earlier also overstate\ntheir prompt length about twice over, so their one-stream speeds belong to a prompt about half as long as labelled. The first-token times and the speeds with more than one request at a time, from 1.0.5 and earlier, were\nnot valid measures. interlock no longer uses them, and its README and handbook carry a correction notice.\n\nThe trial runs so far, three on a CPU and four on the T4, cost about 1.3 of the 800 units set aside for this sweep. With the real timings, the whole sweep is\nnow priced at about 328 to 347 units.\n\n## Three proof products ship\n\nOn 25 October, each of the studio's three proof products had a release:\n\n- **Attestia 2.1.3** cleared every open advisory in its production dependencies by upgrading them, and its chain\n observer now stops with a clear error when a chain returns an integer too large to hold exactly. (2.1.2, earlier\n the same day, carried [issue 30](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/)'s crash fix.)\n- **RepoMesh 2.5.0** adds a required test gate on every change, and its anchor posting stays on the test network: it\n won't connect unless the network is testnet and the server is a known testnet server.\n- **Cognate 1.0.0** reaches 1.0. Every write to its server is signed with the caller's own key, checked for its time\n and for replay, and recorded under the signer's name; every route acts only in the signer's tenant; and decisions\n cite the evaluations they rest on. From 1.0, a breaking change to its API or formats means a new major version.\n\n## The newsletter learns to travel\n\nFrom issue 30 on, every issue publishes itself to four more places as soon as it's live on the site:\n\n| Where | What you get |\n|---|---|\n| Hugging Face: [mcp-tool-shop/rnd-newsletter](https://huggingface.co/datasets/mcp-tool-shop/rnd-newsletter) | every issue as a dataset row and a Markdown page |\n| npm: [`@rnd-news/newsletter`](https://www.npmjs.com/package/@rnd-news/newsletter) | the issues as JSON and Markdown |\n| PyPI: [`mcptoolshop-rnd-newsletter`](https://pypi.org/project/mcptoolshop-rnd-newsletter/) | `rnd_newsletter.latest()`, `issue(n)` and `issues()` |\n| GitHub releases | one per issue, with the issue attached |\n\nAll four are built from the same published files as the site. A run with nothing new publishes nothing, and a\ncorrection to an issue already out becomes a new revision, not a silent change.\n"}
|
|
30
|
+
{"issue": 32, "date": "2026-10-27", "title": "27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp", "description": "A spotlight issue on three tools the studio ships in the open: AI Jam Sessions, which teaches an AI to play, sing and listen; interlock, which went from 1.0.8 to 1.0.10 while it measured its first rented card; and XRPL Camp, which teaches the XRP Ledger in one sitting. Plus one short note from today's runs.", "highlights": ["<strong>AI Jam Sessions.</strong> An MCP server that teaches an AI to play piano and guitar, sing and listen: 109 annotated songs, 56 tools, and three hymns sung whole by a synthetic voice.", "<strong>interlock 1.0.8 to 1.0.10.</strong> Three releases made from what the first rented-card runs showed: a time limit on every cell, a restart when one hangs, and noise recorded instead of thrown away.", "<strong>XRPL Camp 1.5.</strong> Learn the XRP Ledger in about ten minutes on the test network, and keep a proof of what you did that anyone can check."], "url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/", "body_markdown": "Today's Colab work produced little to report, so this issue is a spotlight: three of the studio's tools that are\npublic and shipping, what each one does, and what changed in it lately.\n\n**In this issue:**\n\n1. [AI Jam Sessions: an AI that plays, sings and listens](#ai-jam-sessions-an-ai-that-plays-sings-and-listens)\n2. [interlock 1.0.8 to 1.0.10](#interlock-108-to-1010)\n3. [XRPL Camp: the XRP Ledger in one sitting](#xrpl-camp-the-xrp-ledger-in-one-sitting)\n4. [A short note from today's runs](#a-short-note-from-todays-runs)\n\n## AI Jam Sessions: an AI that plays, sings and listens\n\n[AI Jam Sessions](https://github.com/mcp-tool-shop-org/ai-jam-sessions) is an MCP server that teaches a language model\nto play piano and guitar, and to sing. A model can read text, but it has no ears and no fingers, so the server gives\nit senses it can use:\n\n- **Reading:** 109 songs across 12 genres, every one annotated with its history, its structure bar by bar, and what\n it teaches.\n- **Hearing:** six sound engines, from a sampled piano to a modelled guitar and a vocal tract, playing through real\n speakers.\n- **Seeing:** a piano roll the model can read back, a guitar tablature editor, and a browser cockpit.\n- **Remembering:** a practice journal that carries over from one session to the next.\n- **Listening:** point it at a recording and it measures what is there, with the same kind of tools it uses on the\n score.\n\nIt has 56 MCP tools in all, and installs from npm as `@mcptoolshop/ai-jam-sessions`; the current release is 2.8.0.\n\n### Three hymns, sung whole\n\nVersion 2.8.0 added whole songs sung by a synthetic voice on the piano's clock: Amazing Grace and America the\nBeautiful, every verse. The Battle Hymn of the Republic followed on the [landing\npage](https://mcp-tool-shop-org.github.io/ai-jam-sessions/#listen). The performances were rendered offline on a rented GPU,\nnot by the server's own sound engines. Each is built from sixteen takes of a singer conditioned on the score, with one take chosen for each phrase and checked for timing and pitch by two\ninstruments each, then passed by ear.\n\n| | Amazing Grace | America the Beautiful | Battle Hymn |\n|---|---|---|---|\n| syllables within 40 ms of the score | 107 of 112 | 217 of 224 | 285 of 414 |\n| notes within 50 cents | 135 of 140 | 218 of 224 | 368 of 424 |\n\nThe Battle Hymn's quick dotted notes are left on the singer's own timing, which is where most of its timing misses\ncome from. The voice is [SoulX-Singer](https://github.com/Soul-AILab/SoulX-Singer) (Apache-2.0) with its own example\ntimbre, imitating no real singer, and all three hymns are in the public domain.\n\n### Data others can use\n\nTwo datasets from the same work are public on Hugging Face:\n\n- [jam-actions-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-v0): 57 multi-turn MCP tool-use traces\n over four classical piano works, for teaching a model to answer questions about music by calling tools rather than\n guessing.\n- [jam-actions-acoustic-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-acoustic-v0): 72 records over\n audio rather than notation, each label checked against what the analysis tools actually return. It ships no audio:\n each record carries a recipe and the hash of the waveform it produces.\n\nBoth carry their licences and sources in their cards. Each was corrected on 25 September when an audit found files\nwhose arrangement licence couldn't be established, and the records built from them were withdrawn.\n\n## interlock 1.0.8 to 1.0.10\n\n[interlock](https://github.com/mcp-tool-shop-org/interlock) picks a model, an engine and a device for a job from\nspeeds it has measured. [Issue 31](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/) told how its first runs on a rented T4 card exposed\ndefects in how it measured, fixed in 1.0.5 to 1.0.7. The next three releases came from the same runs:\n\n- **1.0.8: nothing waits for ever.** Every cell has a wall-time cap, and a call that reads the card's memory now has\n a deadline: one run had gone silent for 56 minutes waiting on it. The harness prints a line at each step, so the\n next hang names its own call. Rig checks get their own identity instead of borrowing a Colab card's name. And\n `interlock route --concurrency N` ranks results for the number of requests you'll actually send.\n- **1.0.9: a hung cell is cleaned up.** When a cell's deadline fires, the server is restarted and any late answer\n from the abandoned cell is refused. The fallback reading of the card's memory is checked strictly, and an answer it\n can't trust is recorded as unmeasured.\n- **1.0.10: noise is recorded, not used to discard.** A check meant to catch a cached prompt was also discarding real\n cells, because the time to read the prompt varies from pass to pass. Each measured row now records that spread\n instead, and the cells are kept.\n\ninterlock 1.0.10 is on PyPI as `mcptoolshop-interlock` and needs Python 3.13 or newer.\n\n## XRPL Camp: the XRP Ledger in one sitting\n\n[XRPL Camp](https://github.com/mcp-tool-shop-org/xrpl-camp) teaches the XRP Ledger by doing it: real transactions on\nthe ledger's test network, in about ten minutes, ending with a proof of what you did that anyone can check against\nthe ledger. It's on PyPI as `xrpl-camp` and npm as `@mcptoolshop/xrpl-camp`, both at 1.5.2.\n\n- **1.5.0** wired in the last parts of the course: a walkthrough of the reserve an account owes for the things it\n owns, a plain-text certificate you can keep and share, and dated receipts each time a proof is verified.\n- **1.5.1 and 1.5.2** tightened the edges. A submission whose result comes back missing is no longer taken as a\n success: the camp looks the transaction up on the ledger and reports what it finds. The reserve step says plainly\n when a change couldn't be measured yet. And the standalone downloads no longer bundle a leftover install record from\n their build.\n\n## A short note from today's runs\n\nOn today's T4 runs, the Colab machine's memory limit stopped some of the largest cells: when a model's working cache\nspilled out of the card into the machine's memory, the limit killed the model server at about 11.5 GB of its own\nmemory. In three runs the process running our harness was killed a moment later as well (two seen directly, one\ninferred from its logs); the rows written up to that point were kept, but the rest of that step was lost. The run\nlauncher now notices a dead run and stops it within about two minutes, instead of waiting out its full time. A check\nthat refuses such cells before they load is being built for interlock 1.0.11.\n"}
|
package/index.json
CHANGED
|
@@ -392,5 +392,32 @@
|
|
|
392
392
|
],
|
|
393
393
|
"url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/",
|
|
394
394
|
"file": "issues/issue-30.md"
|
|
395
|
+
},
|
|
396
|
+
{
|
|
397
|
+
"issue": 31,
|
|
398
|
+
"date": "2026-10-26",
|
|
399
|
+
"title": "26 October: our speed table was timing a cache, plus three releases and a newsletter that travels",
|
|
400
|
+
"description": "Three small test runs on a rented T4 showed that interlock's generate ladder timed a prompt cache instead of the prompt, and counted queued requests as parallel ones. We fixed it in three releases and corrected the docs. Also: Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, and this newsletter now publishes itself to Hugging Face, npm, PyPI and GitHub.",
|
|
401
|
+
"highlights": [
|
|
402
|
+
"<strong>A correction.</strong> In interlock 1.0.5 and earlier, the generate ladder's time to first token measured a prompt cache, and its speed with several requests at once was close to the number of requests times too high. Neither is used any more.",
|
|
403
|
+
"<strong>Found cheaply.</strong> Three short runs on a rented T4 card found it, one defect at a time: a prompt that looped, prompts cut in half, a cache, a queue, and cells mislabelled when the card ran out of room.",
|
|
404
|
+
"<strong>Three releases.</strong> Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, Cognate with signed requests.",
|
|
405
|
+
"<strong>The newsletter, everywhere.</strong> Every issue now lands on Hugging Face, npm and PyPI, and gets its own GitHub release."
|
|
406
|
+
],
|
|
407
|
+
"url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/",
|
|
408
|
+
"file": "issues/issue-31.md"
|
|
409
|
+
},
|
|
410
|
+
{
|
|
411
|
+
"issue": 32,
|
|
412
|
+
"date": "2026-10-27",
|
|
413
|
+
"title": "27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp",
|
|
414
|
+
"description": "A spotlight issue on three tools the studio ships in the open: AI Jam Sessions, which teaches an AI to play, sing and listen; interlock, which went from 1.0.8 to 1.0.10 while it measured its first rented card; and XRPL Camp, which teaches the XRP Ledger in one sitting. Plus one short note from today's runs.",
|
|
415
|
+
"highlights": [
|
|
416
|
+
"<strong>AI Jam Sessions.</strong> An MCP server that teaches an AI to play piano and guitar, sing and listen: 109 annotated songs, 56 tools, and three hymns sung whole by a synthetic voice.",
|
|
417
|
+
"<strong>interlock 1.0.8 to 1.0.10.</strong> Three releases made from what the first rented-card runs showed: a time limit on every cell, a restart when one hangs, and noise recorded instead of thrown away.",
|
|
418
|
+
"<strong>XRPL Camp 1.5.</strong> Learn the XRP Ledger in about ten minutes on the test network, and keep a proof of what you did that anyone can check."
|
|
419
|
+
],
|
|
420
|
+
"url": "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/",
|
|
421
|
+
"file": "issues/issue-32.md"
|
|
395
422
|
}
|
|
396
423
|
]
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
---
|
|
2
|
+
issue: 31
|
|
3
|
+
date: "2026-10-26"
|
|
4
|
+
title: "26 October: our speed table was timing a cache, plus three releases and a newsletter that travels"
|
|
5
|
+
description: "Three small test runs on a rented T4 showed that interlock's generate ladder timed a prompt cache instead of the prompt, and counted queued requests as parallel ones. We fixed it in three releases and corrected the docs. Also: Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, and this newsletter now publishes itself to Hugging Face, npm, PyPI and GitHub."
|
|
6
|
+
url: "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/"
|
|
7
|
+
highlights:
|
|
8
|
+
- "<strong>A correction.</strong> In interlock 1.0.5 and earlier, the generate ladder's time to first token measured a prompt cache, and its speed with several requests at once was close to the number of requests times too high. Neither is used any more."
|
|
9
|
+
- "<strong>Found cheaply.</strong> Three short runs on a rented T4 card found it, one defect at a time: a prompt that looped, prompts cut in half, a cache, a queue, and cells mislabelled when the card ran out of room."
|
|
10
|
+
- "<strong>Three releases.</strong> Attestia 2.1.3, RepoMesh 2.5.0 and Cognate 1.0.0 shipped, Cognate with signed requests."
|
|
11
|
+
- "<strong>The newsletter, everywhere.</strong> Every issue now lands on Hugging Face, npm and PyPI, and gets its own GitHub release."
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
Today's lead is a correction to numbers our own tool produced. It's followed by three releases, and a change to how
|
|
15
|
+
this newsletter reaches you.
|
|
16
|
+
|
|
17
|
+
**In this issue:**
|
|
18
|
+
|
|
19
|
+
1. [Our speed table was timing a cache](#our-speed-table-was-timing-a-cache)
|
|
20
|
+
2. [Three proof products ship](#three-proof-products-ship)
|
|
21
|
+
3. [The newsletter learns to travel](#the-newsletter-learns-to-travel)
|
|
22
|
+
|
|
23
|
+
## Our speed table was timing a cache
|
|
24
|
+
|
|
25
|
+
**interlock** picks a model, an engine and a device for a job from speeds it has measured. To measure rented
|
|
26
|
+
cards, it runs a **generate ladder**: the same model at prompt lengths from 512 to 32,768 tokens, with one, four
|
|
27
|
+
and eight requests at a time, recording how long the first token takes and how fast the rest come.
|
|
28
|
+
|
|
29
|
+
Before spending real money on a full sweep, we ran short trial **rungs** on Colab's cheapest card, a T4. Three of
|
|
30
|
+
them found five problems, and every one was in our measurement, not in the card.
|
|
31
|
+
|
|
32
|
+
### Rung 1: a prompt that looped
|
|
33
|
+
|
|
34
|
+
The ladder built its prompt by repeating one word. One model, phi3.5, kept repeating it until Ollama stopped the
|
|
35
|
+
run, so some cells timed a loop instead of an answer. The errors were also filed under the wrong label. **interlock
|
|
36
|
+
1.0.5** builds a varied prompt, calibrated to the length asked for, and gives an unmeasured cell its own label so it
|
|
37
|
+
can never read as a speed.
|
|
38
|
+
|
|
39
|
+
### Rung 2: cut in half, cached, and queued
|
|
40
|
+
|
|
41
|
+
The second rung found three problems at once, all confirmed in Ollama's own source:
|
|
42
|
+
|
|
43
|
+
- **Long prompts were cut in half.** The prompt landed about 1% over its target, the window left no room for that, and
|
|
44
|
+
Ollama halved the prompt at 16k and 32k tokens. Those cells came back unmeasured.
|
|
45
|
+
- **The first token came from a cache.** Ollama keeps the start of the last prompt, and the ladder sent the same
|
|
46
|
+
prompt for its check call, its warm-up and every timed run. So each timed run read a remembered prompt: **15 ms at
|
|
47
|
+
512 tokens and 33 ms at 8,192**, which is too fast to be real.
|
|
48
|
+
- **Requests queued instead of running together.** Ollama serves one request at a time unless told otherwise. The
|
|
49
|
+
ladder sent four or eight at once, they waited in line, and the way it added up their speeds left out the wait. So
|
|
50
|
+
"four at once" read close to four times one stream.
|
|
51
|
+
|
|
52
|
+
**interlock 1.0.6** fixes all three. Each timed call begins with a short prefix of its own,
|
|
53
|
+
so nothing is cached. The server gets as
|
|
54
|
+
many slots as the cell has requests. And combined speed is total tokens over the pass's decode window, from the first decode start to the last
|
|
55
|
+
finish. A prompt-read time that doesn't grow with the prompt now counts as unmeasured.
|
|
56
|
+
|
|
57
|
+
We check every change on our own machine first, where a run costs nothing. Measured that way there (on an RTX
|
|
58
|
+
5090), the time to read the prompt grows as it should: **23 ms at 512 tokens, 100 ms
|
|
59
|
+
at 2k, 463 ms at 8k, 1,244 ms at 16k and 3,791 ms at 32k.**
|
|
60
|
+
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
### Rung 3: out of room, mislabelled
|
|
64
|
+
|
|
65
|
+
On the fixed ladder, some long, busy cells ran out of room. Part of the model didn't fit on the card and ran on
|
|
66
|
+
the processor until the request timed out, or the machine's memory filled and the run was killed. Those are honest
|
|
67
|
+
"too big for this card" results, but they were recorded as a timeout or as unmeasured. **interlock 1.0.7** reads the
|
|
68
|
+
load from Ollama's log before any timed call, records those cells as out of memory, and scales each cell's timeout
|
|
69
|
+
from a calibration call of its own.
|
|
70
|
+
|
|
71
|
+
### What's still valid, and the correction
|
|
72
|
+
|
|
73
|
+
The speed of one stream of tokens and the largest context each card holds were not affected; the embedding
|
|
74
|
+
throughput runs on a different path and wasn't part of these defects. Rows from 1.0.4 and earlier also overstate
|
|
75
|
+
their prompt length about twice over, so their one-stream speeds belong to a prompt about half as long as labelled. The first-token times and the speeds with more than one request at a time, from 1.0.5 and earlier, were
|
|
76
|
+
not valid measures. interlock no longer uses them, and its README and handbook carry a correction notice.
|
|
77
|
+
|
|
78
|
+
The trial runs so far, three on a CPU and four on the T4, cost about 1.3 of the 800 units set aside for this sweep. With the real timings, the whole sweep is
|
|
79
|
+
now priced at about 328 to 347 units.
|
|
80
|
+
|
|
81
|
+
## Three proof products ship
|
|
82
|
+
|
|
83
|
+
On 25 October, each of the studio's three proof products had a release:
|
|
84
|
+
|
|
85
|
+
- **Attestia 2.1.3** cleared every open advisory in its production dependencies by upgrading them, and its chain
|
|
86
|
+
observer now stops with a clear error when a chain returns an integer too large to hold exactly. (2.1.2, earlier
|
|
87
|
+
the same day, carried [issue 30](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-30/)'s crash fix.)
|
|
88
|
+
- **RepoMesh 2.5.0** adds a required test gate on every change, and its anchor posting stays on the test network: it
|
|
89
|
+
won't connect unless the network is testnet and the server is a known testnet server.
|
|
90
|
+
- **Cognate 1.0.0** reaches 1.0. Every write to its server is signed with the caller's own key, checked for its time
|
|
91
|
+
and for replay, and recorded under the signer's name; every route acts only in the signer's tenant; and decisions
|
|
92
|
+
cite the evaluations they rest on. From 1.0, a breaking change to its API or formats means a new major version.
|
|
93
|
+
|
|
94
|
+
## The newsletter learns to travel
|
|
95
|
+
|
|
96
|
+
From issue 30 on, every issue publishes itself to four more places as soon as it's live on the site:
|
|
97
|
+
|
|
98
|
+
| Where | What you get |
|
|
99
|
+
|---|---|
|
|
100
|
+
| Hugging Face: [mcp-tool-shop/rnd-newsletter](https://huggingface.co/datasets/mcp-tool-shop/rnd-newsletter) | every issue as a dataset row and a Markdown page |
|
|
101
|
+
| npm: [`@rnd-news/newsletter`](https://www.npmjs.com/package/@rnd-news/newsletter) | the issues as JSON and Markdown |
|
|
102
|
+
| PyPI: [`mcptoolshop-rnd-newsletter`](https://pypi.org/project/mcptoolshop-rnd-newsletter/) | `rnd_newsletter.latest()`, `issue(n)` and `issues()` |
|
|
103
|
+
| GitHub releases | one per issue, with the issue attached |
|
|
104
|
+
|
|
105
|
+
All four are built from the same published files as the site. A run with nothing new publishes nothing, and a
|
|
106
|
+
correction to an issue already out becomes a new revision, not a silent change.
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
---
|
|
2
|
+
issue: 32
|
|
3
|
+
date: "2026-10-27"
|
|
4
|
+
title: "27 October: a spotlight on AI Jam Sessions, interlock and XRPL Camp"
|
|
5
|
+
description: "A spotlight issue on three tools the studio ships in the open: AI Jam Sessions, which teaches an AI to play, sing and listen; interlock, which went from 1.0.8 to 1.0.10 while it measured its first rented card; and XRPL Camp, which teaches the XRP Ledger in one sitting. Plus one short note from today's runs."
|
|
6
|
+
url: "https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-32/"
|
|
7
|
+
highlights:
|
|
8
|
+
- "<strong>AI Jam Sessions.</strong> An MCP server that teaches an AI to play piano and guitar, sing and listen: 109 annotated songs, 56 tools, and three hymns sung whole by a synthetic voice."
|
|
9
|
+
- "<strong>interlock 1.0.8 to 1.0.10.</strong> Three releases made from what the first rented-card runs showed: a time limit on every cell, a restart when one hangs, and noise recorded instead of thrown away."
|
|
10
|
+
- "<strong>XRPL Camp 1.5.</strong> Learn the XRP Ledger in about ten minutes on the test network, and keep a proof of what you did that anyone can check."
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
Today's Colab work produced little to report, so this issue is a spotlight: three of the studio's tools that are
|
|
14
|
+
public and shipping, what each one does, and what changed in it lately.
|
|
15
|
+
|
|
16
|
+
**In this issue:**
|
|
17
|
+
|
|
18
|
+
1. [AI Jam Sessions: an AI that plays, sings and listens](#ai-jam-sessions-an-ai-that-plays-sings-and-listens)
|
|
19
|
+
2. [interlock 1.0.8 to 1.0.10](#interlock-108-to-1010)
|
|
20
|
+
3. [XRPL Camp: the XRP Ledger in one sitting](#xrpl-camp-the-xrp-ledger-in-one-sitting)
|
|
21
|
+
4. [A short note from today's runs](#a-short-note-from-todays-runs)
|
|
22
|
+
|
|
23
|
+
## AI Jam Sessions: an AI that plays, sings and listens
|
|
24
|
+
|
|
25
|
+
[AI Jam Sessions](https://github.com/mcp-tool-shop-org/ai-jam-sessions) is an MCP server that teaches a language model
|
|
26
|
+
to play piano and guitar, and to sing. A model can read text, but it has no ears and no fingers, so the server gives
|
|
27
|
+
it senses it can use:
|
|
28
|
+
|
|
29
|
+
- **Reading:** 109 songs across 12 genres, every one annotated with its history, its structure bar by bar, and what
|
|
30
|
+
it teaches.
|
|
31
|
+
- **Hearing:** six sound engines, from a sampled piano to a modelled guitar and a vocal tract, playing through real
|
|
32
|
+
speakers.
|
|
33
|
+
- **Seeing:** a piano roll the model can read back, a guitar tablature editor, and a browser cockpit.
|
|
34
|
+
- **Remembering:** a practice journal that carries over from one session to the next.
|
|
35
|
+
- **Listening:** point it at a recording and it measures what is there, with the same kind of tools it uses on the
|
|
36
|
+
score.
|
|
37
|
+
|
|
38
|
+
It has 56 MCP tools in all, and installs from npm as `@mcptoolshop/ai-jam-sessions`; the current release is 2.8.0.
|
|
39
|
+
|
|
40
|
+
### Three hymns, sung whole
|
|
41
|
+
|
|
42
|
+
Version 2.8.0 added whole songs sung by a synthetic voice on the piano's clock: Amazing Grace and America the
|
|
43
|
+
Beautiful, every verse. The Battle Hymn of the Republic followed on the [landing
|
|
44
|
+
page](https://mcp-tool-shop-org.github.io/ai-jam-sessions/#listen). The performances were rendered offline on a rented GPU,
|
|
45
|
+
not by the server's own sound engines. Each is built from sixteen takes of a singer conditioned on the score, with one take chosen for each phrase and checked for timing and pitch by two
|
|
46
|
+
instruments each, then passed by ear.
|
|
47
|
+
|
|
48
|
+
| | Amazing Grace | America the Beautiful | Battle Hymn |
|
|
49
|
+
|---|---|---|---|
|
|
50
|
+
| syllables within 40 ms of the score | 107 of 112 | 217 of 224 | 285 of 414 |
|
|
51
|
+
| notes within 50 cents | 135 of 140 | 218 of 224 | 368 of 424 |
|
|
52
|
+
|
|
53
|
+
The Battle Hymn's quick dotted notes are left on the singer's own timing, which is where most of its timing misses
|
|
54
|
+
come from. The voice is [SoulX-Singer](https://github.com/Soul-AILab/SoulX-Singer) (Apache-2.0) with its own example
|
|
55
|
+
timbre, imitating no real singer, and all three hymns are in the public domain.
|
|
56
|
+
|
|
57
|
+
### Data others can use
|
|
58
|
+
|
|
59
|
+
Two datasets from the same work are public on Hugging Face:
|
|
60
|
+
|
|
61
|
+
- [jam-actions-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-v0): 57 multi-turn MCP tool-use traces
|
|
62
|
+
over four classical piano works, for teaching a model to answer questions about music by calling tools rather than
|
|
63
|
+
guessing.
|
|
64
|
+
- [jam-actions-acoustic-v0](https://huggingface.co/datasets/mcp-tool-shop/jam-actions-acoustic-v0): 72 records over
|
|
65
|
+
audio rather than notation, each label checked against what the analysis tools actually return. It ships no audio:
|
|
66
|
+
each record carries a recipe and the hash of the waveform it produces.
|
|
67
|
+
|
|
68
|
+
Both carry their licences and sources in their cards. Each was corrected on 25 September when an audit found files
|
|
69
|
+
whose arrangement licence couldn't be established, and the records built from them were withdrawn.
|
|
70
|
+
|
|
71
|
+
## interlock 1.0.8 to 1.0.10
|
|
72
|
+
|
|
73
|
+
[interlock](https://github.com/mcp-tool-shop-org/interlock) picks a model, an engine and a device for a job from
|
|
74
|
+
speeds it has measured. [Issue 31](https://mcp-tool-shop-org.github.io/rnd/newsletter/issue-31/) told how its first runs on a rented T4 card exposed
|
|
75
|
+
defects in how it measured, fixed in 1.0.5 to 1.0.7. The next three releases came from the same runs:
|
|
76
|
+
|
|
77
|
+
- **1.0.8: nothing waits for ever.** Every cell has a wall-time cap, and a call that reads the card's memory now has
|
|
78
|
+
a deadline: one run had gone silent for 56 minutes waiting on it. The harness prints a line at each step, so the
|
|
79
|
+
next hang names its own call. Rig checks get their own identity instead of borrowing a Colab card's name. And
|
|
80
|
+
`interlock route --concurrency N` ranks results for the number of requests you'll actually send.
|
|
81
|
+
- **1.0.9: a hung cell is cleaned up.** When a cell's deadline fires, the server is restarted and any late answer
|
|
82
|
+
from the abandoned cell is refused. The fallback reading of the card's memory is checked strictly, and an answer it
|
|
83
|
+
can't trust is recorded as unmeasured.
|
|
84
|
+
- **1.0.10: noise is recorded, not used to discard.** A check meant to catch a cached prompt was also discarding real
|
|
85
|
+
cells, because the time to read the prompt varies from pass to pass. Each measured row now records that spread
|
|
86
|
+
instead, and the cells are kept.
|
|
87
|
+
|
|
88
|
+
interlock 1.0.10 is on PyPI as `mcptoolshop-interlock` and needs Python 3.13 or newer.
|
|
89
|
+
|
|
90
|
+
## XRPL Camp: the XRP Ledger in one sitting
|
|
91
|
+
|
|
92
|
+
[XRPL Camp](https://github.com/mcp-tool-shop-org/xrpl-camp) teaches the XRP Ledger by doing it: real transactions on
|
|
93
|
+
the ledger's test network, in about ten minutes, ending with a proof of what you did that anyone can check against
|
|
94
|
+
the ledger. It's on PyPI as `xrpl-camp` and npm as `@mcptoolshop/xrpl-camp`, both at 1.5.2.
|
|
95
|
+
|
|
96
|
+
- **1.5.0** wired in the last parts of the course: a walkthrough of the reserve an account owes for the things it
|
|
97
|
+
owns, a plain-text certificate you can keep and share, and dated receipts each time a proof is verified.
|
|
98
|
+
- **1.5.1 and 1.5.2** tightened the edges. A submission whose result comes back missing is no longer taken as a
|
|
99
|
+
success: the camp looks the transaction up on the ledger and reports what it finds. The reserve step says plainly
|
|
100
|
+
when a change couldn't be measured yet. And the standalone downloads no longer bundle a leftover install record from
|
|
101
|
+
their build.
|
|
102
|
+
|
|
103
|
+
## A short note from today's runs
|
|
104
|
+
|
|
105
|
+
On today's T4 runs, the Colab machine's memory limit stopped some of the largest cells: when a model's working cache
|
|
106
|
+
spilled out of the card into the machine's memory, the limit killed the model server at about 11.5 GB of its own
|
|
107
|
+
memory. In three runs the process running our harness was killed a moment later as well (two seen directly, one
|
|
108
|
+
inferred from its logs); the rows written up to that point were kept, but the rest of that step was lost. The run
|
|
109
|
+
launcher now notices a dead run and stops it within about two minutes, instead of waiting out its full time. A check
|
|
110
|
+
that refuses such cells before they load is being built for interlock 1.0.11.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@rnd-news/newsletter",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.32.0",
|
|
4
4
|
"description": "The research newsletter of mcp-tool-shop: every published issue as Markdown and JSON.",
|
|
5
5
|
"author": "mcp-tool-shop",
|
|
6
6
|
"license": "MIT",
|
|
@@ -29,5 +29,5 @@
|
|
|
29
29
|
"data/issues.jsonl",
|
|
30
30
|
"issues/"
|
|
31
31
|
],
|
|
32
|
-
"contentSha256": "
|
|
32
|
+
"contentSha256": "bfacf7a5fe936d3ac5110b59c47fd3ae3a9c1fa2c35ba534c17865ea4b9a406d"
|
|
33
33
|
}
|