sf-docs-agent 0.1.1 → 0.1.2

package/README.md

@@ -36,27 +36,39 @@ copies the lines. Every quote is copied word for word from the page, not written
 check it with one `grep` against the cached copy, which saves curly quotes as straight ones and drops image links and
 the padding in table rows.
 
-## 84% less in your context, 63% fewer tokens overall
+## Why not use Salesforce Docs MCP
+
+Salesforce's [salesforce-docs-mcp](https://labs.agentforce.com/docs/salesforce-docs-mcp) works, and behind a
+subagent it was faster and cheaper than sf-docs-agent in my tests. But it searches a closed index you can't see into, with
+no date on its results.
+
+I wanted an alternative I control and can validate against the live sites. sf-docs-agent is open source, reads the page
+Salesforce publishes today, and every quote can be checked against that page with one `grep`. If you're fine
+trusting the index, the MCP in a subagent is a good choice. [The comparison](#compared-with-salesforce-docs-mcp) has
+the numbers and the trade-offs.
+
+## 84% less in your context, 38% fewer tokens than platform-docs-get
 
 On 10 questions against Salesforce's own platform-docs-get skill, the agent
-added 84% fewer tokens to your context and used 63% fewer tokens over the whole session, with the right answer on
-10 of 10.
+added 84% fewer tokens to your context and used 38% fewer tokens over the whole session, with the right answer on
+39 of 40 runs for each. In the second of four rounds, platform-docs-get missed question 5 and sf-docs-agent missed
+question 4.
 
-Below are the measurements, each read as platform-docs-get → sf-docs.
+Below are the medians of those four rounds, each read as platform-docs-get → sf-docs-agent.
 
 | Question | Site | Tokens added to context | Saved | Session token usage | Saved |
 |---|---|---|---|---|---|
-| 1. Scratch org editions | developer | 2,178 → **289** | **86.7%** | 338,852 → **119,355** | **64.8%** |
-| 2. Platform event allocation | developer | 4,107 → **490** | **88.1%** | 387,598 → **120,396** | **68.9%** |
-| 3. QueryLocator batch limits | developer | 2,568 → **581** | **77.4%** | 406,878 → **113,611** | **72.1%** |
-| 4. Integration patterns | architect | 1,104 → **484** | **56.2%** | 154,068 → **125,540** | **18.5%** |
-| 5. Salesforce Connect editions | help | 1,948 → **322** | **83.5%** | 382,751 → **157,746** | **58.8%** |
-| 6. Flow version limits | help | 4,578 → **290** | **93.7%** | 339,464 → **164,888** | **51.4%** |
-| 7. Headless 360 | admin | 3,772 → **380** | **89.9%** | 427,542 → **117,258** | **72.6%** |
-| 8. SLDS tooltips | lightningdesignsystem | 3,876 → **414** | **89.3%** | 340,914 → **112,252** | **67.1%** |
-| 9. Bulk API batch size | developer | 2,134 → **411** | **80.7%** | 506,340 → **125,870** | **75.1%** |
-| 10. Pub/Sub fetch limit | developer | 1,199 → **716** | **40.3%** | 236,766 → **130,110** | **45.0%** |
-| **All** | | 27,464 → **4,377** | **84.1%** | 3,521,173 → **1,287,026** | **63.4%** |
+| 1. Scratch org editions | developer | 2,178 → **289** | **86.7%** | 186,768 → **119,355** | **36.1%** |
+| 2. Platform event allocation | developer | 4,107 → **490** | **88.1%** | 245,482 → **120,396** | **51.0%** |
+| 3. QueryLocator batch limits | developer | 2,568 → **581** | **77.4%** | 217,691 → **113,611** | **47.8%** |
+| 4. Integration patterns | architect | 1,104 → **484** | **56.2%** | 127,043 → **125,540** | **1.2%** |
+| 5. Salesforce Connect editions | help | 1,948 → **322** | **83.5%** | 201,770 → **157,746** | **21.8%** |
+| 6. Flow version limits | help | 4,578 → **290** | **93.7%** | 217,138 → **164,888** | **24.1%** |
+| 7. Headless 360 | admin | 3,772 → **380** | **89.9%** | 250,200 → **105,388** | **57.9%** |
+| 8. SLDS tooltips | lightningdesignsystem | 3,876 → **414** | **89.3%** | 201,906 → **112,252** | **44.4%** |
+| 9. Bulk API batch size | developer | 2,134 → **411** | **80.7%** | 255,340 → **125,870** | **50.7%** |
+| 10. Pub/Sub fetch limit | developer | 1,199 → **716** | **40.3%** | 141,826 → **130,110** | **8.3%** |
+| **All** | | 27,464 → **4,377** | **84.1%** | 2,045,164 → **1,275,156** | **37.7%** |
 
 <details>
 <summary>The 10 questions</summary>
@@ -74,6 +86,68 @@ Below are the measurements, each read as platform-docs-get → sf-docs.
 
 </details>
 
+## Compared with Salesforce Docs MCP
+
+Aside from the platform-docs-get skill, Salesforce also has
+[salesforce-docs-mcp](https://labs.agentforce.com/docs/salesforce-docs-mcp), a closed-source public MCP server that
+searches an index of Help articles, developer guides and release notes. I ran the same 10 questions through it in two
+setups:
+
+a. MCP in main session: the main agent calls the MCP tools directly
+
+b. MCP in a subagent: a subagent whose only tools are the MCP's, with the same report format as sf-docs-agent
+
+Three rounds each, medians, same model.
+
+| | sf-docs-agent | a. MCP in main session | b. MCP in a subagent |
+|---|---|---|---|
+| Tokens added to context | **4,470** | 45,636 | 5,327 |
+| Session token usage | 1,291,504 | 1,052,673 | **937,229** |
+| Seconds, all 10 | 433 | 239 | **218** |
+| Right answers, of 30 | **30** | 28 | 24 |
+| Quotes that match the text the tools returned, all runs | **113 of 113** | 173 of 188 | 172 of 177 |
+| Quotes on today's page, checked with sf-docs-agent's fetcher, all runs | **113 of 113** | 159 of 188 | 135 of 177 |
+
+The MCP in a subagent is faster and uses 27% fewer session tokens than sf-docs-agent, and a. MCP in main session uses
+18% fewer. Its results come back as markdown excerpts, so I suspect Salesforce keeps a `.md` snapshot of the docs taken
+at some point. Because the index is closed, you can't see what it holds or when it was taken, and that has some costs:
+
+- **Coverage.** Across 91 searches it returned no admin.salesforce.com page and no SLDS 2 content page. The
+  subagent answered "not found" or wrong on Headless 360 and SLDS tooltips in all three rounds, and those two
+  questions hold all 8 wrong MCP answers. In the main session, the model also reached for web search or fetch on 27
+  of 30 runs and skipped the MCP on 4; its Headless 360 answers came from admin.salesforce.com through web fetch, not
+  the index.
+- **Freshness.** Results carry no date, and some come from the old atlas copy of a page. On question 2 the quotes were
+  exact, but the source URL now opens a page without them. On question 1 the wording is older than today's page, with
+  the same values. On both questions the values matched the answer key in every MCP run.
+- **Open vs closed.** The index has no published source or refresh schedule. sf-docs-agent reads the live page every time,
+  and every quote can be checked with one `grep`.
+
+Most of the speed gap is page rendering. Of sf-docs-agent's 30 runs, the 18 that read at least one `.md` page took a
+median 26.5 seconds, against 20.5 for the MCP subagent on the same questions and rounds. Help, admin and the
+Apex Developer Guide had no `.md` copy, so sf-docs-agent rendered them in a headless browser, and the 12 runs that
+rendered every page took a median 68.5 seconds, against 20 for the MCP subagent. The Apex Developer Guide and Apex
+Reference had `.md` copies by 2026-10-31, after these runs; Help, admin, Metadata API and Object Reference still have
+none.
+
+<details>
+<summary>Per question, sf-docs-agent / a. MCP in main session / b. MCP in a subagent</summary>
+
+| Question | Tokens added to context | Seconds | Right answers, of 3 |
+|---|---|---|---|
+| 1. Scratch org editions | 248 / 5,011 / 421 | 23 / 22 / 18 | 3 / 3 / 3 |
+| 2. Platform event allocation | 526 / 6,177 / 713 | 29 / 28 / 21 | 3 / 3 / 3 |
+| 3. QueryLocator batch limits | 728 / 8,304 / 531 | 68 / 18 / 20 | 3 / 3 / 3 |
+| 4. Integration patterns | 463 / 4,863 / 788 | 25 / 21 / 22 | 3 / 3 / 3 |
+| 5. Salesforce Connect editions | 300 / 3,556 / 381 | 84 / 17 / 18 | 3 / 3 / 3 |
+| 6. Flow version limits | 307 / 2,817 / 291 | 59 / 20 / 19 | 3 / 3 / 3 |
+| 7. Headless 360 | 374 / 4,232 / 772 | 60 / 31 / 31 | 3 / 3 / 0 |
+| 8. SLDS tooltips | 403 / 1,231 / 617 | 32 / 40 / 33 | 3 / 1 / 0 |
+| 9. Bulk API batch size | 489 / 6,284 / 529 | 27 / 22 / 20 | 3 / 3 / 3 |
+| 10. Pub/Sub fetch limit | 632 / 3,161 / 284 | 26 / 20 / 16 | 3 / 3 / 3 |
+
+</details>
+
 ## How sf-docs-agent reads each site
 
 The agent reads the `.md` copy of a page where the site publishes one. Not every Salesforce docs site does yet.
@@ -86,7 +160,8 @@ The agent finds pages through the site's `llms.txt` index or its sitemaps.
 | Site | How | Where it finds pages |
 |---|---|---|
 | developer.salesforce.com, new docs | the `.md` copy Salesforce publishes | `/docs/llms.txt` and one index per product |
-| developer.salesforce.com, atlas docs (Apex, Apex Reference, Metadata API, Object Reference) | renders the page, code samples included | `/docs/llms-product-docs.txt` |
+| developer.salesforce.com, Apex Developer Guide and Apex Reference | the `.md` copy (atlas URLs redirect to the new docs, by 2026-10-31) | `/docs/llms-product-docs.txt` |
+| developer.salesforce.com, atlas docs (Metadata API, Object Reference) | renders the page, code samples included | `/docs/llms-product-docs.txt` |
 | developer.salesforce.com, API reference pages (`?meta=<operation>`) | renders the page | the product's index |
 | architect.salesforce.com | the `.md` copy | `/llms.txt` |
 | help.salesforce.com, articles and knowledge articles | renders the page | links in other docs and articles, and Help's product sitemaps |
@@ -107,7 +182,7 @@ Name the site or product if you know it, and say what you want to know.
 
 The sample prompt is broad on purpose: I wanted to see the agent decide what "governors and limits" covers, and whether it could find everything in a broad question.
 
-Here are the results of that prompt, one run each on Opus 5.5 and Sonnet 5.5 at effort high. Each session called the sf-docs agent, which returned quotes with their
+Here are the results of that prompt, one run each on Opus 5.5 and Sonnet 5.5 at effort high. Each session called sf-docs-agent, which returned quotes with their
 official URLs, and the session turned them into the answer.
 
 <details>
@@ -291,9 +366,10 @@ live in `~/.claude/sf-docs/` for every scope, so delete that folder last, once n
 
 ## Limitations
 
-- On help, admin, SLDS 1, API reference pages and the atlas guides, the agent finds the article text by where it sits on
-  the page. If Salesforce changes a layout, it reads the largest block of text instead, which can be one section of the
-  article, and an article under 1,500 characters can't be read. A daily check in this repo flags the change.
+- On help, admin, SLDS 1, API reference pages and the atlas guides that still render (Metadata API, Object Reference and
+  more), the agent finds the article text by where it sits on the page. If Salesforce changes a layout, it reads the
+  largest block of text instead, which can be one section of the article, and an article under 1,500 characters can't
+  be read. A daily check in this repo flags the change.
 - A rendered page can take up to 80 seconds before the agent gives up on it (45 s to load, 20 s for content, 15 s for the
   `.md` copy of a page it redirected to).
 - No Help search. Help's `robots.txt` disallows it. Help's sitemaps list article ids but not titles.

package/agents/sf-docs.md

@@ -43,8 +43,8 @@ that, the printed output is the limit, not time.
   `duplicate-of <url>` means the body matches an earlier URL: read only the first copy. `same <url> as <url>` means
   two arguments are one page, fetched once. `(cached)` means it came from disk. Status `0` carries the error.
   `[redirected <url>]` means the page moved and the Source line names where: check that it is the page you asked for,
-  since a mistyped atlas topic redirects to its guide's landing page, which is not evidence. That rule also decides a
-  line that adds `[twin <path>]` (the redirected page read as its `.md`) or `[twin failed: <reason>]` (its rendered
+  since a mistyped atlas topic redirects to its guide's landing page, which is not evidence; in the Apex Developer
+  Guide, which moved to the new docs, it answers 404 and nothing is saved. That rule also decides a line that adds `[twin <path>]` (the redirected page read as its `.md`) or `[twin failed: <reason>]` (its rendered
   text kept); a cached run shows `(cached) [redirected <url>]` and no `[twin]`. `not saved` means a shell, login wall or missing page (Help
   answers 200 for a missing article), or `[api docs did not load]` on an API reference page: not evidence, but a rerun
   may work. `not saved: <code>` means the page loaded but the cache write failed: rerun. `not saved: left the doc

package/fetch-docs.mjs

@@ -451,8 +451,8 @@ try {
     if (got.sel === 'body') got = await page.evaluate(extract, [CONTENT, true])
       .catch(e => { console.error(`fallback failed on ${u.href}: ${e.message.split('\n')[0]}`); return got; });
     const { sel, text, links, hold } = got;
-    // A redirect is shown and cited: a mistyped atlas topic lands on its guide's landing page with status 200, and a
-    // Help article can move to another ?id= on the same path.
+    // A redirect is shown and cited: a mistyped atlas topic lands on its guide's landing page with status 200 (a 404 in
+    // a guide that moved to the new docs), and a Help article can move to another ?id= on the same path.
     const to = new URL(page.url()), hop = sameDoc(to, u) ? '' : ` [redirected ${to.href}]`;
     // Old atlas URLs (and the LDS home) redirect to pages that have a .md twin: keep the rendered links, read the twin.
     const moved = hop && twinOf(to);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sf-docs-agent",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "type": "module",
   "description": "Claude Code subagent that answers from official Salesforce docs (developer, architect, help, admin, lightningdesignsystem.com) with grep-checked quotes",
   "devDependencies": {