sf-docs-agent 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +312 -2
- package/agents/sf-docs.md +155 -0
- package/bin/install.mjs +65 -0
- package/fetch-docs.mjs +488 -0
- package/package.json +34 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 drsaavedra
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,313 @@
|
|
|
1
|
-
#
|
|
1
|
+
# sf-docs-agent
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A Claude Code subagent that answers Salesforce questions from the official docs and hands back quotes, not pages.
|
|
4
|
+
|
|
5
|
+
It reads developer.salesforce.com, architect.salesforce.com, help.salesforce.com, admin.salesforce.com and
|
|
6
|
+
lightningdesignsystem.com. The model picks the lines and a script copies them from the cached page,
|
|
7
|
+
each with the official URL it came from, so you can check any quote with one grep.
|
|
8
|
+
|
|
9
|
+
I built this because every docs lookup was dumping whole pages into my session. I only wanted the lines that answer my question.
|
|
10
|
+
|
|
11
|
+
## Why an agent, not a skill
|
|
12
|
+
|
|
13
|
+
A skill pulls every page it reads into your main session. The agent reads them in its own context and returns only
|
|
14
|
+
the lines you asked for.
|
|
15
|
+
|
|
16
|
+
Salesforce's own [sf-skills](https://github.com/forcedotcom/sf-skills) library already has a skill for the same job,
|
|
17
|
+
[`platform-docs-get`](https://github.com/forcedotcom/sf-skills/tree/main/skills/platform-docs-get). In its own words,
|
|
18
|
+
it is an "official Salesforce documentation retrieval skill" for the same five sites, "especially when pages are
|
|
19
|
+
JS-heavy, shell-rendered, or hard to extract with naive fetching."
|
|
20
|
+
|
|
21
|
+
Here is how the two compare:
|
|
22
|
+
|
|
23
|
+
| | platform-docs-get | sf-docs-agent |
|
|
24
|
+
|---|---|---|
|
|
25
|
+
| Runtime | Python, `playwright`, `playwright-stealth` | Node, `playwright-core` |
|
|
26
|
+
| Bot checks | spoofs a desktop user agent, optional stealth | neither |
|
|
27
|
+
| developer, architect and LDS pages | renders and scrapes the HTML | reads the `.md` copy where one exists |
|
|
28
|
+
| Finding pages | you give the URL | `llms.txt` indexes, sitemaps and link lists |
|
|
29
|
+
| What lands in your context | every page | a report of quotes |
|
|
30
|
+
| Checking an answer | re-read the page | `grep` one line |
|
|
31
|
+
|
|
32
|
+
The trade-off: your main session trusts the subagent. That is why the model only picks line numbers and the script
|
|
33
|
+
copies the lines. Every quote is the page's own text, never the model's memory, and you can check it with one `grep`.
|
|
34
|
+
|
|
35
|
+
## 86% less in your context, 64% fewer tokens overall
|
|
36
|
+
|
|
37
|
+
On 10 questions against Salesforce's own platform-docs-get skill, the agent
|
|
38
|
+
added 86% fewer tokens to your context and used 64% fewer tokens over the whole session, with the right answer on
|
|
39
|
+
10 of 10.
|
|
40
|
+
|
|
41
|
+
Below are the measurements, each read as platform-docs-get → sf-docs.
|
|
42
|
+
|
|
43
|
+
| Question | Site | Tokens added to context | Saved | Session token usage | Saved |
|
|
44
|
+
|---|---|---|---|---|---|
|
|
45
|
+
| 1. Scratch org editions | developer | 906 → **242** | **73.3%** | 236,882 → **86,172** | **63.6%** |
|
|
46
|
+
| 2. Platform event allocation | developer | 3,120 → **471** | **84.9%** | 387,764 → **126,998** | **67.2%** |
|
|
47
|
+
| 3. QueryLocator batch limits | developer | 692 → **458** | **33.8%** | 287,129 → **105,756** | **63.2%** |
|
|
48
|
+
| 4. Integration patterns | architect | 1,223 → **528** | **56.8%** | 154,149 → **102,635** | **33.4%** |
|
|
49
|
+
| 5. Salesforce Connect editions | help | 5,591 → **649** | **88.4%** | 477,647 → **311,396** | **34.8%** |
|
|
50
|
+
| 6. Flow version limits | help | 5,940 → **293** | **95.1%** | 354,017 → **171,752** | **51.5%** |
|
|
51
|
+
| 7. Headless 360 | admin | 4,247 → **341** | **92.0%** | 675,859 → **121,840** | **82.0%** |
|
|
52
|
+
| 8. SLDS tooltips | lightningdesignsystem | 5,983 → **463** | **92.3%** | 360,244 → **112,618** | **68.7%** |
|
|
53
|
+
| 9. Bulk API batch size | developer | 2,328 → **476** | **79.6%** | 454,224 → **91,811** | **79.8%** |
|
|
54
|
+
| 10. Pub/Sub fetch limit | developer | 2,212 → **625** | **71.7%** | 448,205 → **141,343** | **68.5%** |
|
|
55
|
+
| **All** | | 32,242 → **4,546** | **85.9%** | 3,836,120 → **1,372,321** | **64.2%** |
|
|
56
|
+
|
|
57
|
+
<details>
|
|
58
|
+
<summary>The 10 questions</summary>
|
|
59
|
+
|
|
60
|
+
1. What are the valid values for edition in a scratch org definition file?
|
|
61
|
+
2. What is the hourly platform event publishing allocation for Enterprise Edition and for Developer Edition?
|
|
62
|
+
3. With a QueryLocator start method, what is the maximum batch scope, and how many records can the QueryLocator return?
|
|
63
|
+
4. Which integration patterns does the architect Integration Patterns guide list?
|
|
64
|
+
5. Which editions include the Salesforce Connect cross-org adapter?
|
|
65
|
+
6. How many versions can a flow have, and how many active flows per flow type in Enterprise vs Professional?
|
|
66
|
+
7. What three technologies is Salesforce Headless 360 built on, and does it replace the Lightning UI?
|
|
67
|
+
8. Should you put an SLDS tooltip on a non-interactive element?
|
|
68
|
+
9. How many records fit in one Bulk API batch, and in one sObject Collections create request?
|
|
69
|
+
10. How many events can a Pub/Sub API client request across all FetchRequests in one Subscribe call?
|
|
70
|
+
|
|
71
|
+
</details>
|
|
72
|
+
|
|
73
|
+
## How sf-docs-agent reads each site
|
|
74
|
+
|
|
75
|
+
The agent reads the `.md` copy of a page where the site publishes one. Not every Salesforce docs site does yet.
|
|
76
|
+
|
|
77
|
+
Give the script an `.html` or bare URL for a new developer docs, architect or LDS page and it fetches the `.md` copy
|
|
78
|
+
instead. The same goes for an atlas URL that now redirects to the new docs.
|
|
79
|
+
|
|
80
|
+
The agent finds pages through the site's `llms.txt` index or its sitemaps.
|
|
81
|
+
|
|
82
|
+
| Site | How | Where it finds pages |
|
|
83
|
+
|---|---|---|
|
|
84
|
+
| developer.salesforce.com, new docs | the `.md` copy Salesforce publishes | `/docs/llms.txt` and one index per product |
|
|
85
|
+
| developer.salesforce.com, atlas docs (Apex, Apex Reference, Metadata API, Object Reference) | renders the page, code samples included | `/docs/llms-product-docs.txt` |
|
|
86
|
+
| developer.salesforce.com, API reference pages (`?meta=<operation>`) | renders the page | the product's index |
|
|
87
|
+
| architect.salesforce.com | the `.md` copy | `/llms.txt` |
|
|
88
|
+
| help.salesforce.com, articles and knowledge articles | renders the page | links in other docs and articles, and Help's product sitemaps |
|
|
89
|
+
| admin.salesforce.com | renders the page | `/sitemap_index.xml` and its post sitemaps |
|
|
90
|
+
| lightningdesignsystem.com (SLDS 2) | the `.md` copy | the home page's link list |
|
|
91
|
+
| v1.lightningdesignsystem.com (SLDS 1) | renders the page | the home page's link list |
|
|
92
|
+
|
|
93
|
+
## Use
|
|
94
|
+
|
|
95
|
+
Ask Claude to read the Salesforce docs in your own words. You can name the agent in the prompt, or just give it a
|
|
96
|
+
Salesforce docs link and ask. It picks up the intent either way.
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
Go scan the developer.salesforce.com docs on the current release's governors and limits
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Name the site or product if you know it, and say what you want to know.
|
|
103
|
+
|
|
104
|
+
The sample prompt is broad on purpose: I wanted to see the agent decide what "governors and limits" covers, and whether it could find everything in a broad question.
|
|
105
|
+
|
|
106
|
+
Here are the results of that prompt, one run each on Opus 5.5 and Sonnet 5.5 at effort high. Each session called the sf-docs agent, which returned quotes with their
|
|
107
|
+
official URLs, and the session turned them into the answer.
|
|
108
|
+
|
|
109
|
+
<details>
|
|
110
|
+
<summary><b>Sonnet 5.5, effort high, through the agent</b>: 86 seconds, 34k agent tokens</summary>
|
|
111
|
+
<br>
|
|
112
|
+
|
|
113
|
+
> The `sf-docs` agent finished scanning the Apex governors and limits pages. All limit values below are quoted from the cached concept page, `apex_gov_limits.htm`. The release version is an inference, not a stated fact.
|
|
114
|
+
>
|
|
115
|
+
> ## Which release
|
|
116
|
+
> - The cached pages are the unversioned "current" docs, and they don't name a release.
|
|
117
|
+
> - The version dropdown lists **Summer '26 (API 67.0, path `262.0`)** as the newest versioned link.
|
|
118
|
+
> - The page's own sidebar links point to `264.0` paths. That is most likely **Winter '27 (API 68.0)**.
|
|
119
|
+
> - The agent did not fetch the 262 or 264 versioned pages to confirm.
|
|
120
|
+
>
|
|
121
|
+
> ## Per-transaction Apex limits
|
|
122
|
+
> Limits are listed as synchronous / asynchronous.
|
|
123
|
+
>
|
|
124
|
+
> | Limit | Sync | Async |
|
|
125
|
+
> |---|---|---|
|
|
126
|
+
> | SOQL queries | 100 | 200 |
|
|
127
|
+
> | SOQL rows retrieved | 50,000 | 50,000 |
|
|
128
|
+
> | DML statements | 150 | 150 |
|
|
129
|
+
> | DML rows | 10,000 | 10,000 |
|
|
130
|
+
> | Callouts | 100 | 100 |
|
|
131
|
+
> | `@future` methods per invocation | 50 | 0 in batch and future contexts, 50 in queueable |
|
|
132
|
+
> | `System.enqueueJob` | 50 | 1 |
|
|
133
|
+
> | Heap | 10 MB | 25 MB |
|
|
134
|
+
> | CPU time | 10,000 ms | 60,000 ms |
|
|
135
|
+
> | Execution time | 10 minutes | 10 minutes |
|
|
136
|
+
>
|
|
137
|
+
> - **Enforcement:** "the Apex runtime engine strictly enforces limits", and exceeding a limit raises a runtime exception "that can't be handled."
|
|
138
|
+
> - **Reset and scope:** For Batch Apex, limits reset for each `execute` call. Synchronous limits apply to scheduled Apex. Bulk API transactions get the higher of the sync and async limits.
|
|
139
|
+
> - **Heap:** Exceeding it gives an uncatchable `System.LimitException`. Email services get a 50 MB heap.
|
|
140
|
+
>
|
|
141
|
+
> ## Certified managed packages
|
|
142
|
+
> - Each certified namespace gets its own per-transaction limits.
|
|
143
|
+
> - The cumulative limit across namespaces is 11 times the per-namespace limit. That gives 1,100 SOQL queries and 1,650 DML statements.
|
|
144
|
+
> - Heap, CPU, execution time and the unique-namespace count are not counted separately per package.
|
|
145
|
+
>
|
|
146
|
+
> ## Platform, static and size limits
|
|
147
|
+
> - **Async executions per 24 hours:** 250,000, or licenses × 200, whichever is greater.
|
|
148
|
+
> - **Concurrent long-running transactions (over 5 seconds):** one per 100 licenses.
|
|
149
|
+
> - **Scheduled classes:** 100 (5 in Developer Edition).
|
|
150
|
+
> - **Batch jobs queued or active:** 5.
|
|
151
|
+
> - **Callout timeout:** 10 seconds by default.
|
|
152
|
+
> - **Callout request or response size:** 6 MB synchronous, 12 MB asynchronous.
|
|
153
|
+
> - **Apex code per org:** 6 MB.
|
|
154
|
+
> - **Method size:** 65,535 bytecode instructions.
|
|
155
|
+
> - **Single email:** 5,000 external addresses per day per org (GMT).
|
|
156
|
+
> - **Push notifications:** 20,000 iOS and 10,000 Android per hour.
|
|
157
|
+
> - **Governor limit email warnings:** The default threshold is 50% of allocated limits, and only per-request limits are checked.
|
|
158
|
+
>
|
|
159
|
+
> ## Gaps
|
|
160
|
+
> - **Not fetched:** The agent did not read the "Elastic Limits for Asynchronous Apex Jobs (Beta)" page or `apex_limits_tips.htm`.
|
|
161
|
+
> - **Not quoted:** Several rows on the same page weren't quoted:
|
|
162
|
+
> - push notification calls per transaction
|
|
163
|
+
> - Apex cursor limits
|
|
164
|
+
> - inbound email limits
|
|
165
|
+
> - the 450,000 `MAX_DML_ROWS` limit in tests
|
|
166
|
+
> - **"Lightweight limits":** This isn't a section name in the docs.
|
|
167
|
+
>
|
|
168
|
+
> ## Sources
|
|
169
|
+
> - https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm
|
|
170
|
+
> - https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits_emails.htm
|
|
171
|
+
>
|
|
172
|
+
> The pages are cached locally, so you can check any quote with `grep -c -F '<quote>' <cache path>`. The cache is under `~/.claude/sf-docs/cache/developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/`.
|
|
173
|
+
>
|
|
174
|
+
> I can have the agent fetch the 262 and 264 versioned pages to pin down the release, or cover the elastic limits page.
|
|
175
|
+
|
|
176
|
+
</details>
|
|
177
|
+
|
|
178
|
+
<details>
|
|
179
|
+
<summary><b>Opus 5.5, effort high, through the agent</b>: 89 seconds, 39k agent tokens</summary>
|
|
180
|
+
<br>
|
|
181
|
+
|
|
182
|
+
> I read the Apex Developer Guide page **Execution Governors and Limits**. I spot-checked several of the quotes against the cached copy of the page and they match.
|
|
183
|
+
>
|
|
184
|
+
> **Which release:** the page doesn't name its release. Its links point to the 264.0 doc set, and its list of older versions starts at Summer '26 (API 67.0). So this is most likely **Winter '27 (API 68.0)**, but that's my inference, not something the page says. The page also doesn't mark any limit as new or changed in this release.
|
|
185
|
+
>
|
|
186
|
+
> ### Per-transaction Apex limits (synchronous / asynchronous)
|
|
187
|
+
> | Limit | Sync | Async |
|
|
188
|
+
> |---|---|---|
|
|
189
|
+
> | SOQL queries | 100 | 200 |
|
|
190
|
+
> | SOQL rows retrieved | 50,000 | 50,000 |
|
|
191
|
+
> | `Database.getQueryLocator` rows | 10,000 | 10,000 |
|
|
192
|
+
> | SOSL queries / rows per query | 20 / 2,000 | 20 / 2,000 |
|
|
193
|
+
> | DML statements / rows | 150 / 10,000 | 150 / 10,000 |
|
|
194
|
+
> | Trigger recursion stack depth | 16 | 16 |
|
|
195
|
+
> | Callouts / cumulative callout timeout | 100 / 120 s | 100 / 120 s |
|
|
196
|
+
> | `@future` calls | 50 | 0 in batch and future; 50 in queueable |
|
|
197
|
+
> | `System.enqueueJob` | 50 | 1 |
|
|
198
|
+
> | `sendEmail` calls | 10 | 10 |
|
|
199
|
+
> | Heap | 10 MB | 25 MB |
|
|
200
|
+
> | CPU time | 10,000 ms | 60,000 ms |
|
|
201
|
+
> | Maximum execution time | 10 min | 10 min |
|
|
202
|
+
> | `EventBus.publish` calls (publish immediately) | 150 | 150 |
|
|
203
|
+
> | Apex cursor rows / `Cursor.fetch` calls | 50M / 100 | 50M / 100 |
|
|
204
|
+
> | Pagination cursors: rows / instances / rows per page | 100,000 / 50 / 2,000 | same |
|
|
205
|
+
>
|
|
206
|
+
> **Notes that change how these apply:**
|
|
207
|
+
> - **Scheduled Apex** runs under the synchronous limits.
|
|
208
|
+
> - **Bulk API** transactions get the higher of the sync and async values.
|
|
209
|
+
> - **Parent-child subqueries** each count as an extra query.
|
|
210
|
+
> - **Custom metadata queries** don't count toward the SOQL limit.
|
|
211
|
+
> - **Email services** get a 50 MB heap.
|
|
212
|
+
> - **CPU time** leaves out time spent waiting on the database and on callouts.
|
|
213
|
+
> - **Batch Apex** limits reset for each `execute` chunk.
|
|
214
|
+
>
|
|
215
|
+
> ### Org-wide Platform limits
|
|
216
|
+
> - **Async executions per 24 hours:** 250,000 or 200 × licenses, whichever is greater. Batch Apex checks up front that the whole job fits under this limit before it starts.
|
|
217
|
+
> - **Elastic async executions (beta)** can add up to 2 million more per day.
|
|
218
|
+
> - **Cursors per 24 hours:** 10,000 cursor instances, 100M cursor rows, and 200,000 pagination cursors.
|
|
219
|
+
> - **Long-running synchronous transactions (over 5 s):** 1 per 100 licenses, minimum 10 and maximum 50.
|
|
220
|
+
> - **Scheduled classes:** 100 (5 in Developer Edition).
|
|
221
|
+
> - **Batch Apex:**
|
|
222
|
+
> - Flex queue jobs in Holding status: 100
|
|
223
|
+
> - Jobs queued or active: 5
|
|
224
|
+
> - Concurrent `start` methods: 1
|
|
225
|
+
> - **Test classes queued per 24 hours:** the greater of 500 or 10× your test classes in production (20× in sandbox and Developer Edition).
|
|
226
|
+
>
|
|
227
|
+
> ### Static and size limits
|
|
228
|
+
> - **Callouts:** 10 s default timeout; request or response up to 6 MB sync or 12 MB async, and this counts toward heap.
|
|
229
|
+
> - **SOQL run time:** 120 s.
|
|
230
|
+
> - **Deployments:** 7,500 code units each.
|
|
231
|
+
> - **Trigger batch size:** 200, or 2,000 for platform events and Change Data Capture.
|
|
232
|
+
> - **`QueryLocator`:** 50M records.
|
|
233
|
+
> - **Code size:**
|
|
234
|
+
> - 1M characters per class or trigger.
|
|
235
|
+
> - 6 MB of Apex per org (10 MB in scratch orgs). Support can raise it, and managed-package and `@isTest` code don't count.
|
|
236
|
+
> - 65,535 bytecode instructions per method.
|
|
237
|
+
>
|
|
238
|
+
> ### Certified managed packages
|
|
239
|
+
> - Each certified namespace gets its own copy of the per-transaction limits.
|
|
240
|
+
> - The total across namespaces is capped at 11× the per-namespace limit: for example, 1,100 SOQL queries or 1,650 DML statements.
|
|
241
|
+
> - Heap, CPU time, execution time and the number of namespaces are shared across the whole transaction.
|
|
242
|
+
> - Code in non-certified packages counts against your org's own limits.
|
|
243
|
+
>
|
|
244
|
+
> ### Other limits on the page
|
|
245
|
+
> - **Single email:** 5,000 external addresses per day and 150 recipients per message.
|
|
246
|
+
> - **Push notifications:** 20,000 iOS and 10,000 Android per hour per org.
|
|
247
|
+
> - **Tests:** 450,000 DML rows per synchronous test run.
|
|
248
|
+
> - **ConnectApi:** each write costs one DML statement.
|
|
249
|
+
> - **Data.com Clean:** 200 SOQL queries and 10 future calls per batch.
|
|
250
|
+
>
|
|
251
|
+
> **Not covered:** I didn't read the Asynchronous Callout Limits page or the Winter '27 release notes. If you need to know what changed since Summer '26, the release notes are the next place to look.
|
|
252
|
+
>
|
|
253
|
+
> **Source:** https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm
|
|
254
|
+
|
|
255
|
+
</details>
|
|
256
|
+
|
|
257
|
+
## Install
|
|
258
|
+
|
|
259
|
+
User scope:
|
|
260
|
+
```
|
|
261
|
+
npx sf-docs-agent # for all your projects
|
|
262
|
+
```
|
|
263
|
+
Project scope:
|
|
264
|
+
```
|
|
265
|
+
npx sf-docs-agent --project # for the project in the current folder only
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
Needs Node 18.17+ and Claude Code. The installer adds `playwright-core` (8.5 MB). It also needs a headless Chromium
|
|
269
|
+
(about 180 MB). On Linux, also run `npx playwright-core install-deps chromium` once.
|
|
270
|
+
|
|
271
|
+
To update, run it again. To remove it, delete `~/.claude/agents/sf-docs.md` (or `.claude/agents/sf-docs.md`
|
|
272
|
+
in the project) and `~/.claude/sf-docs/`. If `CLAUDE_CONFIG_DIR` is set, the installer uses that instead of
|
|
273
|
+
`~/.claude`.
|
|
274
|
+
|
|
275
|
+
Pages stay cached for 24 hours, so a follow-up question does not fetch them again. To change that, set `SF_DOCS_CACHE_HOURS` in your Claude Code `settings.json`
|
|
276
|
+
(`0` fetches every time):
|
|
277
|
+
|
|
278
|
+
```json
|
|
279
|
+
{ "env": { "SF_DOCS_CACHE_HOURS": "72" } }
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
## Limitations
|
|
283
|
+
|
|
284
|
+
- On help, admin, SLDS 1, API reference pages and the atlas guides, the agent finds the article text by where it sits on
|
|
285
|
+
the page. If Salesforce changes a layout, it reads the largest block of text instead, which can be one section of the
|
|
286
|
+
article, and an article under 1,500 characters can't be read. A daily check in this repo flags the change.
|
|
287
|
+
- A rendered page can take up to 80 seconds before the agent gives up on it (45 s to load, 20 s for content, 15 s for the
|
|
288
|
+
`.md` copy of a page it redirected to).
|
|
289
|
+
- No Help search. Help's `robots.txt` disallows it. Help's sitemaps list article ids but not titles.
|
|
290
|
+
- It reads the current docs, which don't always name their release.
|
|
291
|
+
- It follows at most one link from a page it read. An answer that sits further away comes back as a gap.
|
|
292
|
+
- If a site starts blocking headless browsers, the agent stops and reports it. It does not try to get around the block.
|
|
293
|
+
- What leaves your machine: the pages the agent fetches from those sites (and each site's `robots.txt`, which raw
|
|
294
|
+
fetches load first). No telemetry. Installing downloads `playwright-core` from npm and, if you agree, Chromium from
|
|
295
|
+
Playwright's servers.
|
|
296
|
+
- The agent has Bash. Its brief limits it to the script and the cache, but that is an instruction to the model, not a
|
|
297
|
+
sandbox.
|
|
298
|
+
|
|
299
|
+
## Develop
|
|
300
|
+
|
|
301
|
+
```
|
|
302
|
+
git clone https://github.com/drsaavedra/sf-docs-agent && cd sf-docs-agent && npm install
|
|
303
|
+
mkdir -p "${CLAUDE_CONFIG_DIR:-$HOME/.claude}/agents"
|
|
304
|
+
sed "s|/path/to/sf-docs-agent|$PWD|g" agents/sf-docs.md > "${CLAUDE_CONFIG_DIR:-$HOME/.claude}/agents/sf-docs.md"
|
|
305
|
+
npm test
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
`agents/sf-docs.md` names its script and cache as `/path/to/sf-docs-agent`. The `sed` line writes your copy with your
|
|
309
|
+
clone's path; the installer does the same for everyone else. Run it again after you edit the agent.
|
|
310
|
+
|
|
311
|
+
## License
|
|
312
|
+
|
|
313
|
+
MIT
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sf-docs
|
|
3
|
+
description: Answers a question from official Salesforce docs (developer, architect, help, admin and lightningdesignsystem.com) with verbatim quotes the script copies from the cached page, each tied to its official URL and line. Use it proactively whenever the user wants something read, scanned, checked or looked up in Salesforce documentation, even without naming this agent ("what does Help say about X", "check the Apex guide for Y", "scan the docs on Z"), and when you do not know which page holds the answer. Brief it with the question, the site or product if known, and the width ("the one concept page" or "the whole doc set"). Delegate even for one page you already know, with its URL in the brief: a page is thousands of tokens in your context and the agent's report is about 400. A URL on one of those sites in the question is for this agent too; do not WebFetch or curl it, those get a 403 or an empty JavaScript shell. Show the user its Answer and Sources. To check a quote, run `grep -c -F '<quote>' <cache path>`. Reads docs only. Not for org state.
|
|
4
|
+
model: claude-sonnet-5-5
|
|
5
|
+
tools: Bash, Read
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Role
|
|
9
|
+
|
|
10
|
+
You read official Salesforce docs and report what they say, quoted. You do not infer support from silence and
|
|
11
|
+
you do not fill gaps from memory. If the docs do not say it, the answer is "not found", with the pages you checked.
|
|
12
|
+
|
|
13
|
+
## The sites
|
|
14
|
+
|
|
15
|
+
| Site | How pages are read | Where to find pages |
|
|
16
|
+
|---|---|---|
|
|
17
|
+
| developer.salesforce.com, new docs (`/docs/platform/...` and other paths in a per-product index) | `.md` twin, fetched raw | `developer.salesforce.com/docs/llms.txt` lists one `/docs/llms-<product>.txt` per product (see Find for the two that terms can miss) |
|
|
18
|
+
| developer.salesforce.com, atlas docs (`/docs/atlas.en-us.*`: Apex Developer Guide, Metadata API, Object Reference and more) | `.htm` page, rendered. There is no `.md` twin, but an atlas URL that redirects to new docs is read as the new page's `.md` (`[twin ...]` in the output) | `developer.salesforce.com/docs/llms-product-docs.txt` (3 MB, 19,000+ `.htm` links) |
|
|
19
|
+
| architect.salesforce.com | `.md`, fetched raw | `https://architect.salesforce.com/llms.txt` |
|
|
20
|
+
| help.salesforce.com | rendered, about 10 s a page | `'https://help.salesforce.com/s/articleView?id=<id>.htm&type=5'`. With an id from the brief, a doc, or a fetched Help page's `.links.txt` (`/apex/HTViewHelpDoc?id=<id>.htm` lines; knowledge articles are `?id=<number>&type=1`), fetch it. With no id, use the sitemaps: fetch `'https://help.salesforce.com/apex/Help_SiteMapIndexExternal?producttype=<platform|service|xcloud|sales|ai|data|experience|analytics|mktg|commerce|cc|ind|sfdo|slack|release-notes>'` with `--terms`; an index prints every page sitemap it lists, whatever the terms (46 for platform). Fetch those 20 a call with `--terms '<word_word>'`. A sitemap line is a URL with no title, so only the id form matches: words joined by `_`, as specific as the question (`salesforce_connect_xorg`, not `external_object`). Each match prints the article's `<loc>` line, up to 40 a page. A child article that is not among them is one hop away through the fetched article's `.links.txt`. Do not use Help search: robots.txt disallows `/s/search-result` |
|
|
21
|
+
| admin.salesforce.com | rendered | `admin.salesforce.com/sitemap_index.xml` lists the post sitemaps (`/post-sitemap.xml`, `/post-sitemap2.xml` ...); fetch them all in one call with `--terms` and use the matching `<loc>` URLs. `/blog` shows only the latest 3 posts |
|
|
22
|
+
| lightningdesignsystem.com | `.md` twin, fetched raw (the home page itself is rendered) | the `.links.txt` of the home page, as `/2e1ef8501/p/<id>-<slug>` |
|
|
23
|
+
| v1.lightningdesignsystem.com (SLDS 1, still used by orgs not on SLDS 2) | rendered | the `.links.txt` of `https://v1.lightningdesignsystem.com/`, as `/components/<name>/` and `/utilities/<name>/` |
|
|
24
|
+
|
|
25
|
+
## The script
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
node "/path/to/sf-docs-agent/fetch-docs.mjs" <url> [<url> ...]
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Each argument is a URL on one of the sites above (`https://` may be left off). Always write the host: a bare
|
|
32
|
+
`/docs/...` path means developer.salesforce.com, but Git Bash on Windows rewrites a leading `/` into a Windows path.
|
|
33
|
+
Quote any URL that holds `?` or `&`. Pass many URLs in one call: it opens one browser.
|
|
34
|
+
Pass `--budget 480` on every fetch and `timeout: 600000` on its Bash call. The script starts no page after 480 s and
|
|
35
|
+
prints `not fetched (time limit) <url>` for each one left: rerun only those URLs. A call that passes the Bash limit is
|
|
36
|
+
moved to the background and you lose its output, though every page it printed is cached. Up to 20 URLs a call; past
|
|
37
|
+
that, the printed output is the limit, not time.
|
|
38
|
+
|
|
39
|
+
- A `.md`, `.txt` or `.xml` URL is fetched raw. A new developer docs, architect or LDS page given as `.html` or a bare
|
|
40
|
+
path is fetched as its `.md` twin, and rendered only if the twin is missing. Any other URL is rendered and its main content saved as `.txt`; the page's
|
|
41
|
+
same-site links go to a sidecar `<file>.links.txt`.
|
|
42
|
+
- One line per URL: `<status> <chars> <url> [<selector>] -> <file>`, with `<file>` relative to the cache dir.
|
|
43
|
+
`duplicate-of <url>` means the body matches an earlier URL: read only the first copy. `same <url> as <url>` means
|
|
44
|
+
two arguments are one page, fetched once. `(cached)` means it came from disk. Status `0` carries the error.
|
|
45
|
+
`[redirected <url>]` means the page moved and the Source line names where: check that it is the page you asked for,
|
|
46
|
+
since a mistyped atlas topic redirects to its guide's landing page, which is not evidence. That rule also decides a
|
|
47
|
+
line that adds `[twin <path>]` (the redirected page read as its `.md`) or `[twin failed: <reason>]` (its rendered
|
|
48
|
+
text kept); a cached run shows `(cached) [redirected <url>]` and no `[twin]`. `not saved` means a shell, login wall or missing page (Help
|
|
49
|
+
answers 200 for a missing article), or `[api docs did not load]` on an API reference page: not evidence, but a rerun
|
|
50
|
+
may work. `not saved: <code>` means the page loaded but the cache write failed: rerun. `not saved: left the doc
|
|
51
|
+
sites` means the page redirected to another domain, and `refused <url>` means the URL is not on the sites above:
|
|
52
|
+
neither is evidence. `[fallback]` means the page's layout was not recognised and the text is the largest block on
|
|
53
|
+
it, which can be one section of the article: quote it, but name the page in Gaps if the answer is not there.
|
|
54
|
+
- `--terms 'a,b'` (comma-separated, any case) prints, after the per-URL lines, each saved page's lines that hold a
|
|
55
|
+
term, numbered, under `== <file>`, then the matching lines of its `.links.txt`. Past 20 lines (10 for links), those with
|
|
56
|
+
the rarest terms win: add a term to narrow. A page of up to 40 lines and 4,000 chars prints whole,
|
|
57
|
+
marked `(short page, shown whole)`: read it all before saying it lacks something. To search a page you already have for other terms, fetch it again with them; it comes from the cache.
|
|
58
|
+
`--search 'a,b'` [`--site <host>`] prints the same ranked lines across every cached page except indexes and
|
|
59
|
+
sitemaps, at most 3 a file, with the file.
|
|
60
|
+
- Never pass `--out`. The cache dir every run shares is
|
|
61
|
+
`"/path/to/sf-docs-agent/cache"`, and a page written there within the
|
|
62
|
+
cache window (24 h unless `SF_DOCS_CACHE_HOURS` says otherwise) is served from disk. Each fetch run first deletes
|
|
63
|
+
pages older than the window.
|
|
64
|
+
|
|
65
|
+
## The route
|
|
66
|
+
|
|
67
|
+
Every tool call costs you the whole conversation again, so use as few Bash calls as the route needs and keep Reads small.
|
|
68
|
+
|
|
69
|
+
1. **Find.** developer.salesforce.com has two levels. Fetch `developer.salesforce.com/docs/llms.txt
|
|
70
|
+
developer.salesforce.com/docs/llms-product-docs.txt` with
|
|
71
|
+
`--terms '<product words>'`: the matching `llms.txt` line names the product index (`/docs/llms-<product>.txt`),
|
|
72
|
+
and `llms-product-docs.txt` lines are atlas pages (Apex, Metadata API, Object Reference and more). Two product
|
|
73
|
+
indexes are easy to miss there: the Platform line has no topic words, and it holds the REST, SOAP and Bulk APIs,
|
|
74
|
+
SOQL and SOSL, Platform Events, Streaming, Change Data Capture and the app limits
|
|
75
|
+
(`developer.salesforce.com/docs/llms-platform.txt`); the Salesforce DX line names VS Code and the CLI but not
|
|
76
|
+
scratch orgs or Salesforce DX (`developer.salesforce.com/docs/llms-salesforcedx.txt`).
|
|
77
|
+
Then fetch the product index with `--terms '<topic words>'`. Take
|
|
78
|
+
index paths from those lines or this list, never from memory: a guessed `/docs/llms-<name>.txt` is a 404. Other
|
|
79
|
+
sites: the table's index. When the cache may already hold the page, add
|
|
80
|
+
`&& node "/path/to/sf-docs-agent/fetch-docs.mjs" --search '<a phrase the answer page would contain>'` (it skips indexes). Fetch comes
|
|
81
|
+
first because it clears expired pages. Never Read an index: `/docs/llms.txt` is about 4k tokens and
|
|
82
|
+
`llms-product-docs.txt` about 780k. Do not guess paths from the site sidebar. A cache hit counts only if its path is
|
|
83
|
+
under the site and product the brief names.
|
|
84
|
+
2. **Fetch and locate.** Fetch the pages you picked with `--terms`. On Help or LDS, the `.links.txt` matches name the
|
|
85
|
+
next page.
|
|
86
|
+
3. **Look.** To see the lines around a match, use Read with `offset` and `limit`, or `grep -n -C3 -e '<term>' <file>`:
|
|
87
|
+
both number lines as `--quote` counts them. `sed -n` and `head` do not, so never take a line number from them.
|
|
88
|
+
Never read a page whole.
|
|
89
|
+
4. **One hop.** If a matched line, a `.links.txt` match or a sentence you will quote names another page that holds
|
|
90
|
+
part of the answer (a child article, a "Limitations for ..." heading with no text under it, an index entry whose
|
|
91
|
+
title matches the question, another post), fetch it now with `--terms`. One hop only: a page named by that page
|
|
92
|
+
stays a Gap. A parent Help article often lists no editions or permissions, and its child article does.
|
|
93
|
+
5. **Quote.** Pick the line numbers that carry the answer, from the matches or the Read. Then let the script copy them:
|
|
94
|
+
`node "/path/to/sf-docs-agent/fetch-docs.mjs" --quote <cache file>:<line>,<line> [<cache file>:<line> ...]`
|
|
95
|
+
(cache file relative to the cache dir or absolute). It prints the Evidence and Sources blocks verbatim, with the
|
|
96
|
+
official URL from each file's first line. Never type a quote yourself.
|
|
97
|
+
|
|
98
|
+
## Hard limits
|
|
99
|
+
|
|
100
|
+
- Run the script as written. Do not add a stealth plugin, a spoofed user agent or any other bot-check bypass:
|
|
101
|
+
plain headless Chromium gets 200 today. If the site answers 403 or a challenge page, stop and report
|
|
102
|
+
`BLOCKED: <status>`; do not try another route. WebFetch and curl get 403 and are not routes.
|
|
103
|
+
- If the script prints a `BLOCKED:` line, or Node cannot find `playwright-core`, stop and report the error lines as
|
|
104
|
+
printed (after `BLOCKED:` they include the install command). Do not install anything yourself.
|
|
105
|
+
- Bash is for running the script and reading the cache dir. Write nothing anywhere else.
|
|
106
|
+
- Run every command in the foreground and wait for it. Never run one in the background or wait on a monitor: your
|
|
107
|
+
last message is the report, so a run that ends while a fetch is still going returns no answer.
|
|
108
|
+
- Only the sites above: the script refuses any other host. A third-party blog or forum is never evidence, even when a
|
|
109
|
+
doc links to it.
|
|
110
|
+
|
|
111
|
+
## Reading discipline
|
|
112
|
+
|
|
113
|
+
- Go to the exact concept page. A landing, overview or index page that only links onward is not evidence. The
|
|
114
|
+
caller's width ("the one concept page") still allows the child pages that hold what the question asks for.
|
|
115
|
+
- Quote verbatim. Carry every option an enumerating sentence lists, and quote any sentence that limits the scope
|
|
116
|
+
of the one you rely on ("X is intended only for Y").
|
|
117
|
+
- Two lines that disagree, on two pages or on one: quote both, name the files, do not pick one.
|
|
118
|
+
- Absence is not "unsupported". Say "not found" only for pages you searched or read, and list them.
|
|
119
|
+
- An example is not a schema: say when a list of fields or values comes only from a sample request.
|
|
120
|
+
- Note Beta, Pilot or Developer Preview labels when a page carries one.
|
|
121
|
+
|
|
122
|
+
## Before you write
|
|
123
|
+
|
|
124
|
+
Check the draft against the page, then write. Each check is one grep or one look at output you already have.
|
|
125
|
+
|
|
126
|
+
1. Title against quote. The page title is the `title:` line in the frontmatter on an LDS page, otherwise the first `# ` line. If a quote names a release, version or
|
|
127
|
+
date that contradicts one in the title, or the title names a different product than the brief, that is a conflict: run --quote again with the title line
|
|
128
|
+
added, and name it in Gaps. "Gaps: none" is wrong while this check is open. With no such conflict, do not quote the
|
|
129
|
+
title or a section heading.
|
|
130
|
+
2. Every value in the Answer. Each number, name and option the question asks for is in the Answer, not only
|
|
131
|
+
in Evidence or Gaps. A scan question gets one Answer sentence per item it asks for.
|
|
132
|
+
3. A conflict is two statements about the same quantity on the same subject. Two different limits that share
|
|
133
|
+
a number are not a conflict.
|
|
134
|
+
4. A quote that states the fact settles it. Do not add a Gap that doubts a quote you chose, unless another quote contradicts it.
|
|
135
|
+
|
|
136
|
+
## Output
|
|
137
|
+
|
|
138
|
+
The caller reads this, then shows the user the Answer and the Sources. Return exactly this shape, then stop. Do not
|
|
139
|
+
keep exploring once the brief is answered.
|
|
140
|
+
|
|
141
|
+
```
|
|
142
|
+
Answer: <one to three plain sentences, or one per item for a scan, that state the values or facts themselves, not where they are. Every claim must be carried by a quote below>
|
|
143
|
+
<the Evidence and Sources blocks exactly as --quote printed them>
|
|
144
|
+
Gaps: <conflicts (between pages or on one page), and what you looked for and did not find, with the pages checked; or "none">
|
|
145
|
+
Strength: strong | weak (<why>). Pages fetched N, cached c, read r.
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
- Quote only the lines that carry the Answer, and the title line when check 1 names a conflict. More detail stays on the page the Source links to.
|
|
149
|
+
- Gaps holds at most three lines, and only what limits the Answer: a conflict, something unconfirmed, something
|
|
150
|
+
not found. Do not list what you chose not to quote.
|
|
151
|
+
- Paste the --quote output unchanged. Do not reformat, renumber, add or remove quotes. To drop a quote, run
|
|
152
|
+
--quote again without its line.
|
|
153
|
+
|
|
154
|
+
**strong** means a quote on the exact concept page states the answer. **weak** means the answer comes from an
|
|
155
|
+
example, an index or landing page, a Beta, Pilot or Developer Preview page, or needs two quotes joined.
|
package/bin/install.mjs
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// npx sf-docs-agent [--project] [--yes]: install the sf-docs subagent for Claude Code.
|
|
3
|
+
// The runtime (script, playwright-core, cache home) is always user-level; only the agent file follows the scope.
|
|
4
|
+
import { cpSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
|
5
|
+
import { execFileSync, execSync } from 'node:child_process';
|
|
6
|
+
import { homedir } from 'node:os';
|
|
7
|
+
import { dirname, join, resolve } from 'node:path';
|
|
8
|
+
import { fileURLToPath, pathToFileURL } from 'node:url';
|
|
9
|
+
import { createInterface } from 'node:readline/promises';
|
|
10
|
+
|
|
11
|
+
const pkg = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
|
12
|
+
const config = process.env.CLAUDE_CONFIG_DIR || join(homedir(), '.claude');
|
|
13
|
+
const project = process.argv.includes('--project');
|
|
14
|
+
const runtime = join(config, 'sf-docs');
|
|
15
|
+
const agents = project ? resolve('.claude', 'agents') : join(config, 'agents');
|
|
16
|
+
|
|
17
|
+
mkdirSync(runtime, { recursive: true });
|
|
18
|
+
cpSync(join(pkg, 'fetch-docs.mjs'), join(runtime, 'fetch-docs.mjs'));
|
|
19
|
+
// A devDependency: npx would otherwise download playwright-core once for the installer and again for the runtime.
|
|
20
|
+
const version = JSON.parse(readFileSync(join(pkg, 'package.json'), 'utf8')).devDependencies['playwright-core'];
|
|
21
|
+
writeFileSync(join(runtime, 'package.json'),
|
|
22
|
+
JSON.stringify({ private: true, type: 'module', dependencies: { 'playwright-core': version } }, null, 2) + '\n');
|
|
23
|
+
console.log(`Installing playwright-core ${version} in ${runtime}`);
|
|
24
|
+
// A shell command string: on Windows npm is npm.cmd, which Node only spawns through a shell (CVE-2024-27980),
|
|
25
|
+
// and args alongside shell: true are deprecated (DEP0190). The args are constants, so nothing needs quoting.
|
|
26
|
+
try {
|
|
27
|
+
execSync('npm install --no-audit --no-fund --loglevel=error', { cwd: runtime, stdio: 'inherit' });
|
|
28
|
+
} catch {
|
|
29
|
+
// npm has already printed why; the agent is not installed yet, so a rerun starts clean.
|
|
30
|
+
console.error('\nnpm install failed (see above). The agent was not installed. Run the same command again once npm works.');
|
|
31
|
+
process.exit(1);
|
|
32
|
+
}
|
|
33
|
+
// The browser is the one large download, so it is never fetched without consent: ask at a terminal, take --yes in
|
|
34
|
+
// scripts, and otherwise print the command. If Chromium is missing at first use, the agent reports BLOCKED.
|
|
35
|
+
const cli = join(runtime, 'node_modules/playwright-core/cli.js');
|
|
36
|
+
let present = false;
|
|
37
|
+
try {
|
|
38
|
+
const { chromium } = await import(pathToFileURL(join(runtime, 'node_modules/playwright-core/index.mjs')).href);
|
|
39
|
+
await (await chromium.launch({ headless: true })).close();
|
|
40
|
+
present = true;
|
|
41
|
+
} catch { /* launch failing is the signal that the browser is missing */ }
|
|
42
|
+
const ask = async q => {
|
|
43
|
+
if (!process.stdin.isTTY) return false;
|
|
44
|
+
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
|
45
|
+
const a = await rl.question(q).catch(() => ''); // Ctrl+D or closed input means no
|
|
46
|
+
rl.close();
|
|
47
|
+
return /^y(es)?$/i.test(a.trim());
|
|
48
|
+
};
|
|
49
|
+
if (present) console.log('Headless Chromium is already installed.');
|
|
50
|
+
else if (process.argv.includes('--yes') || await ask('Download headless Chromium (about 180 MB)? [y/N] ')) {
|
|
51
|
+
try { execFileSync(process.execPath, [cli, 'install', '--only-shell', 'chromium'], { stdio: 'inherit' }); }
|
|
52
|
+
catch { console.error(`\nBrowser download failed (see above). Retry later with:\n node "${cli}" install --only-shell chromium`); }
|
|
53
|
+
} else console.log(`Skipped the browser download. Before first use, run:\n node "${cli}" install --only-shell chromium`);
|
|
54
|
+
|
|
55
|
+
// Point the agent at the installed script. rm first: a symlinked agent file would otherwise be written through.
|
|
56
|
+
// The source names its own checkout dir (script and cache); swap every occurrence for the runtime dir.
|
|
57
|
+
const source = readFileSync(join(pkg, 'agents/sf-docs.md'), 'utf8');
|
|
58
|
+
const devDir = source.match(/node "([^"]+)\/fetch-docs\.mjs"/)[1];
|
|
59
|
+
// Forward slashes: the agent runs these paths in Git Bash on Windows, where a backslash is an escape. A function
|
|
60
|
+
// replacement, so a $ in the path is taken literally.
|
|
61
|
+
const agent = source.replaceAll(devDir, () => runtime.replaceAll('\\', '/'));
|
|
62
|
+
mkdirSync(agents, { recursive: true });
|
|
63
|
+
rmSync(join(agents, 'sf-docs.md'), { force: true });
|
|
64
|
+
writeFileSync(join(agents, 'sf-docs.md'), agent);
|
|
65
|
+
console.log(`Installed agent: ${join(agents, 'sf-docs.md')}\nUse subagent_type "sf-docs". If it is not listed yet, start a new Claude Code session.`);
|
package/fetch-docs.mjs
ADDED
|
@@ -0,0 +1,488 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Fetch Salesforce doc pages through headless Chromium and write each unique body to disk.
|
|
3
|
+
// .md/.txt/.xml URLs, Help sitemaps and the .md twins of new docs, architect and LDS pages are fetched raw; any other
|
|
4
|
+
// URL is rendered and its main content extracted as text.
|
|
5
|
+
// Usage: node fetch-docs.mjs [--out <dir>] [--budget <s>] [--terms a,b] <url | /docs/path.md> ... (default out: ./cache here)
|
|
6
|
+
// node fetch-docs.mjs --search a,b [--site <host>] (ranked matching lines across the cached pages)
|
|
7
|
+
// node fetch-docs.mjs --quote <cache file>:<line>[,<line>...] ... (prints Evidence and Sources verbatim)
|
|
8
|
+
import { chromium } from 'playwright-core';
|
|
9
|
+
import { createHash } from 'node:crypto';
|
|
10
|
+
import { closeSync, mkdirSync, openSync, readFileSync, readSync, readdirSync, renameSync, rmSync, statSync, writeFileSync } from 'node:fs';
|
|
11
|
+
import { dirname, isAbsolute, join, relative, sep } from 'node:path';
|
|
12
|
+
import { fileURLToPath } from 'node:url';
|
|
13
|
+
|
|
14
|
+
// First selector with real text wins, so a wrapper of several topics (atlas nested0 or nested1, Help KB blocks)
|
|
15
|
+
// ranks above one topic; doc-amf-reference and doc-content serve pages whose text is all in shadow DOM.
|
|
16
|
+
const CONTENT = ['.slds-text-longform', 'div.container:has(.article-content)', '.post.entry', 'div.nested0',
|
|
17
|
+
'div.container:has(> div.nested1)', 'div.body', 'doc-amf-reference', 'doc-content', 'article', 'main'];
|
|
18
|
+
|
|
19
|
+
const args = process.argv.slice(2);
|
|
20
|
+
let out = join(dirname(fileURLToPath(import.meta.url)), 'cache');
|
|
21
|
+
const oi = args.indexOf('--out');
|
|
22
|
+
if (oi >= 0) out = args.splice(oi, 2)[1];
|
|
23
|
+
// No page starts after the budget: each one left is printed 'not fetched (time limit)', so the caller can rerun it.
|
|
24
|
+
const bi = args.indexOf('--budget');
|
|
25
|
+
const deadline = bi >= 0 ? Date.now() + Number(args.splice(bi, 2)[1]) * 1000 : Infinity;
|
|
26
|
+
// --terms a,b: after the fetch, print each saved page's lines that hold a term, so the caller needs no grep.
|
|
27
|
+
const termsOf = v => (v ?? '').split(',').map(s => s.trim().toLowerCase()).filter(Boolean);
|
|
28
|
+
const ti = args.indexOf('--terms');
|
|
29
|
+
const terms = ti >= 0 ? termsOf(args.splice(ti, 2)[1]) : [];
|
|
30
|
+
if (!args.length || !out || Number.isNaN(deadline) || (ti >= 0 && !terms.length)) { console.error('usage: node fetch-docs.mjs [--out dir] [--budget seconds] [--terms a,b] <url | /docs/path.md> ...\n node fetch-docs.mjs [--out dir] --search a,b [--site <host>]\n node fetch-docs.mjs [--out dir] --quote <cache file>:<line>[,<line>...] ...'); process.exit(2); }
|
|
31
|
+
const fileFor = u => join(out, u.host, (u.pathname + u.search.replace(/[^\w.-]+/g, '_')).replace(/\/$/, '/index') + (/\.(md|txt|xml)$/.test(u.pathname) ? '' : '.txt'));
|
|
32
|
+
// Cache window: SF_DOCS_CACHE_HOURS (default 24; 0 means always fetch). An empty value counts as unset.
|
|
33
|
+
const hours = Number(process.env.SF_DOCS_CACHE_HOURS || 24);
|
|
34
|
+
const MAX_AGE = (Number.isFinite(hours) && hours >= 0 ? hours : 24) * 3600 * 1000;
|
|
35
|
+
// Never delete a page younger than 10 minutes: a short window would otherwise remove pages another run, or this
|
|
36
|
+
// agent's own previous step, fetched moments ago. Serving still follows MAX_AGE.
|
|
37
|
+
const PRUNE_AGE = Math.max(MAX_AGE, 10 * 60 * 1000);
|
|
38
|
+
// A raw fetch that hangs (Help sitemaps do) would otherwise stall the whole call past the caller's tool timeout.
|
|
39
|
+
const RAW_TIMEOUT_MS = 30000;
|
|
40
|
+
// A file from before the Source: line existed is stale, so --quote can always name the official page. Only the first
|
|
41
|
+
// 8 bytes are read, once per URL per run: indexes run to megabytes.
|
|
42
|
+
const fresh = new Map();
|
|
43
|
+
const isFresh = u => {
|
|
44
|
+
if (!fresh.has(u.href)) {
|
|
45
|
+
let ok = false;
|
|
46
|
+
try {
|
|
47
|
+
if (Date.now() - statSync(fileFor(u)).mtimeMs < MAX_AGE) {
|
|
48
|
+
const fd = openSync(fileFor(u), 'r'), b = Buffer.alloc(8);
|
|
49
|
+
try { readSync(fd, b, 0, 8, 0); } finally { closeSync(fd); }
|
|
50
|
+
// A fallback read may be one section or a sidebar of a slow page: refetch it rather than trust it for a day.
|
|
51
|
+
ok = b.toString() === 'Source: ' && !statSync(`${fileFor(u)}.fallback`, { throwIfNoEntry: false });
|
|
52
|
+
}
|
|
53
|
+
} catch { /* missing or unreadable: not fresh */ }
|
|
54
|
+
fresh.set(u.href, ok);
|
|
55
|
+
}
|
|
56
|
+
return fresh.get(u.href);
|
|
57
|
+
};
|
|
58
|
+
// A twin that answered 404 or a shell leaves an empty <file>.missing, so later runs render without asking again;
|
|
59
|
+
// a transient failure (status 0, 5xx) leaves none and the twin is retried.
|
|
60
|
+
const noTwin = t => { try { return Date.now() - statSync(`${fileFor(t)}.missing`).mtimeMs < MAX_AGE; } catch { return false; } };
|
|
61
|
+
// Failing to write a marker only costs a retry later, so it is reported and the run goes on.
|
|
62
|
+
const markMissing = t => {
|
|
63
|
+
const f = `${fileFor(t)}.missing`;
|
|
64
|
+
try { mkdirSync(dirname(f), { recursive: true }); writeFileSync(f, ''); }
|
|
65
|
+
catch (e) { console.error(`could not record missing twin ${slash(f)}: ${e.message}`); }
|
|
66
|
+
};
|
|
67
|
+
// The page a person would open: developer serves .md twins at .html, architect at the bare path.
|
|
68
|
+
const official = u => !u.pathname.endsWith('.md') ? u.href
|
|
69
|
+
: u.href.replace(/\.md$/, u.host === 'developer.salesforce.com' ? '.html' : '');
|
|
70
|
+
const SOURCE = /^Source: [^\n]*\n/;
|
|
71
|
+
// An HTML app shell where text was expected (a .md path with no twin answers 200 with one).
|
|
72
|
+
const isShell = b => /^\s*<(!doctype|html)/i.test(b);
|
|
73
|
+
// A twin is missing (render instead, and remember it) on a 404, or a 200 whose body is an HTML shell or empty.
|
|
74
|
+
const missingTwin = (status, body) => status === 404 || (status === 200 && (isShell(body) || !body.trim()));
|
|
75
|
+
// One page, whatever the #anchor or trailing slash; a Help article is told apart by its ?id=.
|
|
76
|
+
const sameDoc = (a, b) => a.origin === b.origin && a.pathname.replace(/\/$/, '') === b.pathname.replace(/\/$/, '')
|
|
77
|
+
&& a.searchParams.get('id') === b.searchParams.get('id');
|
|
78
|
+
// Paths the agent will paste into Git Bash on Windows: forward slashes, which Node accepts too.
|
|
79
|
+
const slash = p => p.split(sep).join('/');
|
|
80
|
+
|
|
81
|
+
// --terms and --search: lines holding a term, numbered as --quote counts them; past a cap the rarest-term lines win, in
|
|
82
|
+
// page order. Caps set on a clean 20-question sample (looser caps printed twice the old grep's 2k chars a call).
|
|
83
|
+
const MAX_HITS = 20, MAX_LINKS = 10, SHORT_LINES = 40, SHORT_CHARS = 4000, PER_FILE = 3;
|
|
84
|
+
const matches = (name, text, t) => text.split('\n').flatMap((line, i, ls) => {
|
|
85
|
+
const low = i || !ls[0].startsWith('Source: ') ? line.toLowerCase() : '', hit = t.filter(s => low.includes(s)); // a page's line 1 is its Source line; a sidecar has none
|
|
86
|
+
return hit.length ? [{ name, n: i + 1, line, hit, at: Math.min(...hit.map(s => low.indexOf(s))) }] : [];
|
|
87
|
+
});
|
|
88
|
+
const rank = (ms, max = MAX_HITS) => {
|
|
89
|
+
if (ms.length <= max) return ms;
|
|
90
|
+
const df = Object.create(null); // a term like 'constructor' must not read Object.prototype
|
|
91
|
+
for (const m of ms) for (const s of m.hit) df[s] = (df[s] || 0) + 1;
|
|
92
|
+
const score = m => m.hit.reduce((a, s) => a + 1 / df[s], 0);
|
|
93
|
+
return ms.map((m, k) => [score(m), k, m]).sort((a, b) => b[0] - a[0] || a[1] - b[1]).slice(0, max)
|
|
94
|
+
.sort((a, b) => a[1] - b[1]).map(x => x[2]);
|
|
95
|
+
};
|
|
96
|
+
const show = m => `${m.n}: ${m.line.length > 300 ? `...${m.line.slice(Math.max(0, m.at - 100), m.at + 200)}...` : m.line}`;
|
|
97
|
+
const more = (all, top) => all.length > top.length ? [`(${all.length - top.length} more lower-ranked lines: add a term to narrow)`] : [];
|
|
98
|
+
// A page this short is printed whole: its answer can sit on a line no term matches (a Help page's "95%" line).
|
|
99
|
+
// Index files by shape (Help's /apex/Help_*SiteMap* pages, *sitemap*.xml), on the path under the cache dir: a doc
|
|
100
|
+
// page about sitemaps, or an --out dir named for them, is neither.
|
|
101
|
+
const indexKind = rel => /(^|\/)apex\/Help_\w*SiteMapIndex|sitemap_index[^/]*\.xml$/i.test(rel) ? 'index'
|
|
102
|
+
: /(^|\/)apex\/Help_\w*SiteMap|[^/]*sitemap[^/]*\.xml$/i.test(rel) ? 'map' : '';
|
|
103
|
+
const grep = (file, t) => {
|
|
104
|
+
const text = readFileSync(file, 'utf8'), lines = text.split('\n'), kind = indexKind(slash(relative(out, file)));
|
|
105
|
+
if (!file.endsWith('.links.txt') && lines.length <= SHORT_LINES && text.length <= SHORT_CHARS)
|
|
106
|
+
return ['(short page, shown whole)', ...lines.slice(1, text.endsWith('\n') ? -1 : undefined).map((l, i) => `${i + 2}: ${l}`)]; // line 1 is the Source line
|
|
107
|
+
// A sitemap index names page sitemaps, not topics, so no term can pick among them: print every <loc> line.
|
|
108
|
+
if (kind === 'index') return matches(null, text, ['<loc>']).map(show);
|
|
109
|
+
// A sitemap lists each article twice (<loc> and its alternates): keep the <loc> line and show 40 articles.
|
|
110
|
+
const map = kind === 'map', all = matches(null, text, t).filter(m => !map || m.line.includes('<loc>'));
|
|
111
|
+
const top = rank(all, map ? 40 : file.endsWith('.links.txt') ? MAX_LINKS : MAX_HITS);
|
|
112
|
+
return [...top.map(show), ...more(all, top)];
|
|
113
|
+
};
|
|
114
|
+
|
|
115
|
+
// --quote: the agent picks line numbers, the script copies the lines, so quotes are verbatim by construction.
|
|
116
|
+
if (args[0] === '--quote') {
|
|
117
|
+
const files = [], ev = [];
|
|
118
|
+
for (const spec of args.slice(1)) {
|
|
119
|
+
const at = spec.lastIndexOf(':');
|
|
120
|
+
const file = isAbsolute(spec.slice(0, at)) ? spec.slice(0, at) : join(out, spec.slice(0, at));
|
|
121
|
+
// Only cached official pages: inside the cache dir and starting with their Source: line.
|
|
122
|
+
const inside = relative(out, file);
|
|
123
|
+
let lines;
|
|
124
|
+
try { lines = inside.startsWith('..') || isAbsolute(inside) ? [] : readFileSync(file, 'utf8').split('\n'); }
|
|
125
|
+
catch { lines = []; }
|
|
126
|
+
if (!lines[0]?.startsWith('Source: ')) { console.error(`not a cached page: ${slash(file)}`); continue; }
|
|
127
|
+
if (!files.includes(file)) files.push(file);
|
|
128
|
+
for (const n of spec.slice(at + 1).split(',').map(Number)) {
|
|
129
|
+
const text = (lines[n - 1] || '').trim();
|
|
130
|
+
if (text && n > 1) ev.push(`- "${text}" [${files.indexOf(file) + 1}:${n}]`);
|
|
131
|
+
else console.error(`skipped ${slash(relative(out, file))}:${n}: empty, out of range or the Source line`);
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
console.log(['Evidence:', ...ev, 'Sources:', ...files.map((f, i) => {
|
|
135
|
+
const src = (readFileSync(f, 'utf8').match(SOURCE) || ['Source: (unknown) '])[0].slice(8).trim();
|
|
136
|
+
return `[${i + 1}] ${src}\n cache: ${slash(f)}`;
|
|
137
|
+
})].join('\n'));
|
|
138
|
+
process.exit(ev.length ? 0 : 1);
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
// The official doc sites: the only hosts fetched, saved or pruned, so every cached page and quote is Salesforce's own.
|
|
142
|
+
const SITES = ['developer.salesforce.com', 'architect.salesforce.com', 'help.salesforce.com', 'admin.salesforce.com',
|
|
143
|
+
'www.lightningdesignsystem.com', 'v1.lightningdesignsystem.com'];
|
|
144
|
+
// Every file under the given site folders, sorted. A missing folder is a site not cached yet; any other error is printed.
|
|
145
|
+
const cacheFiles = (sites, label) => sites.flatMap(s => {
|
|
146
|
+
try { return readdirSync(join(out, s), { recursive: true }).sort().map(rel => join(out, s, rel)); }
|
|
147
|
+
catch (e) { if (e.code !== 'ENOENT') console.error(`${label}: ${e.message}`); return []; }
|
|
148
|
+
});
|
|
149
|
+
|
|
150
|
+
// --search: ranked lines across the cache, at most PER_FILE a file. It does not fetch, so it can see expired pages:
|
|
151
|
+
// fetch first in a run, which clears them.
|
|
152
|
+
if (args[0] === '--search') {
|
|
153
|
+
const t = termsOf(args[1]?.startsWith('-') ? '' : args[1]);
|
|
154
|
+
const si = args.indexOf('--site'), site = (si >= 0 ? args[si + 1] || '' : '').replace(/^lightningdesignsystem\.com$/, 'www.$&');
|
|
155
|
+
// --site with no value is an error, not a search of every site.
|
|
156
|
+
if (!t.length || (si >= 0 && !SITES.includes(site))) { console.error(`usage: --search a,b [--site <one of ${SITES.join(', ')}>]`); process.exit(2); }
|
|
157
|
+
const all = [];
|
|
158
|
+
let total = 0;
|
|
159
|
+
for (const f of cacheFiles(site ? [site] : SITES, 'search')) {
|
|
160
|
+
const rel = slash(relative(out, f));
|
|
161
|
+
// Sidecars and markers are not pages; a file another run pruned mid-walk is skipped.
|
|
162
|
+
if (/\.(links\.txt|missing|tmp|fallback)$/.test(rel) || !statSync(f, { throwIfNoEntry: false })?.isFile()) continue;
|
|
163
|
+
if (/(^|\/)llms[^/]*\.txt$/i.test(rel) || indexKind(rel)) continue; // indexes: --terms on the index already shows them
|
|
164
|
+
// At most PER_FILE lines a file, so ties across files cannot all come from the one that sorts first.
|
|
165
|
+
try { const ms = matches(rel, readFileSync(f, 'utf8'), t); total += ms.length; all.push(...rank(ms, PER_FILE)); }
|
|
166
|
+
catch (e) { if (e.code !== 'ENOENT') throw e; }
|
|
167
|
+
}
|
|
168
|
+
const top = rank(all), rest = total - top.length;
|
|
169
|
+
console.log([...top.map(m => `${m.name}:${show(m)}`), ...(rest ? [`(${rest} more lower-ranked lines: add a term or --site to narrow)`] : [])].join('\n') || `no match in ${site || 'the cache'}`);
|
|
170
|
+
process.exit(0);
|
|
171
|
+
}
|
|
172
|
+
// A scheme-less host would resolve as a developer.salesforce.com path; sitemap <loc> lines write & as & or &.
|
|
173
|
+
// Anything that is not an http(s) URL (a stray flag, 'https://', ftp:) is refused rather than crashing the call.
|
|
174
|
+
const parse = a => {
|
|
175
|
+
const s = a.replace(/&(#38|amp);/g, '&'), v = /^([\w-]+\.)*(salesforce|lightningdesignsystem)\.com([/?]|$)/i.test(s) ? `https://${s}` : s;
|
|
176
|
+
if (s.startsWith('-') || !URL.canParse(v, 'https://developer.salesforce.com')) return null;
|
|
177
|
+
const u = new URL(v, 'https://developer.salesforce.com');
|
|
178
|
+
// lightningdesignsystem.com serves the same site as www., which the twin route and the cache prune expect.
|
|
179
|
+
if (u.hostname === 'lightningdesignsystem.com') u.hostname = 'www.lightningdesignsystem.com';
|
|
180
|
+
return /^https?:$/.test(u.protocol) ? u : null;
|
|
181
|
+
};
|
|
182
|
+
// One pass over the arguments: a URL on a doc site is fetched, anything else is refused with its reason.
|
|
183
|
+
const given = [], refused = [];
|
|
184
|
+
for (const a of args) {
|
|
185
|
+
const u = parse(a);
|
|
186
|
+
if (u && SITES.includes(u.host)) given.push(u);
|
|
187
|
+
else refused.push(u ? `${u.href}: not an official doc site` : `${a}: not a URL`);
|
|
188
|
+
}
|
|
189
|
+
// New developer docs, architect and LDS pages have a .md twin: fetch it raw rather than render the HTML page.
|
|
190
|
+
const twinOf = u => {
|
|
191
|
+
const p = u.pathname.replace(/\.html$/, '').replace(/\/$/, '');
|
|
192
|
+
const has = !u.search && !/\.(md|txt|xml|htm)$/.test(p) && (
|
|
193
|
+
u.host === 'developer.salesforce.com' ? /^\/docs\/(?!atlas\.|component-library\/)[^/]+\/[^/]+\/[^/]+\/./.test(p)
|
|
194
|
+
: u.host === 'architect.salesforce.com' ? /^\/docs\/architect\/./.test(p)
|
|
195
|
+
: u.host === 'www.lightningdesignsystem.com' && /^\/\w+\/(v\/\d+\/)?p\/./.test(p));
|
|
196
|
+
return has ? new URL(`${u.origin}${p}.md`) : null;
|
|
197
|
+
};
|
|
198
|
+
const asked = new Map(); // twin href -> URL as given, rendered instead when the twin is missing
|
|
199
|
+
// foo.html and foo are one page (one twin): fetch it once, and print 'same <arg> as <first arg>' for the others.
|
|
200
|
+
const urls = [], sameAs = [], first = new Map();
|
|
201
|
+
for (const u of given) {
|
|
202
|
+
const t = twinOf(u), k = t ? t.href : u.href;
|
|
203
|
+
// A .md given before its page: this page becomes the fallback (or is rendered now, if the twin is known missing).
|
|
204
|
+
const mdFirst = t && !asked.has(t.href) && urls.some(x => x.href === t.href);
|
|
205
|
+
if (mdFirst && noTwin(t)) { urls.push(u); continue; }
|
|
206
|
+
if (mdFirst) asked.set(t.href, u);
|
|
207
|
+
if (first.has(k)) { sameAs.push([u.href, first.get(k)]); continue; }
|
|
208
|
+
if (!t || (!isFresh(t) && noTwin(t))) urls.push(u);
|
|
209
|
+
else { asked.set(t.href, u); urls.push(t); }
|
|
210
|
+
first.set(k, urls.at(-1).href); // the key whose result line the 'same' line points to
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
function extract([sels, fallback]) {
|
|
214
|
+
const roots = [document];
|
|
215
|
+
for (let i = 0; i < roots.length; i++) for (const el of roots[i].querySelectorAll('*')) if (el.shadowRoot) roots.push(el.shadowRoot);
|
|
216
|
+
// Same-site links (nav and TOC included) so pages without an llms.txt index can still be discovered.
|
|
217
|
+
const links = new Set();
|
|
218
|
+
for (const r of roots) for (const a of r.querySelectorAll('a[href]')) {
|
|
219
|
+
if (a.host === location.host && a.pathname !== location.pathname) links.add(`${a.pathname}${a.search} ${(a.innerText || '').trim().replace(/\s+/g, ' ').slice(0, 80)}`);
|
|
220
|
+
}
|
|
221
|
+
const page = t => `# ${document.title}\n\n${t}\n`;
|
|
222
|
+
const list = [...links].join('\n');
|
|
223
|
+
// innerText stops at shadow roots (code samples, API references), so walk them in rendered order: hosts, then slots.
|
|
224
|
+
// Like innerText: hidden elements skipped, cells tab-separated; code whitespace held as no-break spaces until the end.
|
|
225
|
+
const SKIP = new Set(['STYLE', 'SCRIPT', 'NOSCRIPT', 'TEMPLATE']);
|
|
226
|
+
const deep = (n, pre) => [...(n.shadowRoot ? n.shadowRoot.childNodes : n.tagName === 'SLOT' ? n.assignedNodes({ flatten: true })
|
|
227
|
+
: n.childNodes)].map(c => {
|
|
228
|
+
if (c.nodeType === 3) return pre ? c.textContent.replace(/ /g, '\u00a0') : c.textContent.replace(/\s+/g, ' ');
|
|
229
|
+
if (c.nodeType !== 1 || SKIP.has(c.tagName)) return '';
|
|
230
|
+
if (c.tagName === 'BR') return '\n';
|
|
231
|
+
const st = getComputedStyle(c);
|
|
232
|
+
if (st.display === 'none' || st.visibility === 'hidden') return '';
|
|
233
|
+
const t = deep(c, pre || st.whiteSpace.startsWith('pre'));
|
|
234
|
+
return /^(inline|contents)/.test(st.display) ? t : st.display === 'table-cell' ? `${t}\t` : `\n${t}\n`;
|
|
235
|
+
}).join('');
|
|
236
|
+
const hosts = roots.slice(1).map(r => r.host);
|
|
237
|
+
// contains() stops at a shadow boundary, so climb each host's roots to see whether it sits anywhere under el.
|
|
238
|
+
const under = (el, h) => { for (let n = h; n; n = n.getRootNode().host) if (el.contains(n)) return true; return false; };
|
|
239
|
+
const text = el => !hosts.some(h => el.contains(h)) ? el.innerText || ''
|
|
240
|
+
: deep(el).replace(/ +/g, ' ').replace(/ ?\t ?/g, '\t').replace(/^ | $|\t+$/gm, '').replace(/\n{2,}/g, '\n')
|
|
241
|
+
.replace(/\u00a0/g, ' ').trim();
|
|
242
|
+
for (const s of sels) for (const r of roots) for (const el of r.querySelectorAll(s)) {
|
|
243
|
+
// Landmarks hold the site's nav components too, so their shadow trees stay out.
|
|
244
|
+
const mark = s === 'main' || s === 'article';
|
|
245
|
+
const t = mark ? el.innerText || '' : text(el);
|
|
246
|
+
// 200 chars skips footer cards that also match div.body; doc-content holds only the topic, and some are one line.
|
|
247
|
+
// pending: components inside it (code samples) not rendered yet. An API reference keeps empty hosts for good, so
|
|
248
|
+
// it holds instead until an api-* documentation component exists.
|
|
249
|
+
const amf = s === 'doc-amf-reference';
|
|
250
|
+
if (t.length > (s === 'doc-content' ? 20 : 200)) return { sel: s, text: page(t), links: list,
|
|
251
|
+
pending: mark || amf ? 0 : hosts.filter(h => under(el, h) && !h.shadowRoot.childElementCount).length,
|
|
252
|
+
hold: amf && !hosts.some(h => h.tagName.startsWith('API-') && under(el, h)) };
|
|
253
|
+
}
|
|
254
|
+
// No known container (a layout change): take the block whose paragraphs hold the most text, outside nav and footers.
|
|
255
|
+
// ponytail: 1,500-char floor keeps shells out (missing pages, Help's outage page: 400 to 1,200); a shorter article still fails.
|
|
256
|
+
if (fallback) {
|
|
257
|
+
const BLOCKS = 'p, li, pre, td, dd, h1, h2, h3', score = new Map(), up = n => n.parentElement || n.getRootNode().host;
|
|
258
|
+
// closest() stops at a shadow root, so climb each host too: a nav component's own template may have no <nav>.
|
|
259
|
+
const inNav = el => { for (let n = el; n; n = n.getRootNode().host) if (n.closest('nav, header, footer, [role=navigation]')) return true; return false; };
|
|
260
|
+
for (const r of roots) for (const el of r.querySelectorAll(BLOCKS)) {
|
|
261
|
+
if (inNav(el) || !el.checkVisibility()) continue; // a hidden block's innerText is its whole textContent
|
|
262
|
+
// Its own visible text: nested blocks score on their own, so a nested list counts once and a step keeps its prose.
|
|
263
|
+
const kids = [...el.querySelectorAll(BLOCKS)].filter(c => c.parentElement?.closest(BLOCKS) === el);
|
|
264
|
+
const len = e => (e.innerText || '').trim().length, n = Math.max(0, len(el) - kids.reduce((a, c) => a + len(c), 0));
|
|
265
|
+
const p = up(el), g = p && up(p);
|
|
266
|
+
if (p) score.set(p, (score.get(p) || 0) + n);
|
|
267
|
+
if (g) score.set(g, (score.get(g) || 0) + n / 2);
|
|
268
|
+
}
|
|
269
|
+
const best = [...score].sort((a, b) => b[1] - a[1])[0]?.[0], t = best ? text(best) : '';
|
|
270
|
+
if (t.length >= 1500) return { sel: 'fallback', text: page(t), links: list };
|
|
271
|
+
}
|
|
272
|
+
return { sel: 'body', text: page(document.body.innerText), links: list };
|
|
273
|
+
}
|
|
274
|
+
|
|
275
|
+
// Delete expired pages first, so a grep or Read over the cache afterwards only sees fresh ones.
|
|
276
|
+
// Not in --quote mode: with a 0-hour window it would delete the pages this run just fetched.
|
|
277
|
+
// Only the site folders: --out may point anywhere, and nothing else in it is ours to delete.
|
|
278
|
+
let pruned = 0;
|
|
279
|
+
for (const f of cacheFiles(SITES, 'cache cleanup')) {
|
|
280
|
+
try {
|
|
281
|
+
const st = statSync(f);
|
|
282
|
+
if (st.isFile() && Date.now() - st.mtimeMs >= PRUNE_AGE) { rmSync(f); pruned++; }
|
|
283
|
+
} catch (e) { if (e.code !== 'ENOENT') console.error(`cache cleanup: ${e.message}`); } // ENOENT: another run removed it
|
|
284
|
+
}
|
|
285
|
+
if (pruned) console.error(`removed ${pruned} expired cache files`);
|
|
286
|
+
|
|
287
|
+
// A stored Source unlike the URL asked for means the page redirected: say so again, as the first run did. A file that
|
|
288
|
+
// vanished or lost its Source line since the freshness check returns null and is fetched again.
|
|
289
|
+
const fromCache = u => {
|
|
290
|
+
let t;
|
|
291
|
+
try { t = readFileSync(fileFor(u), 'utf8'); } catch { return null; }
|
|
292
|
+
const m = t.match(SOURCE);
|
|
293
|
+
if (!m) return null;
|
|
294
|
+
const src = m[0].slice(8).trim();
|
|
295
|
+
return { p: u.origin + u.pathname + u.search, status: 200, body: t.slice(m[0].length), key: u.href, cached: true,
|
|
296
|
+
note: ` (cached)${URL.canParse(src) && !sameDoc(new URL(src), new URL(official(u))) ? ` [redirected ${src}]` : ''}` };
|
|
297
|
+
};
|
|
298
|
+
// A reader never sees half a file: write a temp copy, then rename it into place. The pid in its name keeps two runs
|
|
299
|
+
// saving one page from renaming each other's copy.
|
|
300
|
+
const put = (f, data) => { const t = `${f}.${process.pid}.tmp`; mkdirSync(dirname(f), { recursive: true }); writeFileSync(t, data); renameSync(t, f); };
|
|
301
|
+
for (const r of refused) console.log(`refused ${r}`);
|
|
302
|
+
const results = [], seen = new Map(), saved = new Set(); // saved: files --terms prints matches from
|
|
303
|
+
let ok = 0;
|
|
304
|
+
// A saved page drops what is not doc text: image URLs (LDS signs S3 URLs of about 450 chars; the alt text stays) and
|
|
305
|
+
// the padding in markdown table rows. Code blocks and wording are untouched.
|
|
306
|
+
const lean = text => {
|
|
307
|
+
let code = ''; // the open fence: closed only by the same character, at least as long (CommonMark)
|
|
308
|
+
return text.split('\n').map(l => {
|
|
309
|
+
const f = /^\s*(`{3,}|~{3,})/.exec(l)?.[1];
|
|
310
|
+
if (f && !code) { code = f; return l; }
|
|
311
|
+
if (f && f[0] === code[0] && f.length >= code.length) { code = ''; return l; }
|
|
312
|
+
if (code) return l;
|
|
313
|
+
l = l.replace(/!\[([^\]]*)\]\(https?:\/\/[^)\s]+\)/g, '![$1]');
|
|
314
|
+
return /^\s*\|/.test(l) ? l.replace(/ {2,}/g, ' ') : l;
|
|
315
|
+
}).join('\n');
|
|
316
|
+
};
|
|
317
|
+
// Each page is saved and printed as it lands, so a call stopped part way keeps every page before the stop.
|
|
318
|
+
const emit = r => {
|
|
319
|
+
results.push(r);
|
|
320
|
+
const { p, status, links, key, src, note, cached, shell, offsite } = r;
|
|
321
|
+
// Markdown only (a .md URL, or a twin read as r.md): rendered text has no code fences, so lean could reach a code
|
|
322
|
+
// sample. Cached pages are lean already.
|
|
323
|
+
const md = r.md || /\.md$/.test(new URL(key).pathname);
|
|
324
|
+
const body = status === 200 && !cached && md ? lean(r.body) : r.body;
|
|
325
|
+
const h = createHash('sha1').update(body).digest('hex').slice(0, 12);
|
|
326
|
+
const u = new URL(key);
|
|
327
|
+
const file = fileFor(u);
|
|
328
|
+
let tail = '';
|
|
329
|
+
if (status === 0) tail = ` ${body}`;
|
|
330
|
+
else if (offsite) tail = ' not saved: left the doc sites'; // a redirect to a login page or another domain
|
|
331
|
+
else if (shell) tail = ' not saved: shell, login or missing page';
|
|
332
|
+
else if (status === 200) {
|
|
333
|
+
ok++;
|
|
334
|
+
// Duplicates are written too, so a later run for that URL finds it; written whole (temp then rename), and one
|
|
335
|
+
// unwritable page only loses its own line. Links go to a sidecar so a Read of the page costs only its content.
|
|
336
|
+
let werr = '';
|
|
337
|
+
if (!cached) {
|
|
338
|
+
try { // a fallback marker goes on before its page and comes off only after a full read is on disk, so a failed
|
|
339
|
+
// write never leaves a fallback page that reads as full
|
|
340
|
+
if (r.fallback) put(`${file}.fallback`, '');
|
|
341
|
+
put(file, `Source: ${src || official(u)}\n${body}`); // first line names the official page
|
|
342
|
+
}
|
|
343
|
+
catch (e) { werr = ` not saved: ${e.code || e.message}`; ok--; }
|
|
344
|
+
// The page is saved either way; a marker left behind only means it is fetched again next time.
|
|
345
|
+
try { if (!r.fallback && !werr) rmSync(`${file}.fallback`, { force: true }); }
|
|
346
|
+
catch (e) { console.error(`fallback marker not removed for ${p}: ${e.code || e.message}`); }
|
|
347
|
+
try { if (links && !werr) put(`${file}.links.txt`, `${links}\n`); }
|
|
348
|
+
catch (e) { console.error(`links not saved for ${p}: ${e.code || e.message}`); }
|
|
349
|
+
}
|
|
350
|
+
const rel = slash(relative(out, file));
|
|
351
|
+
if (!werr && !seen.has(h)) saved.add(file); // a duplicate's matches are the first copy's
|
|
352
|
+
tail = werr || (seen.has(h) ? ` duplicate-of ${seen.get(h)} -> ${rel}` : ` -> ${rel}`);
|
|
353
|
+
if (!werr && !seen.has(h)) seen.set(h, p); // an unsaved body can never be the copy others point to
|
|
354
|
+
}
|
|
355
|
+
console.log(`${status} ${body.length} ${p}${note}${tail}`);
|
|
356
|
+
};
|
|
357
|
+
// The URL as given, so the caller reruns the page it asked for rather than its .md twin.
|
|
358
|
+
const skippedAs = new Map(); // key -> the URL its time-limit line printed, for the same lines
|
|
359
|
+
const skip = key => { skippedAs.set(key, asked.get(key)?.href ?? key); console.log(`not fetched (time limit) ${skippedAs.get(key)}`); };
|
|
360
|
+
for (const u of urls.filter(isFresh)) { const r = fromCache(u); if (r) emit(r); else fresh.set(u.href, false); }
|
|
361
|
+
|
|
362
|
+
let browser = { newPage() {}, close() {} };
|
|
363
|
+
if (!urls.every(isFresh) && Date.now() < deadline) { // a spent budget skips every page, so no browser
|
|
364
|
+
try { browser = await chromium.launch({ headless: true }); }
|
|
365
|
+
catch (e) {
|
|
366
|
+
console.error(`BLOCKED: headless Chromium did not start: ${e.message.split('\n')[0]}`);
|
|
367
|
+
// The exact command, not "reinstall": a reinstall without a terminal skips the browser again.
|
|
368
|
+
const cli = slash(join(dirname(fileURLToPath(import.meta.url)), 'node_modules/playwright-core/cli.js'));
|
|
369
|
+
console.error(`Install the browser with: node "${cli}" install --only-shell chromium`);
|
|
370
|
+
console.error('On Linux, also run once: npx playwright-core install-deps chromium');
|
|
371
|
+
process.exit(3);
|
|
372
|
+
}
|
|
373
|
+
}
|
|
374
|
+
try {
|
|
375
|
+
// Cached: a page written in the last 24 h is read from disk, so later agents and sessions skip the fetch.
|
|
376
|
+
const todo = urls.filter(u => !isFresh(u));
|
|
377
|
+
// Help serves its sitemaps (and the KB ones, Help_DocSiteMap...) from extensionless /apex/ URLs; others end .xml.
|
|
378
|
+
const raw = todo.filter(u => /\.(md|txt|xml)$/.test(u.pathname) || /^\/apex\/Help_\w*SiteMap/i.test(u.pathname));
|
|
379
|
+
const rendered = todo.filter(u => !raw.includes(u));
|
|
380
|
+
// A twin that failed (some API reference pages have none) falls back to the URL as given, rendered at the end.
|
|
381
|
+
const settle = r => {
|
|
382
|
+
if (!asked.has(r.key) || (r.status === 200 && !r.shell)) return emit(r);
|
|
383
|
+
if (missingTwin(r.status, r.body)) markMissing(new URL(r.key));
|
|
384
|
+
const u = asked.get(r.key);
|
|
385
|
+
const c = isFresh(u) && fromCache(u);
|
|
386
|
+
if (c) emit(c); else rendered.push(u);
|
|
387
|
+
};
|
|
388
|
+
|
|
389
|
+
// Raw: one same-origin page per host, then in-page fetch() so requests carry the browser's own session.
|
|
390
|
+
for (const origin of new Set(raw.map(u => u.origin))) {
|
|
391
|
+
const paths = raw.filter(u => u.origin === origin).map(u => u.pathname + u.search);
|
|
392
|
+
if (Date.now() >= deadline) { paths.forEach(p => skip(origin + p)); continue; }
|
|
393
|
+
const page = await browser.newPage();
|
|
394
|
+
// light same-origin page; every doc site serves one. A dead host fails its own URLs, not the run.
|
|
395
|
+
const err = await page.goto(`${origin}/robots.txt`).then(() => null, e => e.message.split('\n')[0]);
|
|
396
|
+
if (err) {
|
|
397
|
+
paths.forEach(p => settle({ p, status: 0, body: `error: ${err}`, key: origin + p, note: '' }));
|
|
398
|
+
await page.close();
|
|
399
|
+
continue;
|
|
400
|
+
}
|
|
401
|
+
// Five fetches in flight; a slot takes the next path as soon as it frees, so one slow URL holds only its slot.
|
|
402
|
+
const got = await page.evaluate(async ([ps, TO, DL]) => {
|
|
403
|
+
const res = []; let next = 0;
|
|
404
|
+
await Promise.all(Array.from({ length: 5 }, async () => {
|
|
405
|
+
while (next < ps.length) {
|
|
406
|
+
const k = next++, p = ps[k];
|
|
407
|
+
if (Date.now() >= DL) { res[k] = { p, late: true }; continue; }
|
|
408
|
+
try { const r = await fetch(p, { signal: AbortSignal.timeout(TO) }); res[k] = { p, status: r.status, body: await r.text() }; }
|
|
409
|
+
catch (e) { res[k] = { p, status: 0, body: `error: ${e.message}` }; }
|
|
410
|
+
}
|
|
411
|
+
}));
|
|
412
|
+
return res;
|
|
413
|
+
}, [paths, RAW_TIMEOUT_MS, deadline]);
|
|
414
|
+
// A .md path with no twin (all atlas.en-us.* pages) answers 200 with the HTML app shell: never cache that.
|
|
415
|
+
for (const b of got) {
|
|
416
|
+
if (b.late) skip(origin + b.p);
|
|
417
|
+
else settle({ ...b, key: origin + b.p, note: '', shell: isShell(b.body) || !b.body.trim() });
|
|
418
|
+
}
|
|
419
|
+
await page.close();
|
|
420
|
+
}
|
|
421
|
+
|
|
422
|
+
// Rendered: load, give client-side content up to 20 s to appear, then take the first matching container.
|
|
423
|
+
for (const u of rendered) {
|
|
424
|
+
if (Date.now() >= deadline) { skip(u.href); continue; }
|
|
425
|
+
const page = await browser.newPage();
|
|
426
|
+
let navErr = '';
|
|
427
|
+
const nav = await page.goto(u.href, { waitUntil: 'load', timeout: 45000 }).catch(e => { navErr = e.message.split('\n')[0]; return null; });
|
|
428
|
+
if (navErr) { emit({ p: u.href, status: 0, body: `error: ${navErr}`, key: u.href, note: '' }); await page.close(); continue; }
|
|
429
|
+
// Poll from Node: waitForFunction needs string eval, which some sites' CSP refuses (zeroheight does).
|
|
430
|
+
let got = { sel: 'body', text: '' }, mainAt = 0, matchAt = 0, last = '', same = 0;
|
|
431
|
+
const peak = {};
|
|
432
|
+
for (const end = Date.now() + 20000; Date.now() < end; await page.waitForTimeout(500)) {
|
|
433
|
+
got = await page.evaluate(extract, [CONTENT, false]).catch(e => ({ sel: 'body', text: `extract failed: ${e.message}` }));
|
|
434
|
+
// `main` can be only the site nav while the article still renders (LDS): give a better match 5 s to appear.
|
|
435
|
+
if (got.sel === 'main' && Date.now() - (mainAt ||= Date.now()) < 5000) continue;
|
|
436
|
+
// A container can appear before it is filled (API reference nav first, code samples last): wait for 1 s
|
|
437
|
+
// unchanged with nothing pending, 4 s if a component never renders, 8 s after it last grew if it never settles.
|
|
438
|
+
same = `${got.text.length}/${got.pending}` === last ? same + 1 : 0;
|
|
439
|
+
last = `${got.text.length}/${got.pending}`;
|
|
440
|
+
// The cap restarts only when a container passes its own peak: body text is ignored, and flips do not restart it.
|
|
441
|
+
if (got.sel !== 'body' && got.text.length > (peak[got.sel] || 0)) { peak[got.sel] = got.text.length; matchAt = Date.now(); }
|
|
442
|
+
if (got.sel !== 'body' && !got.hold && (same >= (got.pending ? 8 : 2) || Date.now() - matchAt > 8000)) break;
|
|
443
|
+
}
|
|
444
|
+
// Only after the wait: inside it, the fallback would win before the real container rendered.
|
|
445
|
+
if (got.sel === 'body') got = await page.evaluate(extract, [CONTENT, true])
|
|
446
|
+
.catch(e => { console.error(`fallback failed on ${u.href}: ${e.message.split('\n')[0]}`); return got; });
|
|
447
|
+
const { sel, text, links, hold } = got;
|
|
448
|
+
// A redirect is shown and cited: a mistyped atlas topic lands on its guide's landing page with status 200, and a
|
|
449
|
+
// Help article can move to another ?id= on the same path.
|
|
450
|
+
const to = new URL(page.url()), hop = sameDoc(to, u) ? '' : ` [redirected ${to.href}]`;
|
|
451
|
+
// Old atlas URLs (and the LDS home) redirect to pages that have a .md twin: keep the rendered links, read the twin.
|
|
452
|
+
const moved = hop && twinOf(to);
|
|
453
|
+
const tryTwin = moved && !noTwin(moved);
|
|
454
|
+
// Any failure (late navigation, body error, 15 s hang) keeps the rendered text, and the output line says why.
|
|
455
|
+
// 15 s, not RAW_TIMEOUT_MS: this runs after a full render, so it counts against a rendered page's budget.
|
|
456
|
+
const tw = tryTwin ? await page.evaluate(p => fetch(p, { signal: AbortSignal.timeout(15000) })
|
|
457
|
+
.then(async r => ({ status: r.status, body: await r.text() })).catch(e => ({ why: e.name })), moved.pathname)
|
|
458
|
+
.catch(e => ({ why: e.message.split('\n')[0] })) : {};
|
|
459
|
+
const gone = !tw.why && missingTwin(tw.status, tw.body);
|
|
460
|
+
const md = !tw.why && !gone && tw.status === 200 ? tw.body : '';
|
|
461
|
+
const why = tw.why || (gone ? (tw.status === 404 ? 'status 404' : isShell(tw.body) ? 'shell' : 'empty') : `status ${tw.status}`);
|
|
462
|
+
if (tryTwin && gone) markMissing(moved);
|
|
463
|
+
const failed = tryTwin && !md ? ` [twin failed: ${why}]` : '';
|
|
464
|
+
if (md) {
|
|
465
|
+
emit({ p: u.href, status: 200, body: md, md: true, links, key: u.href, src: official(moved), note: ` [twin ${moved.pathname}] [redirected ${official(moved)}]` });
|
|
466
|
+
await page.close();
|
|
467
|
+
continue;
|
|
468
|
+
}
|
|
469
|
+
// A shell (no container, extract failed, or API docs still not loaded at 20 s) is reported but never cached.
|
|
470
|
+
emit({ p: u.origin + u.pathname + u.search, status: nav?.status() ?? 0, body: text, links, key: u.href,
|
|
471
|
+
src: hop && to.href, note: ` [${sel}]${hold ? ' [api docs did not load]' : ''}${hop}${failed}`, offsite: !SITES.includes(to.host), shell: sel === 'body' || hold, fallback: sel === 'fallback' });
|
|
472
|
+
await page.close();
|
|
473
|
+
}
|
|
474
|
+
|
|
475
|
+
// Name the line actually printed for the first argument: a twin that fell back prints under the page it rendered.
|
|
476
|
+
const shown = new Map(results.map(r => [r.key, r.p]));
|
|
477
|
+
for (const [g, k] of sameAs) console.log(`same ${g} as ${shown.get(k) ?? shown.get(asked.get(k)?.href) ?? skippedAs.get(k) ?? skippedAs.get(asked.get(k)?.href) ?? k}`);
|
|
478
|
+
const late = skippedAs.size;
|
|
479
|
+
console.log(`fetched ${urls.length - late}, 200 ${ok}, unique bodies ${seen.size}${refused.length ? `, refused ${refused.length}` : ''}${late ? `, not fetched ${late} (time limit)` : ''}, out ${slash(out)}`);
|
|
480
|
+
if (!urls.length && refused.length) process.exitCode = 1; // nothing on a doc site was asked for
|
|
481
|
+
for (const f of terms.length ? saved : []) {
|
|
482
|
+
console.log(`\n== ${slash(relative(out, f))}\n${grep(f, terms).join('\n') || 'no match'}`);
|
|
483
|
+
const links = statSync(`${f}.links.txt`, { throwIfNoEntry: false }) ? grep(`${f}.links.txt`, terms) : [];
|
|
484
|
+
if (links.length) console.log(`== ${slash(relative(out, f))}.links.txt\n${links.join('\n')}`);
|
|
485
|
+
}
|
|
486
|
+
} finally {
|
|
487
|
+
await browser.close();
|
|
488
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,36 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sf-docs-agent",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"description": "
|
|
6
|
-
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"type": "module",
|
|
5
|
+
"description": "Claude Code subagent that answers from official Salesforce docs (developer, architect, help, admin, lightningdesignsystem.com) with grep-checked quotes",
|
|
6
|
+
"devDependencies": {
|
|
7
|
+
"playwright-core": "1.56.1"
|
|
8
|
+
},
|
|
9
|
+
"bin": {
|
|
10
|
+
"sf-docs-agent": "bin/install.mjs"
|
|
11
|
+
},
|
|
12
|
+
"files": [
|
|
13
|
+
"bin",
|
|
14
|
+
"agents",
|
|
15
|
+
"fetch-docs.mjs"
|
|
16
|
+
],
|
|
17
|
+
"repository": {
|
|
18
|
+
"type": "git",
|
|
19
|
+
"url": "git+https://github.com/drsaavedra/sf-docs-agent.git"
|
|
20
|
+
},
|
|
21
|
+
"license": "MIT",
|
|
22
|
+
"engines": {
|
|
23
|
+
"node": ">=18.17"
|
|
24
|
+
},
|
|
25
|
+
"keywords": [
|
|
26
|
+
"claude-code",
|
|
27
|
+
"subagent",
|
|
28
|
+
"salesforce",
|
|
29
|
+
"documentation",
|
|
30
|
+
"llms.txt",
|
|
31
|
+
"playwright"
|
|
32
|
+
],
|
|
33
|
+
"scripts": {
|
|
34
|
+
"test": "node --test --test-reporter=spec"
|
|
35
|
+
}
|
|
36
|
+
}
|