neuro-canvas-lms 2.12.0 → 2.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +11 -3
- package/README.md +15 -6
- package/canvas-transport.mjs +4 -1
- package/canvas.mjs +93 -16
- package/download-link.mjs +24 -5
- package/file-content.mjs +38 -8
- package/find-files.mjs +53 -12
- package/i18n.mjs +108 -22
- package/installer/cli.mjs +124 -49
- package/installer/lib.mjs +59 -4
- package/installer/messages.mjs +114 -14
- package/installer/subdomain.mjs +58 -0
- package/module-file.mjs +8 -4
- package/npm-shrinkwrap.json +2779 -0
- package/package.json +7 -2
- package/page-scene.mjs +904 -0
- package/pages.mjs +186 -84
- package/server.json +32 -0
- package/time.mjs +36 -6
- package/update-check.mjs +6 -2
- package/version.mjs +3 -2
- package/worker.mjs +141 -24
- package/zip-content.mjs +45 -14
package/AGENTS.md
CHANGED
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
This repo, **Neuro-CanvasLMS** (`neuro-canvas-lms`), is a read-only Canvas LMS connector (remote MCP
|
|
4
4
|
server on Cloudflare Workers) that each user deploys to their **own** Cloudflare account. It works
|
|
5
|
-
with any Canvas school; nothing school-specific is in the
|
|
6
|
-
It is a community project provided as is.
|
|
5
|
+
with any Canvas school that lets students create access tokens; nothing school-specific is in the
|
|
6
|
+
code (the school comes from Worker vars). It is a community project provided as is.
|
|
7
7
|
|
|
8
8
|
- To install, update, reconnect, uninstall or troubleshoot: follow
|
|
9
9
|
`.claude/skills/canvas-connector-setup/SKILL.md` (Claude Code loads it as a skill; other agents
|
|
@@ -30,9 +30,17 @@ Code map:
|
|
|
30
30
|
`download-link.mjs` (one-time, minutes-long Canvas storage link so coding agents can curl an original
|
|
31
31
|
file to disk).
|
|
32
32
|
- `owner.mjs`: owner binding, allowed OAuth clients. `i18n.mjs` + `pages.mjs`: web pages.
|
|
33
|
+
`page-scene.mjs`: their one script (allowed by its hash in the CSP; pages stay off the network):
|
|
34
|
+
the black-hole scene and the step-by-step authorization. Everything works without it.
|
|
33
35
|
- `update-check.mjs`: once a day reads the newest version number from npm; when it is newer the AI
|
|
34
36
|
tells the user how to update. Also the `canvas_connector_version` tool.
|
|
35
|
-
- `installer/`: the setup wizard.
|
|
37
|
+
- `installer/`: the setup wizard. On a new Cloudflare account it registers a random workers.dev
|
|
38
|
+
subdomain through Cloudflare's API. Re-running it asks once (y/n), then updates with the saved
|
|
36
39
|
settings (without a terminal it does not ask). `installer/verify.mjs`: the post-deploy online
|
|
37
40
|
check, also run alone by `npx -y neuro-canvas-lms check`.
|
|
38
41
|
- `version.mjs`: shown on the home page and /healthz, compared with npm by update-check.mjs.
|
|
42
|
+
- `server.json`: the entry in the official MCP Registry (one remote whose host each user fills in).
|
|
43
|
+
Same version as `version.mjs` and package.json (`version.test.mjs`).
|
|
44
|
+
- Releases: bump those three, push to main, then push a tag `vX.Y.Z`;
|
|
45
|
+
`.github/workflows/publish.yml` tests, publishes to npm and updates the MCP Registry, with
|
|
46
|
+
no stored secrets (OIDC trusted publishing).
|
package/README.md
CHANGED
|
@@ -2,9 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
**English** · [简体中文](docs/README.zh-CN.md) · [Español](docs/README.es.md) · [日本語](docs/README.ja.md) · [हिन्दी](docs/README.hi.md)
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
[](https://www.npmjs.com/package/neuro-canvas-lms) [](https://github.com/BloodyReginaa/Neuro-CanvasLMS/actions/workflows/test.yml) [](https://github.com/BloodyReginaa/Neuro-CanvasLMS/blob/main/LICENSE)
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+

|
|
8
|
+
|
|
9
|
+
**Wire your Canvas LMS straight into Claude and ChatGPT.** Your courses, deadlines, grades and lecture files become part of what your AI knows, in any chat and on any device. It is a read-only remote MCP server that one command deploys to your own free Cloudflare account. Works at any Canvas school that lets students create access tokens ([30-second check](#can-you-use-it)).
|
|
10
|
+
|
|
11
|
+
> "What's due this week?" · "Summarize Lecture 5 in MATH 132." · "What feedback did I get on HW 3?" · "What's in the code ZIP for Lecture 12?" · "Set up the PA2 starter code in this folder."
|
|
8
12
|
|
|
9
13
|
## Why Neuro-CanvasLMS
|
|
10
14
|
|
|
@@ -15,7 +19,7 @@
|
|
|
15
19
|
| 💸 **Free** | It runs on Cloudflare's free plan, which is far more than one student needs. |
|
|
16
20
|
| 🧠 **Your files live in the conversation** | Lecture notes and PDFs the AI reads stay with that chat on your AI provider's servers. Nothing to download. |
|
|
17
21
|
|
|
18
|
-
|
|
22
|
+
Local Canvas tools only work while your own computer runs them. Hosted services keep your Canvas token, or a copy of your courses, on their servers. Neuro-CanvasLMS runs in **your own** Cloudflare account: always on, usable from your phone and in ChatGPT, and your token goes only to your own Worker and your school's Canvas.
|
|
19
23
|
|
|
20
24
|
## What you get
|
|
21
25
|
|
|
@@ -23,20 +27,25 @@ Most Canvas integrations for AI are **local** tools: they only work while your o
|
|
|
23
27
|
- **Real file reading:** text from PDF, Word, PowerPoint and Excel, about 90 kinds of code and text files, and Jupyter notebooks (cells and outputs); ZIP archives unpacked on the server (listing plus every code/text file); scanned or handwritten PDFs are recognised and their original pages handed to the AI to look at.
|
|
24
28
|
- **Linked files too:** course files that a module links on Google Docs/Slides/Sheets, Google Drive, Dropbox (files and folders) and GitHub (files and repositories) are read as text, as long as they are shared publicly ("Anyone with the link").
|
|
25
29
|
- **Finds files the course hides:** many courses hide their Files list from students. The connector searches the front page, syllabus, modules, assignments, announcements and the pages they link to, so the AI can still find "the Lecture 12 slides" by name.
|
|
30
|
+
- **Starter code onto your computer:** in Claude Code, Codex and other coding agents, ask "set up PA2 from Canvas". The agent gets a one-time download link that expires in minutes, saves the assignment's starter files in your folder and unpacks them.
|
|
26
31
|
- **Read-only by design:** only fixed `GET` requests. It never submits work, sends messages, starts quizzes or changes anything in Canvas.
|
|
27
32
|
- **Yours only:** deployed in your own Cloudflare account and locked to your own Canvas account. Nobody else, including the authors, ever sees your Canvas token.
|
|
28
33
|
|
|
34
|
+
## Can you use it?
|
|
35
|
+
|
|
36
|
+
Check in 30 seconds: in Canvas, open **Account → Settings** and scroll to **Approved Integrations**. If you see a **+ New Access Token** button, you're set. If it is missing, or creating a token is refused, your school doesn't let students make personal tokens, and this connector can't work there yet.
|
|
37
|
+
|
|
29
38
|
## Install (about 5 minutes)
|
|
30
39
|
|
|
31
|
-
You need [Node.js 22+](https://nodejs.org), a free [Cloudflare account](https://dash.cloudflare.com/sign-up) and your Canvas login. In a terminal, run:
|
|
40
|
+
You need [Node.js 22+](https://nodejs.org), a free [Cloudflare account](https://dash.cloudflare.com/sign-up) (verify its email address) and your Canvas login. In a terminal, run:
|
|
32
41
|
|
|
33
42
|
```bash
|
|
34
43
|
npx -y neuro-canvas-lms
|
|
35
44
|
```
|
|
36
45
|
|
|
37
|
-
The installer signs you in to Cloudflare (a browser page opens; click **Authorize**), asks for your school's Canvas address, deploys the connector to your own account and checks that it works. Then:
|
|
46
|
+
The installer signs you in to Cloudflare (a browser page opens; click **Authorize**), asks for your school's Canvas address, sets up your free workers.dev address if the account is new, deploys the connector to your own account and checks that it works. A new address can take a few minutes to come online; meanwhile the installer opens Canvas so you can create your token. Then:
|
|
38
47
|
|
|
39
|
-
1. **Create a Canvas token
|
|
48
|
+
1. **Create a Canvas token** (if you haven't yet): Canvas → Account → Settings → **+ New Access Token**. Your school may cap its lifetime (30 days at UMass Amherst). Copy it right away; Canvas shows it only once.
|
|
40
49
|
2. **Claude:** Settings → Connectors → **Add custom connector** → paste the URL the installer printed (it is already copied) → **Connect** → paste the Canvas token → Authorize. It then works everywhere you use Claude, including the mobile app and Claude Code.
|
|
41
50
|
3. **ChatGPT:** add a custom connector (developer mode) with the same URL.
|
|
42
51
|
4. Start a new chat and ask: "What assignments do I have this week?"
|
package/canvas-transport.mjs
CHANGED
|
@@ -135,9 +135,12 @@ export async function parseCanvasResponse(bytes) {
|
|
|
135
135
|
}
|
|
136
136
|
// A JSON answer is Canvas itself speaking; HTML (or a 406) means something in front of Canvas
|
|
137
137
|
// rejected the request, which is when the socket path is worth trying.
|
|
138
|
+
// A rate limit (429, or Canvas's own 403 carrying X-Rate-Limit-Remaining) is Canvas speaking too:
|
|
139
|
+
// asking again at once over the socket would only add to the count.
|
|
138
140
|
export function shouldFallBack(response) {
|
|
139
141
|
if (response.status === 406) return true;
|
|
140
|
-
if (response.status < 400) return false;
|
|
142
|
+
if (response.status < 400 || response.status === 429) return false;
|
|
143
|
+
if (response.headers.has('x-rate-limit-remaining')) return false;
|
|
141
144
|
return !(response.headers.get('content-type') ?? '').toLowerCase().includes('json');
|
|
142
145
|
}
|
|
143
146
|
|
package/canvas.mjs
CHANGED
|
@@ -229,27 +229,87 @@ export async function callCanvas(name, args, token, fetcher = fetch, opts = {})
|
|
|
229
229
|
return out;
|
|
230
230
|
}
|
|
231
231
|
|
|
232
|
+
// Canvas could not be reached at all: say so (and that the connection is fine), so the AI retries
|
|
233
|
+
// instead of changing its arguments. Our own transport messages already start with "Canvas".
|
|
234
|
+
export function unreachable(e) {
|
|
235
|
+
if (/^Canvas/.test(e?.message ?? '')) return e;
|
|
236
|
+
const timeout = e?.name === 'TimeoutError' || e?.name === 'AbortError';
|
|
237
|
+
return Object.assign(
|
|
238
|
+
new Error(
|
|
239
|
+
timeout
|
|
240
|
+
? 'Canvas did not answer within 20 seconds; it may be slow or down right now. Try again in a minute. The token and the connection are fine: do not reconnect.'
|
|
241
|
+
: `Canvas could not be reached (network error: ${e?.name ?? 'Error'}). Try again in a minute. The token and the connection are fine: do not reconnect.`,
|
|
242
|
+
),
|
|
243
|
+
{ unreachable: true, cause: e },
|
|
244
|
+
);
|
|
245
|
+
}
|
|
246
|
+
|
|
232
247
|
// One GET to the Canvas API: fixed headers, no redirects, Canvas errors turned into safe messages
|
|
233
248
|
// (never the upstream body). `e.status` is set when Canvas refused this item (not a token problem).
|
|
249
|
+
// Canvas throttles API use per token: it answers "403 Forbidden (Rate Limit Exceeded)" (its own
|
|
250
|
+
// answers carry X-Rate-Limit-Remaining), and a proxy in front may answer 429. That is not "this
|
|
251
|
+
// course hides its files" (no find_files hint, no retry over the socket): the AI is told to wait.
|
|
252
|
+
// The error has no `status`, so nothing treats it as a hidden resource.
|
|
253
|
+
async function rateLimit(r) {
|
|
254
|
+
if (r.status !== 429 && r.status !== 403) return null;
|
|
255
|
+
let limited = r.status === 429;
|
|
256
|
+
if (!limited) {
|
|
257
|
+
const remaining = r.headers.get('x-rate-limit-remaining');
|
|
258
|
+
if (remaining !== null && Number(remaining) <= 0) limited = true;
|
|
259
|
+
else if (!(r.headers.get('content-type') ?? '').includes('json'))
|
|
260
|
+
try {
|
|
261
|
+
limited = /rate limit/i.test((await r.clone().text()).slice(0, 2000));
|
|
262
|
+
} catch {}
|
|
263
|
+
}
|
|
264
|
+
if (!limited) return null;
|
|
265
|
+
await r.body?.cancel().catch(() => {});
|
|
266
|
+
const seconds = Number(r.headers.get('retry-after'));
|
|
267
|
+
const wait =
|
|
268
|
+
Number.isFinite(seconds) && seconds > 0
|
|
269
|
+
? `${Math.min(Math.ceil(seconds), 3600)} seconds`
|
|
270
|
+
: 'about a minute';
|
|
271
|
+
return Object.assign(
|
|
272
|
+
new Error(
|
|
273
|
+
`Canvas rate limit reached (HTTP ${r.status}): too many Canvas requests in a short time. Wait ${wait} before the next Canvas call and do not retry in a loop. The token and the connection are fine; this is not a permission problem.`,
|
|
274
|
+
),
|
|
275
|
+
{ rateLimited: true },
|
|
276
|
+
);
|
|
277
|
+
}
|
|
278
|
+
|
|
234
279
|
export async function canvasGet(u, token, fetcher = fetch) {
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
280
|
+
let r;
|
|
281
|
+
try {
|
|
282
|
+
r = await fetcher(u, {
|
|
283
|
+
method: 'GET',
|
|
284
|
+
headers: {
|
|
285
|
+
Authorization: `Bearer ${token}`,
|
|
286
|
+
Accept: 'application/json',
|
|
287
|
+
'User-Agent': 'Neuro-CanvasLMS/2.0 (read-only personal integration)',
|
|
288
|
+
},
|
|
289
|
+
redirect: 'manual',
|
|
290
|
+
signal: AbortSignal.timeout(20000),
|
|
291
|
+
});
|
|
292
|
+
} catch (e) {
|
|
293
|
+
throw unreachable(e);
|
|
294
|
+
}
|
|
295
|
+
const limited = await rateLimit(r);
|
|
296
|
+
if (limited) throw limited;
|
|
245
297
|
if (
|
|
246
298
|
r.status === 406 ||
|
|
247
299
|
(r.status >= 500 && !(r.headers.get('content-type') ?? '').includes('json'))
|
|
248
300
|
) {
|
|
249
|
-
// Canvas answers its API with JSON. An HTML error page means
|
|
250
|
-
//
|
|
251
|
-
|
|
252
|
-
|
|
301
|
+
// Canvas answers its API with JSON. An HTML error page means Canvas is down or in maintenance
|
|
302
|
+
// (502/503/504 from the load balancer), a firewall in front of Canvas rejected the request, or
|
|
303
|
+
// Canvas failed on a malformed token (an HTML 500; only possible on the authorization page,
|
|
304
|
+
// since a connected token already passed). The phrase "HTML error page" stops canvas_find_files.
|
|
305
|
+
await r.body?.cancel().catch(() => {});
|
|
306
|
+
throw Object.assign(
|
|
307
|
+
new Error(
|
|
308
|
+
[502, 503, 504].includes(r.status)
|
|
309
|
+
? `Canvas is temporarily unavailable (HTTP ${r.status}, an HTML error page instead of JSON). Try again in a few minutes. The token and the connection are fine: do not reconnect.`
|
|
310
|
+
: `Canvas returned an HTML error page (HTTP ${r.status}) instead of JSON: usually a temporary Canvas problem or a firewall in front of Canvas. Try again in a few minutes; reconnecting does not help.`,
|
|
311
|
+
),
|
|
312
|
+
{ htmlError: r.status },
|
|
253
313
|
);
|
|
254
314
|
}
|
|
255
315
|
if (r.status === 401) {
|
|
@@ -276,7 +336,14 @@ export async function canvasGet(u, token, fetcher = fetch) {
|
|
|
276
336
|
),
|
|
277
337
|
{ status: r.status },
|
|
278
338
|
);
|
|
279
|
-
|
|
339
|
+
let data;
|
|
340
|
+
try {
|
|
341
|
+
data = await r.json();
|
|
342
|
+
} catch {
|
|
343
|
+
throw new Error(
|
|
344
|
+
`Canvas answered HTTP ${r.status} but not with JSON (a maintenance or login page in between?). Try again in a few minutes; the token and the connection are fine.`,
|
|
345
|
+
);
|
|
346
|
+
}
|
|
280
347
|
const next = nextPage(r.headers.get('link'));
|
|
281
348
|
// api_url is the API request itself (it shows JSON); links for the user are the html_url fields.
|
|
282
349
|
return { data, has_more: next !== null, next_page: next, api_url: u.href };
|
|
@@ -299,12 +366,22 @@ const FEED = /https?:\/\/[^\s"'<>()]*\/feeds\/[^\s"'<>()]+/gi;
|
|
|
299
366
|
const LINK = /(?:https?:\/\/|\/api\/v1\/(?:canvadoc|crocodoc)_session)[^\s"'<>()]*/gi;
|
|
300
367
|
const SIGNED =
|
|
301
368
|
/(?:canvadoc|crocodoc)_session|[?&](?:amp;)?(?:sf_verifier|hmac|x-amz-signature|signature)=|[?&](?:amp;)?token=[\w-]+\.[\w-]+\.[\w-]+/i;
|
|
302
|
-
|
|
369
|
+
// A file's `uuid` is the verifier itself (Canvas accepts ?verifier=<uuid>) and its thumbnail_url
|
|
370
|
+
// carries it in the path; a course's uuid is the key of its calendar feed. Raw results lose them.
|
|
371
|
+
const SIGNED_KEYS = new Set([
|
|
372
|
+
'canvadoc_session_url',
|
|
373
|
+
'crocodoc_session_url',
|
|
374
|
+
'uuid',
|
|
375
|
+
'thumbnail_url',
|
|
376
|
+
]);
|
|
377
|
+
// The same pass URL-encoded inside another link (e.g. a login page's return_to=).
|
|
378
|
+
const ENCODED_VERIFIER = /verifier%3D[^&#"'\s<>()%]*/gi;
|
|
303
379
|
export function withoutVerifiers(value) {
|
|
304
380
|
if (typeof value === 'string') {
|
|
305
381
|
let v = value;
|
|
306
382
|
if (v.includes('verifier='))
|
|
307
383
|
v = v.replace(VERIFIER, (m, before, after) => (after ? before : ''));
|
|
384
|
+
if (/verifier%3D/i.test(v)) v = v.replace(ENCODED_VERIFIER, 'verifier%3D');
|
|
308
385
|
if (v.includes('/feeds/')) v = v.replace(FEED, '[private feed link removed]');
|
|
309
386
|
if (/[?]|_session/.test(v))
|
|
310
387
|
v = v.replace(LINK, link => (SIGNED.test(link) ? '[signed link removed]' : link));
|
package/download-link.mjs
CHANGED
|
@@ -160,12 +160,20 @@ export function newDownloadId() {
|
|
|
160
160
|
.replace(/=+$/, '');
|
|
161
161
|
}
|
|
162
162
|
|
|
163
|
+
// Cloudflare KV failed (briefly unavailable, or the free plan's 1000 writes a day are used up).
|
|
164
|
+
export const STORAGE_DOWN =
|
|
165
|
+
"Connector storage (Cloudflare KV) is unavailable right now, so no download link could be made. It is often the free plan's daily KV limit, which resets at 00:00 UTC. Meanwhile use canvas_file_original or canvas_file_content; the Canvas connection itself is fine.";
|
|
166
|
+
|
|
163
167
|
// Store the Canvas address for 5 minutes and return the Worker link that leads to it.
|
|
164
168
|
export async function issueDownload(kv, origin, target, now = Date.now) {
|
|
165
169
|
const id = newDownloadId();
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
170
|
+
try {
|
|
171
|
+
await kv.put('dl:' + id, JSON.stringify({ target, exp: now() + DOWNLOAD_TTL * 1000 }), {
|
|
172
|
+
expirationTtl: DOWNLOAD_TTL,
|
|
173
|
+
});
|
|
174
|
+
} catch {
|
|
175
|
+
throw new Error(STORAGE_DOWN);
|
|
176
|
+
}
|
|
169
177
|
return `${origin}/dl/${id}`;
|
|
170
178
|
}
|
|
171
179
|
|
|
@@ -188,11 +196,22 @@ export async function redeemDownload(request, kv, id, now = Date.now) {
|
|
|
188
196
|
{ status: 403, headers: plain },
|
|
189
197
|
);
|
|
190
198
|
const key = 'dl:' + id;
|
|
199
|
+
let stored;
|
|
200
|
+
try {
|
|
201
|
+
stored = await kv.get(key);
|
|
202
|
+
} catch {
|
|
203
|
+
// not "expired": the link may be fine, the storage is not
|
|
204
|
+
return new Response(
|
|
205
|
+
"The connector's storage (Cloudflare KV) is unavailable right now. Try again in a minute; if it keeps failing, the free plan's daily limit may be used up (it resets at 00:00 UTC).\n",
|
|
206
|
+
{ status: 503, headers: plain },
|
|
207
|
+
);
|
|
208
|
+
}
|
|
191
209
|
let entry = null;
|
|
192
210
|
try {
|
|
193
|
-
entry = JSON.parse(
|
|
211
|
+
entry = JSON.parse(stored ?? 'null');
|
|
194
212
|
} catch {}
|
|
195
|
-
|
|
213
|
+
// Used up before it is used: a failed delete does not stop this download (it still expires).
|
|
214
|
+
if (entry) await kv.delete(key).catch(() => {});
|
|
196
215
|
// KV expiry can lag a little: the stored deadline is checked too.
|
|
197
216
|
if (!entry?.target || !(now() < entry.exp))
|
|
198
217
|
return new Response(
|
package/file-content.mjs
CHANGED
|
@@ -226,6 +226,20 @@ export async function extractBytes(bytes, shown, mime = '', opts = {}) {
|
|
|
226
226
|
'This format is not supported for text extraction. Call canvas_file_original with this file_id to get the original file.',
|
|
227
227
|
};
|
|
228
228
|
}
|
|
229
|
+
// extractBytes, but a parser failing on the file itself (damaged, password-protected, or not really
|
|
230
|
+
// a PDF/Office file) becomes a note with a way on instead of an error the AI reads as "check your
|
|
231
|
+
// arguments". Our own messages (limits, a wrong `entry`) start with "Canvas" and go through.
|
|
232
|
+
export async function extractOrExplain(bytes, name, mime, opts, wayOn) {
|
|
233
|
+
try {
|
|
234
|
+
return await extractBytes(bytes, name, mime, opts);
|
|
235
|
+
} catch (e) {
|
|
236
|
+
if (/^Canvas/.test(e?.message ?? '')) throw e;
|
|
237
|
+
return {
|
|
238
|
+
text: null,
|
|
239
|
+
extraction_note: `Could not read the text of this file (${e?.name ?? 'Error'}): it may be damaged, password-protected, or not really the type its name says. ${wayOn}`,
|
|
240
|
+
};
|
|
241
|
+
}
|
|
242
|
+
}
|
|
229
243
|
export async function readFileContent(meta, args, fetcher = fetch) {
|
|
230
244
|
// `hidden` files are only hidden from the Files list ("only available with link"), which is how
|
|
231
245
|
// many courses share files; students may open them. Only a lock stops a download.
|
|
@@ -237,7 +251,14 @@ export async function readFileContent(meta, args, fetcher = fetch) {
|
|
|
237
251
|
html_url: `https://${CANVAS_HOST}/files/${meta.id}`,
|
|
238
252
|
mime_type: meta['content-type'],
|
|
239
253
|
};
|
|
240
|
-
if (!meta.url
|
|
254
|
+
if (!meta.url)
|
|
255
|
+
return {
|
|
256
|
+
...base,
|
|
257
|
+
text: null,
|
|
258
|
+
extraction_note:
|
|
259
|
+
'Canvas gives no download address for this file (it may only be viewable inside Canvas); open html_url.',
|
|
260
|
+
};
|
|
261
|
+
if (!allowedDownload(meta.url))
|
|
241
262
|
return {
|
|
242
263
|
...base,
|
|
243
264
|
text: null,
|
|
@@ -262,14 +283,22 @@ export async function readFileContent(meta, args, fetcher = fetch) {
|
|
|
262
283
|
throw new Error(
|
|
263
284
|
'Canvas file redirect host is not allowed: ' + [...hops, where(url)].join(' → '),
|
|
264
285
|
);
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
286
|
+
try {
|
|
287
|
+
response = await fetcher(url, {
|
|
288
|
+
method: 'GET',
|
|
289
|
+
redirect: 'manual',
|
|
290
|
+
headers: { Accept: '*/*', 'User-Agent': DOWNLOAD_UA },
|
|
291
|
+
signal: AbortSignal.timeout(20000),
|
|
292
|
+
});
|
|
293
|
+
} catch (e) {
|
|
294
|
+
if (/^Canvas/.test(e?.message ?? '')) throw e;
|
|
295
|
+
throw new Error(
|
|
296
|
+
`Canvas file download did not finish (${e?.name === 'TimeoutError' || e?.name === 'AbortError' ? 'no answer within 20 seconds' : 'network error: ' + (e?.name ?? 'Error')}; ${where(url)}). Try again in a minute; the connection is fine.`,
|
|
297
|
+
);
|
|
298
|
+
}
|
|
271
299
|
hops.push(response.status + ' ' + where(url));
|
|
272
300
|
if (response.status >= 300 && response.status < 400) {
|
|
301
|
+
await response.body?.cancel().catch(() => {});
|
|
273
302
|
const location = response.headers.get('location');
|
|
274
303
|
if (!location) throw new Error('Canvas download redirect is missing its destination.');
|
|
275
304
|
url = new URL(location, url).href;
|
|
@@ -293,11 +322,12 @@ export async function readFileContent(meta, args, fetcher = fetch) {
|
|
|
293
322
|
(response.headers.get('content-type') ?? 'unknown').split(';')[0] +
|
|
294
323
|
').',
|
|
295
324
|
);
|
|
296
|
-
const extracted = await
|
|
325
|
+
const extracted = await extractOrExplain(
|
|
297
326
|
await boundedBytes(response),
|
|
298
327
|
base.name ?? '',
|
|
299
328
|
base.mime_type ?? response.headers.get('content-type') ?? '',
|
|
300
329
|
{ entry: args.entry, storedName: meta.filename },
|
|
330
|
+
'Call canvas_file_original with this file_id to get the file itself, or open html_url.',
|
|
301
331
|
);
|
|
302
332
|
return paginated(base, extracted, args);
|
|
303
333
|
}
|
package/find-files.mjs
CHANGED
|
@@ -180,14 +180,26 @@ export async function findFiles(args, token, fetcher) {
|
|
|
180
180
|
return canvasGet(u, token, fetcher);
|
|
181
181
|
};
|
|
182
182
|
const searched = {};
|
|
183
|
-
//
|
|
183
|
+
// A source being hidden or not enabled (401/403/404) is normal. A source that failed (timeout,
|
|
184
|
+
// network, rate limit, Canvas error) was not searched: the result is then incomplete, and if
|
|
185
|
+
// nothing could be searched at all the error is passed on instead of an empty "complete" list.
|
|
186
|
+
// An expired token, a rate limit or an HTML error page (Canvas down, a firewall) stops everything.
|
|
187
|
+
const failures = [];
|
|
188
|
+
const hidden = e => [401, 403, 404].includes(e.status);
|
|
189
|
+
const stops = e =>
|
|
190
|
+
e.rateLimited || /authorization expired|HTML error page/.test(e?.message ?? '');
|
|
184
191
|
const attempt = async (source, run) => {
|
|
185
192
|
try {
|
|
186
193
|
const n = await run();
|
|
187
194
|
searched[source] = n === undefined ? 'ok' : `ok (${n})`;
|
|
188
195
|
} catch (e) {
|
|
189
|
-
if (
|
|
190
|
-
searched[source] =
|
|
196
|
+
if (stops(e)) throw e;
|
|
197
|
+
if (hidden(e)) searched[source] = `not available (HTTP ${e.status})`;
|
|
198
|
+
else {
|
|
199
|
+
failures.push(e);
|
|
200
|
+
complete = false;
|
|
201
|
+
searched[source] = e.status ? `failed (HTTP ${e.status})` : 'failed';
|
|
202
|
+
}
|
|
191
203
|
}
|
|
192
204
|
};
|
|
193
205
|
const pagesOf = async (path, query, maxPages, each) => {
|
|
@@ -337,7 +349,8 @@ export async function findFiles(args, token, fetcher) {
|
|
|
337
349
|
|
|
338
350
|
// Pages linked from everything above, breadth first, never outside this course.
|
|
339
351
|
let pagesRead = 0,
|
|
340
|
-
pagesUnavailable = 0
|
|
352
|
+
pagesUnavailable = 0,
|
|
353
|
+
pagesFailed = 0;
|
|
341
354
|
while (queue.length) {
|
|
342
355
|
const batch = queue.splice(0, PAGE_BATCH).filter(slug => !read.has(slug));
|
|
343
356
|
await Promise.all(
|
|
@@ -354,8 +367,13 @@ export async function findFiles(args, token, fetcher) {
|
|
|
354
367
|
page_url: r.data?.url ?? slug,
|
|
355
368
|
});
|
|
356
369
|
} catch (e) {
|
|
357
|
-
if (
|
|
358
|
-
pagesUnavailable++;
|
|
370
|
+
if (stops(e)) throw e;
|
|
371
|
+
if (hidden(e)) pagesUnavailable++;
|
|
372
|
+
else {
|
|
373
|
+
failures.push(e);
|
|
374
|
+
complete = false;
|
|
375
|
+
pagesFailed++;
|
|
376
|
+
}
|
|
359
377
|
}
|
|
360
378
|
}),
|
|
361
379
|
);
|
|
@@ -365,7 +383,20 @@ export async function findFiles(args, token, fetcher) {
|
|
|
365
383
|
}
|
|
366
384
|
}
|
|
367
385
|
searched.linked_pages =
|
|
368
|
-
`${pagesRead} read` +
|
|
386
|
+
`${pagesRead} read` +
|
|
387
|
+
(pagesUnavailable ? `, ${pagesUnavailable} not available` : '') +
|
|
388
|
+
(pagesFailed ? `, ${pagesFailed} failed` : '');
|
|
389
|
+
const sources = Object.entries(searched).filter(([k]) => k !== 'linked_pages');
|
|
390
|
+
// Nothing could be searched: say why instead of "no files, complete".
|
|
391
|
+
if (failures.length && !pagesRead && !sources.some(([, v]) => v.startsWith('ok'))) {
|
|
392
|
+
const e = failures.find(x => /^Canvas/.test(x.message ?? '')) ?? failures[0];
|
|
393
|
+
throw /^Canvas/.test(e.message ?? '')
|
|
394
|
+
? e
|
|
395
|
+
: new Error(`Canvas could not be searched (${e.name}). Try again in a minute.`);
|
|
396
|
+
}
|
|
397
|
+
// Every source 404: most likely not a course this user is in.
|
|
398
|
+
const wrongCourse =
|
|
399
|
+
!files.size && sources.length && sources.every(([, v]) => v === 'not available (HTTP 404)');
|
|
369
400
|
|
|
370
401
|
let list = [...files.values()];
|
|
371
402
|
let matchedOn = null;
|
|
@@ -393,11 +424,21 @@ export async function findFiles(args, token, fetcher) {
|
|
|
393
424
|
external_links: ext,
|
|
394
425
|
searched,
|
|
395
426
|
complete,
|
|
396
|
-
...(
|
|
397
|
-
? {
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
427
|
+
...(failures.length
|
|
428
|
+
? {
|
|
429
|
+
note: `Some places could not be searched (they failed, see searched: ${failures.length} failed request${failures.length > 1 ? 's' : ''}), so a file missing here may still exist. Try again in a minute before telling the user it is not there.`,
|
|
430
|
+
}
|
|
431
|
+
: complete
|
|
432
|
+
? {}
|
|
433
|
+
: {
|
|
434
|
+
note: `Stopped after ${requests} Canvas requests; some linked pages or lists were not searched. Open more pages with canvas_context_page or canvas_module_items if the file is not listed.`,
|
|
435
|
+
}),
|
|
436
|
+
...(wrongCourse
|
|
437
|
+
? {
|
|
438
|
+
course_note:
|
|
439
|
+
'Every part of this course answered 404: check the course_id (canvas_courses lists the user’s courses).',
|
|
440
|
+
}
|
|
441
|
+
: {}),
|
|
401
442
|
how_to_read:
|
|
402
443
|
'Read a file_id with canvas_file_content (text from PDF/Office/code, ZIP listing and contents) or canvas_file_original (original bytes, e.g. scanned PDFs). An external link with read_with (a module item): call canvas_module_file_content with those IDs for its text (Google Docs/Slides/Sheets/Drive, Dropbox, GitHub), or canvas_module_file for the original file. Other external links cannot be read by this connector: give the user the link.',
|
|
403
444
|
};
|