ntu-cool-cli 1.3.1__tar.gz

ntu_cool_cli-1.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Gary Xie and ntu-cool-cli contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ntu_cool_cli-1.3.1/PKG-INFO

@@ -0,0 +1,505 @@
+Metadata-Version: 2.4
+Name: ntu-cool-cli
+Version: 1.3.1
+Summary: Read-only CLI for NTU Cool: course announcements, course files, handouts, and submission status.
+Author-email: Gary Xie <gary.1020.xie@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Gary1020-byte/ntu-cool-cli
+Project-URL: Repository, https://github.com/Gary1020-byte/ntu-cool-cli
+Project-URL: Issues, https://github.com/Gary1020-byte/ntu-cool-cli/issues
+Project-URL: Changelog, https://github.com/Gary1020-byte/ntu-cool-cli/blob/main/CHANGELOG.md
+Keywords: ntu,cool,ceiba,cli,announcements,course,files,homework,handout,student,scraper,read-only
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Education
+Classifier: Natural Language :: English
+Classifier: Natural Language :: Chinese (Traditional)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Education
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28
+Dynamic: license-file
+
+# ntu-cool-cli
+
+A CLI and an agent skill for read-only access to NTU Cool: course
+announcements, course files, CA handouts, and submission status.
+
+Use it from a shell, or install it as a skill and let an agent drive it.
+
+> 中文版: [README.zh-TW.md](README.zh-TW.md)
+
+One CLI, four stores. Cool keeps a course's material in more than one place, and
+the obvious one is often empty:
+
+| what you want | where it actually is |
+|---|---|
+| what a course announced | `discussion_topics` — announcements and their attachments |
+| course files: handouts, worksheets, solutions, syllabus | the Files tree (`folders`) |
+| a CA handout, when the Files tree looks empty | the **assignment description** (`assignments`) |
+| lectures grouped by week | **modules** (`modules` → `modules/{id}/items`) |
+| whether your submission was accepted | `/submissions/self` |
+
+```bash
+# no credentials — is the platform reachable?
+python3 scripts/ntu_fetch.py --probe
+
+# what courses am I taking this term?
+python3 scripts/ntu_fetch.py --list-files --course <COURSE> --cred-file ~/.ntu-creds.json
+```
+
+**Read-only.** It logs in to read and never writes, posts, or comments on
+cool.ntu.edu.tw.
+
+---
+
+## Why not a service
+
+A daemon that polls on a schedule and pushes to Telegram needs a persistent
+store, an ORM, and a migration surface — to answer a one-endpoint question.
+On Python 3.14 that stack fails three ways before it reaches the endpoint:
+
+```
+ImportError: cannot import name 'Markup' from 'jinja2'          # Flask 1.0.2
+AttributeError: module 'sqlite3.dbapi2' has no attribute 'version_info'   # SQLAlchemy 1.3.8
+TypeError: Mapper.__init__() got an unexpected keyword argument 'order_by'
+```
+
+Upgrading Flask cascades into the SQLAlchemy break, which cascades into a
+mapper-API removal. The total patch surface is one file and four lines — which
+tells you the service is the problem, not the pins.
+
+This script drops Flask, SQLAlchemy, and `beautifulsoup4` entirely. **The only
+third-party import is `requests`.** HTML parsing uses `html.parser` from the
+stdlib, because the two things needed from BeautifulSoup — "find a form, read
+its inputs" and "flatten visible text" — are twenty lines of `HTMLParser`.
+
+Fetching is also the honest scope: a script that reads and exits has nothing to
+keep in sync. Storage, dedupe, and delivery belong to whatever runs it.
+
+---
+
+## Install
+
+As a command — this gives you `ntu-cool-cli` on your PATH.
+
+```bash
+pipx install ntu-cool-cli     # or: uv tool install ntu-cool-cli
+ntu-cool-cli --probe
+```
+
+Or with plain pip, into your user site:
+
+```bash
+python3 -m pip install --user ntu-cool-cli
+```
+
+**From Git instead** — if you want the unreleased tip, or to pin a commit:
+
+```bash
+uv tool install 'git+https://github.com/Gary1020-byte/ntu-cool-cli.git'
+pipx install 'git+https://github.com/Gary1020-byte/ntu-cool-cli.git'
+```
+
+**From a local checkout:**
+
+```bash
+git clone https://github.com/Gary1020-byte/ntu-cool-cli.git
+cd ntu-cool-cli
+pipx install .          # or: uv tool install .  /  python3 -m pip install --user .
+ntu-cool-cli --probe
+```
+
+The module is `ntu_fetch`, the command is `ntu-cool-cli`, and both do exactly the
+same thing — `ntu-cool-cli --help` and `python3 scripts/ntu_fetch.py --help` are
+identical. Use whichever suits; the script needs no install at all.
+
+Requires Python 3.9+ and `requests`. Verify:
+
+```bash
+ntu-cool-cli --probe
+```
+
+### Tests
+
+```bash
+python3 -m unittest discover -s tests -v
+```
+
+52 offline tests, no network and no credentials. They cover the JSON prefix
+handling, HTML form extraction without BeautifulSoup, the credential-file
+permission guard, and — the one that matters — that a trap form parked in
+`<head>` can never become a POST target.
+
+Two classes guard the bug that was silently losing files: one drives
+`_paged_get` across a multi-page Link chain, one proves the walk enters through
+the root and still works when `/folders/root` answers 403.
+
+### Note
+
+The console script is a thin wrapper: `ntu_fetch:main`, with `main()` returning
+the exit code. Importing the module has no side effects, so the tests can import
+it without logging in or reaching the network.
+
+> **Test with the interpreter you will actually use.** A virtualenv's Python and the
+> system Python have separate `site-packages`. A bug report that says "the import
+> fails" is often a report about the wrong interpreter.
+> See `references/testing-and-interpreters.md`.
+
+---
+
+## Credentials
+
+Two platforms need your NTU account. **This tool never stores, requests, or prints a
+password.**
+
+Either export them for one shell:
+
+```bash
+export NTU_USER=bXXXXXXXX
+export NTU_PASS='...'
+```
+
+…or, better, keep them in a file the script will only read when it is mode `600`:
+
+```bash
+umask 077
+echo '{"ntu_user":"bXXXXXXXX","ntu_pass":"..."}' > ~/.ntu-creds.json
+```
+
+A group- or world-readable credential file is **refused** with the `chmod` command to
+fix it. Note that environment variables exported in one terminal are not visible to
+another — a credential file is what makes this script usable unattended (cron, etc.).
+
+### Where the password can and cannot leak
+
+The password is submitted to ADFS as a form field. Four things it could have
+done, and what it actually does:
+
+| leak surface | what the tool does |
+|---|---|
+| printed in an error message | it doesn't — the login failure names the hop and the HTTP status, never the submitted values |
+| dumped by `requests` debug logging | it doesn't — `enable_http_debugging()` and `debuglevel` are absent, and a test asserts they stay absent |
+| left in the output as a traceback | it doesn't — the traceback would carry the *call*, not the value; the value only ever exists in the POST body |
+| sitting in a world-readable file | it refuses to read one |
+
+**The real risks are on your side of the boundary:**
+
+- **The credential file's path.** `--cred-file ~/.ntu-creds.json` goes into
+  your shell history, and the path is visible in the process list. That is
+  why the environment-variable path is the better one when you are typing
+  the command by hand — and the file path is the better one when a cron job
+  needs it unattended.
+- **The signed download URLs.** A file's `url` from the API authenticates.
+  The tool never logs one, and it never writes one into a file it produces.
+  Don't paste one into an issue.
+- **The session cookies.** They live in the `requests.Session` for the length
+  of the run and are gone when the process exits. Nothing is written to disk.
+
+---
+
+## The three traps this encodes
+
+### 1. `/api/v1/courses` is not your current course list
+
+It returns courses the account has **ever** been enrolled in.
+
+```
+API said:      10 courses, newest announcement 2026-10-13, oldest 2025-09-05
+               → omitted 3 current-term courses, one of them your main math course
+               → included 3 dead courses as {"access_restricted_by_date": true}
+
+Home page says: 7 courses, term "<term>"  ← correct
+```
+
+`fetch_courses()` parses the roster out of the home page's embedded JSON. The
+historical list is still reachable via `fetch_all_courses_historical()` when you want
+last term's exam schedule — never as the current roster.
+
+### 2. Course material is not in the announcement body
+
+Measured on one term:
+
+```
+20 announcement bodies    2,041 characters   0 file links
+course Files tree         91 files
+announcement attachments   5 files           <- a third set
+```
+
+An announcement whose entire body is the word `attached` is **not** a missing-
+attachment case to report — it is the normal shape, and the files are fully
+retrievable. Reporting "this course has no attachments" after reading announcements
+alone reads as complete and misses every worksheet.
+
+### 3. Enter through the root, and follow `Link` — three separate traps
+
+```
+GET /api/v1/courses/{id}/folders/root   → the root folder        <- enter here
+folder["files_url"]                     → files in that folder   (paginated)
+folder["folders_url"]                   → child folders; recurse (paginated)
+```
+
+**(a) The URLs come off the objects.** Six plausible spellings return 404 or 400 —
+including `/api/v1/courses/{id}/files`, which looks obviously right and is dead.
+
+**(b) `/folders` is a flat list, not a tree.** It enumerates folders at every depth.
+Iterating it *and* recursing `folders_url` walks each non-root folder twice — three
+times when its parent is listed too. A naive walk reported **110 files where 64 were
+retrievable**. Enter through the root instead.
+
+**(c) Every list endpoint is paginated at 10 by default.** This is the trap that
+actually loses data: a folder holding 29 files returns 10, with no error and no
+empty-looking tail. Measured across one term — **27 of 91 files silently missing**.
+
+```
+projected from Link header:  <...>; rel="next", <...>; rel="last"
+bare request:                page=1&per_page=10
+```
+
+Follow `Link: rel="next"`. Do not "fix" it with a bigger `per_page` — the API caps
+that parameter, and the Link header is the only pagination contract it honours.
+
+`fetch_course_files()` enters through `/folders/root` (falling back to the
+parentless entries of `/folders`, because `/folders/root` answers **403** on at
+least one non-credit course), paginates both directions, and dedupes by file `id`
+as a last resort. There are tests that fail if any of the three is removed.
+
+Announcement attachments are a **third** set, separate from both: one term had 5 of
+them and 91 in the Files tree. `--list-files` and `--download-files` collect both.
+
+---
+
+## Usage
+
+### Announcements
+
+```bash
+# current term roster
+python3 scripts/ntu_fetch.py --list-files --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# one course's announcements to stdout
+python3 scripts/ntu_fetch.py --course <COURSE> --since 2026-09-01 --stdout
+
+# write markdown into a notes directory
+python3 scripts/ntu_fetch.py --course <COURSE> --since 2026-09-01 --out ./captures/
+```
+
+### Course files
+
+```bash
+# list everything in the Files tree
+python3 scripts/ntu_fetch.py --list-files --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# download, skipping anything over 20 MB
+python3 scripts/ntu_fetch.py --download-files --course <COURSE> \
+    --cred-file ~/.ntu-creds.json --files-out ./files/ --max-mb 20
+```
+
+Downloads land at `<files-out>/<course_id>/<file_id>/<filename>`, so two files with
+the same display name never collide.
+
+### Assignments, submissions, pages, modules
+
+```bash
+# the homework roster: name, due_at, lock_at, points
+python3 scripts/ntu_fetch.py --list-assignments --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# your own submission status per assignment (read-only)
+python3 scripts/ntu_fetch.py --list-submissions --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# course pages: syllabus, calendar, late-submission policy
+python3 scripts/ntu_fetch.py --list-pages --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# modules — lectures grouped by week
+python3 scripts/ntu_fetch.py --list-modules --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# handout links that live in assignment descriptions, not the Files tree
+python3 scripts/ntu_fetch.py --list-handouts --course <COURSE> --cred-file ~/.ntu-creds.json
+
+# one course page's content, by slug (see --list-pages for the slugs)
+python3 scripts/ntu_fetch.py --show-page ke-cheng-da-gang --course <COURSE> \
+    --cred-file ~/.ntu-creds.json
+
+# download the handouts themselves
+python3 scripts/ntu_fetch.py --download-handout --course <COURSE> \
+    --cred-file ~/.ntu-creds.json --files-out ./handouts/
+```
+
+`lock_at` is the real deadline; `due_at` is the nominal one, and the gap between
+them is the buffer the course page promises. A `lock_at` in the past means the
+upload window is closed.
+
+`--show-page` needs exactly one course — it prints the page's title, its URL,
+and the body as plain text. `--download-handout` follows the two-hop path
+(the endpoint answers with JSON describing the file; the signed URL is inside
+it) and writes the file under its real display name.
+
+---
+
+## Installing this as an agent skill
+
+Copy the block below into your agent (Claude Code, Hermes, a custom assistant —
+anything that can run a command and read a file).
+
+**The agent must ask which capabilities are needed, then install only those.**
+Each capability maps to a specific flag or a specific section of `SKILL.md`, so
+a partial install is the normal case, not a compromise.
+
+````text
+Install the NTU Cool skill: https://github.com/Gary1020-byte/ntu-cool-cli
+
+Before installing anything, ask me which capabilities I need. Do NOT install
+all of them by default. The capabilities are:
+
+  1. Announcements        — course announcements, deadlines, TA notices
+                            flag: --course / --all-courses
+  2. Course files         — the Files tree: handouts, worksheets, solutions,
+                            syllabus PDFs
+                            flags: --list-files, --download-files
+  3. Announcement attachments — files bound to one announcement (a separate,
+                            smaller set from the Files tree)
+                            included in --list-files / --download-files
+  4. Homework roster      — /assignments: what is assigned, due_at, lock_at,
+                            points, and the handout link
+                            flag: --list-assignments
+  5. Submission status    — my own submission per assignment: unsubmitted /
+                            submitted / graded, score, late policy
+                            flag: --list-submissions
+  6. Course pages         — syllabus, calendar, late-submission policy,
+                            submission instructions
+                            flag: --list-pages
+  7. Modules              — lectures grouped by week
+                            flag: --list-modules
+  8. CA handouts          — handout links that live in assignment descriptions
+                            rather than the Files tree (the Files tree is often
+                            nearly empty for CA courses)
+                            flags: --list-handouts, --download-handout
+  9. Page content         — the body of one course page: syllabus, calendar,
+                            late-submission policy, submission instructions
+                            flag: --show-page <slug>
+
+Show me this list, ask which ones I want, and wait for my answer.
+
+Then:
+  - clone the repo
+  - install the `ntu-cool-cli` package:
+    `pipx install ntu-cool-cli` (or `uv tool install ntu-cool-cli`, or
+    `python3 -m pip install --user ntu-cool-cli`). It is on PyPI. If the
+    user wants the unreleased tip instead, install from the Git URL:
+    `uv tool install 'git+https://github.com/Gary1020-byte/ntu-cool-cli.git'`
+  - read SKILL.md, and keep only the sections matching the capabilities I chose
+  - tell me how to supply credentials. Prefer the environment variables
+    NTU_USER and NTU_PASS — nothing lands on disk and nothing stays in the
+    shell history. The alternative is a JSON file with ntu_user / ntu_pass
+    keys, passed with --cred-file; that file must be mode 600, because the
+    tool refuses to read a world-readable one. Never write credentials into
+    the repo, and never echo them back to me.
+  - run `ntu-cool-cli --probe` first. It needs no credentials and tells you
+    whether the platform is reachable. Do not debug credentials until the
+    probe is clean.
+
+This tool is READ-ONLY. It logs in to read and never writes, posts, or
+comments on cool.ntu.edu.tw. It cannot upload a submission — if I need to
+hand something in, tell me what is due and where the file is, and let me do
+the upload in the browser. Never attempt a write against the platform.
+````
+
+### Why the agent should ask first
+
+Cool keeps a course's material in four different places, and which one holds
+what is not guessable from the outside — the Files tree is nearly empty for
+some CA courses, and a handout that is missing from it is usually in the
+assignment description. Installing all nine capabilities means the agent
+reads a skill file that describes stores it will never query, and every
+capability it does not need is a way for it to answer a question with the
+wrong source.
+
+The list also doubles as a map: an agent that knows "the Files tree is empty
+for this course" knows to ask for `--list-handouts` instead of reporting
+"no attachments".
+
+---
+
+## What the output is worth trusting
+
+| claim | status |
+|---|---|
+| NTU Cool endpoints still exist | verified — 401 + structured body, not 404 |
+| the SAML login sequence | verified against the live site |
+| `/api/v1/courses` as current roster | **false** — see trap 1 |
+| the Files-tree endpoint shape | verified, 91 files enumerated across a term |
+| downloads are really PDFs | magic-checked per file |
+| nothing is silently dropped | verified — pagination followed via `Link` |
+| Ceiba reachability | **varies by network** — see `references/ceiba-unreachable.md` |
+| Ceiba scraping | not implemented; the positional table parser is unverified |
+
+**A download reporting `OK` has passed both checks.** Cool answers an expired signed
+URL with HTTP 200 carrying the login page, so status code alone proves nothing —
+the script sniffs for `<!DOCTYPE html` and verifies `%PDF-` against the declared
+content type.
+
+Signed file URLs are credential-bearing. The script never logs them; don't paste them
+anywhere.
+
+---
+
+## Many course PDFs have no text layer
+
+Of 10 files in one course, **9 had a zero-length text layer** — they are rasterised
+scans or vector drawings. That is not data loss. Render at ~160 dpi and read the
+image:
+
+```python
+import pymupdf
+d = pymupdf.open("worksheet.pdf")
+d[0].get_pixmap(dpi=160).save("page1.png")   # then read it
+```
+
+Never file a PDF as unusable on the strength of one extraction path.
+
+---
+
+## Ceiba
+
+`ceiba.ntu.edu.tw` is not implemented. On some networks it is unreachable (DNS
+resolves, three sibling NTU hosts answer in <150 ms, ceiba times out on connect).
+That is a network-access question, not a code one — see
+`references/ceiba-unreachable.md` for the control test that distinguishes them.
+
+---
+
+## Files
+
+```
+SKILL.md                              the rulebook: verification status, traps, workflow
+scripts/ntu_fetch.py                  the whole tool
+pyproject.toml                        makes `ntu-cool-cli` installable
+tests/test_ntu_fetch.py               52 offline tests, no network
+ACKNOWLEDGEMENTS.md                   attribution and licensing notes
+references/api-findings.md           what the platform actually returns
+references/ceiba-unreachable.md       reachability control test
+references/testing-and-interpreters.md  the venv-vs-system-python trap
+```
+
+`SKILL.md` is written for an AI agent reading it before acting. It is readable by
+humans too, but its shape is "what is verified, what is not, what will bite you".
+
+---
+
+## Ethics and scope
+
+Read-only. Every request this script makes is a `GET` except the two login form
+submissions required to obtain a session — there is no upload, no write, no delete,
+and no course modification of any kind. It obeys the same rule as the account
+holder: fetch what you can already see by logging in, and nothing more.
+
+Use it against your own account.