arcus-cli 0.1.0__tar.gz → 0.2.0__tar.gz

arcus_cli-0.2.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,47 @@
+name: Bug report
+description: Something in arcus isn't working as expected
+labels: ["bug"]
+body:
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened
+      description: What did you run, and what went wrong?
+      placeholder: |
+        Ran `arcus "..."` and got a traceback / wrong answer / hang, etc.
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: What you expected instead
+    validations:
+      required: true
+
+  - type: input
+    id: version
+    attributes:
+      label: arcus-cli version
+      description: Output of `pip show arcus-cli` or `uv tool list`
+    validations:
+      required: true
+
+  - type: dropdown
+    id: mode
+    attributes:
+      label: Which mode were you using?
+      options:
+        - "arcus \"...\" (one-shot)"
+        - "arcus chat"
+        - "arcus stats"
+        - "arcus --random"
+        - Not sure / other
+    validations:
+      required: true
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Full error output, if any
+      render: shell

arcus_cli-0.2.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,25 @@
+name: Feature request
+description: Suggest something arcus should do
+labels: ["enhancement"]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: What problem does this solve?
+      description: What are you trying to do that arcus doesn't support today?
+    validations:
+      required: true
+
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What you'd like to see
+      description: A rough idea of the behavior or interface is fine, doesn't need to be a full spec.
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives you've considered
+      description: Any workarounds you're using today, or other approaches you thought about.

{arcus_cli-0.1.0 → arcus_cli-0.2.0}/.github/workflows/release.yml

@@ -48,3 +48,16 @@ jobs:
 
       - name: Publish to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
+
+  github_release:
+    needs: publish
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write # required to create the GitHub Release
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Create GitHub Release
+        uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true

arcus_cli-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,48 @@
+# Contributing to Arcus
+
+## Setup
+
+```bash
+git clone https://github.com/ShakirFarhan/Arcus.git
+cd Arcus
+uv sync
+```
+
+You'll need an ARC API key to run the tool live (`llm.arc.vt.edu` > User
+profile > Settings > Account > API keys). Most of the test suite runs
+without one; a handful of live tests in `tests/adapters/test_arc_adapter_live.py`
+only run if `ARC_API_KEY` is set in your shell:
+
+```bash
+uv run pytest tests/ -q
+```
+
+## Before opening a PR
+
+- Add a test for any new business logic (routing, reward calculation,
+  cache scoring, quality checks). Bug fixes should include a test that
+  fails without the fix.
+- Keep PRs focused, one change per PR rather than bundling unrelated
+  fixes together.
+- Run the full test suite locally before pushing; CI runs it again on
+  every PR but catching it early saves a round trip.
+- Algorithmic functions (bandit logic, reward math, cache scoring)
+  should have a docstring explaining *why* a design choice was made,
+  not just what the code does.
+
+## Code style
+
+- No new dependencies without discussing them in the PR description
+  first, this project deliberately keeps its dependency footprint
+  small.
+- Comments should explain non-obvious reasoning (a workaround, a
+  constraint, a subtle invariant), not restate what the code already
+  says.
+- Prefer editing existing files and reusing existing patterns over
+  introducing new abstractions.
+
+## Reporting bugs
+
+Open an issue with what you ran, what you expected, and what actually
+happened. If it's reproducible with `arcus --random`, mention that too,
+it helps narrow down whether it's routing-specific.

{arcus_cli-0.1.0 → arcus_cli-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: arcus-cli
-Version: 0.1.0
+Version: 0.2.0
 Summary: Adaptive routing, a quality gate, and a correctness-aware cache on top of Virginia Tech ARC's LLM API
 Project-URL: Homepage, https://github.com/ShakirFarhan/Arcus
 Project-URL: Repository, https://github.com/ShakirFarhan/Arcus
@@ -30,6 +30,11 @@ Description-Content-Type: text/markdown
 
 # Arcus
 
+[![CI](https://github.com/ShakirFarhan/Arcus/actions/workflows/ci.yml/badge.svg)](https://github.com/ShakirFarhan/Arcus/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/arcus-cli.svg)](https://pypi.org/project/arcus-cli/)
+[![Python](https://img.shields.io/pypi/pyversions/arcus-cli.svg)](https://pypi.org/project/arcus-cli/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
+
 A CLI that sits on top of Virginia Tech ARC's LLM API and makes it
 smarter: it picks which of ARC's four open-weight models to route a
 request to, checks the response before handing it back to you, and
@@ -40,11 +45,17 @@ machine with your own ARC key. Nothing goes through a shared server.
 arcus "explain how binary search works"
 ```
 
+**Contents:** [Why](#why) · [How it works](#how-it-works) · [Install](#install)
+· [Usage](#usage) · [Status](#status) · [Security & privacy](#security--privacy)
+
 ## Why
 
 ARC gives every VT user free access to four open-weight models
 (gpt-oss-120b, GLM-5.3, Kimi-K3, DeepSeek-V4-Flash) through one
-OpenAI-compatible endpoint. Picking a model by hand every time is
+OpenAI-compatible endpoint ([ARC's own docs](https://www.docs.arc.vt.edu/ai/011_llm_api_arc_vt_edu.html)
+cover the service itself, including its rate limits and data-handling
+approval, arcus is a client built on top of it, not affiliated with
+ARC). Picking a model by hand every time is
 tedious, and a plain HTTP 200 doesn't tell you whether the response
 inside it was actually any good, a truncated answer or a flat refusal
 comes back looking the same as a correct one unless something reads the
@@ -127,6 +138,26 @@ by replaying the local request log, which works because a bandit's
 `update()` is just an associative accumulation of pull counts and reward
 sums.
 
+The four model ids arcus routes to live in `ArcModel`, but ARC runs its
+own model catalog independently and can rename or retire an entry at
+any time. `src/arcus/routing/model_catalog.py` checks the configured
+list against what ARC is actually serving (cached for a few hours so
+this doesn't cost a network round trip on every call) and quietly drops
+anything that's no longer live, rather than routing to a model
+guaranteed to fail. Local history logged under a since-renamed model id
+is skipped the same way when the bandit's state gets rebuilt.
+
+Optionally, code, math, and long-document questions can also route
+across ARC's `-thinking-*` reasoning-effort model variants, not just
+the base four (`arcus config set enable_reasoning_variants true`,
+default off). Everyday questions stay on the fast base four either way.
+**This is built and unit tested but hasn't been run against a real ARC
+key from this environment**, ARC's docs list these as separate catalog
+model ids rather than a parameter on the base id, the same pattern
+already confirmed for web search's legacy-tool-calling variants below,
+but that specific assumption is unverified. Ask a code or math question
+after turning it on and confirm it actually answers before trusting it.
+
 ### Quality gate
 
 Five independent checks run over every response: truncation
@@ -137,6 +168,17 @@ negative reward for that model in that context and retries with a
 different one, up to once per available arm, before giving up and
 returning the last attempt. See `src/arcus/quality/gate.py`.
 
+ARC caps concurrent requests per account rather than per model, so a
+429 doesn't mean the model that was just called is bad, switching to a
+different arm wouldn't help either. A rate limit gets a few short
+retries against the same model before it's treated as a real failure,
+so one busy moment doesn't unfairly tank that model's learned reward.
+
+Similarly, ARC's access restriction (see Install below) applies to the
+whole account, not one model, so hitting it stops the request
+immediately with a clear message instead of cycling through every arm
+against the same wall, and doesn't count against any model's reward.
+
 ### Semantic cache
 
 Local `sentence-transformers` embeddings (`all-MiniLM-L6-v2`), cosine
@@ -202,6 +244,29 @@ All three real algorithms land well below the random baseline, which is
 the actual point: they're spending far less time on worse-than-best
 arms than picking blindly would.
 
+### Document Q&A and web search
+
+Both build directly on capabilities ARC's own API already provides,
+rather than reimplementing them:
+
+- **`arcus --doc <path> "question"`** uploads the file to ARC's RAG
+  endpoint, attaches it to the request, and deletes it from your ARC
+  account again once you have an answer. Works across all four core
+  models, confirmed live against the real API.
+- **`arcus --web "question"`** routes to ARC's `server:websearch` tool
+  through three of its "legacy-tool-calling" model variants
+  (`gpt-oss-120b`, `Kimi-K3`, and the older `glm-52` variant) confirmed
+  to actually perform a real search and cite sources. A fourth,
+  DeepSeek's legacy variant, accepts the same request without erroring
+  but doesn't reliably act on it, live testing caught it answering a
+  time-sensitive question wrong with no citation, so it's left out.
+
+Both skip the semantic cache: a cached answer keyed on question text
+alone would risk answering about the wrong document, or serving a
+web-search answer that's since gone stale. See `src/arcus/cli.py`
+(`run_doc_ask`, `run_web_ask`) and `ArcAdapter.upload_file`/
+`delete_file` in `src/arcus/adapters/arc_adapter.py`.
+
 ## Install
 
 ```bash
@@ -225,6 +290,23 @@ First run walks you through a one-time setup: it asks for your ARC key
 `~/.config/arcus/config.toml` with `chmod 600`. No separate setup
 command to remember.
 
+ARC restricts the API to VT's campus network, so this (and every
+`arcus` call after it) needs either an on-campus connection or VT's
+VPN. Arcus surfaces this as a clear message rather than the generic
+"no usable response" error when it happens.
+
+For tab completion on the `chat`/`stats`/`eval`/`models`/`config`/
+`--random`/`--model`/`--image`/`--doc`/`--web` words, add one of these
+to your shell config:
+
+```bash
+# zsh, in ~/.zshrc
+eval "$(arcus --completion zsh)"
+
+# bash, in ~/.bashrc
+eval "$(arcus --completion bash)"
+```
+
 ## Usage
 
 ```bash
@@ -240,39 +322,145 @@ python broken.py 2>&1 | arcus "why is this failing"
 # force the random-routing baseline instead of the learned bandit policy
 arcus --random "explain how binary search works"
 
+# skip the bandit entirely and pin a specific model for this one call
+arcus --model GLM-5.3 "explain how binary search works"
+
 # see how it's doing
 arcus stats
 
+# compare the routing policy actually run against offline alternatives
+arcus eval
+
+# see every model ARC is currently serving, and which ones arcus routes to
+arcus models
+
 # hold a multi-turn conversation instead of a single question
 arcus chat
+
+# inside chat, --doc/--web/--image/--model all work inline, one
+# attachment per turn: "you: --doc paper.pdf summarize this"
+
+# save the conversation to a file when you leave
+arcus chat --save transcript.md
+
+# ask about an image (routes to Kimi-K3, the one ARC model documented
+# as vision-capable)
+arcus --image screenshot.png "what's wrong with this code?"
+
+# ask a question about a document, ARC handles the retrieval
+arcus --doc syllabus.pdf "when is the midterm?"
+
+# ask something that needs current information
+arcus --web "what's the latest release of Python?"
+
+# view or change local settings
+arcus config
+arcus config set bandit_algorithm ucb1
+
+# check which version is installed
+arcus --version
 ```
 
+Quick reference, details for each are below:
+
+| Command | What it does |
+| --- | --- |
+| `arcus "<question>"` | Ask something, routed through the bandit + quality gate. |
+| `arcus --random "<question>"` | Same, but routes randomly instead of using the learned policy. |
+| `arcus --model NAME "<question>"` | Skip routing, pin one specific model. |
+| `arcus --image PATH "<question>"` | Ask about an image (vision-capable model only). |
+| `arcus --doc PATH "<question>"` | Ask about an uploaded document (RAG). |
+| `arcus --web "<question>"` | Ask something needing current information (web search). |
+| `arcus chat [--save PATH]` | Multi-turn conversation; `--doc`/`--web`/`--image`/`--model` all work inline per turn. |
+| `arcus stats` | Local routing performance so far. |
+| `arcus eval` | Offline comparison of the routing policy against alternatives. |
+| `arcus models` | ARC's live model catalog vs. what arcus routes to. |
+| `arcus config [set ...]` | View or change local settings. |
+| `arcus --version` | Installed version. |
+
 `arcus chat` opens a REPL that remembers everything said earlier in that
 session (resending the growing transcript each turn, since ARC's API has
 no session concept of its own) and routes each turn through the same
 bandit/quality-gate/logging pipeline as a one-shot `arcus "..."` call.
 Type `exit` or press ctrl-d to leave. The conversation only lives for
-that one run, closing the terminal loses it.
+that one run, closing the terminal loses it, unless you pass `--save
+<path>`, which writes the full transcript (not just whatever's still in
+the trimmed context window) to a markdown file when you exit.
+
+`--doc PATH`, `--web`, `--image PATH`, and `--model NAME` all work
+inline inside `arcus chat` too, typed as part of a turn (`you: --doc
+paper.pdf summarize this`), one attachment per turn, the same rules as
+below apply. The attachment only applies to that one turn, a later turn
+that wants to keep asking about the same document attaches it again.
+
+`arcus --image <path> "question"` attaches an image to a one-shot
+question. It always goes to Kimi-K3 rather than through the usual
+bandit comparison, confirmed directly against the API to be the only
+one of the four models that can actually see an image, GLM-5.3 and
+DeepSeek-V4-Flash both reject image content outright and gpt-oss-120b
+accepts the request but reports it can't see anything. Skips the
+semantic cache entirely too, matching on the question text alone would
+risk serving back an answer about a completely different image.
+
+`arcus --doc <path> "question"` and `arcus --web "question"` work the
+same way as `--image`, cache skipped, see "Document Q&A and web search"
+above for what each actually does. Only one of `--image`, `--doc`, or
+`--web` can be used at a time.
+
+`arcus --model NAME "question"` skips the bandit entirely and always
+uses that model, checked against ARC's live catalog first. Since
+there's only one arm, the quality gate's checks (empty, truncated,
+repetitive, refusal) still run and still get reported, there's just no
+other model left to fall back to if it fails, that's the point of an
+explicit override. Combine with `--web` or `--image` and the name has
+to be one of the models valid for that mode.
+
+`arcus config` shows your current settings (the API key masked) and the
+path to the config file. `arcus config set bandit_algorithm <algo>`
+changes which bandit algorithm arcus uses without hand-editing the TOML
+file. `arcus config set enable_reasoning_variants <true|false>` turns
+the reasoning-effort routing described above on or off. Re-keying isn't
+supported here on purpose, delete the config file and run `arcus` again
+to go through setup fresh.
 
 `arcus stats` reads your local SQLite log and prints a `rich`-formatted
 table: request count, average reward, average latency, and cost score
 per model per mode, plus your cache hit rate and how many attempts the
 quality gate has caught and retried. Entirely local, no network call.
 
+`arcus eval` runs the offline policy evaluation described above against
+your own logged history and prints the comparison table (IPS and
+doubly-robust estimates with 95% confidence intervals for the greedy
+policy and each "always use model X" baseline, against what actually
+ran). Below 30 logged bandit-mode requests it still prints the table but
+flags the numbers as illustrative only, a bootstrap confidence interval
+on a handful of rows isn't a reliable comparison yet.
+
 ## Status
 
 Everything described above is implemented and working: the ARC adapter,
 context classification, all three bandit algorithms with propensity
-tracking, the reward function, the quality gate, the semantic cache and
-its benchmark, the CLI (ask command, chat mode, first-run wizard, error
-piping, stats), and the offline evaluation + regret benchmarking layer.
+tracking, the reward function, the quality gate (including rate-limit
+backoff and VPN-restriction handling), the semantic cache and its
+benchmark, the CLI (ask command with a manual `--model` override, chat
+mode with inline attachments and transcript export, image input,
+document Q&A, web search, a config command, first-run wizard, error
+piping, stats, models, and offline eval), and the offline evaluation +
+regret benchmarking layer.
 
 Verified live against a real ARC key: all four models respond correctly
-(`tests/adapters/test_arc_adapter_live.py`), and a full end-to-end
+(`tests/adapters/test_arc_adapter_live.py`), a full end-to-end
 `arcus "..."` run exercises the whole pipeline (context classification,
 cache miss, bandit routing, a real ARC call, the quality gate, logging,
-and caching the result) against real traffic. Test suite: 178 passing
-with a key set (174 plus 4 live-only tests), 4 skipped without one.
+and caching the result) against real traffic, and image input, document
+Q&A, and web search have each been run against real responses too, not
+just unit tested. Test suite: 273 passing with a key set (269 plus 4
+live-only tests), 4 skipped without one.
+
+The one exception: reasoning-effort variant routing
+(`enable_reasoning_variants`) is unit tested against a fake adapter only,
+not yet confirmed against a real ARC key, and defaults off for exactly
+that reason. See "Adaptive routing" above.
 
 Worth knowing: ARC's models are reasoning models under the hood, they
 write to a hidden `reasoning` field before `content`, so a small
@@ -284,19 +472,27 @@ budget of your own.
 
 What's still open:
 
-- Not published to PyPI yet.
-- Real logged usage is still thin (a handful of manual runs). Once
-  there's a real query history, `arcus/eval/offline.py`'s
-  `evaluate_policies()` is what turns it into the comparison table
-  described above.
+- Real logged usage is still thin (a handful of manual runs). `arcus
+  eval` runs the comparison today, it's just not resting on enough
+  data yet to trust the numbers, it says so when that's the case rather
+  than presenting a false-confidence table.
+- Reasoning-effort variant routing needs a live-key run to confirm ARC
+  actually serves the `-thinking-*` ids the way its docs describe,
+  before it's safe to turn on by default.
 
 ## Security & privacy
 
 - Each install uses its own user's ARC key. Keys are never shared,
   bundled, or sent anywhere but ARC's own endpoint.
-- No data leaves your machine by default. Request logs, cache entries,
-  and stats are all local SQLite, nothing is aggregated or reported
-  anywhere.
+- No data leaves your machine except to ARC itself, with your own key.
+  Request logs, cache entries, and stats are all local SQLite, nothing
+  is aggregated or reported anywhere else.
+- `arcus --doc` uploads the whole file to your ARC account temporarily
+  (deleted again once you have an answer), and `arcus --web` sends your
+  question through ARC's own web search tool. Both stay within ARC,
+  same as every other request, but a document leaving your machine
+  entirely (even briefly, even to your own account) is worth knowing
+  about explicitly.
 - This tool hasn't been through ARC's security review for regulated
   data (FERPA records, health data, etc.) the way ARC's own web
   interface has. Don't route sensitive regulated data through it.