gemcatch 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -7,6 +7,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-09-24
+
+### Added
+
+- **Research over your own data.** A Deep Research agent could only read the
+  public web through gemcatch, while the API has always let it read much more.
+  Five new flags on `research` and `batch` open that up, listed under their own
+  heading in `--help`. All of them need `--agent`; without it gemcatch exits
+  with one line naming the flag and writes nothing.
+
+  - `--attach <path|url>` (repeatable) gives the agent a PDF, CSV or image.
+    Local files go inline, in order, while the request stays under the API's
+    inline limit (100 MB, or 50 MB once a PDF is in it). The file that would
+    cross it, and every one after, is uploaded through the Files API and sent by
+    URI. Uploads expire after 48 hours, and a `--plan` run prints when. An https
+    URL is sent by reference, its query string masked everywhere but the wire,
+    and one without a supported extension is typed from a HEAD request (a
+    one-byte GET if HEAD is refused). Unknown types, missing and empty files are refused in one
+    line before the spend confirmation, with a hint to export Word files as PDF
+    and spreadsheets as CSV. A file that changes size between that check and
+    the send is refused too. A `batch` of more than one prompt uploads every
+    local file once instead of sending it inline with each prompt.
+  - `--mcp <url>` (repeatable) adds a remote MCP server, named after its host
+    unless `--mcp-name` says otherwise. `--mcp-header 'Name: value'` and
+    `--mcp-allow tool,tool` apply to the `--mcp` before them. Header values
+    and URL credentials are masked in the confirmation, `--dry-run`,
+    `list --json`, `get --raw` and API errors. A header value can read
+    `${VAR}` from the environment at send time, so the token never reaches
+    `tasks.db`; a value written out in full is kept there, since a later turn
+    has to resend it. On POSIX systems the data directory and the images
+    folder are now 0700, and `tasks.db` and each image 0600. A local or
+    private address, or credentials over plain http, get a warning.
+  - `--file-search <store>` (repeatable) lets the agent search File Search
+    stores, all in one `file_search` tool.
+  - `--no-web` drops Google Search and URL Context so the agent reads only what
+    you gave it. Code Execution stays, because it's how the agent works through a
+    CSV and draws a chart. It's refused when nothing else was given to read.
+  - `--visualize` sets `agent_config.visualization: "auto"`, alongside
+    `collaborative_planning` when `--plan` is also given. Charts in the final
+    report are written to the data directory's `images/` folder as
+    `<task-id>-<n>.<ext>`, recorded in the store, and listed under the report by
+    `get`, `watch` and `digest` and as `images` in `--json`. `export -o` copies
+    them next to the export and links them; `rm` and `prune` delete them.
+
+  Once a tool flag is given, the request lists Google Search, URL Context and
+  Code Execution alongside the new sources, because an explicit `tools` list
+  replaces the agent's defaults. `--attach` on its own sends no `tools` field.
+  `refine` and `approve` resend the plan's tools and visualization setting;
+  attachments go with the first turn only. `--dry-run` and the spend
+  confirmation list the sources above the cost, and `list` gains a SOURCES
+  column when a listed task used any of this.
+- An unknown `--flag=value` option is reported without its value, which could
+  be a token.
+- Additive schema migration: `tools_json`, `attachments_json`, `visualization`
+  and `images_json`, all nullable. A 0.5.0 `tasks.db` upgrades in place and
+  behaves exactly as it did.
+
+### Notes
+
+- A run with none of the new flags sends the same request body, byte for byte,
+  as 0.5.0 did, and a result without charts prints the same text.
+
+## [0.5.0] - 2026-09-04
+
+### Added
+
+- **Collaborative planning.** Until now the only agent flow was a blind one-shot
+  bet: gemcatch showed a cost band, asked y/N, and fired. The cost band was
+  everything you knew before spending, and there was no way to check the agent
+  had read the prompt the way you meant it. A 20-line `batch` against
+  `deep-research-max` is a $60–$140 command placed sight unseen. The Deep
+  Research docs have always documented a second mode for exactly this, and
+  gemcatch could not reach it. Now it can:
+
+  - `--plan` on `research` and `batch` submits with
+    `agent_config.collaborative_planning: true`, so **the agent returns a
+    research plan instead of a report**. Still `background: true`, still stored,
+    still collected by the daemon. The row is stored with `kind='plan'`.
+  - `gemcatch get <id>` on a plan prints the plan and then the literal next
+    command, `Approve with: gemcatch approve 8f3a1c04 · Refine with: gemcatch
+    refine 8f3a1c04 "..."`, on stderr, so `get <plan> > plan.md` still captures
+    just the plan.
+  - `gemcatch refine <id> "<instruction>"` sends the instruction back with
+    `previous_interaction_id` and planning still on, and stores the revised plan
+    linked to the one it came from, inheriting its agent and tag.
+  - `gemcatch approve <id>` is the turn that commits: `previous_interaction_id`
+    with `collaborative_planning: false`, stored as `kind='report'` linked to the
+    plan. It shows the cost band and asks, honours `--yes` when stdin is not a
+    TTY, and honours `--dry-run`.
+
+  **Planning is not cheaper.** The docs give one band per task and do not price a
+  planning turn separately, so a planning turn is quoted at the same band and the
+  line says so: *"the docs price per task and do not price a planning turn
+  separately"*. What you get for it is a look at the plan before you commit to
+  the research run, not a discount.
+- `gemcatch list` renders a chain indented under its root, in submission order,
+  and grows a `KIND` column when a listing contains plans or reports (a store
+  with neither keeps the layout it had). `gemcatch export` follows a chain to its
+  report and leaves the intermediate plans out unless `--include-plans` is
+  passed; the Markdown sections and the JSON rows now name the turn.
+- `gemcatch stats` now tallies plan and report turns, and totals the estimated
+  spend across the agent turns that were actually billed, from the same
+  published bands the guard quotes before each one. A chain bills per turn, so a
+  release that turns one submission into three owes the user a running total. It
+  is the documented bands applied to what was submitted, not a reading of your
+  bill, and it errs toward telling you rather than flattering you: only runs
+  that reached the server are priced (a submit that failed before it left the
+  machine is counted as an attempt but costs nothing), and an agent with no
+  published band totals to `unknown`, never to `$0.00`. A store with no agent
+  runs prints neither line.
+- Additive schema migration: `collaborative_planning`, `previous_interaction_id`,
+  `kind` (defaulting to `'task'`) and `parent_id`. A 0.4.0 `tasks.db` upgrades in
+  place, keeps every row, and behaves exactly as it did.
+
+### Fixed
+
+- `approve` and `refine` fail before sending anything when the target is not a
+  plan, or is a plan that has not completed. A plan whose interaction the free
+  tier has already dropped names the retention window (1 day free, 55 days paid)
+  instead of sending a `previous_interaction_id` the server would reject. A plan
+  that expires *after* it completed locally gets the same explanation rather
+  than a bare 404 for an id the user never typed.
+- `--plan` without `--agent` (or with `--model`) is a clean error rather than a
+  flag that quietly does nothing: collaborative planning is an agent feature.
+- Approving the same plan twice submits a second run rather than silently
+  reusing the first, and `list` shows both under their plan.
+
 ## [0.4.0] - 2026-08-08
 
 ### Added
@@ -204,7 +331,9 @@ seen a task complete, the text is cached locally and survives that expiry — bu
 something has to poll inside that window for it to be seen at all, which is what
 `gemcatch daemon` exists to do.
 
-[Unreleased]: https://github.com/Booyaka101/gemcatch/compare/v0.4.0...HEAD
+[Unreleased]: https://github.com/Booyaka101/gemcatch/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/Booyaka101/gemcatch/compare/v0.5.0...v0.6.0
+[0.5.0]: https://github.com/Booyaka101/gemcatch/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/Booyaka101/gemcatch/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/Booyaka101/gemcatch/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/Booyaka101/gemcatch/compare/v0.1.1...v0.2.0

package/README.md

@@ -81,8 +81,10 @@ This week in AI: ...
 | `gemcatch research "<prompt>"` | Submits with `background: true`, stores the interaction ID, exits immediately. |
 | `gemcatch batch <file>` | Submits many prompts from a file at once, tagged as one collectable batch. |
 | `gemcatch status <id>` | Polls the API and prints the current state. |
-| `gemcatch get <id>` | Prints the full response if complete, otherwise the current status. |
-| `gemcatch list` | All tasks, newest first: id, age, status, prompt. |
+| `gemcatch get <id>` | Prints the full response if complete, otherwise the current status. On a [plan](#see-the-plan-before-you-pay-for-the-run), the plan plus the approve/refine commands. |
+| `gemcatch refine <id> "<instruction>"` | Sends an instruction back to a plan and returns a revised plan. |
+| `gemcatch approve <id>` | Approves a plan and submits the research run it describes. |
+| `gemcatch list` | All tasks, newest first: id, age, status, prompt. Plan chains are indented under their root. |
 | `gemcatch export` | Concatenates finished results, each under its prompt, to stdout or a file (Markdown or JSON). |
 | `gemcatch digest` | Feeds a tag's completed results through one Gemini call into a single summary. |
 | `gemcatch watch <id>` | Polls until the task finishes, then prints the result. |
@@ -91,7 +93,7 @@ This week in AI: ...
 | `gemcatch cancel <id>` | Asks the API to stop an in-flight task. |
 | `gemcatch rm <ids...>` | Forgets tasks locally. `--remote` deletes them server-side too. |
 | `gemcatch prune` | Drops finished tasks older than `--days` (default 30). |
-| `gemcatch stats` | Where the store lives and what's in it. |
+| `gemcatch stats` | Where the store lives, what's in it, and what the agent runs have plausibly cost. |
 
 Useful flags:
 
@@ -100,7 +102,13 @@ Useful flags:
 | `--json` | most commands | Machine-readable output. |
 | `-m, --model <id>` | `research`, `batch` | Override the model. |
 | `-a, --agent <id>` | `research`, `batch` | Submit to a [research agent](#research-agents) instead of a model. Mutually exclusive with `--model`. |
-| `--yes` | `research`, `batch` | Confirm the agent cost without asking. Required for `--agent` when stdin is not a TTY. |
+| `--plan` | `research`, `batch` | Ask the agent for a [research plan](#see-the-plan-before-you-pay-for-the-run) first, to refine and approve. Needs `--agent`. |
+| `--attach <path\|url>` | `research`, `batch` | Give the agent a PDF, CSV or image ([your own data](#research-over-your-own-data)). Repeatable. Needs `--agent`. |
+| `--mcp <url>` | `research`, `batch` | Let the agent call a remote MCP server. Repeatable; `--mcp-header`, `--mcp-allow` and `--mcp-name` apply to the one before them. Needs `--agent`. |
+| `--file-search <store>` | `research`, `batch` | Let the agent search a File Search store. Repeatable. Needs `--agent`. |
+| `--no-web` | `research`, `batch` | Drop Google Search and URL Context, so the agent reads only what you gave it. Needs `--agent`. |
+| `--visualize` | `research`, `batch` | Let the agent draw charts; they're saved as image files. Needs `--agent`. |
+| `--yes` | `research`, `batch`, `refine`, `approve` | Confirm the agent cost without asking. Required for `--agent` when stdin is not a TTY. |
 | `-s, --system <text>` | `research`, `batch` | Set a system instruction. |
 | `-f, --file <path>` | `research` | Read the prompt from a file. |
 | `-t, --tag <tag>` | `research`, `batch`, `list` | Label tasks and filter them. |
@@ -112,7 +120,8 @@ Useful flags:
 | `-n, --limit <n>` | `list` | Cap the rows (non-negative; `0` shows none). |
 | `--format <md\|json>` | `export` | Output format. Default `md`. |
 | `-o, --out <file>` | `export` | Write to a file instead of stdout. |
-| `--dry-run` | `research`, `batch`, `prune` | Show what would go — including the projected agent spend; submit/delete nothing. |
+| `--include-plans` | `export` | Also emit a chain's plan turns, not just its report. |
+| `--dry-run` | `research`, `batch`, `refine`, `approve`, `prune` | Show what would go — including the projected agent spend; submit/delete nothing. |
 | `--raw` | `get` | Dump the raw interaction JSON. |
 
 IDs are the first 8 characters of a UUID. Any unique prefix works, so `gemcatch get 8f3a` is fine.
@@ -187,14 +196,172 @@ The report lands like any other result — final answer only, none of the agent'
 
 An agent run can also come back `incomplete` — that is what a `max_total_tokens` budget cap produces when the run "safely pauses" — which `gemcatch` treats as terminal, exactly like the API does: the daemon retires it and moves on.
 
+### See the plan before you pay for the run
+
+A cost band tells you what a run will cost. It tells you nothing about whether the agent understood the question. Add `--plan` and it doesn't start researching: with `agent_config.collaborative_planning: true`, *"the agent returns a research plan instead of a full report"*. You read it, push back on it, and approve it when it's right.
+
+```console
+$ gemcatch research "map the EU AI Act high-risk obligations against the UK approach" --agent deep-research --plan -t euuk
+Agent deep-research-preview-04-2026 (planning turn) — estimated $1.00–$3.00 for this task (preview rates, subject to change; the docs price per task and do not price a planning turn separately).
+Submit? [y/N] y
+Plan task d014e21b submitted. Run: gemcatch get d014e21b when ready.
+
+$ gemcatch get d014e21b
+Research plan
+
+1. Scope the EU AI Act high-risk regime: Annex III use cases, Article 6 classification,
+   and the Chapter III obligations (risk management, data governance, logging, human
+   oversight, conformity assessment) with their August 2026 / August 2027 dates.
+2. Scope the UK approach: the five cross-sector principles, the regulator-led model
+   (ICO, FCA, MHRA, Ofcom), and what is guidance rather than statute.
+3. Build an obligation-by-obligation comparison table: EU requirement, nearest UK
+   equivalent, whether it is binding, and who enforces it.
+4. Flag the gaps in both directions and the compliance implications for a firm
+   operating in both.
+Approve with: gemcatch approve d014e21b   ·   Refine with: gemcatch refine d014e21b "..."
+
+$ gemcatch refine d014e21b "cut the history, and add enforcement penalties on both sides"
+Plan task 2140e699 submitted (refines d014e21b). Run: gemcatch get 2140e699 when ready.
+
+$ gemcatch approve 2140e699
+Agent deep-research-preview-04-2026 — estimated $1.00–$3.00 for this task (preview rates, subject to change).
+Submit? [y/N] y
+Task 96e8209b submitted (approves plan 2140e699).
+```
+
+Every turn is an ordinary background task: stored, polled, and collected by the daemon before the 1-day expiry, same as everything else. `refine` chains as many times as you like; each revision inherits the agent and tag of the plan it came from.
+
+**A planning turn is a task, and it is billed as one.** The docs publish one band per task and price no planning turn separately, so `gemcatch` quotes the same band for it and says exactly that on the line. What `--plan` buys you is a look at the plan before you commit to the research run, not a discount. Budget for the plan, each refinement, and the run.
+
+Since a chain bills per turn, `gemcatch stats` keeps a running total across all of them:
+
+```console
+$ gemcatch stats
+Store: ~/.gemcatch/tasks.db
+Tasks: 3
+  completed         3
+Agent runs:
+  deep-research-preview-04-2026      3
+Plan chains: 2 plan, 1 report
+Estimated spend: $3.00–$9.00 across 3 billed task(s) (preview rates, subject to change).
+```
+
+That's the chain above: one plan, one refinement, one run, three tasks at the same band. It's the published bands applied to what you actually submitted, not a reading of your bill, and it errs toward telling you rather than flattering you:
+
+- Only runs that **reached the server** are priced. A submit that failed before it left the machine (bad key, rejected agent id) is still counted under "Agent runs" as an attempt, but it costs nothing and isn't billed.
+- An agent with **no published band** totals to `unknown`, never to `$0.00`. Quoting zero for a run that costs real money is the one thing a spend guard must not do.
+
+`list` shows the chain as one thing, and `export` follows it to the report:
+
+```console
+$ gemcatch list
+ID        AGE   STATUS           KIND    AGENT              PROMPT
+d014e21b  22s   completed        plan    deep-research      map the EU AI Act high-risk obligations against the UK ap...
+2140e699  12s   completed        plan    deep-research      └─ cut the history, and add enforcement penalties on both sides
+96e8209b  7s    completed        report  deep-research        └─ map the EU AI Act high-risk obligations against the UK ap...
+
+$ gemcatch export --tag euuk -o report.md
+Wrote 1 result(s) to report.md.
+```
+
+The plans are working notes on the way to the report, so `export` leaves them out; pass `--include-plans` if you want the whole chain in the document. The report is filed under the question that started the chain rather than the one-line approval actually sent to the API, so an exported document reads as research.
+
+A few things that will bite otherwise:
+
+- `--plan` needs `--agent`. Collaborative planning is an agent feature, so `--plan --model ...` (or `--plan` on its own) is an error, not a no-op.
+- `approve` only works on a plan that has completed. Anything else fails before a request goes out.
+- On the free tier the plan's interaction is dropped after a day. Once that happens the chain cannot be continued. `approve` says so and names the window instead of sending a `previous_interaction_id` the server will reject. Run the daemon, or approve the same day.
+- Approving twice submits twice. There is no dedupe, and both runs show up under the plan in `list`.
+- `--plan` and `--watch` work together: `gemcatch research "..." --agent deep-research --plan -w` waits for the plan and then prints it.
+
 The agent recipe, end to end:
 
 ```bash
-$ gemcatch batch questions.txt --agent deep-research --yes   # bands shown, N × total quoted
-$ gemcatch daemon --exit-when-idle                           # catch reports before the 1-day expiry
-$ gemcatch export --tag batch-1a2b3c -o reports.md           # every report, with its sources
+$ gemcatch batch questions.txt --agent deep-research --plan --yes   # a plan per prompt, N × band quoted
+$ gemcatch daemon --exit-when-idle                                  # catch the plans
+$ gemcatch list --tag batch-1a2b3c                                  # read them, approve the good ones
+$ gemcatch approve 8f3a1c04 --yes
+$ gemcatch daemon --exit-when-idle                                  # catch the reports before the 1-day expiry
+$ gemcatch export --tag batch-1a2b3c -o reports.md                  # every report, with its sources
 ```
 
+Drop `--plan` and the first two lines become the 0.4.0 one-shot flow, which still works exactly as it did.
+
+## Research over your own data
+
+Out of the box a Deep Research agent reads the public web. These flags hand it your files, your MCP servers and your File Search stores as well, and `--visualize` lets it draw charts. They are agent features, so every one of them needs `--agent`. Without it gemcatch stops with one line naming the flag and sends nothing.
+
+### Files: `--attach`
+
+```console
+$ gemcatch research "Which region is growing fastest, and does the board pack agree?" --agent deep-research \
+    --attach sales-2026.csv --attach q3-board-pack.pdf --attach whiteboard.png \
+    --attach customer-contracts-2026.pdf --yes
+Attachments: sales-2026.csv (inline, 1 KB); q3-board-pack.pdf (inline, 1 KB); whiteboard.png (inline, 1 KB); customer-contracts-2026.pdf (upload, 40.0 MB)
+Agent deep-research-preview-04-2026 — estimated $1.00–$3.00 for this task (preview rates, subject to change).
+Uploading customer-contracts-2026.pdf (40.0 MB) to the Files API...
+Task 7cf00c5b submitted. Run: gemcatch get 7cf00c5b when ready.
+```
+
+PDF and CSV go in as documents; PNG, JPEG, WebP, HEIC, HEIF, GIF and BMP as images. The type comes from the extension, and anything else is refused before you're asked to pay, with a nudge to export a Word file as PDF or a spreadsheet as CSV. `--attach` also takes an https URL, which is passed to the agent by reference. A URL with no supported extension (arxiv's `https://arxiv.org/pdf/1706.03762`, say) gets one HEAD request to read its Content-Type, or a one-byte GET if the server refuses HEAD. The query string of a signed URL is sent as given and shown as `***` everywhere else, the store included.
+
+Local files are sent inline, in the order given, as long as the request stays under the API's inline limit: 100 MB of base64, or 50 MB once a PDF is in it. The first file that doesn't fit, and every file after it, goes up through the Files API instead and is referenced by URI. Google keeps uploads for 48 hours. On a `--plan` run gemcatch prints when they expire, and a `refine` or `approve` after that gets a warning. A file given twice is sent once.
+
+In a `batch` with more than one prompt, every local file is uploaded, once, and each prompt points at the upload. Inline bytes would otherwise go over the wire again with every prompt.
+
+Files go with the first turn only. A `refine` or `approve` continues the same conversation, and the files are already in it.
+
+### MCP servers: `--mcp`
+
+```console
+$ gemcatch research "Which accounts are at risk of churn this quarter?" --agent deep-research --plan --yes \
+    --mcp https://mcp.example-crm.com/mcp \
+    --mcp-header 'Authorization: Bearer ${CRM_TOKEN}' \
+    --mcp-allow search_accounts,get_account
+Tools: google_search; url_context; code_execution; MCP mcp.example-crm.com https://mcp.example-crm.com/mcp (Authorization: ***) [search_accounts, get_account]
+Agent deep-research-preview-04-2026 (planning turn) — estimated $1.00–$3.00 for this task (preview rates, subject to change; the docs price per task and do not price a planning turn separately).
+Plan task 5451f8b9 submitted. Run: gemcatch get 5451f8b9 when ready.
+```
+
+`--mcp` is repeatable, and `--mcp-header`, `--mcp-allow` and `--mcp-name` apply to the `--mcp` just before them. The server is named after its host unless you pass `--mcp-name`, and two servers on one host become `host` and `host-2`. Giving two servers the same `--mcp-name`, or one server the same header twice, is refused. Google's servers make the call, not your machine, so the URL has to be reachable from the internet; a localhost or private address gets a warning, and so does a plain `http://` URL that carries credentials.
+
+Header values, and any user, password, query string or fragment in an MCP URL, never appear in anything gemcatch prints: the confirmation, `--dry-run`, `list --json`, `get --raw` and API errors all show `***`.
+
+Note the single quotes in the example. gemcatch reads `${CRM_TOKEN}` from the environment itself, each time a turn is sent, so `tasks.db` only ever holds the reference. A `refine` or `approve` needs the variable set too, and stops in one line naming it if it isn't. A value written out in full, or expanded by your shell, is stored as given, because a later turn has to send it again. On Linux and macOS gemcatch keeps `~/.gemcatch` readable by you alone.
+
+### File Search stores: `--file-search`
+
+```console
+$ gemcatch research "What does our playbook say about discounting?" --agent deep-research \
+    --file-search sales-playbooks --no-web --dry-run
+Tools: code_execution; File Search fileSearchStores/sales-playbooks
+Agent deep-research-preview-04-2026 — estimated $1.00–$3.00 for this task. Nothing submitted (--dry-run).
+```
+
+Repeat it for more stores; they all go in one `file_search` tool. A bare name gets `fileSearchStores/` put in front of it. gemcatch doesn't create or fill stores, [the File Search docs](https://ai.google.dev/gemini-api/docs/file-search) cover that.
+
+### Charts: `--visualize`
+
+```console
+$ gemcatch research "Which region is growing fastest, and does the board pack agree?" --agent deep-research \
+    --attach sales-2026.csv --visualize --yes -w
+...
+(report text)
+
+Images:
+  /home/you/.gemcatch/images/7cf00c5b-1.png
+```
+
+This sets `agent_config.visualization` to `auto`, next to `collaborative_planning` when you also pass `--plan`. The charts in the final report are written to the data directory's `images/` folder as `<task-id>-<n>.<ext>`, and listed under the report by `get`, `watch` and `digest`, and as `images` in `--json`. `export -o report.md` copies them next to the export and links them in the Markdown. `rm` and `prune` delete them with the task.
+
+### What the agent can reach: `--no-web`
+
+With no source flags gemcatch sends no `tools` field and the agent gets its defaults: Google Search, URL Context and Code Execution. The API reads an explicit list as the whole set, so once you add `--mcp` or `--file-search`, gemcatch lists those three back in alongside your sources. `--attach` alone sends no list at all, since a file isn't a tool.
+
+`--no-web` drops Google Search and URL Context, so the agent works only from what you gave it. Code Execution stays: it can't reach the web, and it's what the agent uses to crunch a CSV and draw a chart. On its own `--no-web` is refused, because it would leave the agent nothing to read.
+
+`refine` and `approve` send the plan's tools and visualization setting again, so every turn of a chain reaches the same sources. Once a task in the listing used any of this, `list` grows a SOURCES column (`mcp`, `store`, `no-web`, `2 files`, `charts`).
+
 ## How it works
 
 Tasks live in SQLite at `~/.gemcatch/tasks.db` (override with `GEMCATCH_HOME`):
@@ -202,7 +369,9 @@ Tasks live in SQLite at `~/.gemcatch/tasks.db` (override with `GEMCATCH_HOME`):
 ```sql
 CREATE TABLE tasks (id TEXT PRIMARY KEY, prompt TEXT, interaction_id TEXT,
                     status TEXT DEFAULT 'pending', result TEXT, created_at INTEGER);
--- plus model, system_instruction, tag, error, usage, updated_at, agent, citations
+-- plus model, system_instruction, tag, error, usage, updated_at, agent, citations,
+--      collaborative_planning, previous_interaction_id, kind, parent_id,
+--      tools_json, attachments_json, visualization, images_json
 ```
 
 `research` calls `interactions.create({model, input, background: true})` via [`@google/genai`](https://www.npmjs.com/package/@google/genai) and keeps the returned `id`. The polling commands call `interactions.get(id)` and write the status back. Once a task completes, the text is cached in the `result` column — `gemcatch get` then answers from disk without touching the network.
@@ -252,6 +421,7 @@ Transient failures are retried with exponential backoff and full jitter, honouri
 | `GEMCATCH_HOME` | Where `tasks.db` lives. Default `~/.gemcatch`. |
 | `GEMCATCH_MODEL` | Default model. Default `gemini-3.5-flash-lite`. |
 | `GEMCATCH_POLL_MS` | `watch` poll interval in ms. Default `10000`. |
+| `GEMCATCH_UPLOAD_POLL_MS` | How often an upload still being processed by the Files API is checked, in ms. Default `2000`. |
 | `GEMCATCH_DAEMON_S` | `daemon` interval in seconds. Default `300`. |
 | `GEMCATCH_RPM` | Requests/minute ceiling. Default `15` (the free tier). `0` disables pacing. |
 | `GEMCATCH_MAX_RETRIES` | Extra attempts on a transient failure. Default `4`. `0` disables retries. |

package/db.js

@@ -31,6 +31,26 @@ const MIGRATIONS = [
   // the JSON array of sources an agent run returned alongside its report.
   ['agent', 'TEXT'],
   ['citations', 'TEXT'],
+  // 0.5.0: collaborative planning. `collaborative_planning` is the agent_config
+  // flag the row was submitted with (1 plan turn, 0 report turn, NULL for every
+  // run that sent no agent_config at all, including every pre-0.5.0 row);
+  // `previous_interaction_id` is the interaction this turn continues;
+  // `kind` is 'task' | 'plan' | 'report'; `parent_id` is the local task this
+  // turn continues. The DEFAULT backfills 'task' for older rows, so a 0.4.0
+  // store keeps behaving exactly as it did.
+  ['collaborative_planning', 'INTEGER'],
+  ['previous_interaction_id', 'TEXT'],
+  ['kind', "TEXT DEFAULT 'task'"],
+  ['parent_id', 'TEXT'],
+  // 0.6.0: sources. `tools_json` is the tools array the row was submitted with
+  // (header values included, so it is masked wherever it is printed);
+  // `attachments_json` records what was attached on this turn, never the bytes;
+  // `visualization` is the agent_config value; `images_json` lists the image
+  // files saved from the result. All NULL for a run that used none of them.
+  ['tools_json', 'TEXT'],
+  ['attachments_json', 'TEXT'],
+  ['visualization', 'TEXT'],
+  ['images_json', 'TEXT'],
 ];
 
 let _db = null;
@@ -46,8 +66,18 @@ function migrate(d) {
 
 function db() {
   if (_db) return _db;
-  fs.mkdirSync(HOME, { recursive: true });
+  const created = fs.mkdirSync(HOME, { recursive: true, mode: 0o700 });
   _db = new Database(DB_PATH);
+  // Rows can hold MCP header values. mkdir's mode only applies to a new
+  // directory, so an existing ~/.gemcatch is tightened too; a GEMCATCH_HOME the
+  // user already had is left alone. SQLite gives the -wal and -shm files the
+  // database file's mode. Best effort: some mounts refuse chmod.
+  if (process.platform !== 'win32') {
+    try {
+      if (!created && !process.env.GEMCATCH_HOME) fs.chmodSync(HOME, 0o700);
+      fs.chmodSync(DB_PATH, 0o600);
+    } catch (_) {}
+  }
   _db.pragma('journal_mode = WAL');
   _db.exec(BASE_SCHEMA);
   migrate(_db);
@@ -64,8 +94,12 @@ function createTask(fields) {
   const now = Date.now();
   db()
     .prepare(
-      'INSERT INTO tasks (id, prompt, status, created_at, updated_at, model, system_instruction, tag, agent) ' +
-        'VALUES (@id, @prompt, @status, @now, @now, @model, @system_instruction, @tag, @agent)'
+      'INSERT INTO tasks (id, prompt, status, created_at, updated_at, model, system_instruction, tag, agent, ' +
+        'kind, parent_id, collaborative_planning, previous_interaction_id, ' +
+        'tools_json, attachments_json, visualization, images_json) ' +
+        'VALUES (@id, @prompt, @status, @now, @now, @model, @system_instruction, @tag, @agent, ' +
+        '@kind, @parent_id, @collaborative_planning, @previous_interaction_id, ' +
+        '@tools_json, @attachments_json, @visualization, @images_json)'
     )
     .run({
       id,
@@ -76,6 +110,16 @@ function createTask(fields) {
       system_instruction: t.systemInstruction || null,
       tag: t.tag || null,
       agent: t.agent || null,
+      kind: t.kind || 'task',
+      parent_id: t.parentId || null,
+      // Presence, not truthiness: `false` is the report turn's real flag and
+      // must be stored as 0, while a run that sends no agent_config stores NULL.
+      collaborative_planning: t.collaborativePlanning === undefined ? null : Number(!!t.collaborativePlanning),
+      previous_interaction_id: t.previousInteractionId || null,
+      tools_json: t.toolsJson || null,
+      attachments_json: t.attachmentsJson || null,
+      visualization: t.visualization || null,
+      images_json: t.imagesJson || null,
     });
   return id;
 }
@@ -107,7 +151,7 @@ function setStatus(id, status, extra) {
   const e = extra || {};
   const sets = ['status = @status', 'updated_at = @now'];
   const params = { id, status, now: Date.now() };
-  for (const key of ['result', 'error', 'usage', 'citations']) {
+  for (const key of ['result', 'error', 'usage', 'citations', 'images_json']) {
     if (e[key] !== undefined) {
       sets.push(`${key} = @${key}`);
       params[key] = e[key];
@@ -181,6 +225,27 @@ function agentCounts() {
     .all();
 }
 
+// Agent runs that actually reached the server, keyed by agent. A submit that
+// never got an interaction_id (a bad key, a rejected agent id, a network
+// failure) was never billed, so it must not appear in a spend total -- unlike
+// agentCounts(), which tallies every attempt.
+function billedAgentCounts() {
+  return db()
+    .prepare(
+      'SELECT agent, COUNT(*) AS n FROM tasks WHERE agent IS NOT NULL AND interaction_id IS NOT NULL GROUP BY agent'
+    )
+    .all();
+}
+
+// Plan/report totals for `stats`. Ordinary tasks are not a row here; they are
+// already accounted for in counts(), and a store that has never planned reports
+// nothing at all.
+function kindCounts() {
+  return db()
+    .prepare("SELECT kind, COUNT(*) AS n FROM tasks WHERE kind IS NOT NULL AND kind != 'task' GROUP BY kind")
+    .all();
+}
+
 function close() {
   if (_db) _db.close();
   _db = null;
@@ -200,5 +265,7 @@ module.exports = {
   prunableTasks,
   counts,
   agentCounts,
+  billedAgentCounts,
+  kindCounts,
   close,
 };