mailshift 1.0.0__tar.gz

mailshift-1.0.0/.claudeignore

@@ -0,0 +1,70 @@
+# Claude ignore: exclude common build, cache, env, logs, and sensitive files
+
+# Node
+node_modules/
+npm-debug.log*
+yarn-debug.log*
+pnpm-debug.log*
+
+# Python virtual environments and caches
+.venv/
+venv/
+env/
+ENV/
+__pycache__/
+*.py[cod]
+*$py.class
+.pytest_cache/
+.mypy_cache/
+
+# Build and packaging
+build/
+dist/
+*.egg-info/
+.eggs/
+pip-wheel-metadata/
+target/
+src-tauri/target/
+
+# Logs, databases and runtime files
+logs/
+logs/**
+*.log
+mailshift.db
+*.sqlite
+*.sqlite3
+*.db
+
+# Coverage and test artifacts
+.coverage
+coverage.xml
+htmlcov/
+
+# Editor/IDE directories
+.vscode/
+.idea/
+*.sublime-workspace
+*.sublime-project
+
+# OS and temp files
+.DS_Store
+Thumbs.db
+tmp/
+temp/
+*.tmp
+*.bak
+*.swp
+
+# Notebooks
+.ipynb_checkpoints/
+
+# Secrets and local env
+.env
+.env.*
+secrets/
+.secrets/
+local_settings.py
+
+# Misc caches and tools
+.cache/
+.tox/

mailshift-1.0.0/.gitignore

@@ -0,0 +1,35 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+*.egg-info/
+dist/
+build/
+.eggs/
+*.db
+*.log
+*.txt
+credentials.json
+worker_profiles.json
+power_user_settings.json
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Pytest
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Editors
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Logs
+logs/

mailshift-1.0.0/AGENTS.md

@@ -0,0 +1,71 @@
+# AGENTS.md
+
+## Must-follow constraints
+
+- Decision strings are **Turkish**: `"SIL"` (delete) and `"TUT"` (keep). Never use English equivalents anywhere in `ScanResult.decision`.
+- `dry_run=True` is the default. Actual deletion only happens with `--no-dry-run`. Never change this default.
+- On any LLM or analysis error, fall back to `"TUT"` — never `"SIL"`. Safety-first.
+- Emails with `has_attachment=True` must always return `"TUT"` (enforced in `fast_analyzer.py`). Do not remove this guard.
+- Only update `AGENTS.md` and `README.md` if the changes introduce new conventions, logic, or significant architectural shifts. Minor bugfixes or trivial cleanups do not require documentation updates.
+- `IMAPConfig.password` Pydantic kütüphanesi ile `SecretStr` olarak tanımlanmıştır. Değerine erişmek için her zaman `.get_secret_value()` kullanılmalıdır, doğrudan öznitelik erişimi (`.password`) yasaktır. Bu sayede parola loglarda veya hata çıktılarında açık şekilde görünmez.
+
+## Validation before finishing
+
+```
+py -3.14 -m pytest tests/
+```
+
+No other build or lint step is configured.
+
+- Gmail delete/move-to-trash behavior is covered by `tests/test_google_delete_trash.py`.
+
+## Repo-specific conventions
+
+- **Python interpreter**: `py -3.14` — not `python`, not `python3`.
+- **Keyword lists**: `whitelist.json` and `blacklist.json` are plain JSON arrays at the project root. `JUNK_PATTERN` and `WHITELIST_PATTERN` are compiled once at import time in `config.py`. Runtime mutations via `add_to_blacklist()` / `add_to_whitelist()` update the JSON file but do **not** update the in-memory regex — a process restart is required for changes to take effect in analysis.
+- **Fast analyzer safety guards**: `fast_analyzer.py` has a built-in keep-guard that forces `TUT` for premium lifecycle expiry notices, verification-code/OTP mails, and Drive/cloud storage fullness alerts before junk matching. Keep this guard intact unless there is a stronger safety replacement.
+- **Fast analyzer order**: Decision flow is `has_attachment -> whitelist(TUT) -> safety-guard(TUT) -> blacklist(SIL) -> no match(TUT)`. Preserve this order unless a requirement explicitly changes it.
+- **Fast analyzer text split**: Whitelist and safety-guard checks run against `full_text` (subject + sender + body_preview). Blacklist matching runs against `content_text` (subject + body_preview only, **excluding sender**) to prevent false positives from legitimate automated senders such as `no-reply@github.com` matching the `"no-reply"` rule.
+- **Turkish case normalization**: `fast_analyzer._normalize()` replaces `İ` (U+0130) with `i` then lowercases before matching. Both `full_text` and `content_text` pass through this function, so all regex matches return lowercase tokens in `ScanResult.reason`.
+- **`analyzer.py`** is a re-export shim only. Do not add logic there.
+- **Console output**: Use `from ui import console` (Rich console) for any new terminal output. Do not use bare `print()` — stdout is wrapped for UTF-8 on Windows and `print()` bypasses Rich's rendering.
+- **Progress UI stability**: Keep per-item progress labels short/sanitized (single-line, no control chars) and prefer ASCII status tags (`SIL`/`TUT`) over emoji to avoid wrapped/duplicated-looking progress bars in narrow Windows terminals.
+- **Logger stream**: Keep the console logger on `sys.stderr` so Rich progress output on stdout is not visually disrupted by warning/error logs.
+- **No YAML**: The project uses JSON throughout. `requirements.txt` does not include pyyaml.
+
+## Important locations
+
+- `mailshift.db` — SQLite cache, created at **CWD** (not a fixed path). Contains `mails_cache` (header/body cache) and `fetch_checkpoint` (resume table). Persists across runs. Delete it or call `clear_mails_cache()` to force a fresh scan.
+- `whitelist.json` / `blacklist.json` — project root, not in a subdirectory.
+- **OS Keyring** (Windows Credential Manager) — Kimlik bilgileri `keyring` kütüphanesi ile OS düzeyinde şifreli olarak saklanır. Proje dizininde düz metin / JSON şifre dosyası tutulmaz.
+
+## Change safety rules
+
+- Pro mode is **two-phase**: heuristic runs first; LLM is called only for `"SIL"` candidates. Any change to fast analysis directly affects what the LLM sees.
+- Proton Mail requires Bridge running locally at `127.0.0.1:1143` (no SSL). Gmail is `imap.gmail.com:993` (SSL). These are set via `PROVIDER_DEFAULTS` in `config.py`.
+- `delete_mails()` and `move_to_trash()` both call `self._conn.expunge()` after flagging. Removing expunge silently leaves messages flagged but not deleted.
+- Delete/trash chunk operations and expunge are retry-protected; on socket/SSL EOF-style errors MailEngine reconnects and re-selects `INBOX` before the next retry.
+- `fetch_headers_concurrent` is sequential at the IMAP level (single connection, chunked). The name is misleading — do not add actual per-thread IMAP connections without careful locking.
+- `AppConfig`, `IMAPConfig`, and `OllamaConfig` are frozen Pydantic models. Do not attempt in-place mutation.
+- `hardware.py` detects NVIDIA first, then Intel/AMD GPUs on Windows via `Win32_VideoController` (PowerShell CIM query). Shared VRAM can be estimated from system RAM when `AdapterRAM` is not usable.
+
+## Known gotchas
+
+- `JUNK_PATTERN` / `WHITELIST_PATTERN` are `None` if the respective JSON file is empty — always guard with `if JUNK_PATTERN:` before calling `.search()`.
+- `ScanStats.errors` uses a mutable default in a dataclass — it is initialised via `__post_init__`. Do not set it as a class-level default list.
+- `database.py` resolves `DB_FILE = Path("mailshift.db")` relative to CWD at import time. If tests change directory, the DB lands in the wrong place.
+- `list_uids()` returns UIDs in **reverse order** (newest first) due to `[::-1]` before applying `scan_limit`. Ordering assumptions matter when reasoning about which emails are skipped by `scan_limit`.
+- `pro_analyzer.py` decision parsing accepts plain-text and JSON-like LLM outputs, and normalizes Turkish dotted/dotless `i` so `SİL/SIL/sıl` are treated consistently. Keep the safety fallback to `"TUT"` when no valid decision is found.
+- `pro_analyzer.py` uses Ollama `/api/chat` with a JSON decision schema (`decision: SIL|TUT`) and reads `message.content` first (then `response` fallback). This avoids empty-output failures seen with some small models on `/api/generate`.
+- `pro_analyzer.py` sends `think: false` and a higher `num_predict` budget for `qwen3.5:2B/4B` so the model does not spend output tokens on hidden reasoning and leave `message.content` empty.
+- **LM Studio Offloading**: Pro mode attempts to unload models from LM Studio VRAM immediately after analysis completion via `POST /api/v1/models/unload` to free up system resources.
+- **Ollama Visibility**: Ollama on Windows often runs without a visible window or system tray icon. To completely terminate it, users must use the Task Manager.
+- **LLM Provider Selection**: Pro mode now supports two backends — Ollama (default, recommended NVIDIA GPU users, "Easy Setup") and LM Studio (recommended all users). Selected via `prompt_llm_backend()` in `cli_utils.py`. `prompt_mode()` returns `(Mode, model_name, llm_backend)`.
+- **Ollama launch helpers**: `_launch_ollama_process()`, `stop_ollama()`, and `_is_ollama_running()` in `cli_utils.py` are reusable primitives used by `start_ollama()`. `stop_ollama()` uses `taskkill /F /IM ollama.exe` on Windows and `pkill -f ollama` on other platforms.
+- **Post-install UX**: When Ollama is newly installed from interactive Pro-mode flow, MailShift prints a "next steps" panel (restart terminal, `ollama serve`, rerun Pro mode). Keep this guidance visible on fallback-to-fast paths and do not require manual `ollama pull` there.
+- **Model selection UX**: Recommended model list includes `qwen3.5:0.8B` with `%95 Accurate` label, and missing recommended models are auto-downloaded/verified from the selection screen.
+- **Worker policy source-of-truth**: Resolve worker decisions via `resolve_worker_plan()` in `hardware.py` and pass the resolved value through runtime; `calculate_optimal_workers()` is kept as a compatibility wrapper.
+- **Manual worker safety clamp**: Manual worker values must be capped to the computed safe upper limit (backend/VRAM/RAM/CPU) and CLI should show a clear warning when clamping occurs.
+- **Fast mode worker UX**: Fast mode does not use worker-based parallel analysis; keep configuration output explicit as not-used to avoid user confusion.
+- **Worker profile learning cache**: Auto worker runs in Pro mode can persist learned recommendations in `worker_profiles.json` using hashed device signature + backend/model keys; warm-start applies only to auto plans and still obeys safe upper limits/manual override rules.
+- **Power-user worker probe**: `--power-worker-probe` enables a persisted hardware probe preference (`power_user_settings.json`) and can seed initial worker recommendation as `auto-probe` when no profile warm-start exists.

mailshift-1.0.0/CLAUDE.md

@@ -0,0 +1,70 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Run all tests
+py -3.14 -m pytest tests/
+
+# Run a single test file
+py -3.14 -m pytest tests/test_fast_analyzer.py
+
+# Run the app
+python main.py
+```
+
+**Python interpreter**: Always use `py -3.14`, not `python` or `python3`. No lint step is configured.
+
+## Architecture
+
+MailShift is a privacy-first, AI-assisted Gmail/Proton/IMAP mail cleaner. Entry point: `main.py` (root) → `src/mailshift/main.py` (Click CLI).
+
+**Data flow for a scan**:
+1. `main.py` + `ui/cli.py` — collect provider, credentials (stored in OS Keyring), scan mode, LLM backend
+2. `config/config.py` — assembles frozen Pydantic models (`AppConfig`, `IMAPConfig`, `OllamaConfig`/`LMStudioConfig`); `PROVIDER_DEFAULTS` maps provider names to IMAP hosts
+3. `core/engine.py` (`MailEngine`) — IMAP connect, `list_uids()` (returns newest-first), `fetch_headers_concurrent()` (chunked, single connection despite the name), SQLite checkpoint resume via `db/database.py`
+4. **Fast mode** (`core/analyzers/fast.py`): heuristic-only. Decision order: `has_attachment → whitelist(TUT) → safety-guard(TUT) → blacklist(SIL) → no-match(TUT)`
+5. **Pro mode** (`core/analyzers/pro.py`): two-phase — Fast runs first, then LLM (Ollama or LM Studio) is called only for `SIL` candidates
+6. `core/session.py` — `LLMWorker` (threaded) + Rich progress handlers
+7. `core/engine.py` — `delete_mails()` / `move_to_trash()`: chunk, flag `\Deleted`, expunge, retry on SSL/EOF
+8. `utils/history.py` — JSON logs under `logs/`, CSV/JSON export; `ui/styles.py` — Rich result tables
+9. **Unsubscribe flow** (`utils/unsubscribe.py`): after scan results are shown, `_prompt_unsubscribe()` in `main.py` collects all `SIL` results that carry a `List-Unsubscribe` HTTP URL, deduplicates by URL, and presents a 4-option menu (auto-all / select / export / skip). Triggered in both dry-run and live paths.
+
+**Key files**:
+- `blacklist.json` / `whitelist.json` — project root, compiled to regex at import time in `config.py`. `JUNK_PATTERN`/`WHITELIST_PATTERN` may be `None` if empty — always guard with `if pattern:` before `.search()`
+- `mailshift.db` — SQLite cache at CWD (not a fixed path); delete to force fresh scan
+- `utils/unsubscribe.py` — `UnsubscribeEntry`, `build_unsubscribe_entries()`, `perform_unsubscribe()`, `export_unsubscribe_links()`
+
+## Critical Conventions
+
+- **Decision strings are Turkish**: `"SIL"` (delete), `"TUT"` (keep). Never use English equivalents in `ScanResult.decision`.
+- **`dry_run=True` is the immutable default.** Actual deletion only occurs with `--no-dry-run`.
+- **Any LLM or analysis error must fall back to `"TUT"`**, never `"SIL"`. Safety-first.
+- **Emails with `has_attachment=True` always return `"TUT"`** — do not remove this guard in `fast_analyzer.py`.
+- **`IMAPConfig.password` is `SecretStr`** — access via `.get_secret_value()` only, never direct attribute.
+- **Fast analyzer text split**: whitelist + safety-guard check `full_text` (subject + sender + body); blacklist checks `content_text` (subject + body only, sender excluded) to prevent false positives from senders like `no-reply@github.com`.
+- **Turkish normalization**: `fast_analyzer._normalize()` replaces `İ` (U+0130) → `i` before lowercasing. All regex tokens in `ScanResult.reason` are lowercase.
+- **Console output**: Use `from ui import console` (Rich). Never use bare `print()` — it bypasses Rich's UTF-8 wrapper.
+- **Logger on `sys.stderr`**: keeps Rich progress on stdout visually clean.
+- **No YAML**: JSON throughout (`blacklist.json`, `whitelist.json`, logs).
+- `analyzer.py` is a re-export shim only — do not add logic there.
+- Pydantic models `AppConfig`, `IMAPConfig`, `OllamaConfig` are frozen — no in-place mutation.
+
+## Gotchas
+
+- `MailMeta.unsubscribe_url` is populated from the `List-Unsubscribe` header during `_fetch_mails_bulk()` using `_UNSUB_URL_RE` (HTTP-only; `mailto:` entries are silently ignored). The second body-fetch pass only updates `body_preview`, so the URL from the first header fetch is preserved.
+- `database.py` auto-migrates old DBs: `init_db()` runs `ALTER TABLE mails_cache ADD COLUMN unsubscribe_url TEXT DEFAULT ''` if the column is absent.
+- `perform_unsubscribe()` tries GET first, then falls back to an RFC 8058 one-click POST (`List-Unsubscribe=One-Click`). Never import `requests` for this — use stdlib `urllib.request` only.
+- `fetch_headers_concurrent` is sequential at the IMAP level (single connection). Do not add per-thread IMAP connections without careful locking.
+- Runtime mutations via `add_to_blacklist()` / `add_to_whitelist()` update JSON but **do not update the in-memory regex** — a process restart is needed.
+- `list_uids()` reverses UIDs (`[::-1]`) so newest-first; `scan_limit` skips oldest emails.
+- `database.py` resolves `DB_FILE` relative to CWD at import time — if tests change directory, the DB lands in the wrong place.
+- `ScanStats.errors` is initialized in `__post_init__`, not as a class-level default list.
+- Pro mode sends `think: false` and higher `num_predict` for `qwen3.5` models to avoid empty `message.content` from hidden reasoning tokens.
+- LM Studio: after analysis, Pro mode unloads the model from VRAM via `POST /api/v1/models/unload`.
+- Proton Mail requires Bridge at `127.0.0.1:1143` (no SSL); Gmail is `imap.gmail.com:993` (SSL).