fetch2gmail 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,25 @@
1
+ MIT License (Non-Commercial)
2
+
3
+ Copyright (c) 2025 Fetch2Gmail Contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, and/or sublicense the
9
+ Software, and to permit persons to whom the Software is furnished to do so,
10
+ subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ The Software may not be sold or otherwise commercialized. It may not be used
16
+ as part of a paid product or service without explicit written permission from
17
+ the copyright holder.
18
+
19
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
20
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
21
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
22
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
23
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
24
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
25
+ SOFTWARE.
@@ -0,0 +1,660 @@
1
+ Metadata-Version: 2.4
2
+ Name: fetch2gmail
3
+ Version: 1.0.0
4
+ Summary: Self-hosted email fetcher: IMAP to Gmail API import with idempotent state tracking
5
+ Author: Fetch2Gmail Contributors
6
+ License: MIT
7
+ Keywords: email,imap,gmail,fetch,self-hosted
8
+ Classifier: Development Status :: 4 - Beta
9
+ Classifier: Environment :: Console
10
+ Classifier: Intended Audience :: End Users/Desktop
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Operating System :: OS Independent
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Topic :: Communications :: Email
16
+ Requires-Python: >=3.11
17
+ Description-Content-Type: text/markdown
18
+ License-File: LICENSE
19
+ Requires-Dist: google-auth>=2.23.0
20
+ Requires-Dist: google-auth-oauthlib>=1.1.0
21
+ Requires-Dist: google-api-python-client>=2.100.0
22
+ Requires-Dist: fastapi>=0.104.0
23
+ Requires-Dist: uvicorn[standard]>=0.24.0
24
+ Requires-Dist: python-dotenv>=1.0.0
25
+ Requires-Dist: python-multipart>=0.0.6
26
+ Requires-Dist: cryptography>=42.0.0
27
+ Provides-Extra: dev
28
+ Requires-Dist: pytest>=7.0; extra == "dev"
29
+ Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
30
+ Dynamic: license-file
31
+
32
+ # Fetch2Gmail
33
+
34
+ Self-hosted email fetcher: **IMAP (ISP mailbox) → Gmail API import**. Replaces Gmail’s deprecated POP3 fetch. Runs on Debian (e.g. Odroid HC4) or any Linux/macOS with Python 3.11+.
35
+
36
+ - Polls an ISP mailbox via **IMAPS** (port 993).
37
+ - Imports messages with **Gmail API `users.messages.import`** (not SMTP).
38
+ - Applies a Gmail label (e.g. **"ISP Mail"**), preserves headers and date.
39
+ - **Deletes from ISP only after** Gmail confirms import (keeps limited ISP storage clear).
40
+ - **Idempotent**: tracks by IMAP UID and SHA256 message hash; safe across crashes and UIDVALIDITY changes.
41
+
42
+ **Two ways to get started:**
43
+
44
+ - **Headless server (e.g. Odroid, Raspberry Pi):** [Part 1](#part-1-get-the-oauth-token-on-a-computer-windows-or-linux) — get the OAuth token on a Windows or Linux computer; [Part 2](#part-2-install-on-the-server-odroid-raspberry-pi-etc-and-run-as-a-system-service) — install on the server, put config and token there, run as a system service.
45
+ - **All on one machine:** [Try locally first](#try-locally-first-step-by-step) — run everything (UI and fetch) on your laptop or desktop.
46
+
47
+ ## Requirements
48
+
49
+ - **Python 3.11+**
50
+ - IMAP credentials (ISP mailbox).
51
+ - Google Cloud project with Gmail API and OAuth2 credentials (refresh token after one-time consent).
52
+
53
+ ---
54
+
55
+ ## Part 1: Get the OAuth token on a computer (Windows or Linux)
56
+
57
+ Do this on a **laptop or desktop** that has a browser. You’ll get **credentials.json** (from Google) and **token.json** (from one-time sign-in). You’ll copy both to your server later. Google OAuth does not allow redirect URIs that use an IP address, so the token must be obtained on a machine where the app can use `http://127.0.0.1:8765`.
58
+
59
+ ### Step 1. Install Python 3.11+
60
+
61
+ - **Windows**: Download and install from [python.org](https://www.python.org/downloads/). Ensure “Add Python to PATH” is checked.
62
+ - **Linux**: e.g. `sudo apt install python3 python3-pip python3-venv` (Debian/Ubuntu).
63
+
64
+ ### Step 2. Install fetch2gmail
65
+
66
+ Either use PyPI (if the package is published) or clone the repo:
67
+
68
+ **Option A — from PyPI:**
69
+ ```bash
70
+ pip install fetch2gmail
71
+ ```
72
+ (On Linux you may prefer `pip install --user fetch2gmail` so you don’t need admin.)
73
+
74
+ **Option B — from source:**
75
+ ```bash
76
+ git clone https://github.com/yourusername/fetch2gmail.git
77
+ cd fetch2gmail
78
+ python3 -m venv .venv
79
+ # Linux/macOS:
80
+ source .venv/bin/activate
81
+ # Windows (PowerShell):
82
+ # .venv\Scripts\Activate.ps1
83
+ # Windows (Command Prompt):
84
+ # .venv\Scripts\activate.bat
85
+ pip install -e .
86
+ ```
87
+
88
+ ### Step 3. Create Google OAuth credentials (Web application)
89
+
90
+ 1. Go to [Google Cloud Console](https://console.cloud.google.com/) → create or select a project.
91
+ 2. Enable **Gmail API**: APIs & Services → Library → search “Gmail API” → Enable.
92
+ 3. **OAuth consent screen**: APIs & Services → OAuth consent screen → External → add app name, add scope `https://www.googleapis.com/auth/gmail.modify`, add yourself as **Test user**.
93
+ 4. **Credentials**: APIs & Services → Credentials → Create credentials → **OAuth client ID** → Application type **Web application**.
94
+ 5. Under **Authorized redirect URIs** add: `http://127.0.0.1:8765/auth/gmail/callback` and `http://localhost:8765/auth/gmail/callback`.
95
+ 6. Create → download the JSON. Save it as **credentials.json** in a folder you’ll use for the next step (e.g. your desktop or `~/fetch2gmail-auth`).
96
+
97
+ ### Step 4. Run the auth command to get token.json
98
+
99
+ Open a terminal in the **same folder** where you saved **credentials.json**. Then run:
100
+
101
+ ```bash
102
+ fetch2gmail auth
103
+ ```
104
+
105
+ (If you installed from source with a venv, activate the venv first, then run `fetch2gmail auth`.)
106
+
107
+ - A browser will open at http://127.0.0.1:8765. Sign in with the **Gmail account that should receive the imported mail**.
108
+ - Click “Allow” when asked for permission.
109
+ - **token.json** is saved in that same folder. You can stop the auth server with **Ctrl+C**.
110
+
111
+ You can optionally specify paths:
112
+ `fetch2gmail auth --credentials /path/to/credentials.json --token /path/to/token.json`
113
+
114
+ ### Step 5. Keep these two files for the server
115
+
116
+ You now have:
117
+
118
+ - **credentials.json** — from Google Cloud (Step 3).
119
+ - **token.json** — from `fetch2gmail auth` (Step 4).
120
+
121
+ Copy both to your server and put them in the **data directory** where the app will run (see Part 2). Do not commit them to git; keep them private.
122
+
123
+ ---
124
+
125
+ ## Part 2: Install on the server (Odroid, Raspberry Pi, etc.) and run as a system service
126
+
127
+ Do this on the **headless server** (e.g. Odroid, Raspberry Pi) where fetch2gmail will run. You need **config.json**, **credentials.json**, and **token.json** in one directory; the app will create **state.db** and (if you use the UI) **.cookie_secret** there.
128
+
129
+ ### Step 1. Create a data directory
130
+
131
+ Pick one directory that will hold all config and secrets (e.g. `/opt/fetch2gmail` or `/home/odroid/fetch2gmail`). Create it and go there:
132
+
133
+ ```bash
134
+ sudo mkdir -p /opt/fetch2gmail
135
+ sudo chown "$USER" /opt/fetch2gmail
136
+ cd /opt/fetch2gmail
137
+ ```
138
+
139
+ (Replace `/opt/fetch2gmail` with your choice; use the same path in the steps below.)
140
+
141
+ ### Step 2. Put config and token files in that directory
142
+
143
+ - **config.json** — Create from the example in the repo: copy `config.example.json` to `config.json` and edit **imap** (host, username, mailbox) and **gmail** (label, `credentials_path`, `token_path`). Use paths relative to this directory, e.g. `credentials.json` and `token.json`.
144
+ - **credentials.json** — Copy from the computer where you ran Part 1 (Step 4).
145
+ - **token.json** — Copy from the same place.
146
+ - **.env** (optional) — If you don’t set the IMAP password in the systemd unit, create `.env` here with:
147
+ `IMAP_PASSWORD=your_imap_password`
148
+
149
+ So this directory should contain at least: **config.json**, **credentials.json**, **token.json**, and optionally **.env**.
150
+
151
+ ### Step 3. Install fetch2gmail on the server
152
+
153
+ **Option A — global install (simplest):**
154
+ ```bash
155
+ pip install fetch2gmail
156
+ # or, without sudo, so it’s per-user:
157
+ pip install --user fetch2gmail
158
+ ```
159
+ Use the binary path in systemd: `/usr/local/bin/fetch2gmail` or `$HOME/.local/bin/fetch2gmail` if you used `--user`.
160
+
161
+ **Option B — venv in the data directory (isolated):**
162
+ ```bash
163
+ cd /opt/fetch2gmail
164
+ python3 -m venv .venv
165
+ .venv/bin/pip install fetch2gmail
166
+ ```
167
+ Use in systemd: `ExecStart=/opt/fetch2gmail/.venv/bin/fetch2gmail run`.
168
+
169
+ ### Step 4. Install and edit the systemd units
170
+
171
+ Copy the timer and service into systemd:
172
+
173
+ ```bash
174
+ sudo cp /path/to/fetch2gmail/systemd/fetch2gmail.service /path/to/fetch2gmail/systemd/fetch2gmail.timer /etc/systemd/system/
175
+ ```
176
+
177
+ Edit the service (replace paths and user with yours):
178
+
179
+ ```bash
180
+ sudo systemctl edit --full fetch2gmail.service
181
+ ```
182
+
183
+ Set at least:
184
+
185
+ | Setting | Example |
186
+ |--------|---------|
187
+ | **User=** | `odroid` (user that owns the data directory) |
188
+ | **Group=** | `odroid` |
189
+ | **WorkingDirectory=** | `/opt/fetch2gmail` (your data directory) |
190
+ | **Environment=FETCH2GMAIL_CONFIG=** | `/opt/fetch2gmail/config.json` |
191
+ | **ExecStart=** | `/usr/local/bin/fetch2gmail run` (global) or `/opt/fetch2gmail/.venv/bin/fetch2gmail run` (venv) |
192
+ | **Environment=IMAP_PASSWORD=** | (optional) your IMAP password if you don’t use `.env` |
193
+
194
+ Save and exit.
195
+
196
+ ### Step 5. Enable and start the timer
197
+
198
+ ```bash
199
+ sudo systemctl daemon-reload
200
+ sudo systemctl enable fetch2gmail.timer
201
+ sudo systemctl start fetch2gmail.timer
202
+ ```
203
+
204
+ The timer runs the fetch every 5 minutes. To watch logs:
205
+
206
+ ```bash
207
+ journalctl -u fetch2gmail.service -f
208
+ ```
209
+
210
+ ### Step 6. (Optional) Run the web UI on the server
211
+
212
+ If you want the dashboard on the server (e.g. at http://192.168.1.38:8765), run the UI there with the same data directory:
213
+
214
+ ```bash
215
+ cd /opt/fetch2gmail
216
+ FETCH2GMAIL_CONFIG=/opt/fetch2gmail/config.json fetch2gmail serve --host 0.0.0.0
217
+ ```
218
+
219
+ You can run this in a separate systemd service or in a terminal. You don’t need to sign in with Google on the device; **token.json** is already there.
220
+
221
+ ---
222
+
223
+ ## Try locally first (step-by-step)
224
+
225
+ Follow these steps if you want to run fetch2gmail **on one machine** (laptop or desktop) with the web UI and manual or scheduled fetch. You’ll use a **virtual environment** so the project’s dependencies don’t touch your system Python.
226
+
227
+ *If you’re setting up a **headless server** (Odroid, Raspberry Pi, etc.) instead, use [Part 1](#part-1-get-the-oauth-token-on-a-computer-windows-or-linux) to get the token on a PC, then [Part 2](#part-2-install-on-the-server-odroid-raspberry-pi-etc-and-run-as-a-system-service) to install and run as a system service on the server.*
228
+
229
+ ### 1. Open a terminal
230
+
231
+ - **Linux / macOS**: Open “Terminal” (or any terminal app).
232
+ - **Windows**: Open “Command Prompt” or “PowerShell”, or use the terminal inside your editor.
233
+
234
+ ### 2. Go to the project folder
235
+
236
+ ```bash
237
+ cd /path/to/fetch2gmail
238
+ ```
239
+
240
+ Use the real path where you cloned or unpacked Fetch2Gmail (e.g. `cd ~/dev/fetch2gmail` or `cd C:\Users\You\fetch2gmail`).
241
+
242
+ ### 3. Create a virtual environment
243
+
244
+ A virtual environment is an isolated Python environment for this project.
245
+
246
+ ```bash
247
+ python3 -m venv .venv
248
+ ```
249
+
250
+ If that fails, try:
251
+
252
+ ```bash
253
+ python -m venv .venv
254
+ ```
255
+
256
+ You should see no errors. A folder named `.venv` will appear in the project.
257
+
258
+ ### 4. Activate the virtual environment
259
+
260
+ - **Linux / macOS**:
261
+ ```bash
262
+ source .venv/bin/activate
263
+ ```
264
+ - **Windows (Command Prompt)**:
265
+ ```cmd
266
+ .venv\Scripts\activate.bat
267
+ ```
268
+ - **Windows (PowerShell)**:
269
+ ```powershell
270
+ .venv\Scripts\Activate.ps1
271
+ ```
272
+
273
+ When it’s active, your prompt usually starts with `(.venv)`.
274
+
275
+ ### 5. Install the project
276
+
277
+ Still in the same terminal, with the venv active:
278
+
279
+ ```bash
280
+ pip install -e .
281
+ ```
282
+
283
+ Wait until it finishes. You should see “Successfully installed fetch2gmail…”.
284
+
285
+ ### 6. Create Google OAuth credentials (Web application)
286
+
287
+ Do this once (see [OAuth setup](#oauth-setup) below):
288
+
289
+ - Create a Google Cloud project, enable Gmail API, then set up the **OAuth consent screen** (add the Gmail scope and yourself as a **Test user** so you can sign in without publishing the app).
290
+ - Create **OAuth client ID** with application type **Web application** (not Desktop — only Web application has the redirect URI field).
291
+ - Add **Authorized redirect URIs**: **`http://127.0.0.1:8765/auth/gmail/callback`** and **`http://localhost:8765/auth/gmail/callback`** (so either URL works).
292
+ - Download the JSON and save it in the project folder as **`credentials.json`**.
293
+
294
+ ### 7. Start the web UI and finish setup there (recommended)
295
+
296
+ ```bash
297
+ fetch2gmail serve
298
+ ```
299
+
300
+ Open **http://127.0.0.1:8765** in your browser.
301
+ **If the app will run on a headless device** (e.g. Odroid): use **`fetch2gmail auth`** on a laptop/PC to get **token.json**, then copy **credentials.json** and **token.json** to the device — see [Headless or LAN-only (e.g. Odroid)](#headless-or-lan-only-eg-odroid).
302
+
303
+ - **If you see “Initial setup”**: enter your IMAP host, username, password, mailbox, and Gmail label, then click **Create config**. Your password is stored **encrypted** in a `.env` file next to the config (not plain text, not in the config file).
304
+ - **If you already have `config.json`**: you’ll see the dashboard. You can enter or change your IMAP password in the **Config** section and click **Save config** (it’s stored encrypted in `.env`).
305
+ - Click **Connect Gmail (OAuth)** to sign in with Google in the browser. After you allow access, you’re connected and don’t need to do it again.
306
+ - Use **Run fetch now** or **Dry run** to test.
307
+
308
+ So: sign in with Google first, then create config (or use the dashboard if config already exists).
309
+
310
+ ### 8. Or create config by hand (alternative to step 7)
311
+
312
+ Create your config file:
313
+
314
+ ```bash
315
+ cp config.example.json config.json
316
+ ```
317
+
318
+ Edit `config.json`: set **imap.host**, **imap.username**, **imap.mailbox**, **gmail.credentials_path**, **gmail.token_path**. Do not put your IMAP password in the file. Set it in the environment (next step) or later in the UI.
319
+
320
+ ### 9. Set your IMAP password (if not using the UI)
321
+
322
+ In the same terminal (venv still active):
323
+
324
+ - **Linux / macOS**:
325
+ ```bash
326
+ export IMAP_PASSWORD='your_actual_imap_password'
327
+ ```
328
+ - **Windows (Command Prompt)**:
329
+ ```cmd
330
+ set IMAP_PASSWORD=your_actual_imap_password
331
+ ```
332
+ - **Windows (PowerShell)**:
333
+ ```powershell
334
+ $env:IMAP_PASSWORD = 'your_actual_imap_password'
335
+ ```
336
+
337
+ Replace `your_actual_imap_password` with the real password. This only applies to that terminal session. (If you used the UI in step 7, you already set the password there and can skip this.)
338
+
339
+ ### 10. One-time Gmail sign-in (if you didn’t use “Connect Gmail” in the UI)
340
+
341
+ If you didn’t connect Gmail in the web UI, run a fetch once so the app can open a browser and get a refresh token:
342
+
343
+ ```bash
344
+ fetch2gmail run
345
+ ```
346
+
347
+ - A browser window should open.
348
+ - Sign in with the **Gmail account that should receive the imported mail**.
349
+ - Click “Allow” when asked for permission.
350
+ - After that, **`token.json`** is created. You won’t need to sign in again.
351
+
352
+ If you see “Environment variable IMAP_PASSWORD is not set”, set it (step 9) or set it in the UI and save config.
353
+
354
+ ### 11. Try a dry run (recommended)
355
+
356
+ A dry run connects to your ISP and would import mail, but **does not** send anything to Gmail and **does not** delete anything from the ISP:
357
+
358
+ ```bash
359
+ fetch2gmail run --dry-run
360
+ ```
361
+
362
+ Check the output for errors. If it lists “Would import …”, the connection and config are working.
363
+
364
+ ### 12. Run a real fetch
365
+
366
+ When you’re ready to actually import mail into Gmail:
367
+
368
+ ```bash
369
+ fetch2gmail run
370
+ ```
371
+
372
+ Messages are imported into Gmail with the label you set in `config.json` (e.g. “ISP Mail”), and only then deleted from the ISP mailbox.
373
+
374
+ ### 13. Use the web UI anytime
375
+
376
+ Whenever you want to change settings or trigger a fetch from the browser, start the UI (with venv active):
377
+
378
+ ```bash
379
+ fetch2gmail serve
380
+ ```
381
+
382
+ Open **http://127.0.0.1:8765**. You can change IMAP/Gmail settings (including password, stored encrypted in `.env`), connect Gmail, run fetch or dry run, and see recent logs. Stop the server with **Ctrl+C** when you’re done.
383
+
384
+ ---
385
+
386
+ ## Quick start (reference)
387
+
388
+ If you already use virtual environments and know the basics:
389
+
390
+ 1. **Clone and install** (with a venv):
391
+ ```bash
392
+ cd fetch2gmail
393
+ python3 -m venv .venv && source .venv/bin/activate # or .venv\Scripts\activate on Windows
394
+ pip install -e .
395
+ ```
396
+
397
+ 2. **Create Google OAuth credentials** (see [OAuth setup](#oauth-setup)).
398
+
399
+ 3. **Config**: `cp config.example.json config.json`, edit it, and set `IMAP_PASSWORD` in the environment.
400
+
401
+ 4. **One-time OAuth**: `fetch2gmail run` (browser opens; then `token.json` is saved).
402
+
403
+ 5. **Run**: `fetch2gmail run` or `fetch2gmail serve` for the UI at http://127.0.0.1:8765.
404
+
405
+ 6. **Dry-run**: `fetch2gmail run --dry-run`.
406
+
407
+ ---
408
+
409
+ ## OAuth setup
410
+
411
+ Use a **Web application** OAuth client (not Desktop) so you can set the redirect URI for the web UI’s “Connect Gmail” flow.
412
+
413
+ ### 1. Google Cloud project and Gmail API
414
+
415
+ 1. Go to [Google Cloud Console](https://console.cloud.google.com/).
416
+ 2. Create a project (or select one) → **APIs & Services** → **Library**.
417
+ 3. Search for **Gmail API** → **Enable**.
418
+
419
+ ### 2. OAuth consent screen (do this before creating credentials)
420
+
421
+ 1. **APIs & Services** → **OAuth consent screen**.
422
+ 2. Choose **External** (so you can use your personal Gmail). Click **Create**.
423
+ 3. **App information**: fill App name, User support email, Developer contact. Save.
424
+ 4. **Scopes** (Data access): click **Add or remove scopes**. Search for “Gmail API” and add:
425
+ - **`https://www.googleapis.com/auth/gmail.modify`**
426
+ (View and modify but not delete your email.)
427
+ Save.
428
+ 5. **Test users** (so you can sign in without publishing the app):
429
+ - Under **Test users**, click **Add users**.
430
+ - Add the Gmail address that will receive the imported mail (e.g. yourself).
431
+ - Only these addresses can sign in while the app is in **Testing**.
432
+ 6. Leave the app in **Testing** — you do **not** need to publish it. Up to 100 test users can sign in.
433
+
434
+ ### 3. OAuth client (Web application) and redirect URI
435
+
436
+ 1. **APIs & Services** → **Credentials** → **Create credentials** → **OAuth client ID**.
437
+ 2. **Application type**: choose **Web application** (not Desktop).
438
+ (Desktop apps don’t show a redirect URI field; the web UI needs a fixed callback URL.)
439
+ 3. **Name**: e.g. “Fetch2Gmail”.
440
+ 4. **Authorized redirect URIs** → **Add URI** and add:
441
+ - **`http://127.0.0.1:8765/auth/gmail/callback`**
442
+ - **`http://localhost:8765/auth/gmail/callback`**
443
+ Google does **not** allow redirect URIs that use an IP address (e.g. `http://192.168.1.38:8765/...`). For a **headless or LAN-only** device (e.g. Odroid) that you access at `http://<ip>:8765`, see [Headless or LAN-only (e.g. Odroid)](#headless-or-lan-only-eg-odroid) below: you do the one-time Gmail sign-in on a computer with a browser, then copy `token.json` (and `credentials.json`) to the device.
444
+ 5. Click **Create**.
445
+ 6. Download the JSON (click the download icon for the new client) and save it as **`credentials.json`** in your project folder (same place as `config.json`).
446
+
447
+ ### 4. Refresh token (one-time)
448
+
449
+ Either use the web UI (“Connect Gmail”) or the CLI:
450
+
451
+ - **Web UI**: Put `credentials.json` in the project folder, add the redirect URI above, then run `fetch2gmail serve`, open http://127.0.0.1:8765, and click **Connect Gmail (OAuth)**. Sign in with the Gmail account you added as a test user; after you allow access, `token.json` is saved.
452
+ - **CLI**: Put `credentials.json` in place and set paths in `config.json`. Run `fetch2gmail run`; a browser opens → sign in (with a test user) → allow access → `token.json` is written.
453
+
454
+ Keep `token.json` and `credentials.json` **private** (do not commit; they are in `.gitignore`).
455
+
456
+ ---
457
+
458
+ ## Configuration
459
+
460
+ - **config.json** (see `config.example.json`):
461
+ - **imap**: `host`, `port` (993), `username`, `password_env` (e.g. `IMAP_PASSWORD`), `mailbox`, `use_ssl: true`.
462
+ - **gmail**: `label`, `credentials_path`, `token_path`.
463
+ - **state**: `db_path` (SQLite path).
464
+ - **ui**: `host`, `port` for the web UI.
465
+ - **poll_interval_minutes**: used by the UI/documentation; actual polling is via systemd timer or manual/UI trigger.
466
+
467
+ - **Secrets**: Put IMAP password in environment (e.g. `IMAP_PASSWORD`) or set it in the UI (stored encrypted in `.env`). Do not put OAuth tokens or passwords in config.
468
+
469
+ ---
470
+
471
+ ## Deployment
472
+
473
+ ### Where to put config and secrets on the server
474
+
475
+ Use **one directory** as your app data directory (e.g. `/opt/fetch2gmail` or `/home/odroid/fetch2gmail`). Put everything there:
476
+
477
+ | File | Purpose |
478
+ |------|--------|
479
+ | **config.json** | IMAP/Gmail settings (paths below are relative to this file’s directory) |
480
+ | **credentials.json** | Google OAuth client (from GCP) |
481
+ | **token.json** | Gmail refresh token (from `fetch2gmail auth` on a machine with a browser) |
482
+ | **.env** | Optional: `IMAP_PASSWORD=...` or set via systemd `Environment=` |
483
+
484
+ When the app runs, it will create in that same directory:
485
+
486
+ - **state.db** — SQLite state (last UID, message hashes)
487
+ - **.cookie_secret** — Web UI session signing (if you use `fetch2gmail serve`)
488
+
489
+ So: **same folder as `config.json`** is the single place for config, credentials, token, and generated state. Run the app with that directory as the **working directory** and set **`FETCH2GMAIL_CONFIG`** to the full path to `config.json` (e.g. `/opt/fetch2gmail/config.json`).
490
+
491
+ ### Install on the server: venv vs global
492
+
493
+ - **Global (simplest):** `pip install fetch2gmail` (or `pip install --user fetch2gmail`). Then run `fetch2gmail run` or `fetch2gmail serve` from your data directory, or point systemd at the global binary (e.g. `/usr/local/bin/fetch2gmail`).
494
+ - **Venv (isolated):** Create a venv inside your data directory so the app and its deps don’t touch system Python:
495
+ ```bash
496
+ mkdir -p /opt/fetch2gmail && cd /opt/fetch2gmail
497
+ python3 -m venv .venv
498
+ .venv/bin/pip install fetch2gmail
499
+ ```
500
+ Put `config.json`, `credentials.json`, `token.json` (and optionally `.env`) in `/opt/fetch2gmail`. Run with `/opt/fetch2gmail/.venv/bin/fetch2gmail run` and set `WorkingDirectory=/opt/fetch2gmail` in systemd.
501
+
502
+ Either way, **WorkingDirectory** must be that data directory so the app finds config and writes `state.db` and `.cookie_secret` there.
503
+
504
+ ### systemd (Debian / Odroid)
505
+
506
+ - **Service**: oneshot run per cycle.
507
+ - **Timer**: every 5 minutes.
508
+
509
+ 1. Copy units and edit the service for your paths and user:
510
+ ```bash
511
+ sudo cp systemd/fetch2gmail.service systemd/fetch2gmail.timer /etc/systemd/system/
512
+ sudo systemctl edit --full fetch2gmail.service
513
+ ```
514
+ Set:
515
+ - **User=** and **Group=** — user that owns the data directory (e.g. `odroid`).
516
+ - **WorkingDirectory=** — your data directory (e.g. `/opt/fetch2gmail` or `/home/odroid/fetch2gmail`).
517
+ - **Environment=FETCH2GMAIL_CONFIG=** — full path to `config.json` (e.g. `/opt/fetch2gmail/config.json`).
518
+ - **ExecStart=** — path to `fetch2gmail run`:
519
+ - Global install: `ExecStart=/usr/local/bin/fetch2gmail run` (or `ExecStart=/home/odroid/.local/bin/fetch2gmail run` if you used `pip install --user`).
520
+ - Venv in data dir: `ExecStart=/opt/fetch2gmail/.venv/bin/fetch2gmail run`.
521
+ - Optionally **Environment=IMAP_PASSWORD=** if you don’t use `.env`.
522
+
523
+ 2. Reload, enable and start the timer:
524
+ ```bash
525
+ sudo systemctl daemon-reload
526
+ sudo systemctl enable fetch2gmail.timer
527
+ sudo systemctl start fetch2gmail.timer
528
+ ```
529
+
530
+ Logs go to the **systemd journal**:
531
+ ```bash
532
+ journalctl -u fetch2gmail.service -f
533
+ ```
534
+
535
+ See **systemd/README.md** for instance units (e.g. one service per user).
536
+
537
+ ### Headless or LAN-only (e.g. Odroid)
538
+
539
+ Google OAuth **does not accept redirect URIs that use an IP address** (e.g. `http://192.168.1.38:8765/auth/gmail/callback`). So you cannot complete “Sign in with Google” when the only way to reach the app is via a LAN IP (e.g. **http://192.168.1.38:8765** on a headless Odroid).
540
+
541
+ **Use the `auth` command on a machine with a browser (like rclone’s authorize flow):**
542
+
543
+ 1. **On any Linux or Windows machine** (laptop, desktop) where you can open a browser:
544
+ - **Get the app**: clone the repo and install (e.g. **`git clone <repo-url> && cd fetch2gmail && pip install .`**). If the package is available on PyPI, **`pip install fetch2gmail`** instead.
545
+ - In GCP, create an OAuth **Web application** client and add **only** **`http://127.0.0.1:8765/auth/gmail/callback`** (and optionally `http://localhost:8765/auth/gmail/callback`). Download the JSON and save as **credentials.json** in a folder (e.g. your desktop or home).
546
+ - In that folder, run: **`fetch2gmail auth`**
547
+ A browser will open at http://127.0.0.1:8765. Sign in with Google; when done, **token.json** is saved in the same folder. Press Ctrl+C to stop the auth server.
548
+ - Optional: **`fetch2gmail auth --credentials /path/to/credentials.json --token /path/to/token.json`** to choose paths.
549
+ 2. **Copy to the Odroid** (or other headless device):
550
+ - **credentials.json**
551
+ - **token.json**
552
+ Put them in the **data directory** where the app runs (same folder as `config.json`; see [Where to put config and secrets on the server](#where-to-put-config-and-secrets-on-the-server)).
553
+ 3. **On the Odroid**, run Fetch2Gmail (e.g. systemd timer for fetch, and optionally `fetch2gmail serve` for the UI). Open the UI at **http://192.168.1.38:8765** (if the UI is bound to that host). Use the dashboard (config, fetch, logs) as usual. No “Sign in with Google” on the device; **token.json** is used for Gmail. If you ever click “Reconnect Gmail”, run **`fetch2gmail auth`** again on the laptop and copy **token.json** back.
554
+
555
+ So: **Get Fetch2Gmail on a laptop/PC (clone and pip install, or pip install from PyPI if available), run `fetch2gmail auth`, then copy the two files to the headless device.**
556
+
557
+ ---
558
+
559
+ ## Web UI and CLI
560
+
561
+ - **Web UI** (`fetch2gmail serve`): localhost only. **OAuth only** (no username/password). Flow:
562
+ 1. **Add credentials.json first**: Get it from Google Cloud (OAuth client, Web application). If you open the UI without it, you’ll see a message asking you to add **credentials.json** to the app folder, then refresh.
563
+ 2. **Sign in with Google**: Once credentials exist, opening the UI sends you to **Sign in with Google**. That **one sign-in** both logs you into the app and connects your Gmail account (saves `token.json`). No second step.
564
+ 3. **Configure ISP email**: After sign-in, if you don’t have a config yet you’ll see the **Configure your ISP email** form (IMAP host, username, password, mailbox, Gmail label). Create config, then run fetch or dry run.
565
+ - If you already have **config.json**, after sign-in you see the dashboard. Use **Reconnect Gmail** only to switch to a different Google account.
566
+ - **No database for auth**: UI session is a signed cookie; `.cookie_secret` stores the signing secret.
567
+ - **Redirect URI**: In your Google Cloud OAuth client, add both `http://127.0.0.1:8765/auth/gmail/callback` and `http://localhost:8765/auth/gmail/callback`.
568
+ - **CLI**:
569
+ - `fetch2gmail run` — one fetch cycle.
570
+ - `fetch2gmail run --dry-run` — fetch from ISP only, no import/delete.
571
+ - **`fetch2gmail auth`** — get **token.json** on a machine with a browser (for headless setup). Opens http://127.0.0.1:8765, you sign in with Google, token is saved; then copy **credentials.json** and **token.json** to the Odroid.
572
+ - `fetch2gmail config --init` — create `config.json` from template.
573
+ - `fetch2gmail config --validate` — validate config.
574
+ - `fetch2gmail wizard` — interactive config wizard.
575
+
576
+ ---
577
+
578
+ ## Switching Gmail account and multiple accounts
579
+
580
+ ### Signing in with a different Google account
581
+
582
+ Each config has **one** `token.json` (path set in `config.json`). If you already have a Gmail account connected and you **Reconnect Gmail** (or run OAuth again), the app **overwrites** that token with the new account. All future fetches will go to the **new** account; the previous account is no longer used.
583
+
584
+ - The UI shows **Connected as you@gmail.com** and, when you click **Reconnect Gmail (switch account)**, asks for confirmation before starting OAuth.
585
+ - **State** (last UID, message hashes) is stored per config directory, not per Gmail account. So after switching, the same IMAP mailbox is still “resumed” from the same UID; messages are imported into the new Gmail account. If you switch back later, you’d need to run OAuth again and the old account would receive only **new** messages (from the current UID onward).
586
+
587
+ ### Multiple Gmail accounts or multiple ISP mailboxes
588
+
589
+ The app is **one config = one IMAP mailbox → one Gmail account**. To use multiple combinations (e.g. ISP1 → Gmail A, ISP2 → Gmail B):
590
+
591
+ - **Run multiple instances**, each with its own directory and config:
592
+ - Directory 1: `config.json` (IMAP for ISP1, `token_path`: `token_a.json`), `credentials.json`, `token_a.json`, `state.db`.
593
+ - Directory 2: `config.json` (IMAP for ISP2, `token_path`: `token_b.json`), same or different `credentials.json`, `token_b.json`, `state.db`.
594
+ - Use **different config file paths** (e.g. `FETCH2GMAIL_CONFIG=/path/to/config_a.json` and `FETCH2GMAIL_CONFIG=/path/to/config_b.json`) and run two systemd services/timers.
595
+ - You can use the **same** Google Cloud OAuth client and `credentials.json` for all; each instance has its own `token_*.json` so each can be connected to a different Google account.
596
+
597
+ ---
598
+
599
+ ## Idempotency and safety
600
+
601
+ - **UID + UIDVALIDITY**: State is stored per mailbox and per IMAP `UIDVALIDITY`. If the server resets UIDs (new UIDVALIDITY), we do not reuse old `last_processed_uid`; we still avoid duplicates via hashes.
602
+ - **Message hash**: Before import, we compute **SHA256(raw message)** and store it. If a message is seen again (same or different UID after reset), we skip import and can still delete from ISP to free space.
603
+ - **Order**: For each message: (1) fetch, (2) check hash → skip if already imported, (3) import to Gmail, (4) record hash + UID → Gmail ID in DB, (5) update `last_processed_uid`, (6) delete from ISP and expunge. **We only delete after** a successful Gmail import (or after confirming duplicate by hash).
604
+ - **Crashes**: If the process dies after import but before delete, the next run will see the same UID again; the hash is already in the DB, so we skip import and can delete from ISP. No duplicate in Gmail.
605
+ - **Network/API failures**: On Gmail API failure we do not update state and do not delete; the same message will be retried next run. Exponential backoff is used for transient API errors.
606
+
607
+ ---
608
+
609
+ ## Security considerations
610
+
611
+ - **Secrets**: Store IMAP password in environment variables or set it in the UI (stored encrypted in `.env` using the same key as session cookies). Never commit `config.json` with passwords, or `credentials.json` / `token.json`.
612
+ - **Web UI**: Bind to **127.0.0.1** only so the UI is not exposed on the network.
613
+ - **Gmail scope**: Only `gmail.modify` is requested (read and modify labels/messages); no send or full account access.
614
+ - **Files**: Restrict permissions on `config.json`, `token.json`, `credentials.json`, `.cookie_secret`, and `state.db` to the user running the service.
615
+
616
+ ---
617
+
618
+ ## Project layout
619
+
620
+ ```
621
+ fetch2gmail/
622
+ ├── src/fetcher/
623
+ │ ├── __init__.py
624
+ │ ├── cli.py # CLI entrypoint
625
+ │ ├── config.py # Config load
626
+ │ ├── gmail_client.py # Gmail API import, backoff
627
+ │ ├── imap_client.py # IMAPS fetch, delete
628
+ │ ├── log_buffer.py # In-memory logs for UI
629
+ │ ├── run.py # Main run loop, dry-run
630
+ │ ├── state.py # SQLite state (UID, hash)
631
+ │ └── web_ui.py # FastAPI UI
632
+ ├── systemd/
633
+ │ ├── fetch2gmail.service
634
+ │ ├── fetch2gmail.timer
635
+ │ └── README.md
636
+ ├── config.example.json
637
+ ├── pyproject.toml
638
+ ├── requirements.txt
639
+ ├── README.md
640
+ ├── LICENSE
641
+ └── .gitignore
642
+ ```
643
+
644
+ ---
645
+
646
+ ## Fork and run on your own Debian / Odroid
647
+
648
+ 1. Clone: `git clone https://github.com/yourusername/fetch2gmail.git && cd fetch2gmail`
649
+ 2. Install: create a venv and `pip install -e .` (or from PyPI: `pip install fetch2gmail`).
650
+ 3. Create Google Cloud project, enable Gmail API, create OAuth **Web application** credentials → save as `credentials.json` in your data directory.
651
+ 4. Run once to get refresh token: from the directory that has `config.json`, run `fetch2gmail run` (browser opens for sign-in; then `token.json` is created there). Or use `fetch2gmail auth` on a laptop and copy `credentials.json` and `token.json` to the server (see [Headless or LAN-only](#headless-or-lan-only-eg-odroid)).
652
+ 5. Copy and edit systemd units from `systemd/`; set `User`, `Group`, `WorkingDirectory` (your data directory), `FETCH2GMAIL_CONFIG`, and `ExecStart` (path to `fetch2gmail run`). See [Deployment](#deployment).
653
+ 6. Enable timer: `sudo systemctl enable fetch2gmail.timer && sudo systemctl start fetch2gmail.timer`
654
+ 7. Optional: run the web UI with `fetch2gmail serve` (e.g. via SSH tunnel) to change settings and trigger fetches.
655
+
656
+ ---
657
+
658
+ ## License
659
+
660
+ MIT. See **LICENSE**.