imap-backup 6.3.0 → 7.0.0.rc1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +51 -161
- data/docs/configuration.md +23 -0
- data/docs/development.md +6 -0
- data/imap-backup.gemspec +3 -1
- data/lib/imap/backup/account/folder.rb +5 -4
- data/lib/imap/backup/cli/utils.rb +1 -1
- data/lib/imap/backup/configuration.rb +1 -1
- data/lib/imap/backup/downloader.rb +2 -1
- data/lib/imap/backup/migrator.rb +2 -2
- data/lib/imap/backup/serializer/appender.rb +6 -5
- data/lib/imap/backup/serializer/imap.rb +39 -14
- data/lib/imap/backup/serializer/mbox.rb +7 -0
- data/lib/imap/backup/serializer/mbox_enumerator.rb +9 -3
- data/lib/imap/backup/serializer/message_enumerator.rb +8 -9
- data/lib/imap/backup/serializer/version2_migrator.rb +90 -0
- data/lib/imap/backup/serializer.rb +18 -8
- data/lib/imap/backup/setup/asker.rb +0 -1
- data/lib/imap/backup/uploader.rb +4 -4
- data/lib/imap/backup/version.rb +3 -3
- metadata +35 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d858a14c5bbb9300fd99fe6338cc2ff83fd596071dc2db48ca792f45eee430a3
|
4
|
+
data.tar.gz: c28f18cd459cd9c2c9a62fb5a55234615218bcff2c3c46e0422af9c6893e6043
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c669d9a170ec1d500e898324bbdc2072cc42597e9c2d30b6b284b7095151afecd0451c9fa7cb0adb93bbf46f1f2b1ae188554507141161acb6ba125349c1cee9
|
7
|
+
data.tar.gz: c53ecb54ed6d33ab574caa18e62ae57e214e76975af65c72b11b8b2612d45487422be234c025966c792260d86d20e5431bf27845772e40d82cc8fea0132f9963
|
data/README.md
CHANGED
@@ -5,218 +5,108 @@
|
|
5
5
|
|
6
6
|
# imap-backup
|
7
7
|
|
8
|
-
|
8
|
+
Backup, restore and migrate email accounts.
|
9
|
+
|
10
|
+
The backups can then be restored, used to migrate to another service,
|
11
|
+
inspected or exported.
|
9
12
|
|
10
13
|
* [Source Code]
|
11
|
-
* [
|
14
|
+
* [Documentation]
|
12
15
|
* [Rubygem]
|
13
16
|
* [CI Status]
|
14
17
|
|
15
18
|
[Source Code]: https://github.com/joeyates/imap-backup "Source code at GitHub"
|
16
|
-
[
|
19
|
+
[Documentation]: https://rubydoc.info/gems/imap-backup/frames "RDoc API Documentation at Rubydoc.info"
|
17
20
|
[Rubygem]: https://rubygems.org/gems/imap-backup "Ruby gem at rubygems.org"
|
18
21
|
[CI Status]: https://github.com/joeyates/imap-backup/actions/workflows/main.yml
|
19
22
|
|
20
|
-
#
|
23
|
+
# Backup Emails
|
21
24
|
|
22
|
-
|
23
|
-
$ gem install 'imap-backup'
|
24
|
-
```
|
25
|
+
imap-backup downloads emails and stores them on disk.
|
25
26
|
|
26
|
-
|
27
|
+
The backup is incremental and interruptable, so backups won't get messed if your connection goes down during an operation.
|
27
28
|
|
28
|
-
|
29
|
+
# Installation
|
29
30
|
|
30
|
-
|
31
|
-
$ imap-backup help
|
32
|
-
```
|
31
|
+
## Homebrew (macOS)
|
33
32
|
|
34
|
-
|
33
|
+
If you have [Homebrew](https://brew.sh/), do this:
|
35
34
|
|
36
|
-
```
|
37
|
-
|
35
|
+
```sh
|
36
|
+
brew install imap-backup
|
38
37
|
```
|
39
38
|
|
40
|
-
|
41
|
-
|
42
|
-
In order to do backups, you need to add accounts via a menu-driven command
|
43
|
-
line program:
|
39
|
+
## As a Ruby Gem
|
44
40
|
|
45
|
-
|
46
|
-
|
47
|
-
```shell
|
48
|
-
$ imap-backup setup
|
41
|
+
```sh
|
42
|
+
gem install imap-backup
|
49
43
|
```
|
50
44
|
|
51
|
-
|
52
|
-
|
53
|
-
To use imap-backup with GMail, you will need to enable 'App passwords' on your account.
|
54
|
-
|
55
|
-
## Folders
|
56
|
-
|
57
|
-
By default, all folders are backed-up. You can override this by choosing
|
58
|
-
specific folders.
|
45
|
+
If that doesn't work, see the [detailed installation instructions](docs/installation/rubygem.md).
|
59
46
|
|
60
|
-
##
|
47
|
+
## From Source Code
|
61
48
|
|
62
|
-
|
49
|
+
If you want to use imap-backup directly from the source code, see [here](docs/installation/source.md).
|
63
50
|
|
64
|
-
|
51
|
+
# Setup
|
65
52
|
|
66
|
-
|
67
|
-
|
68
|
-
"accounts": [
|
69
|
-
{
|
70
|
-
"username": "my.user@gmail.com",
|
71
|
-
"password": "secret",
|
72
|
-
"local_path": "/path/to/backup/root",
|
73
|
-
"folders":
|
74
|
-
[
|
75
|
-
{"name": "[Gmail]/All Mail"},
|
76
|
-
{"name": "my_folder"}
|
77
|
-
]
|
78
|
-
}
|
79
|
-
]
|
80
|
-
}
|
81
|
-
```
|
53
|
+
As a first step, you need to add accounts via a menu-driven command
|
54
|
+
line program:
|
82
55
|
|
83
|
-
|
84
|
-
|
85
|
-
```json
|
86
|
-
{
|
87
|
-
"accounts": [
|
88
|
-
{
|
89
|
-
"username": "my.user@gmail.com",
|
90
|
-
"password": "secret",
|
91
|
-
"server": "my.imap.example.com",
|
92
|
-
"local_path": "/path/to/backup/root",
|
93
|
-
"folders":
|
94
|
-
[
|
95
|
-
{"name": "[Gmail]/All Mail"},
|
96
|
-
{"name": "my_folder"}
|
97
|
-
]
|
98
|
-
}
|
99
|
-
]
|
100
|
-
}
|
101
|
-
```
|
56
|
+
Run:
|
102
57
|
|
103
|
-
|
104
|
-
|
105
|
-
You can override the parameters passed to `Net::IMAP` with `connection_options`.
|
106
|
-
|
107
|
-
Specifically, if you are using a self-signed certificate and get SSL errors, e.g.
|
108
|
-
`certificate verify failed`, you can choose to not verify the TLS connection:
|
109
|
-
|
110
|
-
```json
|
111
|
-
{
|
112
|
-
"accounts": [
|
113
|
-
{
|
114
|
-
"username": "my.user@gmail.com",
|
115
|
-
"password": "secret",
|
116
|
-
"server": "my.imap.example.com",
|
117
|
-
"local_path": "/path/to/backup/root",
|
118
|
-
"folders": [
|
119
|
-
{"name": "[Gmail]/All Mail"},
|
120
|
-
{"name": "my_folder"}
|
121
|
-
],
|
122
|
-
"connection_options": {
|
123
|
-
"ssl": {"verify_mode": 0},
|
124
|
-
"port": 993
|
125
|
-
}
|
126
|
-
}
|
127
|
-
]
|
128
|
-
}
|
58
|
+
```sh
|
59
|
+
imap-backup setup
|
129
60
|
```
|
130
61
|
|
131
|
-
|
132
|
-
|
133
|
-
Note that email usernames and passwords are held in plain text
|
134
|
-
in the configuration file.
|
135
|
-
|
136
|
-
The directory ~/.imap-backup, the configuration file and all backup
|
137
|
-
directories have their access permissions set to only allow access
|
138
|
-
by your user. This is not done on Windows - see below.
|
139
|
-
|
140
|
-
## Windows
|
141
|
-
|
142
|
-
Due to the complexity of managing permissions on Windows,
|
143
|
-
directory and file access permissions are not set explicity.
|
62
|
+
## GMail
|
144
63
|
|
145
|
-
|
146
|
-
would be welcome!
|
64
|
+
To use imap-backup with GMail, you will need to enable 'App passwords' on your account.
|
147
65
|
|
148
66
|
# Run Backup
|
149
67
|
|
150
68
|
Manually, from the command line:
|
151
69
|
|
152
|
-
```
|
153
|
-
|
70
|
+
```sh
|
71
|
+
imap-backup
|
154
72
|
```
|
155
73
|
|
156
74
|
Alternatively, add it to your crontab.
|
157
75
|
|
158
|
-
|
76
|
+
Emails are stored on disk in [Mbox files](./docs/files/mboxrd.md) for each folder, with metadata
|
77
|
+
stored in [Imap files](./docs/files/imap.md).
|
159
78
|
|
160
|
-
|
161
|
-
Alongside each mbox is a file with extension '.imap', which lists the source IMAP
|
162
|
-
UIDs to allow a full restore.
|
163
|
-
|
164
|
-
# Local commands
|
79
|
+
# Commands
|
165
80
|
|
166
|
-
|
81
|
+
* [folders](./commands/folders.md)
|
82
|
+
* [restore](./commands/restore.md)
|
83
|
+
* [status](./commands/status.md)
|
167
84
|
|
168
|
-
|
85
|
+
For a full list of available commands, run
|
169
86
|
|
170
|
-
```
|
171
|
-
|
87
|
+
```sh
|
88
|
+
imap-backup help
|
172
89
|
```
|
173
90
|
|
174
|
-
|
175
|
-
|
176
|
-
If you have problems:
|
91
|
+
For more information about a command, run
|
177
92
|
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
```json
|
182
|
-
{
|
183
|
-
"accounts":
|
184
|
-
[
|
185
|
-
...
|
186
|
-
],
|
187
|
-
"debug": true
|
188
|
-
}
|
93
|
+
```sh
|
94
|
+
imap-backup help COMMAND
|
189
95
|
```
|
190
96
|
|
191
|
-
|
192
|
-
|
193
|
-
All missing messages are pushed to the IMAP server.
|
194
|
-
Existing messages are left unchanged.
|
97
|
+
## Configuration
|
195
98
|
|
196
|
-
|
197
|
-
extension to IMAP4.
|
99
|
+
`imap-backup setup` creates the file `~/.imap-backup/config.json`.
|
198
100
|
|
199
|
-
|
200
|
-
|
201
|
-
List IMAP folders:
|
202
|
-
|
203
|
-
```shell
|
204
|
-
$ imap-backup folders
|
205
|
-
```
|
101
|
+
[More information about configuration is available in the specific documentation](./docs/configuration.md).
|
206
102
|
|
207
|
-
|
208
|
-
|
209
|
-
```shell
|
210
|
-
$ imap-backup status
|
211
|
-
```
|
103
|
+
# Troubleshooting
|
212
104
|
|
213
|
-
|
105
|
+
If you have problems:
|
214
106
|
|
215
|
-
|
216
|
-
|
217
|
-
* Standalone - do not rely on an email client or MTA
|
107
|
+
1. ensure that you have the latest release,
|
108
|
+
2. turn on debugging output via the `imap-backup setup` main menu.
|
218
109
|
|
219
|
-
#
|
110
|
+
# Development
|
220
111
|
|
221
|
-
|
222
|
-
* [Restore](./docs/restore.md)
|
112
|
+
See the [Development documentation](./docs/development.md)
|
@@ -0,0 +1,23 @@
|
|
1
|
+
Configuration is stored in a JSON file.
|
2
|
+
|
3
|
+
The format is documented [here](./docs/files/config.json).
|
4
|
+
|
5
|
+
# Folders
|
6
|
+
|
7
|
+
By default, all folders are backed-up. You can override this by choosing
|
8
|
+
specific folders.
|
9
|
+
|
10
|
+
# Connection options
|
11
|
+
|
12
|
+
You can override the parameters passed to `Net::IMAP` with `connection_options`.
|
13
|
+
|
14
|
+
Specifically, if you are using a self-signed certificate and get SSL errors, e.g.
|
15
|
+
`certificate verify failed`, you can choose to not verify the TLS connection.
|
16
|
+
|
17
|
+
Connection options can be entered via `imap-backup setup` as JSON.
|
18
|
+
|
19
|
+
Choose the account, then 'modify connection options'.
|
20
|
+
|
21
|
+
For example:
|
22
|
+
|
23
|
+
![Entering connection options as JSON](./images/entering-connection-options-as-json.png "Entering connection options as JSON")
|
data/docs/development.md
CHANGED
data/imap-backup.gemspec
CHANGED
@@ -19,10 +19,12 @@ Gem::Specification.new do |gem|
|
|
19
19
|
|
20
20
|
gem.executables = gem.files.grep(%r{^bin/}).map { |f| File.basename(f) }
|
21
21
|
gem.require_paths = ["lib"]
|
22
|
-
gem.required_ruby_version = ">= 2.
|
22
|
+
gem.required_ruby_version = ">= 2.6"
|
23
23
|
|
24
24
|
gem.add_runtime_dependency "highline"
|
25
25
|
gem.add_runtime_dependency "mail"
|
26
|
+
gem.add_runtime_dependency "net-imap"
|
27
|
+
gem.add_runtime_dependency "net-smtp"
|
26
28
|
gem.add_runtime_dependency "os"
|
27
29
|
gem.add_runtime_dependency "rake"
|
28
30
|
gem.add_runtime_dependency "thor", "~> 1.1"
|
@@ -73,7 +73,7 @@ module Imap::Backup
|
|
73
73
|
examine
|
74
74
|
fetch_data_items =
|
75
75
|
retry_on_error(errors: UID_FETCH_RETRY_CLASSES) do
|
76
|
-
client.uid_fetch(uids, [BODY_ATTRIBUTE])
|
76
|
+
client.uid_fetch(uids, [BODY_ATTRIBUTE, "FLAGS"])
|
77
77
|
end
|
78
78
|
return nil if fetch_data_items.nil?
|
79
79
|
|
@@ -82,18 +82,19 @@ module Imap::Backup
|
|
82
82
|
|
83
83
|
{
|
84
84
|
uid: attributes["UID"],
|
85
|
-
body: attributes[BODY_ATTRIBUTE]
|
85
|
+
body: attributes[BODY_ATTRIBUTE],
|
86
|
+
flags: attributes["FLAGS"]
|
86
87
|
}
|
87
88
|
end
|
88
89
|
rescue FolderNotFound
|
89
90
|
nil
|
90
91
|
end
|
91
92
|
|
92
|
-
def append(message)
|
93
|
+
def append(message, flags: nil)
|
93
94
|
body = message.imap_body
|
94
95
|
date = message.date&.to_time
|
95
96
|
retry_on_error(errors: APPEND_RETRY_CLASSES, limit: 3) do
|
96
|
-
response = client.append(utf7_encoded_name, body,
|
97
|
+
response = client.append(utf7_encoded_name, body, flags, date)
|
97
98
|
extract_uid(response)
|
98
99
|
end
|
99
100
|
end
|
@@ -79,6 +79,7 @@ module Imap::Backup
|
|
79
79
|
def handle_uid_and_body(uid_and_body, index)
|
80
80
|
uid = uid_and_body[:uid]
|
81
81
|
body = uid_and_body[:body]
|
82
|
+
flags = uid_and_body[:flags]
|
82
83
|
case
|
83
84
|
when !body
|
84
85
|
info("Fetch returned empty body - skipping")
|
@@ -86,7 +87,7 @@ module Imap::Backup
|
|
86
87
|
info("Fetch returned empty UID - skipping")
|
87
88
|
else
|
88
89
|
debug("uid: #{uid} (#{index}/#{uids.count}) - #{body.size} bytes")
|
89
|
-
serializer.append uid, body
|
90
|
+
serializer.append uid, body, flags
|
90
91
|
end
|
91
92
|
end
|
92
93
|
|
data/lib/imap/backup/migrator.rb
CHANGED
@@ -16,7 +16,7 @@ module Imap::Backup
|
|
16
16
|
ensure_destination_empty!
|
17
17
|
|
18
18
|
Logger.logger.debug "[#{folder.name}] #{count} to migrate"
|
19
|
-
serializer.each_message(serializer.uids).with_index do |(uid, message), i|
|
19
|
+
serializer.each_message(serializer.uids).with_index do |(uid, message, flags), i|
|
20
20
|
next if message.nil?
|
21
21
|
|
22
22
|
log_prefix = "[#{folder.name}] uid: #{uid} (#{i + 1}/#{count}) -"
|
@@ -25,7 +25,7 @@ module Imap::Backup
|
|
25
25
|
)
|
26
26
|
|
27
27
|
begin
|
28
|
-
folder.append(message)
|
28
|
+
folder.append(message, flags: flags)
|
29
29
|
rescue StandardError => e
|
30
30
|
Logger.logger.warn "#{log_prefix} append error: #{e}"
|
31
31
|
end
|
@@ -10,7 +10,7 @@ module Imap::Backup
|
|
10
10
|
@mbox = mbox
|
11
11
|
end
|
12
12
|
|
13
|
-
def run(uid:, message:)
|
13
|
+
def run(uid:, message:, flags:)
|
14
14
|
raise "Can't add messages without uid_validity" if !imap.uid_validity
|
15
15
|
|
16
16
|
uid = uid.to_i
|
@@ -21,19 +21,20 @@ module Imap::Backup
|
|
21
21
|
return
|
22
22
|
end
|
23
23
|
|
24
|
-
do_append uid, message
|
24
|
+
do_append uid, message, flags
|
25
25
|
end
|
26
26
|
|
27
27
|
private
|
28
28
|
|
29
|
-
def do_append(uid, message)
|
29
|
+
def do_append(uid, message, flags)
|
30
30
|
mboxrd_message = Email::Mboxrd::Message.new(message)
|
31
31
|
initial = mbox.length || 0
|
32
32
|
mbox_appended = false
|
33
33
|
begin
|
34
|
-
|
34
|
+
serialized = mboxrd_message.to_serialized
|
35
|
+
mbox.append serialized
|
35
36
|
mbox_appended = true
|
36
|
-
imap.append uid
|
37
|
+
imap.append uid, serialized.length, flags
|
37
38
|
rescue StandardError => e
|
38
39
|
mbox.rewind(initial) if mbox_appended
|
39
40
|
|
@@ -2,7 +2,7 @@ require "json"
|
|
2
2
|
|
3
3
|
module Imap::Backup
|
4
4
|
class Serializer::Imap
|
5
|
-
CURRENT_VERSION =
|
5
|
+
CURRENT_VERSION = 3
|
6
6
|
|
7
7
|
attr_reader :folder_path
|
8
8
|
attr_reader :loaded
|
@@ -11,7 +11,7 @@ module Imap::Backup
|
|
11
11
|
@folder_path = folder_path
|
12
12
|
@loaded = false
|
13
13
|
@uid_validity = nil
|
14
|
-
@
|
14
|
+
@messages = nil
|
15
15
|
@version = nil
|
16
16
|
end
|
17
17
|
|
@@ -23,21 +23,38 @@ module Imap::Backup
|
|
23
23
|
true
|
24
24
|
end
|
25
25
|
|
26
|
-
def append(uid)
|
27
|
-
|
26
|
+
def append(uid, length, flags = [])
|
27
|
+
offset =
|
28
|
+
if messages.empty?
|
29
|
+
0
|
30
|
+
else
|
31
|
+
last_message = messages[-1]
|
32
|
+
last_message[:offset] + last_message[:length]
|
33
|
+
end
|
34
|
+
messages << {uid: uid, offset: offset, length: length, flags: flags}
|
28
35
|
save
|
29
36
|
end
|
30
37
|
|
38
|
+
def get(uid)
|
39
|
+
messages.find { |m| m[:uid] == uid }
|
40
|
+
end
|
41
|
+
|
31
42
|
def delete
|
32
43
|
return if !exist?
|
33
44
|
|
34
45
|
File.unlink(pathname)
|
46
|
+
@loaded = false
|
47
|
+
@messages = nil
|
48
|
+
@uid_validity = nil
|
49
|
+
@version = nil
|
35
50
|
end
|
36
51
|
|
52
|
+
# Deprecated
|
37
53
|
def include?(uid)
|
38
54
|
uids.include?(uid)
|
39
55
|
end
|
40
56
|
|
57
|
+
# Deprecated
|
41
58
|
def index(uid)
|
42
59
|
uids.find_index(uid)
|
43
60
|
end
|
@@ -60,20 +77,26 @@ module Imap::Backup
|
|
60
77
|
def uid_validity=(value)
|
61
78
|
ensure_loaded
|
62
79
|
@uid_validity = value
|
63
|
-
@uids ||= []
|
64
80
|
save
|
65
81
|
end
|
66
82
|
|
67
|
-
|
83
|
+
# Make private
|
84
|
+
def messages
|
68
85
|
ensure_loaded
|
69
|
-
@
|
86
|
+
@messages
|
87
|
+
end
|
88
|
+
|
89
|
+
# Deprecated
|
90
|
+
def uids
|
91
|
+
messages.map { |m| m[:uid] }
|
70
92
|
end
|
71
93
|
|
72
94
|
def update_uid(old, new)
|
73
|
-
index =
|
95
|
+
index = messages.find_index { |m| m[:uid] == old }
|
74
96
|
return if index.nil?
|
75
97
|
|
76
|
-
|
98
|
+
updated = messages[index].merge({uid: new})
|
99
|
+
messages[index] = updated
|
77
100
|
save
|
78
101
|
end
|
79
102
|
|
@@ -97,11 +120,11 @@ module Imap::Backup
|
|
97
120
|
|
98
121
|
data = load
|
99
122
|
if data
|
100
|
-
@
|
123
|
+
@messages = data[:messages]
|
101
124
|
@uid_validity = data[:uid_validity]
|
102
125
|
@version = data[:version]
|
103
126
|
else
|
104
|
-
@
|
127
|
+
@messages = []
|
105
128
|
@uid_validity = nil
|
106
129
|
@version = CURRENT_VERSION
|
107
130
|
end
|
@@ -121,8 +144,10 @@ module Imap::Backup
|
|
121
144
|
|
122
145
|
return nil if !data.key?(:version)
|
123
146
|
return nil if !data.key?(:uid_validity)
|
124
|
-
return nil if !data.key?(:
|
125
|
-
return nil if !data[:
|
147
|
+
return nil if !data.key?(:messages)
|
148
|
+
return nil if !data[:messages].is_a?(Array)
|
149
|
+
|
150
|
+
data[:messages].each { |m| m[:flags] = m[:flags].map(&:to_sym) }
|
126
151
|
|
127
152
|
data
|
128
153
|
end
|
@@ -135,7 +160,7 @@ module Imap::Backup
|
|
135
160
|
data = {
|
136
161
|
version: @version,
|
137
162
|
uid_validity: @uid_validity,
|
138
|
-
|
163
|
+
messages: @messages
|
139
164
|
}
|
140
165
|
content = data.to_json
|
141
166
|
File.open(pathname, "w") { |f| f.write content }
|
@@ -8,8 +8,8 @@ module Imap::Backup
|
|
8
8
|
@mbox_pathname = mbox_pathname
|
9
9
|
end
|
10
10
|
|
11
|
-
def each
|
12
|
-
return enum_for(:each) if !
|
11
|
+
def each(&block)
|
12
|
+
return enum_for(:each) if !block
|
13
13
|
|
14
14
|
File.open(mbox_pathname, "rb") do |f|
|
15
15
|
lines = []
|
@@ -26,8 +26,14 @@ module Imap::Backup
|
|
26
26
|
end
|
27
27
|
end
|
28
28
|
|
29
|
-
|
29
|
+
block.call(lines.join) if lines.count.positive?
|
30
30
|
end
|
31
31
|
end
|
32
|
+
|
33
|
+
def map(&block)
|
34
|
+
return enum_for(:map) if !block
|
35
|
+
|
36
|
+
each.map { |line| block.call(line) }
|
37
|
+
end
|
32
38
|
end
|
33
39
|
end
|
@@ -12,17 +12,16 @@ module Imap::Backup
|
|
12
12
|
end
|
13
13
|
|
14
14
|
def run(uids:)
|
15
|
-
|
15
|
+
uids.each do |uid_maybe_string|
|
16
16
|
uid = uid_maybe_string.to_i
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
next if !uid
|
17
|
+
message = imap.get(uid)
|
18
|
+
|
19
|
+
next if !message
|
20
|
+
|
21
|
+
raw = mbox.read(message[:offset], message[:length])
|
22
|
+
body = Email::Mboxrd::Message.from_serialized(raw)
|
24
23
|
|
25
|
-
yield uid,
|
24
|
+
yield message[:uid], body, message[:flags]
|
26
25
|
end
|
27
26
|
end
|
28
27
|
end
|
@@ -0,0 +1,90 @@
|
|
1
|
+
require "json"
|
2
|
+
|
3
|
+
require "imap/backup/serializer/mbox_enumerator"
|
4
|
+
|
5
|
+
module Imap::Backup
|
6
|
+
class Serializer::Version2Migrator
|
7
|
+
attr_reader :folder_path
|
8
|
+
|
9
|
+
def initialize(folder_path)
|
10
|
+
@folder_path = folder_path
|
11
|
+
end
|
12
|
+
|
13
|
+
def required?
|
14
|
+
return false if !mbox_exists?
|
15
|
+
return false if !imap_exists?
|
16
|
+
return false if !data
|
17
|
+
return false if data[:version] != 2
|
18
|
+
return false if !data[:uid_validity]
|
19
|
+
return false if !uids.is_a?(Array)
|
20
|
+
|
21
|
+
true
|
22
|
+
end
|
23
|
+
|
24
|
+
def run
|
25
|
+
return false if !required?
|
26
|
+
|
27
|
+
messages = message_uids_and_lengths
|
28
|
+
|
29
|
+
return false if !messages
|
30
|
+
|
31
|
+
imap.delete
|
32
|
+
imap.uid_validity = data[:uid_validity]
|
33
|
+
messages.map { |m| imap.append(m[:uid], m[:length]) }
|
34
|
+
|
35
|
+
true
|
36
|
+
end
|
37
|
+
|
38
|
+
private
|
39
|
+
|
40
|
+
def imap_pathname
|
41
|
+
"#{folder_path}.imap"
|
42
|
+
end
|
43
|
+
|
44
|
+
def imap_exists?
|
45
|
+
File.exist?(imap_pathname)
|
46
|
+
end
|
47
|
+
|
48
|
+
def mbox_pathname
|
49
|
+
"#{folder_path}.mbox"
|
50
|
+
end
|
51
|
+
|
52
|
+
def mbox_exists?
|
53
|
+
File.exist?(mbox_pathname)
|
54
|
+
end
|
55
|
+
|
56
|
+
def data
|
57
|
+
@data ||=
|
58
|
+
begin
|
59
|
+
content = File.read(imap_pathname)
|
60
|
+
JSON.parse(content, symbolize_names: true)
|
61
|
+
rescue JSON::ParserError
|
62
|
+
nil
|
63
|
+
end
|
64
|
+
end
|
65
|
+
|
66
|
+
def uids
|
67
|
+
data[:uids]
|
68
|
+
end
|
69
|
+
|
70
|
+
def message_uids_and_lengths
|
71
|
+
enumerator = Serializer::MboxEnumerator.new(mbox_pathname)
|
72
|
+
messages = enumerator.map.with_index do |raw, i|
|
73
|
+
length = raw.length
|
74
|
+
message = {
|
75
|
+
uid: uids[i],
|
76
|
+
length: length
|
77
|
+
}
|
78
|
+
message
|
79
|
+
end
|
80
|
+
|
81
|
+
return nil if messages.count != uids.count
|
82
|
+
|
83
|
+
messages
|
84
|
+
end
|
85
|
+
|
86
|
+
def imap
|
87
|
+
@imap ||= Serializer::Imap.new(folder_path)
|
88
|
+
end
|
89
|
+
end
|
90
|
+
end
|
@@ -6,6 +6,7 @@ require "imap/backup/serializer/imap"
|
|
6
6
|
require "imap/backup/serializer/mbox"
|
7
7
|
require "imap/backup/serializer/mbox_enumerator"
|
8
8
|
require "imap/backup/serializer/message_enumerator"
|
9
|
+
require "imap/backup/serializer/version2_migrator"
|
9
10
|
require "imap/backup/serializer/unused_name_finder"
|
10
11
|
|
11
12
|
module Imap::Backup
|
@@ -31,6 +32,8 @@ module Imap::Backup
|
|
31
32
|
# Returns true if there are existing, valid files
|
32
33
|
# false otherwise (in which case any existing files are deleted)
|
33
34
|
def validate!
|
35
|
+
optionally_migrate2to3
|
36
|
+
|
34
37
|
return true if imap.valid? && mbox.valid?
|
35
38
|
|
36
39
|
imap.delete
|
@@ -62,11 +65,11 @@ module Imap::Backup
|
|
62
65
|
internal_force_uid_validity(value)
|
63
66
|
end
|
64
67
|
|
65
|
-
def append(uid, message)
|
68
|
+
def append(uid, message, flags)
|
66
69
|
validate!
|
67
70
|
|
68
71
|
appender = Serializer::Appender.new(folder: folder, imap: imap, mbox: mbox)
|
69
|
-
appender.run(uid: uid, message: message)
|
72
|
+
appender.run(uid: uid, message: message, flags: flags)
|
70
73
|
end
|
71
74
|
|
72
75
|
def load(uid_maybe_string)
|
@@ -79,12 +82,6 @@ module Imap::Backup
|
|
79
82
|
internal_load_nth(message_index)
|
80
83
|
end
|
81
84
|
|
82
|
-
def load_nth(index)
|
83
|
-
validate!
|
84
|
-
|
85
|
-
internal_load_nth(index)
|
86
|
-
end
|
87
|
-
|
88
85
|
def each_message(required_uids, &block)
|
89
86
|
validate!
|
90
87
|
|
@@ -134,6 +131,19 @@ module Imap::Backup
|
|
134
131
|
end
|
135
132
|
end
|
136
133
|
|
134
|
+
def optionally_migrate2to3
|
135
|
+
migrator = Version2Migrator.new(folder_path)
|
136
|
+
return if !migrator.required?
|
137
|
+
|
138
|
+
Logger.logger.info <<~MESSAGE
|
139
|
+
Local metadata for folder '#{folder_path}' is currently stored in the version 2 format.
|
140
|
+
|
141
|
+
This will now be transformed into the version 3 format.
|
142
|
+
MESSAGE
|
143
|
+
|
144
|
+
migrator.run
|
145
|
+
end
|
146
|
+
|
137
147
|
def folder_path
|
138
148
|
self.class.folder_path_for(path: path, folder: folder)
|
139
149
|
end
|
data/lib/imap/backup/uploader.rb
CHANGED
@@ -19,14 +19,14 @@ module Imap::Backup
|
|
19
19
|
return if count.zero?
|
20
20
|
|
21
21
|
Logger.logger.debug "[#{folder.name}] #{count} to restore"
|
22
|
-
serializer.each_message(missing_uids).with_index do |(uid, message), i|
|
23
|
-
upload_message uid, message, i + 1
|
22
|
+
serializer.each_message(missing_uids).with_index do |(uid, message, flags), i|
|
23
|
+
upload_message uid, message, flags, i + 1
|
24
24
|
end
|
25
25
|
end
|
26
26
|
|
27
27
|
private
|
28
28
|
|
29
|
-
def upload_message(uid, message, index)
|
29
|
+
def upload_message(uid, message, flags, index)
|
30
30
|
return if message.nil?
|
31
31
|
|
32
32
|
log_prefix = "[#{folder.name}] uid: #{uid} (#{index}/#{count}) -"
|
@@ -35,7 +35,7 @@ module Imap::Backup
|
|
35
35
|
)
|
36
36
|
|
37
37
|
begin
|
38
|
-
new_uid = folder.append(message)
|
38
|
+
new_uid = folder.append(message, flags: flags)
|
39
39
|
serializer.update_uid(uid, new_uid)
|
40
40
|
rescue StandardError => e
|
41
41
|
Logger.logger.warn "#{log_prefix} append error: #{e}"
|
data/lib/imap/backup/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: imap-backup
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 7.0.0.rc1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Joe Yates
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-08-
|
11
|
+
date: 2022-08-30 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: highline
|
@@ -38,6 +38,34 @@ dependencies:
|
|
38
38
|
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
40
|
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: net-imap
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: net-smtp
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
41
69
|
- !ruby/object:Gem::Dependency
|
42
70
|
name: os
|
43
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -189,6 +217,7 @@ files:
|
|
189
217
|
- LICENSE
|
190
218
|
- README.md
|
191
219
|
- bin/imap-backup
|
220
|
+
- docs/configuration.md
|
192
221
|
- docs/development.md
|
193
222
|
- docs/restore.md
|
194
223
|
- docs/setup.md
|
@@ -237,6 +266,7 @@ files:
|
|
237
266
|
- lib/imap/backup/serializer/mbox_enumerator.rb
|
238
267
|
- lib/imap/backup/serializer/message_enumerator.rb
|
239
268
|
- lib/imap/backup/serializer/unused_name_finder.rb
|
269
|
+
- lib/imap/backup/serializer/version2_migrator.rb
|
240
270
|
- lib/imap/backup/setup.rb
|
241
271
|
- lib/imap/backup/setup/account.rb
|
242
272
|
- lib/imap/backup/setup/account/header.rb
|
@@ -264,12 +294,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
264
294
|
requirements:
|
265
295
|
- - ">="
|
266
296
|
- !ruby/object:Gem::Version
|
267
|
-
version: '2.
|
297
|
+
version: '2.6'
|
268
298
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
269
299
|
requirements:
|
270
|
-
- - "
|
300
|
+
- - ">"
|
271
301
|
- !ruby/object:Gem::Version
|
272
|
-
version:
|
302
|
+
version: 1.3.1
|
273
303
|
requirements: []
|
274
304
|
rubygems_version: 3.1.6
|
275
305
|
signing_key:
|