mclone 0.1.1 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGES.md +8 -4
- data/README.md +409 -369
- data/bin/mclone +23 -8
- data/lib/mclone.rb +223 -126
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: cb414361ce06f3977f87735131a0c8de7d392b469e55b6c49554b2ac1766baae
|
4
|
+
data.tar.gz: 60691ed2d154162a68c8a9262bf7ae90af6023a8bcf0148776d1ed2d00dfa987
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e2c1cbfa056b6ca56bd88c2b7abe93c550a6f1c616d2eb93c98037c3197afc60ae30e7ca060997836ea5ae003da278c86a04989d6c1cef22abd04c93cb8d3e98
|
7
|
+
data.tar.gz: 395f81ed736663c5309eb52a5d1b63f10c7a546e360bd2827038a6dd6eb7b7881fa770b70a386d51b86acb95bcbf9d5422bdda990319c9c670d2e3f2aa66535e
|
data/CHANGES.md
CHANGED
data/README.md
CHANGED
@@ -1,370 +1,410 @@
|
|
1
|
-
# Mclone
|
2
|
-
|
3
|
-
[Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
|
4
|
-
[Rclone](https://rclone.org) as a backend for doing actual file transfer.
|
5
|
-
|
6
|
-
## Purpose
|
7
|
-
|
8
|
-
Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
|
9
|
-
For example, consider a terabyte private media archive one can not afford to lose.
|
10
|
-
|
11
|
-
As the data gets periodically updated, there is a need for regular synchronization.
|
12
|
-
When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
|
13
|
-
backing up comes back into play.
|
14
|
-
|
15
|
-
A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
|
16
|
-
computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
|
17
|
-
a mirror storage which holds the backup, and a portable storage (USB flash disc,
|
18
|
-
serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
|
19
|
-
|
20
|
-
In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
|
21
|
-
employing portable storage as a "shuttle" or a "ferry".
|
22
|
-
|
23
|
-
All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
|
24
|
-
since portable storage is involved, the actual file paths may change between synchronizations as a storage
|
25
|
-
mounted under different mount points on *NIX system or change the disk drive on Windows system.
|
26
|
-
|
27
|
-
While the Rclone itself is a great tool for local file synchronization, typing
|
28
|
-
becomes tedious and error prone where
|
29
|
-
|
30
|
-
This is where the Mclone comes in.
|
31
|
-
It is designed to automatize the Rclone synchronization process by memorizing
|
32
|
-
|
33
|
-
|
34
|
-
## Installation
|
35
|
-
|
36
|
-
Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
|
37
|
-
|
38
|
-
Once the Ruby runtime is properly set, the Mclone itself is installed with
|
39
|
-
|
40
|
-
```shell
|
41
|
-
$ gem install mclone
|
42
|
-
```
|
43
|
-
|
44
|
-
Obviously, the Rclone installation is also required.
|
45
|
-
The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
|
46
|
-
the `PATH` environment variable to locate the `rclone` executable.
|
47
|
-
|
48
|
-
Once properly installed, the Mclone provides the `mclone` command line utility.
|
49
|
-
|
50
|
-
```shell
|
51
|
-
$ mclone -h
|
52
|
-
```
|
53
|
-
## Basic use case
|
54
|
-
|
55
|
-
Let's start with the simplest case.
|
56
|
-
|
57
|
-
Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
|
58
|
-
into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
|
59
|
-
|
60
|
-
### 1. Create volumes
|
61
|
-
|
62
|
-
Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
|
63
|
-
for all Mclone operations.
|
64
|
-
|
65
|
-
By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
|
66
|
-
all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
|
67
|
-
in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
|
68
|
-
on *NIX systems or the semicolon `;` on Windows system.
|
69
|
-
|
70
|
-
If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
|
71
|
-
the environment for later reuse
|
72
|
-
|
73
|
-
```shell
|
74
|
-
export MCLONE_PATH=/data
|
75
|
-
```
|
76
|
-
|
77
|
-
On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
|
78
|
-
therefore there is no need to put it there.
|
79
|
-
|
80
|
-
Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
|
81
|
-
|
82
|
-
```shell
|
83
|
-
$ mclone volume create /data
|
84
|
-
$ mclone volume create /mnt/backup
|
85
|
-
```
|
86
|
-
|
87
|
-
After that, `mclone info` can be used to review the recognized volumes
|
88
|
-
|
89
|
-
```shell
|
90
|
-
$ mclone info
|
91
|
-
|
92
|
-
# Mclone version 0.1.0
|
93
|
-
|
94
|
-
## Volumes
|
95
|
-
|
96
|
-
* [6bfa4a2d] :: (/data)
|
97
|
-
* [7443e311] :: (/mnt/backup)
|
98
|
-
```
|
99
|
-
|
100
|
-
Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
|
101
|
-
_Obviously, the tags will be different in your case._
|
102
|
-
|
103
|
-
### 2. Create a task
|
104
|
-
|
105
|
-
A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
|
106
|
-
the source and destination subdirectories _relative to the respective volumes_,
|
107
|
-
as well as additional Rclone command line arguments to be used.
|
108
|
-
|
109
|
-
_There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
|
110
|
-
|
111
|
-
A task with all defaults is created with
|
112
|
-
|
113
|
-
```shell
|
114
|
-
$ mclone task create /data/files /mnt/backup/files
|
115
|
-
```
|
116
|
-
|
117
|
-
Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
|
118
|
-
|
119
|
-
Again, use the `mclone info` to review the changes
|
120
|
-
|
121
|
-
```shell
|
122
|
-
# Mclone version 0.1.0
|
123
|
-
|
124
|
-
## Volumes
|
125
|
-
|
126
|
-
* [6bfa4a2d] :: (/data)
|
127
|
-
* [7443e311] :: (/mnt/backup)
|
128
|
-
|
129
|
-
## Intact tasks
|
130
|
-
|
131
|
-
* [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
|
132
|
-
```
|
133
|
-
|
134
|
-
The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
|
135
|
-
`6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
|
136
|
-
including `**` all files and subdirectories.
|
137
|
-
|
138
|
-
Again, the task's tag is randomly generated and will be different in your case.
|
139
|
-
|
140
|
-
There are two kinds of tasks to encounter - intact and stale.
|
141
|
-
|
142
|
-
An intact task is a task which is fully ready for processing with the Rclone.
|
143
|
-
As with the volumes, its tag is shown in the square brackets `[...]`
|
144
|
-
|
145
|
-
Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
|
146
|
-
A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
|
147
|
-
the angle brackets.
|
148
|
-
|
149
|
-
Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
|
150
|
-
portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
|
151
|
-
|
152
|
-
The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
|
153
|
-
be detectable by the Mclone.
|
154
|
-
|
155
|
-
### 3. Modify the task
|
156
|
-
|
157
|
-
Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
|
158
|
-
Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
|
159
|
-
|
160
|
-
A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
|
161
|
-
|
162
|
-
Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
|
163
|
-
|
164
|
-
```shell
|
165
|
-
$ mclone task modify -m sync -x '*.bak' cef
|
166
|
-
```
|
167
|
-
|
168
|
-
This time the task is identified by its tag instead of a directory.
|
169
|
-
|
170
|
-
Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
|
171
|
-
and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
|
172
|
-
Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
|
173
|
-
otherwise the Mclone will bail out with error.
|
174
|
-
|
175
|
-
The abbreviations are supported for operation mode, volume and task tags.
|
176
|
-
|
177
|
-
Behold the changes
|
178
|
-
|
179
|
-
```shell
|
180
|
-
$ mclone info
|
181
|
-
|
182
|
-
# Mclone version 0.1.0
|
183
|
-
|
184
|
-
## Volumes
|
185
|
-
|
186
|
-
* [6bfa4a2d] :: (/data)
|
187
|
-
* [7443e311] :: (/mnt/backup)
|
188
|
-
|
189
|
-
## Intact tasks
|
190
|
-
|
191
|
-
* [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
|
192
|
-
```
|
193
|
-
|
194
|
-
### 4. Process the tasks
|
195
|
-
|
196
|
-
Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
|
197
|
-
|
198
|
-
```shell
|
199
|
-
$ mclone task process
|
200
|
-
```
|
201
|
-
|
202
|
-
If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
|
203
|
-
|
204
|
-
|
205
|
-
```shell
|
206
|
-
$ mclone task process cef
|
207
|
-
```
|
208
|
-
|
209
|
-
Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
|
210
|
-
volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
|
211
|
-
|
212
|
-
Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
|
213
|
-
for every occasion.
|
214
|
-
|
215
|
-
## Advanced use case
|
216
|
-
|
217
|
-
Now back to the triple storage scenario outlined above.
|
218
|
-
|
219
|
-
Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
|
220
|
-
be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
|
221
|
-
|
222
|
-
In this case the full data propagation graph is **S->P->D**.
|
223
|
-
|
224
|
-
### 1. Set up the S->P route
|
225
|
-
|
226
|
-
1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
|
227
|
-
|
228
|
-
1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
|
229
|
-
|
230
|
-
1.3. Unplug **P**.
|
231
|
-
|
232
|
-
At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
|
233
|
-
|
234
|
-
### 2. Set up the P->D route
|
235
|
-
|
236
|
-
2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
|
237
|
-
|
238
|
-
Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
|
239
|
-
|
240
|
-
2.2. Create the **D**'s volume, then create a **P->D** task.
|
241
|
-
Note that **P** at this point already contains a volume and therefore must not be formatted.
|
242
|
-
|
243
|
-
2.3. Unplug **P**.
|
244
|
-
|
245
|
-
Now **S** and **D** are formatted and carry the respective tasks.
|
246
|
-
**P** contains its own copies of both **S->P** and **P->D** tasks.
|
247
|
-
|
248
|
-
### 3. Process the **S->P->D** route
|
249
|
-
|
250
|
-
3.1. Plug in **P** to the **S**'s computer and mount it.
|
251
|
-
|
252
|
-
3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
|
253
|
-
|
254
|
-
3.3. Unplug **P**.
|
255
|
-
|
256
|
-
**P** now carries its own copy of the **S**'s data.
|
257
|
-
|
258
|
-
3.4. Plug in **P** to the **D**'s computer and mount it.
|
259
|
-
|
260
|
-
3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
|
261
|
-
|
262
|
-
3.6. Unplug **P**.
|
263
|
-
|
264
|
-
_Voilà!_
|
265
|
-
Both **P** and **D** now carry a copy of the **S**'s data.
|
266
|
-
|
267
|
-
There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
|
268
|
-
storage in the above way.
|
269
|
-
|
270
|
-
Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
|
271
|
-
|
272
|
-
##
|
273
|
-
|
274
|
-
|
275
|
-
|
276
|
-
|
277
|
-
|
278
|
-
|
279
|
-
|
280
|
-
|
281
|
-
|
282
|
-
|
283
|
-
|
284
|
-
|
285
|
-
|
286
|
-
|
287
|
-
|
288
|
-
|
289
|
-
|
290
|
-
|
291
|
-
|
292
|
-
|
293
|
-
|
294
|
-
|
295
|
-
|
296
|
-
|
297
|
-
|
298
|
-
|
299
|
-
|
300
|
-
|
301
|
-
|
302
|
-
|
303
|
-
|
304
|
-
|
305
|
-
|
306
|
-
|
307
|
-
|
308
|
-
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
314
|
-
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
|
319
|
-
|
320
|
-
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
|
330
|
-
|
331
|
-
|
332
|
-
|
333
|
-
|
334
|
-
|
335
|
-
|
336
|
-
|
337
|
-
|
338
|
-
|
339
|
-
|
340
|
-
Rclone
|
341
|
-
|
342
|
-
|
343
|
-
|
344
|
-
|
345
|
-
|
346
|
-
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
|
351
|
-
|
352
|
-
|
353
|
-
|
354
|
-
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
|
359
|
-
|
360
|
-
*
|
361
|
-
|
362
|
-
*
|
363
|
-
|
364
|
-
|
365
|
-
|
366
|
-
|
367
|
-
|
368
|
-
|
369
|
-
|
1
|
+
# Mclone
|
2
|
+
|
3
|
+
[Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
|
4
|
+
[Rclone](https://rclone.org) as a backend for doing actual file transfer.
|
5
|
+
|
6
|
+
## Purpose
|
7
|
+
|
8
|
+
Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
|
9
|
+
For example, consider a terabyte private media archive one can not afford to lose.
|
10
|
+
|
11
|
+
As the data gets periodically updated, there is a need for regular synchronization.
|
12
|
+
When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
|
13
|
+
backing up comes back into play.
|
14
|
+
|
15
|
+
A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
|
16
|
+
computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
|
17
|
+
a mirror storage which holds the backup, and a portable storage (USB flash disc, external HDD or SSD - whatever) which
|
18
|
+
serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
|
19
|
+
|
20
|
+
In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
|
21
|
+
employing portable storage as a "shuttle" or a "ferry".
|
22
|
+
|
23
|
+
All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
|
24
|
+
since portable storage is involved, the actual file paths may change between synchronizations as a storage can be
|
25
|
+
mounted under different mount points on *NIX system or change the disk drive on Windows system.
|
26
|
+
|
27
|
+
While the Rclone itself is a great tool for local file synchronization, typing command line to be executed in this case
|
28
|
+
becomes tedious and error prone where possible cost of error is a backup corruption due to wrong paths or misspelled flags.
|
29
|
+
|
30
|
+
This is where the Mclone comes in.
|
31
|
+
It is designed to automatize the Rclone synchronization process by memorizing command line options and detecting
|
32
|
+
proper source and destination locations wherever they are.
|
33
|
+
|
34
|
+
## Installation
|
35
|
+
|
36
|
+
Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
|
37
|
+
|
38
|
+
Once the Ruby runtime is properly set, the Mclone itself is installed with
|
39
|
+
|
40
|
+
```shell
|
41
|
+
$ gem install mclone
|
42
|
+
```
|
43
|
+
|
44
|
+
Obviously, the Rclone installation is also required.
|
45
|
+
The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
|
46
|
+
the `PATH` environment variable to locate the `rclone` executable.
|
47
|
+
|
48
|
+
Once properly installed, the Mclone provides the `mclone` command line utility.
|
49
|
+
|
50
|
+
```shell
|
51
|
+
$ mclone -h
|
52
|
+
```
|
53
|
+
## Basic use case
|
54
|
+
|
55
|
+
Let's start with the simplest case.
|
56
|
+
|
57
|
+
Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
|
58
|
+
into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
|
59
|
+
|
60
|
+
### 1. Create volumes
|
61
|
+
|
62
|
+
Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
|
63
|
+
for all Mclone operations.
|
64
|
+
|
65
|
+
By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
|
66
|
+
all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
|
67
|
+
in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
|
68
|
+
on *NIX systems or the semicolon `;` on Windows system.
|
69
|
+
|
70
|
+
If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
|
71
|
+
the environment for later reuse
|
72
|
+
|
73
|
+
```shell
|
74
|
+
export MCLONE_PATH=/data
|
75
|
+
```
|
76
|
+
|
77
|
+
On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
|
78
|
+
therefore there is no need to put it there.
|
79
|
+
|
80
|
+
Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
|
81
|
+
|
82
|
+
```shell
|
83
|
+
$ mclone volume create /data
|
84
|
+
$ mclone volume create /mnt/backup
|
85
|
+
```
|
86
|
+
|
87
|
+
After that, `mclone info` can be used to review the recognized volumes
|
88
|
+
|
89
|
+
```shell
|
90
|
+
$ mclone info
|
91
|
+
|
92
|
+
# Mclone version 0.1.0
|
93
|
+
|
94
|
+
## Volumes
|
95
|
+
|
96
|
+
* [6bfa4a2d] :: (/data)
|
97
|
+
* [7443e311] :: (/mnt/backup)
|
98
|
+
```
|
99
|
+
|
100
|
+
Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
|
101
|
+
_Obviously, the tags will be different in your case._
|
102
|
+
|
103
|
+
### 2. Create a task
|
104
|
+
|
105
|
+
A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
|
106
|
+
the source and destination subdirectories _relative to the respective volumes_,
|
107
|
+
as well as additional Rclone command line arguments to be used.
|
108
|
+
|
109
|
+
_There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
|
110
|
+
|
111
|
+
A task with all defaults is created with
|
112
|
+
|
113
|
+
```shell
|
114
|
+
$ mclone task create /data/files /mnt/backup/files
|
115
|
+
```
|
116
|
+
|
117
|
+
Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
|
118
|
+
|
119
|
+
Again, use the `mclone info` to review the changes
|
120
|
+
|
121
|
+
```shell
|
122
|
+
# Mclone version 0.1.0
|
123
|
+
|
124
|
+
## Volumes
|
125
|
+
|
126
|
+
* [6bfa4a2d] :: (/data)
|
127
|
+
* [7443e311] :: (/mnt/backup)
|
128
|
+
|
129
|
+
## Intact tasks
|
130
|
+
|
131
|
+
* [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
|
132
|
+
```
|
133
|
+
|
134
|
+
The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
|
135
|
+
`6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
|
136
|
+
including `**` all files and subdirectories.
|
137
|
+
|
138
|
+
Again, the task's tag is randomly generated and will be different in your case.
|
139
|
+
|
140
|
+
There are two kinds of tasks to encounter - intact and stale.
|
141
|
+
|
142
|
+
An intact task is a task which is fully ready for processing with the Rclone.
|
143
|
+
As with the volumes, its tag is shown in the square brackets `[...]`
|
144
|
+
|
145
|
+
Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
|
146
|
+
A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
|
147
|
+
the angle brackets.
|
148
|
+
|
149
|
+
Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
|
150
|
+
portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
|
151
|
+
|
152
|
+
The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
|
153
|
+
be detectable by the Mclone.
|
154
|
+
|
155
|
+
### 3. Modify the task
|
156
|
+
|
157
|
+
Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
|
158
|
+
Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
|
159
|
+
|
160
|
+
A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
|
161
|
+
|
162
|
+
Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
|
163
|
+
|
164
|
+
```shell
|
165
|
+
$ mclone task modify -m sync -x '*.bak' cef
|
166
|
+
```
|
167
|
+
|
168
|
+
This time the task is identified by its tag instead of a directory.
|
169
|
+
|
170
|
+
Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
|
171
|
+
and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
|
172
|
+
Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
|
173
|
+
otherwise the Mclone will bail out with error.
|
174
|
+
|
175
|
+
The abbreviations are supported for operation mode, volume and task tags.
|
176
|
+
|
177
|
+
Behold the changes
|
178
|
+
|
179
|
+
```shell
|
180
|
+
$ mclone info
|
181
|
+
|
182
|
+
# Mclone version 0.1.0
|
183
|
+
|
184
|
+
## Volumes
|
185
|
+
|
186
|
+
* [6bfa4a2d] :: (/data)
|
187
|
+
* [7443e311] :: (/mnt/backup)
|
188
|
+
|
189
|
+
## Intact tasks
|
190
|
+
|
191
|
+
* [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
|
192
|
+
```
|
193
|
+
|
194
|
+
### 4. Process the tasks
|
195
|
+
|
196
|
+
Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
|
197
|
+
|
198
|
+
```shell
|
199
|
+
$ mclone task process
|
200
|
+
```
|
201
|
+
|
202
|
+
If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
|
203
|
+
|
204
|
+
|
205
|
+
```shell
|
206
|
+
$ mclone task process cef
|
207
|
+
```
|
208
|
+
|
209
|
+
Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
|
210
|
+
volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
|
211
|
+
|
212
|
+
Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
|
213
|
+
for every occasion.
|
214
|
+
|
215
|
+
## Advanced use case
|
216
|
+
|
217
|
+
Now back to the triple storage scenario outlined above.
|
218
|
+
|
219
|
+
Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
|
220
|
+
be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
|
221
|
+
|
222
|
+
In this case the full data propagation graph is **S->P->D**.
|
223
|
+
|
224
|
+
### 1. Set up the S->P route
|
225
|
+
|
226
|
+
1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
|
227
|
+
|
228
|
+
1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
|
229
|
+
|
230
|
+
1.3. Unplug **P**.
|
231
|
+
|
232
|
+
At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
|
233
|
+
|
234
|
+
### 2. Set up the P->D route
|
235
|
+
|
236
|
+
2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
|
237
|
+
|
238
|
+
Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
|
239
|
+
|
240
|
+
2.2. Create the **D**'s volume, then create a **P->D** task.
|
241
|
+
Note that **P** at this point already contains a volume and therefore must not be formatted.
|
242
|
+
|
243
|
+
2.3. Unplug **P**.
|
244
|
+
|
245
|
+
Now **S** and **D** are formatted and carry the respective tasks.
|
246
|
+
**P** contains its own copies of both **S->P** and **P->D** tasks.
|
247
|
+
|
248
|
+
### 3. Process the **S->P->D** route
|
249
|
+
|
250
|
+
3.1. Plug in **P** to the **S**'s computer and mount it.
|
251
|
+
|
252
|
+
3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
|
253
|
+
|
254
|
+
3.3. Unplug **P**.
|
255
|
+
|
256
|
+
**P** now carries its own copy of the **S**'s data.
|
257
|
+
|
258
|
+
3.4. Plug in **P** to the **D**'s computer and mount it.
|
259
|
+
|
260
|
+
3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
|
261
|
+
|
262
|
+
3.6. Unplug **P**.
|
263
|
+
|
264
|
+
_Voilà!_
|
265
|
+
Both **P** and **D** now carry a copy of the **S**'s data.
|
266
|
+
|
267
|
+
There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
|
268
|
+
storage in the above way.
|
269
|
+
|
270
|
+
Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
|
271
|
+
|
272
|
+
## Encryption
|
273
|
+
|
274
|
+
Encryption is an essential part of the Mclone as it is all about handling portable storage which may by compromised
|
275
|
+
while holding confidential data. Mclone fully relies on [encryption capabilities](https://rclone.org/crypt/) of Rclone,
|
276
|
+
that is an encrypted directory structure can be further treated with the Rclone itself.
|
277
|
+
|
278
|
+
The encryption operation in Mclone is activated during task creation time.
|
279
|
+
The encryption mode is activated with `-e` or `-d` command line flag for encryption or decryption, respectively.
|
280
|
+
It no either flag is specified, the encryption gets turned off.
|
281
|
+
|
282
|
+
When in encryption mode, Mclone recursively takes plain files and directories under the source root and creates encrypted
|
283
|
+
files and directories under the destination root.
|
284
|
+
Conversely, when in decryption mode, Mclone takes encrypted source root and decrypts it into the destination root.
|
285
|
+
Mclone is set up to encrypt not only the files' contents but also the file and directory names themselves. The file sizes,
|
286
|
+
modification times as well as some other metadata are not encrypted, though, as they are required for proper operation
|
287
|
+
the file synchronization mechanism.
|
288
|
+
Note that the encrypted root is a regular directory hierarchy (just with fancy file names) and thus can be treated as such.
|
289
|
+
|
290
|
+
***Be wary that file name encryption has a serious implication on the file name length.***
|
291
|
+
The Rclone [crypt](https://rclone.org/crypt/) documentation states the the individual file or directory name length can
|
292
|
+
not exceed ~143 charactes (although ***bytes*** here would be more correct).
|
293
|
+
As the Rclone accepts UTF-8 encoded names, this estimate generally holds true for the Latin charset only, where
|
294
|
+
a character is encoded with a single byte.
|
295
|
+
For non-Latin characters, which can be encoded with two or even more bytes, the maximum allowed name length be
|
296
|
+
much lower.
|
297
|
+
When Rclone encounters a file name too long to hold, it will refuse to process it.
|
298
|
+
|
299
|
+
Rclone employs symmetric cryptography to do its job, which requires some form of password to be supplied upon task
|
300
|
+
creation.
|
301
|
+
This is done by the `-p` command line flag, which specifies a plain text password used to derive the real encryption key.
|
302
|
+
There is another password-related `-t` command line flag which can be used to directly specify
|
303
|
+
an [Rclone-obscured](https://rclone.org/commands/rclone_obscure/) token.
|
304
|
+
Once created, a task memorizes the encryption key on the ***unencrypted end*** of the source/destination volume pair,
|
305
|
+
so there will be no need to pass it during the task processing.
|
306
|
+
|
307
|
+
## Whats next
|
308
|
+
|
309
|
+
### On-screen help
|
310
|
+
|
311
|
+
Every `mclone` (sub)command has its own help page which can be shown with `--help` option
|
312
|
+
|
313
|
+
```shell
|
314
|
+
$ mclone task create --help
|
315
|
+
|
316
|
+
Usage:
|
317
|
+
mclone task new [OPTIONS] SOURCE DESTINATION
|
318
|
+
|
319
|
+
Parameters:
|
320
|
+
SOURCE Source path
|
321
|
+
DESTINATION Destination path
|
322
|
+
|
323
|
+
Options:
|
324
|
+
-m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
|
325
|
+
-i, --include PATTERN Include paths pattern
|
326
|
+
-x, --exclude PATTERN Exclude paths pattern
|
327
|
+
-d, --decrypt Decrypt source
|
328
|
+
-e, --encrypt Encrypt destination
|
329
|
+
-p, --password PASSWORD Plain text password
|
330
|
+
-t, --token TOKEN Rclone crypt token (obscured password)
|
331
|
+
-f, --force Insist on potentially dangerous actions (default: false)
|
332
|
+
-n, --dry-run Simulation mode with no on-disk modifications (default: false)
|
333
|
+
-v, --verbose Verbose operation (default: false)
|
334
|
+
-V, --version Show version
|
335
|
+
-h, --help print help
|
336
|
+
```
|
337
|
+
|
338
|
+
### File filtering
|
339
|
+
|
340
|
+
The Mclone passes its include and exclude options to the Rclone.
|
341
|
+
The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
|
342
|
+
documentation [section](https://rclone.org/filtering).
|
343
|
+
|
344
|
+
### Dry run
|
345
|
+
|
346
|
+
The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
|
347
|
+
no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
|
348
|
+
The Rclone is run during task processing but in turn is supplied with this option.
|
349
|
+
|
350
|
+
### Force mode
|
351
|
+
|
352
|
+
The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
|
353
|
+
or overwriting existing task.
|
354
|
+
In this case a `--force` command line option should be used to pass through.
|
355
|
+
|
356
|
+
### Task operation modes
|
357
|
+
|
358
|
+
#### Update
|
359
|
+
|
360
|
+
* Copy source files which are newer than the destination's or have different size or checksum.
|
361
|
+
|
362
|
+
* Do not delete destination files which are nonexistent in the source.
|
363
|
+
|
364
|
+
* Do not copy source files which are older than the destination's.
|
365
|
+
|
366
|
+
A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
|
367
|
+
|
368
|
+
Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
|
369
|
+
|
370
|
+
#### Synchronize
|
371
|
+
|
372
|
+
* Copy source files which are newer than the destination's or have different size or checksum.
|
373
|
+
|
374
|
+
* Delete destination files which are nonexistent in the source.
|
375
|
+
|
376
|
+
* Copy source files which are older than the destination's.
|
377
|
+
|
378
|
+
This is the mirroring mode which makes destination completely identical to the source.
|
379
|
+
|
380
|
+
Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
|
381
|
+
|
382
|
+
#### Copy
|
383
|
+
|
384
|
+
* Copy source files which are newer than the destination's or have different size or checksum.
|
385
|
+
|
386
|
+
* Do not delete destination files which are nonexistent in the source.
|
387
|
+
|
388
|
+
* Do not copy source files which are older than the destination's.
|
389
|
+
|
390
|
+
This mode is much like synchronize with only difference that it does not delete files.
|
391
|
+
|
392
|
+
Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
|
393
|
+
|
394
|
+
#### Move
|
395
|
+
|
396
|
+
* Copy source files which are newer than the destination's or have different size or checksum.
|
397
|
+
|
398
|
+
* Do not delete destination files which are nonexistent in the source.
|
399
|
+
|
400
|
+
* Do not copy source files which are older than the destination's.
|
401
|
+
|
402
|
+
* Delete source files after successful copy to the destination.
|
403
|
+
|
404
|
+
Rclone [command](https://rclone.org/commands/rclone_move): `move`.
|
405
|
+
|
406
|
+
## The end
|
407
|
+
|
408
|
+
Cheers,
|
409
|
+
|
370
410
|
Oleg A. Khlybov <fougas@mail.ru>
|