mclone 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGES.md +8 -4
  3. data/README.md +409 -369
  4. data/bin/mclone +23 -8
  5. data/lib/mclone.rb +223 -126
  6. metadata +3 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c89cb8e633183fd633ff44b2e87fdc73c5cd348bb377e61f77d12cde4d389e61
4
- data.tar.gz: 1844a2921e78e5cc2057b0aa906494e978883ffda60a487f1881743fbe5598cd
3
+ metadata.gz: cb414361ce06f3977f87735131a0c8de7d392b469e55b6c49554b2ac1766baae
4
+ data.tar.gz: 60691ed2d154162a68c8a9262bf7ae90af6023a8bcf0148776d1ed2d00dfa987
5
5
  SHA512:
6
- metadata.gz: dec529f91301c96fd64b839aa4fdcd4cb733f076d8e56ab506ab86ebe0ec075821623d5e5bf8e91a871d72498b2a77db70b5328837764f9d95987b2a24a62150
7
- data.tar.gz: fac7f8a99b14bdacbe1048d7ba5e7d325863677bf3149a6f2c2a23665996ec75790caed9b51dde2260300c4afd838795c05134284b3f738c94c268d601108c21
6
+ metadata.gz: e2c1cbfa056b6ca56bd88c2b7abe93c550a6f1c616d2eb93c98037c3197afc60ae30e7ca060997836ea5ae003da278c86a04989d6c1cef22abd04c93cb8d3e98
7
+ data.tar.gz: 395f81ed736663c5309eb52a5d1b63f10c7a546e360bd2827038a6dd6eb7b7881fa770b70a386d51b86acb95bcbf9d5422bdda990319c9c670d2e3f2aa66535e
data/CHANGES.md CHANGED
@@ -1,5 +1,9 @@
1
- # 0.1.1 (2021-06-24)
2
-
3
- * Switch from FFI to Fiddle.
4
-
1
+ # 0.2.0 (2021-08-08)
2
+
3
+ * Support for encryption.
4
+
5
+ # 0.1.1 (2021-06-24)
6
+
7
+ * Switch from FFI to Fiddle.
8
+
5
9
  # 0.1.0 (2021-06-22)
data/README.md CHANGED
@@ -1,370 +1,410 @@
1
- # Mclone
2
-
3
- [Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
4
- [Rclone](https://rclone.org) as a backend for doing actual file transfer.
5
-
6
- ## Purpose
7
-
8
- Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
9
- For example, consider a terabyte private media archive one can not afford to lose.
10
-
11
- As the data gets periodically updated, there is a need for regular synchronization.
12
- When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
13
- backing up comes back into play.
14
-
15
- A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
16
- computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
17
- a mirror storage which holds the backup, and a portable storage (USB flash disc, exteral HDD or SSD - whatever) which
18
- serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
19
-
20
- In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
21
- employing portable storage as a "shuttle" or a "ferry".
22
-
23
- All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
24
- since portable storage is involved, the actual file paths may change between synchronizations as a storage device can be
25
- mounted under different mount points on *NIX system or change the disk drive on Windows system.
26
-
27
- While the Rclone itself is a great tool for local file synchronization, typing the command line for execution in this case
28
- becomes tedious and error prone where the possible cost of error is a backup corruption due to wrong paths or misspelled flags.
29
-
30
- This is where the Mclone comes in.
31
- It is designed to automatize the Rclone synchronization process by memorizing the command line options and detecting
32
- the proper source and destination locations wherever they are.
33
-
34
- ## Installation
35
-
36
- Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
37
-
38
- Once the Ruby runtime is properly set, the Mclone itself is installed with
39
-
40
- ```shell
41
- $ gem install mclone
42
- ```
43
-
44
- Obviously, the Rclone installation is also required.
45
- The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
46
- the `PATH` environment variable to locate the `rclone` executable.
47
-
48
- Once properly installed, the Mclone provides the `mclone` command line utility.
49
-
50
- ```shell
51
- $ mclone -h
52
- ```
53
- ## Basic use case
54
-
55
- Let's start with the simplest case.
56
-
57
- Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
58
- into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
59
-
60
- ### 1. Create volumes
61
-
62
- Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
63
- for all Mclone operations.
64
-
65
- By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
66
- all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
67
- in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
68
- on *NIX systems or the semicolon `;` on Windows system.
69
-
70
- If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
71
- the environment for later reuse
72
-
73
- ```shell
74
- export MCLONE_PATH=/data
75
- ```
76
-
77
- On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
78
- therefore there is no need to put it there.
79
-
80
- Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
81
-
82
- ```shell
83
- $ mclone volume create /data
84
- $ mclone volume create /mnt/backup
85
- ```
86
-
87
- After that, `mclone info` can be used to review the recognized volumes
88
-
89
- ```shell
90
- $ mclone info
91
-
92
- # Mclone version 0.1.0
93
-
94
- ## Volumes
95
-
96
- * [6bfa4a2d] :: (/data)
97
- * [7443e311] :: (/mnt/backup)
98
- ```
99
-
100
- Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
101
- _Obviously, the tags will be different in your case._
102
-
103
- ### 2. Create a task
104
-
105
- A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
106
- the source and destination subdirectories _relative to the respective volumes_,
107
- as well as additional Rclone command line arguments to be used.
108
-
109
- _There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
110
-
111
- A task with all defaults is created with
112
-
113
- ```shell
114
- $ mclone task create /data/files /mnt/backup/files
115
- ```
116
-
117
- Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
118
-
119
- Again, use the `mclone info` to review the changes
120
-
121
- ```shell
122
- # Mclone version 0.1.0
123
-
124
- ## Volumes
125
-
126
- * [6bfa4a2d] :: (/data)
127
- * [7443e311] :: (/mnt/backup)
128
-
129
- ## Intact tasks
130
-
131
- * [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
132
- ```
133
-
134
- The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
135
- `6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
136
- including `**` all files and subdirectories.
137
-
138
- Again, the task's tag is randomly generated and will be different in your case.
139
-
140
- There are two kinds of tasks to encounter - intact and stale.
141
-
142
- An intact task is a task which is fully ready for processing with the Rclone.
143
- As with the volumes, its tag is shown in the square brackets `[...]`
144
-
145
- Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
146
- A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
147
- the angle brackets.
148
-
149
- Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
150
- portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
151
-
152
- The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
153
- be detectable by the Mclone.
154
-
155
- ### 3. Modify the task
156
-
157
- Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
158
- Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
159
-
160
- A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
161
-
162
- Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
163
-
164
- ```shell
165
- $ mclone task modify -m sync -x '*.bak' cef
166
- ```
167
-
168
- This time the task is identified by its tag instead of a directory.
169
-
170
- Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
171
- and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
172
- Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
173
- otherwise the Mclone will bail out with error.
174
-
175
- The abbreviations are supported for operation mode, volume and task tags.
176
-
177
- Behold the changes
178
-
179
- ```shell
180
- $ mclone info
181
-
182
- # Mclone version 0.1.0
183
-
184
- ## Volumes
185
-
186
- * [6bfa4a2d] :: (/data)
187
- * [7443e311] :: (/mnt/backup)
188
-
189
- ## Intact tasks
190
-
191
- * [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
192
- ```
193
-
194
- ### 4. Process the tasks
195
-
196
- Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
197
-
198
- ```shell
199
- $ mclone task process
200
- ```
201
-
202
- If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
203
-
204
-
205
- ```shell
206
- $ mclone task process cef
207
- ```
208
-
209
- Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
210
- volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
211
-
212
- Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
213
- for every occasion.
214
-
215
- ## Advanced use case
216
-
217
- Now back to the triple storage scenario outlined above.
218
-
219
- Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
220
- be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
221
-
222
- In this case the full data propagation graph is **S->P->D**.
223
-
224
- ### 1. Set up the S->P route
225
-
226
- 1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
227
-
228
- 1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
229
-
230
- 1.3. Unplug **P**.
231
-
232
- At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
233
-
234
- ### 2. Set up the P->D route
235
-
236
- 2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
237
-
238
- Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
239
-
240
- 2.2. Create the **D**'s volume, then create a **P->D** task.
241
- Note that **P** at this point already contains a volume and therefore must not be formatted.
242
-
243
- 2.3. Unplug **P**.
244
-
245
- Now **S** and **D** are formatted and carry the respective tasks.
246
- **P** contains its own copies of both **S->P** and **P->D** tasks.
247
-
248
- ### 3. Process the **S->P->D** route
249
-
250
- 3.1. Plug in **P** to the **S**'s computer and mount it.
251
-
252
- 3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
253
-
254
- 3.3. Unplug **P**.
255
-
256
- **P** now carries its own copy of the **S**'s data.
257
-
258
- 3.4. Plug in **P** to the **D**'s computer and mount it.
259
-
260
- 3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
261
-
262
- 3.6. Unplug **P**.
263
-
264
- _Voilà!_
265
- Both **P** and **D** now carry a copy of the **S**'s data.
266
-
267
- There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
268
- storage in the above way.
269
-
270
- Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
271
-
272
- ## Whats next
273
-
274
- ### On-screen help
275
-
276
- Every `mclone` (sub)command has its own help page which can be shown with `--help` option
277
-
278
- ```shell
279
- $ mclone task create --help
280
-
281
- Usage:
282
- mclone task create [OPTIONS] SOURCE DESTINATION
283
-
284
- Parameters:
285
- SOURCE Source path
286
- DESTINATION Destination path
287
-
288
- Options:
289
- -m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
290
- -i, --include PATTERN Include paths pattern (default: "**")
291
- -x, --exclude PATTERN Exclude paths pattern
292
- -f, --force Insist on potentially dangerous activities (default: false)
293
- -n, --dry-run Simulation mode with no on-disk modifications (default: false)
294
- -v, --verbose Verbose operation (default: false)
295
- -h, --help print help
296
- ```
297
-
298
- ### File filtering
299
-
300
- The Mclone passes its include and exclude options to the Rclone.
301
- The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
302
- documentation [section](https://rclone.org/filtering).
303
-
304
- ### Dry run
305
-
306
- The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
307
- no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
308
- The Rclone is run during task processing but in turn is supplied with this option.
309
-
310
- ### Force mode
311
-
312
- The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
313
- or overwriting existing task.
314
- In this case a `--force` command line option should be used to pass through.
315
-
316
- ### Task operation modes
317
-
318
- #### Update
319
-
320
- * Copy source files which are newer than the destination's or have different size or checksum.
321
-
322
- * Do not delete destination files which are nonexistent in the source.
323
-
324
- * Do not copy source files which are older than the destination's.
325
-
326
- A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
327
-
328
- Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
329
-
330
- #### Synchronize
331
-
332
- * Copy source files which are newer than the destination's or have different size or checksum.
333
-
334
- * Delete destination files which are nonexistent in the source.
335
-
336
- * Copy source files which are older than the destination's.
337
-
338
- This is the mirroring mode which makes destination completely identical to the source.
339
-
340
- Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
341
-
342
- #### Copy
343
-
344
- * Copy source files which are newer than the destination's or have different size or checksum.
345
-
346
- * Do not delete destination files which are nonexistent in the source.
347
-
348
- * Do not copy source files which are older than the destination's.
349
-
350
- This mode is much like synchronize with only difference that it does not delete files.
351
-
352
- Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
353
-
354
- #### Move
355
-
356
- * Copy source files which are newer than the destination's or have different size or checksum.
357
-
358
- * Do not delete destination files which are nonexistent in the source.
359
-
360
- * Do not copy source files which are older than the destination's.
361
-
362
- * Delete source files after successful copy to the destination.
363
-
364
- Rclone [command](https://rclone.org/commands/rclone_move): `move`.
365
-
366
- ## The end
367
-
368
- Cheers,
369
-
1
+ # Mclone
2
+
3
+ [Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
4
+ [Rclone](https://rclone.org) as a backend for doing actual file transfer.
5
+
6
+ ## Purpose
7
+
8
+ Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
9
+ For example, consider a terabyte private media archive one can not afford to lose.
10
+
11
+ As the data gets periodically updated, there is a need for regular synchronization.
12
+ When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
13
+ backing up comes back into play.
14
+
15
+ A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
16
+ computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
17
+ a mirror storage which holds the backup, and a portable storage (USB flash disc, external HDD or SSD - whatever) which
18
+ serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
19
+
20
+ In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
21
+ employing portable storage as a "shuttle" or a "ferry".
22
+
23
+ All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
24
+ since portable storage is involved, the actual file paths may change between synchronizations as a storage can be
25
+ mounted under different mount points on *NIX system or change the disk drive on Windows system.
26
+
27
+ While the Rclone itself is a great tool for local file synchronization, typing command line to be executed in this case
28
+ becomes tedious and error prone where possible cost of error is a backup corruption due to wrong paths or misspelled flags.
29
+
30
+ This is where the Mclone comes in.
31
+ It is designed to automatize the Rclone synchronization process by memorizing command line options and detecting
32
+ proper source and destination locations wherever they are.
33
+
34
+ ## Installation
35
+
36
+ Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
37
+
38
+ Once the Ruby runtime is properly set, the Mclone itself is installed with
39
+
40
+ ```shell
41
+ $ gem install mclone
42
+ ```
43
+
44
+ Obviously, the Rclone installation is also required.
45
+ The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
46
+ the `PATH` environment variable to locate the `rclone` executable.
47
+
48
+ Once properly installed, the Mclone provides the `mclone` command line utility.
49
+
50
+ ```shell
51
+ $ mclone -h
52
+ ```
53
+ ## Basic use case
54
+
55
+ Let's start with the simplest case.
56
+
57
+ Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
58
+ into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
59
+
60
+ ### 1. Create volumes
61
+
62
+ Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
63
+ for all Mclone operations.
64
+
65
+ By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
66
+ all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
67
+ in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
68
+ on *NIX systems or the semicolon `;` on Windows system.
69
+
70
+ If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
71
+ the environment for later reuse
72
+
73
+ ```shell
74
+ export MCLONE_PATH=/data
75
+ ```
76
+
77
+ On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
78
+ therefore there is no need to put it there.
79
+
80
+ Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
81
+
82
+ ```shell
83
+ $ mclone volume create /data
84
+ $ mclone volume create /mnt/backup
85
+ ```
86
+
87
+ After that, `mclone info` can be used to review the recognized volumes
88
+
89
+ ```shell
90
+ $ mclone info
91
+
92
+ # Mclone version 0.1.0
93
+
94
+ ## Volumes
95
+
96
+ * [6bfa4a2d] :: (/data)
97
+ * [7443e311] :: (/mnt/backup)
98
+ ```
99
+
100
+ Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
101
+ _Obviously, the tags will be different in your case._
102
+
103
+ ### 2. Create a task
104
+
105
+ A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
106
+ the source and destination subdirectories _relative to the respective volumes_,
107
+ as well as additional Rclone command line arguments to be used.
108
+
109
+ _There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
110
+
111
+ A task with all defaults is created with
112
+
113
+ ```shell
114
+ $ mclone task create /data/files /mnt/backup/files
115
+ ```
116
+
117
+ Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
118
+
119
+ Again, use the `mclone info` to review the changes
120
+
121
+ ```shell
122
+ # Mclone version 0.1.0
123
+
124
+ ## Volumes
125
+
126
+ * [6bfa4a2d] :: (/data)
127
+ * [7443e311] :: (/mnt/backup)
128
+
129
+ ## Intact tasks
130
+
131
+ * [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
132
+ ```
133
+
134
+ The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
135
+ `6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
136
+ including `**` all files and subdirectories.
137
+
138
+ Again, the task's tag is randomly generated and will be different in your case.
139
+
140
+ There are two kinds of tasks to encounter - intact and stale.
141
+
142
+ An intact task is a task which is fully ready for processing with the Rclone.
143
+ As with the volumes, its tag is shown in the square brackets `[...]`
144
+
145
+ Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
146
+ A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
147
+ the angle brackets.
148
+
149
+ Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
150
+ portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
151
+
152
+ The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
153
+ be detectable by the Mclone.
154
+
155
+ ### 3. Modify the task
156
+
157
+ Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
158
+ Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
159
+
160
+ A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
161
+
162
+ Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
163
+
164
+ ```shell
165
+ $ mclone task modify -m sync -x '*.bak' cef
166
+ ```
167
+
168
+ This time the task is identified by its tag instead of a directory.
169
+
170
+ Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
171
+ and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
172
+ Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
173
+ otherwise the Mclone will bail out with error.
174
+
175
+ The abbreviations are supported for operation mode, volume and task tags.
176
+
177
+ Behold the changes
178
+
179
+ ```shell
180
+ $ mclone info
181
+
182
+ # Mclone version 0.1.0
183
+
184
+ ## Volumes
185
+
186
+ * [6bfa4a2d] :: (/data)
187
+ * [7443e311] :: (/mnt/backup)
188
+
189
+ ## Intact tasks
190
+
191
+ * [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
192
+ ```
193
+
194
+ ### 4. Process the tasks
195
+
196
+ Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
197
+
198
+ ```shell
199
+ $ mclone task process
200
+ ```
201
+
202
+ If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
203
+
204
+
205
+ ```shell
206
+ $ mclone task process cef
207
+ ```
208
+
209
+ Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
210
+ volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
211
+
212
+ Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
213
+ for every occasion.
214
+
215
+ ## Advanced use case
216
+
217
+ Now back to the triple storage scenario outlined above.
218
+
219
+ Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
220
+ be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
221
+
222
+ In this case the full data propagation graph is **S->P->D**.
223
+
224
+ ### 1. Set up the S->P route
225
+
226
+ 1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
227
+
228
+ 1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
229
+
230
+ 1.3. Unplug **P**.
231
+
232
+ At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
233
+
234
+ ### 2. Set up the P->D route
235
+
236
+ 2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
237
+
238
+ Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
239
+
240
+ 2.2. Create the **D**'s volume, then create a **P->D** task.
241
+ Note that **P** at this point already contains a volume and therefore must not be formatted.
242
+
243
+ 2.3. Unplug **P**.
244
+
245
+ Now **S** and **D** are formatted and carry the respective tasks.
246
+ **P** contains its own copies of both **S->P** and **P->D** tasks.
247
+
248
+ ### 3. Process the **S->P->D** route
249
+
250
+ 3.1. Plug in **P** to the **S**'s computer and mount it.
251
+
252
+ 3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
253
+
254
+ 3.3. Unplug **P**.
255
+
256
+ **P** now carries its own copy of the **S**'s data.
257
+
258
+ 3.4. Plug in **P** to the **D**'s computer and mount it.
259
+
260
+ 3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
261
+
262
+ 3.6. Unplug **P**.
263
+
264
+ _Voilà!_
265
+ Both **P** and **D** now carry a copy of the **S**'s data.
266
+
267
+ There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
268
+ storage in the above way.
269
+
270
+ Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
271
+
272
+ ## Encryption
273
+
274
+ Encryption is an essential part of the Mclone as it is all about handling portable storage which may by compromised
275
+ while holding confidential data. Mclone fully relies on [encryption capabilities](https://rclone.org/crypt/) of Rclone,
276
+ that is an encrypted directory structure can be further treated with the Rclone itself.
277
+
278
+ The encryption operation in Mclone is activated during task creation time.
279
+ The encryption mode is activated with `-e` or `-d` command line flag for encryption or decryption, respectively.
280
+ It no either flag is specified, the encryption gets turned off.
281
+
282
+ When in encryption mode, Mclone recursively takes plain files and directories under the source root and creates encrypted
283
+ files and directories under the destination root.
284
+ Conversely, when in decryption mode, Mclone takes encrypted source root and decrypts it into the destination root.
285
+ Mclone is set up to encrypt not only the files' contents but also the file and directory names themselves. The file sizes,
286
+ modification times as well as some other metadata are not encrypted, though, as they are required for proper operation
287
+ the file synchronization mechanism.
288
+ Note that the encrypted root is a regular directory hierarchy (just with fancy file names) and thus can be treated as such.
289
+
290
+ ***Be wary that file name encryption has a serious implication on the file name length.***
291
+ The Rclone [crypt](https://rclone.org/crypt/) documentation states the the individual file or directory name length can
292
+ not exceed ~143 charactes (although ***bytes*** here would be more correct).
293
+ As the Rclone accepts UTF-8 encoded names, this estimate generally holds true for the Latin charset only, where
294
+ a character is encoded with a single byte.
295
+ For non-Latin characters, which can be encoded with two or even more bytes, the maximum allowed name length be
296
+ much lower.
297
+ When Rclone encounters a file name too long to hold, it will refuse to process it.
298
+
299
+ Rclone employs symmetric cryptography to do its job, which requires some form of password to be supplied upon task
300
+ creation.
301
+ This is done by the `-p` command line flag, which specifies a plain text password used to derive the real encryption key.
302
+ There is another password-related `-t` command line flag which can be used to directly specify
303
+ an [Rclone-obscured](https://rclone.org/commands/rclone_obscure/) token.
304
+ Once created, a task memorizes the encryption key on the ***unencrypted end*** of the source/destination volume pair,
305
+ so there will be no need to pass it during the task processing.
306
+
307
+ ## Whats next
308
+
309
+ ### On-screen help
310
+
311
+ Every `mclone` (sub)command has its own help page which can be shown with `--help` option
312
+
313
+ ```shell
314
+ $ mclone task create --help
315
+
316
+ Usage:
317
+ mclone task new [OPTIONS] SOURCE DESTINATION
318
+
319
+ Parameters:
320
+ SOURCE Source path
321
+ DESTINATION Destination path
322
+
323
+ Options:
324
+ -m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
325
+ -i, --include PATTERN Include paths pattern
326
+ -x, --exclude PATTERN Exclude paths pattern
327
+ -d, --decrypt Decrypt source
328
+ -e, --encrypt Encrypt destination
329
+ -p, --password PASSWORD Plain text password
330
+ -t, --token TOKEN Rclone crypt token (obscured password)
331
+ -f, --force Insist on potentially dangerous actions (default: false)
332
+ -n, --dry-run Simulation mode with no on-disk modifications (default: false)
333
+ -v, --verbose Verbose operation (default: false)
334
+ -V, --version Show version
335
+ -h, --help print help
336
+ ```
337
+
338
+ ### File filtering
339
+
340
+ The Mclone passes its include and exclude options to the Rclone.
341
+ The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
342
+ documentation [section](https://rclone.org/filtering).
343
+
344
+ ### Dry run
345
+
346
+ The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
347
+ no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
348
+ The Rclone is run during task processing but in turn is supplied with this option.
349
+
350
+ ### Force mode
351
+
352
+ The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
353
+ or overwriting existing task.
354
+ In this case a `--force` command line option should be used to pass through.
355
+
356
+ ### Task operation modes
357
+
358
+ #### Update
359
+
360
+ * Copy source files which are newer than the destination's or have different size or checksum.
361
+
362
+ * Do not delete destination files which are nonexistent in the source.
363
+
364
+ * Do not copy source files which are older than the destination's.
365
+
366
+ A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
367
+
368
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
369
+
370
+ #### Synchronize
371
+
372
+ * Copy source files which are newer than the destination's or have different size or checksum.
373
+
374
+ * Delete destination files which are nonexistent in the source.
375
+
376
+ * Copy source files which are older than the destination's.
377
+
378
+ This is the mirroring mode which makes destination completely identical to the source.
379
+
380
+ Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
381
+
382
+ #### Copy
383
+
384
+ * Copy source files which are newer than the destination's or have different size or checksum.
385
+
386
+ * Do not delete destination files which are nonexistent in the source.
387
+
388
+ * Do not copy source files which are older than the destination's.
389
+
390
+ This mode is much like synchronize with only difference that it does not delete files.
391
+
392
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
393
+
394
+ #### Move
395
+
396
+ * Copy source files which are newer than the destination's or have different size or checksum.
397
+
398
+ * Do not delete destination files which are nonexistent in the source.
399
+
400
+ * Do not copy source files which are older than the destination's.
401
+
402
+ * Delete source files after successful copy to the destination.
403
+
404
+ Rclone [command](https://rclone.org/commands/rclone_move): `move`.
405
+
406
+ ## The end
407
+
408
+ Cheers,
409
+
370
410
  Oleg A. Khlybov <fougas@mail.ru>