mclone 0.1.1 → 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (6) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGES.md +8 -4
  3. data/README.md +409 -369
  4. data/bin/mclone +23 -8
  5. data/lib/mclone.rb +223 -126
  6. metadata +3 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c89cb8e633183fd633ff44b2e87fdc73c5cd348bb377e61f77d12cde4d389e61
4
- data.tar.gz: 1844a2921e78e5cc2057b0aa906494e978883ffda60a487f1881743fbe5598cd
3
+ metadata.gz: cb414361ce06f3977f87735131a0c8de7d392b469e55b6c49554b2ac1766baae
4
+ data.tar.gz: 60691ed2d154162a68c8a9262bf7ae90af6023a8bcf0148776d1ed2d00dfa987
5
5
  SHA512:
6
- metadata.gz: dec529f91301c96fd64b839aa4fdcd4cb733f076d8e56ab506ab86ebe0ec075821623d5e5bf8e91a871d72498b2a77db70b5328837764f9d95987b2a24a62150
7
- data.tar.gz: fac7f8a99b14bdacbe1048d7ba5e7d325863677bf3149a6f2c2a23665996ec75790caed9b51dde2260300c4afd838795c05134284b3f738c94c268d601108c21
6
+ metadata.gz: e2c1cbfa056b6ca56bd88c2b7abe93c550a6f1c616d2eb93c98037c3197afc60ae30e7ca060997836ea5ae003da278c86a04989d6c1cef22abd04c93cb8d3e98
7
+ data.tar.gz: 395f81ed736663c5309eb52a5d1b63f10c7a546e360bd2827038a6dd6eb7b7881fa770b70a386d51b86acb95bcbf9d5422bdda990319c9c670d2e3f2aa66535e
data/CHANGES.md CHANGED
@@ -1,5 +1,9 @@
1
- # 0.1.1 (2021-06-24)
2
-
3
- * Switch from FFI to Fiddle.
4
-
1
+ # 0.2.0 (2021-08-08)
2
+
3
+ * Support for encryption.
4
+
5
+ # 0.1.1 (2021-06-24)
6
+
7
+ * Switch from FFI to Fiddle.
8
+
5
9
  # 0.1.0 (2021-06-22)
data/README.md CHANGED
@@ -1,370 +1,410 @@
1
- # Mclone
2
-
3
- [Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
4
- [Rclone](https://rclone.org) as a backend for doing actual file transfer.
5
-
6
- ## Purpose
7
-
8
- Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
9
- For example, consider a terabyte private media archive one can not afford to lose.
10
-
11
- As the data gets periodically updated, there is a need for regular synchronization.
12
- When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
13
- backing up comes back into play.
14
-
15
- A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
16
- computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
17
- a mirror storage which holds the backup, and a portable storage (USB flash disc, exteral HDD or SSD - whatever) which
18
- serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
19
-
20
- In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
21
- employing portable storage as a "shuttle" or a "ferry".
22
-
23
- All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
24
- since portable storage is involved, the actual file paths may change between synchronizations as a storage device can be
25
- mounted under different mount points on *NIX system or change the disk drive on Windows system.
26
-
27
- While the Rclone itself is a great tool for local file synchronization, typing the command line for execution in this case
28
- becomes tedious and error prone where the possible cost of error is a backup corruption due to wrong paths or misspelled flags.
29
-
30
- This is where the Mclone comes in.
31
- It is designed to automatize the Rclone synchronization process by memorizing the command line options and detecting
32
- the proper source and destination locations wherever they are.
33
-
34
- ## Installation
35
-
36
- Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
37
-
38
- Once the Ruby runtime is properly set, the Mclone itself is installed with
39
-
40
- ```shell
41
- $ gem install mclone
42
- ```
43
-
44
- Obviously, the Rclone installation is also required.
45
- The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
46
- the `PATH` environment variable to locate the `rclone` executable.
47
-
48
- Once properly installed, the Mclone provides the `mclone` command line utility.
49
-
50
- ```shell
51
- $ mclone -h
52
- ```
53
- ## Basic use case
54
-
55
- Let's start with the simplest case.
56
-
57
- Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
58
- into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
59
-
60
- ### 1. Create volumes
61
-
62
- Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
63
- for all Mclone operations.
64
-
65
- By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
66
- all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
67
- in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
68
- on *NIX systems or the semicolon `;` on Windows system.
69
-
70
- If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
71
- the environment for later reuse
72
-
73
- ```shell
74
- export MCLONE_PATH=/data
75
- ```
76
-
77
- On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
78
- therefore there is no need to put it there.
79
-
80
- Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
81
-
82
- ```shell
83
- $ mclone volume create /data
84
- $ mclone volume create /mnt/backup
85
- ```
86
-
87
- After that, `mclone info` can be used to review the recognized volumes
88
-
89
- ```shell
90
- $ mclone info
91
-
92
- # Mclone version 0.1.0
93
-
94
- ## Volumes
95
-
96
- * [6bfa4a2d] :: (/data)
97
- * [7443e311] :: (/mnt/backup)
98
- ```
99
-
100
- Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
101
- _Obviously, the tags will be different in your case._
102
-
103
- ### 2. Create a task
104
-
105
- A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
106
- the source and destination subdirectories _relative to the respective volumes_,
107
- as well as additional Rclone command line arguments to be used.
108
-
109
- _There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
110
-
111
- A task with all defaults is created with
112
-
113
- ```shell
114
- $ mclone task create /data/files /mnt/backup/files
115
- ```
116
-
117
- Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
118
-
119
- Again, use the `mclone info` to review the changes
120
-
121
- ```shell
122
- # Mclone version 0.1.0
123
-
124
- ## Volumes
125
-
126
- * [6bfa4a2d] :: (/data)
127
- * [7443e311] :: (/mnt/backup)
128
-
129
- ## Intact tasks
130
-
131
- * [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
132
- ```
133
-
134
- The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
135
- `6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
136
- including `**` all files and subdirectories.
137
-
138
- Again, the task's tag is randomly generated and will be different in your case.
139
-
140
- There are two kinds of tasks to encounter - intact and stale.
141
-
142
- An intact task is a task which is fully ready for processing with the Rclone.
143
- As with the volumes, its tag is shown in the square brackets `[...]`
144
-
145
- Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
146
- A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
147
- the angle brackets.
148
-
149
- Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
150
- portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
151
-
152
- The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
153
- be detectable by the Mclone.
154
-
155
- ### 3. Modify the task
156
-
157
- Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
158
- Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
159
-
160
- A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
161
-
162
- Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
163
-
164
- ```shell
165
- $ mclone task modify -m sync -x '*.bak' cef
166
- ```
167
-
168
- This time the task is identified by its tag instead of a directory.
169
-
170
- Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
171
- and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
172
- Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
173
- otherwise the Mclone will bail out with error.
174
-
175
- The abbreviations are supported for operation mode, volume and task tags.
176
-
177
- Behold the changes
178
-
179
- ```shell
180
- $ mclone info
181
-
182
- # Mclone version 0.1.0
183
-
184
- ## Volumes
185
-
186
- * [6bfa4a2d] :: (/data)
187
- * [7443e311] :: (/mnt/backup)
188
-
189
- ## Intact tasks
190
-
191
- * [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
192
- ```
193
-
194
- ### 4. Process the tasks
195
-
196
- Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
197
-
198
- ```shell
199
- $ mclone task process
200
- ```
201
-
202
- If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
203
-
204
-
205
- ```shell
206
- $ mclone task process cef
207
- ```
208
-
209
- Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
210
- volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
211
-
212
- Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
213
- for every occasion.
214
-
215
- ## Advanced use case
216
-
217
- Now back to the triple storage scenario outlined above.
218
-
219
- Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
220
- be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
221
-
222
- In this case the full data propagation graph is **S->P->D**.
223
-
224
- ### 1. Set up the S->P route
225
-
226
- 1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
227
-
228
- 1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
229
-
230
- 1.3. Unplug **P**.
231
-
232
- At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
233
-
234
- ### 2. Set up the P->D route
235
-
236
- 2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
237
-
238
- Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
239
-
240
- 2.2. Create the **D**'s volume, then create a **P->D** task.
241
- Note that **P** at this point already contains a volume and therefore must not be formatted.
242
-
243
- 2.3. Unplug **P**.
244
-
245
- Now **S** and **D** are formatted and carry the respective tasks.
246
- **P** contains its own copies of both **S->P** and **P->D** tasks.
247
-
248
- ### 3. Process the **S->P->D** route
249
-
250
- 3.1. Plug in **P** to the **S**'s computer and mount it.
251
-
252
- 3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
253
-
254
- 3.3. Unplug **P**.
255
-
256
- **P** now carries its own copy of the **S**'s data.
257
-
258
- 3.4. Plug in **P** to the **D**'s computer and mount it.
259
-
260
- 3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
261
-
262
- 3.6. Unplug **P**.
263
-
264
- _Voilà!_
265
- Both **P** and **D** now carry a copy of the **S**'s data.
266
-
267
- There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
268
- storage in the above way.
269
-
270
- Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
271
-
272
- ## Whats next
273
-
274
- ### On-screen help
275
-
276
- Every `mclone` (sub)command has its own help page which can be shown with `--help` option
277
-
278
- ```shell
279
- $ mclone task create --help
280
-
281
- Usage:
282
- mclone task create [OPTIONS] SOURCE DESTINATION
283
-
284
- Parameters:
285
- SOURCE Source path
286
- DESTINATION Destination path
287
-
288
- Options:
289
- -m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
290
- -i, --include PATTERN Include paths pattern (default: "**")
291
- -x, --exclude PATTERN Exclude paths pattern
292
- -f, --force Insist on potentially dangerous activities (default: false)
293
- -n, --dry-run Simulation mode with no on-disk modifications (default: false)
294
- -v, --verbose Verbose operation (default: false)
295
- -h, --help print help
296
- ```
297
-
298
- ### File filtering
299
-
300
- The Mclone passes its include and exclude options to the Rclone.
301
- The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
302
- documentation [section](https://rclone.org/filtering).
303
-
304
- ### Dry run
305
-
306
- The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
307
- no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
308
- The Rclone is run during task processing but in turn is supplied with this option.
309
-
310
- ### Force mode
311
-
312
- The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
313
- or overwriting existing task.
314
- In this case a `--force` command line option should be used to pass through.
315
-
316
- ### Task operation modes
317
-
318
- #### Update
319
-
320
- * Copy source files which are newer than the destination's or have different size or checksum.
321
-
322
- * Do not delete destination files which are nonexistent in the source.
323
-
324
- * Do not copy source files which are older than the destination's.
325
-
326
- A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
327
-
328
- Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
329
-
330
- #### Synchronize
331
-
332
- * Copy source files which are newer than the destination's or have different size or checksum.
333
-
334
- * Delete destination files which are nonexistent in the source.
335
-
336
- * Copy source files which are older than the destination's.
337
-
338
- This is the mirroring mode which makes destination completely identical to the source.
339
-
340
- Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
341
-
342
- #### Copy
343
-
344
- * Copy source files which are newer than the destination's or have different size or checksum.
345
-
346
- * Do not delete destination files which are nonexistent in the source.
347
-
348
- * Do not copy source files which are older than the destination's.
349
-
350
- This mode is much like synchronize with only difference that it does not delete files.
351
-
352
- Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
353
-
354
- #### Move
355
-
356
- * Copy source files which are newer than the destination's or have different size or checksum.
357
-
358
- * Do not delete destination files which are nonexistent in the source.
359
-
360
- * Do not copy source files which are older than the destination's.
361
-
362
- * Delete source files after successful copy to the destination.
363
-
364
- Rclone [command](https://rclone.org/commands/rclone_move): `move`.
365
-
366
- ## The end
367
-
368
- Cheers,
369
-
1
+ # Mclone
2
+
3
+ [Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
4
+ [Rclone](https://rclone.org) as a backend for doing actual file transfer.
5
+
6
+ ## Purpose
7
+
8
+ Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
9
+ For example, consider a terabyte private media archive one can not afford to lose.
10
+
11
+ As the data gets periodically updated, there is a need for regular synchronization.
12
+ When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
13
+ backing up comes back into play.
14
+
15
+ A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
16
+ computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
17
+ a mirror storage which holds the backup, and a portable storage (USB flash disc, external HDD or SSD - whatever) which
18
+ serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
19
+
20
+ In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
21
+ employing portable storage as a "shuttle" or a "ferry".
22
+
23
+ All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
24
+ since portable storage is involved, the actual file paths may change between synchronizations as a storage can be
25
+ mounted under different mount points on *NIX system or change the disk drive on Windows system.
26
+
27
+ While the Rclone itself is a great tool for local file synchronization, typing command line to be executed in this case
28
+ becomes tedious and error prone where possible cost of error is a backup corruption due to wrong paths or misspelled flags.
29
+
30
+ This is where the Mclone comes in.
31
+ It is designed to automatize the Rclone synchronization process by memorizing command line options and detecting
32
+ proper source and destination locations wherever they are.
33
+
34
+ ## Installation
35
+
36
+ Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
37
+
38
+ Once the Ruby runtime is properly set, the Mclone itself is installed with
39
+
40
+ ```shell
41
+ $ gem install mclone
42
+ ```
43
+
44
+ Obviously, the Rclone installation is also required.
45
+ The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
46
+ the `PATH` environment variable to locate the `rclone` executable.
47
+
48
+ Once properly installed, the Mclone provides the `mclone` command line utility.
49
+
50
+ ```shell
51
+ $ mclone -h
52
+ ```
53
+ ## Basic use case
54
+
55
+ Let's start with the simplest case.
56
+
57
+ Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
58
+ into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
59
+
60
+ ### 1. Create volumes
61
+
62
+ Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
63
+ for all Mclone operations.
64
+
65
+ By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
66
+ all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
67
+ in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
68
+ on *NIX systems or the semicolon `;` on Windows system.
69
+
70
+ If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
71
+ the environment for later reuse
72
+
73
+ ```shell
74
+ export MCLONE_PATH=/data
75
+ ```
76
+
77
+ On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
78
+ therefore there is no need to put it there.
79
+
80
+ Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
81
+
82
+ ```shell
83
+ $ mclone volume create /data
84
+ $ mclone volume create /mnt/backup
85
+ ```
86
+
87
+ After that, `mclone info` can be used to review the recognized volumes
88
+
89
+ ```shell
90
+ $ mclone info
91
+
92
+ # Mclone version 0.1.0
93
+
94
+ ## Volumes
95
+
96
+ * [6bfa4a2d] :: (/data)
97
+ * [7443e311] :: (/mnt/backup)
98
+ ```
99
+
100
+ Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
101
+ _Obviously, the tags will be different in your case._
102
+
103
+ ### 2. Create a task
104
+
105
+ A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
106
+ the source and destination subdirectories _relative to the respective volumes_,
107
+ as well as additional Rclone command line arguments to be used.
108
+
109
+ _There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
110
+
111
+ A task with all defaults is created with
112
+
113
+ ```shell
114
+ $ mclone task create /data/files /mnt/backup/files
115
+ ```
116
+
117
+ Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
118
+
119
+ Again, use the `mclone info` to review the changes
120
+
121
+ ```shell
122
+ # Mclone version 0.1.0
123
+
124
+ ## Volumes
125
+
126
+ * [6bfa4a2d] :: (/data)
127
+ * [7443e311] :: (/mnt/backup)
128
+
129
+ ## Intact tasks
130
+
131
+ * [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
132
+ ```
133
+
134
+ The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
135
+ `6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
136
+ including `**` all files and subdirectories.
137
+
138
+ Again, the task's tag is randomly generated and will be different in your case.
139
+
140
+ There are two kinds of tasks to encounter - intact and stale.
141
+
142
+ An intact task is a task which is fully ready for processing with the Rclone.
143
+ As with the volumes, its tag is shown in the square brackets `[...]`
144
+
145
+ Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
146
+ A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
147
+ the angle brackets.
148
+
149
+ Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
150
+ portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
151
+
152
+ The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
153
+ be detectable by the Mclone.
154
+
155
+ ### 3. Modify the task
156
+
157
+ Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
158
+ Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
159
+
160
+ A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
161
+
162
+ Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
163
+
164
+ ```shell
165
+ $ mclone task modify -m sync -x '*.bak' cef
166
+ ```
167
+
168
+ This time the task is identified by its tag instead of a directory.
169
+
170
+ Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
171
+ and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
172
+ Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
173
+ otherwise the Mclone will bail out with error.
174
+
175
+ The abbreviations are supported for operation mode, volume and task tags.
176
+
177
+ Behold the changes
178
+
179
+ ```shell
180
+ $ mclone info
181
+
182
+ # Mclone version 0.1.0
183
+
184
+ ## Volumes
185
+
186
+ * [6bfa4a2d] :: (/data)
187
+ * [7443e311] :: (/mnt/backup)
188
+
189
+ ## Intact tasks
190
+
191
+ * [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
192
+ ```
193
+
194
+ ### 4. Process the tasks
195
+
196
+ Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
197
+
198
+ ```shell
199
+ $ mclone task process
200
+ ```
201
+
202
+ If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
203
+
204
+
205
+ ```shell
206
+ $ mclone task process cef
207
+ ```
208
+
209
+ Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
210
+ volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
211
+
212
+ Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
213
+ for every occasion.
214
+
215
+ ## Advanced use case
216
+
217
+ Now back to the triple storage scenario outlined above.
218
+
219
+ Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
220
+ be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
221
+
222
+ In this case the full data propagation graph is **S->P->D**.
223
+
224
+ ### 1. Set up the S->P route
225
+
226
+ 1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
227
+
228
+ 1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
229
+
230
+ 1.3. Unplug **P**.
231
+
232
+ At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
233
+
234
+ ### 2. Set up the P->D route
235
+
236
+ 2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
237
+
238
+ Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
239
+
240
+ 2.2. Create the **D**'s volume, then create a **P->D** task.
241
+ Note that **P** at this point already contains a volume and therefore must not be formatted.
242
+
243
+ 2.3. Unplug **P**.
244
+
245
+ Now **S** and **D** are formatted and carry the respective tasks.
246
+ **P** contains its own copies of both **S->P** and **P->D** tasks.
247
+
248
+ ### 3. Process the **S->P->D** route
249
+
250
+ 3.1. Plug in **P** to the **S**'s computer and mount it.
251
+
252
+ 3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
253
+
254
+ 3.3. Unplug **P**.
255
+
256
+ **P** now carries its own copy of the **S**'s data.
257
+
258
+ 3.4. Plug in **P** to the **D**'s computer and mount it.
259
+
260
+ 3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
261
+
262
+ 3.6. Unplug **P**.
263
+
264
+ _Voilà!_
265
+ Both **P** and **D** now carry a copy of the **S**'s data.
266
+
267
+ There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
268
+ storage in the above way.
269
+
270
+ Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
271
+
272
+ ## Encryption
273
+
274
+ Encryption is an essential part of the Mclone as it is all about handling portable storage which may by compromised
275
+ while holding confidential data. Mclone fully relies on [encryption capabilities](https://rclone.org/crypt/) of Rclone,
276
+ that is an encrypted directory structure can be further treated with the Rclone itself.
277
+
278
+ The encryption operation in Mclone is activated during task creation time.
279
+ The encryption mode is activated with `-e` or `-d` command line flag for encryption or decryption, respectively.
280
+ It no either flag is specified, the encryption gets turned off.
281
+
282
+ When in encryption mode, Mclone recursively takes plain files and directories under the source root and creates encrypted
283
+ files and directories under the destination root.
284
+ Conversely, when in decryption mode, Mclone takes encrypted source root and decrypts it into the destination root.
285
+ Mclone is set up to encrypt not only the files' contents but also the file and directory names themselves. The file sizes,
286
+ modification times as well as some other metadata are not encrypted, though, as they are required for proper operation
287
+ the file synchronization mechanism.
288
+ Note that the encrypted root is a regular directory hierarchy (just with fancy file names) and thus can be treated as such.
289
+
290
+ ***Be wary that file name encryption has a serious implication on the file name length.***
291
+ The Rclone [crypt](https://rclone.org/crypt/) documentation states the the individual file or directory name length can
292
+ not exceed ~143 charactes (although ***bytes*** here would be more correct).
293
+ As the Rclone accepts UTF-8 encoded names, this estimate generally holds true for the Latin charset only, where
294
+ a character is encoded with a single byte.
295
+ For non-Latin characters, which can be encoded with two or even more bytes, the maximum allowed name length be
296
+ much lower.
297
+ When Rclone encounters a file name too long to hold, it will refuse to process it.
298
+
299
+ Rclone employs symmetric cryptography to do its job, which requires some form of password to be supplied upon task
300
+ creation.
301
+ This is done by the `-p` command line flag, which specifies a plain text password used to derive the real encryption key.
302
+ There is another password-related `-t` command line flag which can be used to directly specify
303
+ an [Rclone-obscured](https://rclone.org/commands/rclone_obscure/) token.
304
+ Once created, a task memorizes the encryption key on the ***unencrypted end*** of the source/destination volume pair,
305
+ so there will be no need to pass it during the task processing.
306
+
307
+ ## Whats next
308
+
309
+ ### On-screen help
310
+
311
+ Every `mclone` (sub)command has its own help page which can be shown with `--help` option
312
+
313
+ ```shell
314
+ $ mclone task create --help
315
+
316
+ Usage:
317
+ mclone task new [OPTIONS] SOURCE DESTINATION
318
+
319
+ Parameters:
320
+ SOURCE Source path
321
+ DESTINATION Destination path
322
+
323
+ Options:
324
+ -m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
325
+ -i, --include PATTERN Include paths pattern
326
+ -x, --exclude PATTERN Exclude paths pattern
327
+ -d, --decrypt Decrypt source
328
+ -e, --encrypt Encrypt destination
329
+ -p, --password PASSWORD Plain text password
330
+ -t, --token TOKEN Rclone crypt token (obscured password)
331
+ -f, --force Insist on potentially dangerous actions (default: false)
332
+ -n, --dry-run Simulation mode with no on-disk modifications (default: false)
333
+ -v, --verbose Verbose operation (default: false)
334
+ -V, --version Show version
335
+ -h, --help print help
336
+ ```
337
+
338
+ ### File filtering
339
+
340
+ The Mclone passes its include and exclude options to the Rclone.
341
+ The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
342
+ documentation [section](https://rclone.org/filtering).
343
+
344
+ ### Dry run
345
+
346
+ The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
347
+ no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
348
+ The Rclone is run during task processing but in turn is supplied with this option.
349
+
350
+ ### Force mode
351
+
352
+ The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
353
+ or overwriting existing task.
354
+ In this case a `--force` command line option should be used to pass through.
355
+
356
+ ### Task operation modes
357
+
358
+ #### Update
359
+
360
+ * Copy source files which are newer than the destination's or have different size or checksum.
361
+
362
+ * Do not delete destination files which are nonexistent in the source.
363
+
364
+ * Do not copy source files which are older than the destination's.
365
+
366
+ A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
367
+
368
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
369
+
370
+ #### Synchronize
371
+
372
+ * Copy source files which are newer than the destination's or have different size or checksum.
373
+
374
+ * Delete destination files which are nonexistent in the source.
375
+
376
+ * Copy source files which are older than the destination's.
377
+
378
+ This is the mirroring mode which makes destination completely identical to the source.
379
+
380
+ Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
381
+
382
+ #### Copy
383
+
384
+ * Copy source files which are newer than the destination's or have different size or checksum.
385
+
386
+ * Do not delete destination files which are nonexistent in the source.
387
+
388
+ * Do not copy source files which are older than the destination's.
389
+
390
+ This mode is much like synchronize with only difference that it does not delete files.
391
+
392
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
393
+
394
+ #### Move
395
+
396
+ * Copy source files which are newer than the destination's or have different size or checksum.
397
+
398
+ * Do not delete destination files which are nonexistent in the source.
399
+
400
+ * Do not copy source files which are older than the destination's.
401
+
402
+ * Delete source files after successful copy to the destination.
403
+
404
+ Rclone [command](https://rclone.org/commands/rclone_move): `move`.
405
+
406
+ ## The end
407
+
408
+ Cheers,
409
+
370
410
  Oleg A. Khlybov <fougas@mail.ru>