mclone 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +370 -0
  3. data/bin/mclone +144 -0
  4. data/lib/mclone.rb +647 -0
  5. metadata +76 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 55aeb4153e4c0c1387458da22ac441d0a53dfb7778845f94ccc8e6c48ef699db
4
+ data.tar.gz: 32d565d1ea60267cd3da77ae03b408a8a42d02a7a6bf3ab6d4ff30ac9e9d3c3c
5
+ SHA512:
6
+ metadata.gz: 00eef197ed719146957579af92909974433514ebfeff86d6ca4f97cd1a9ff0969fbed7848a468166d1ca20049d014576bc2ddb9027154a91383aed9860c54cc7
7
+ data.tar.gz: 130f20d592e4c73f320e6e7680f4b100c840f06258eca96c28f7c036e230f305de760f908187b88a8b40cf36dc3834af93724798c7bdb0bfa1b4c017414b517f
data/README.md ADDED
@@ -0,0 +1,370 @@
1
+ # Mclone
2
+
3
+ [Mclone](https://github.com/okhlybov/mclone) is a utility for offline file synchronization utilizing the
4
+ [Rclone](https://rclone.org) as a backend for doing actual file transfer.
5
+
6
+ ## Purpose
7
+
8
+ Suppose you have a (large amount of) data which needs to be either distributed across many storages or simply backed up.
9
+ For example, consider a terabyte private media archive one can not afford to lose.
10
+
11
+ As the data gets periodically updated, there is a need for regular synchronization.
12
+ When the use of online cloud storage is not an option due storage space or security reasons, the good ol' offline
13
+ backing up comes back into play.
14
+
15
+ A sane backup strategy mandates the data copies to be physically separated - be it a next room (building, city or planet)
16
+ computer or just an external drive. Or, even better, the two computers' storages - a primary, where all activity takes place,
17
+ a mirror storage which holds the backup, and a portable storage (USB flash disc, exteral HDD or SSD - whatever) which
18
+ serves as both an intermediate storage and a means of propagating the changes between the primary and the mirror.
19
+
20
+ In a more complex scenario there may be multiple one-way or two-way point-to-point data transfer routes between the storages,
21
+ employing portable storage as a "shuttle" or a "ferry".
22
+
23
+ All in all the synchronization task boils down to copying or synchronizing the contents of two local directories. However,
24
+ since portable storage is involved, the actual file paths may change between synchronizations as a storage device can be
25
+ mounted under different mount points on *NIX system or change the disk drive on Windows system.
26
+
27
+ While the Rclone itself is a great tool for local file synchronization, typing the command line for execution in this case
28
+ becomes tedious and error prone where the possible cost of error is a backup corruption due to wrong paths or misspelled flags.
29
+
30
+ This is where the Mclone comes in.
31
+ It is designed to automatize the Rclone synchronization process by memorizing the command line options and detecting
32
+ the proper source and destination locations wherever they are.
33
+
34
+ ## Installation
35
+
36
+ Mclone is written in [Ruby](https://www.ruby-lang.org) language and is distributed in the form of the Ruby [GEM](https://rubygems.org).
37
+
38
+ Once the Ruby runtime is properly set, the Mclone itself is installed with
39
+
40
+ ```shell
41
+ $ gem install mclone
42
+ ```
43
+
44
+ Obviously, the Rclone installation is also required.
45
+ The Mclone will use either the contents of the `RCLONE` environment variable if exists or look though
46
+ the `PATH` environment variable to locate the `rclone` executable.
47
+
48
+ Once properly installed, the Mclone provides the `mclone` command line utility.
49
+
50
+ ```shell
51
+ $ mclone -h
52
+ ```
53
+ ## Basic use case
54
+
55
+ Let's start with the simplest case.
56
+
57
+ Suppose you have a data directory `/data` and you'd want to set up the backup of the `/data/files` subdirectory
58
+ into a backup directory `/mnt/backup`. The latter may be an ordinary directory or a mounted portable storage, or whatever.
59
+
60
+ ### 1. Format the Mclone volumes
61
+
62
+ Mclone has a notion of a volume - a file system directory containing the `.mclone` file, which is used as a root directory
63
+ for all Mclone operations.
64
+
65
+ By default, in order to detect currently available volumes the Mclone scans all mount points on *NIX systems and
66
+ all available disk drives on Windows system. Additionally, a static volume directories list to consider can be specified
67
+ in the `MCLONE_PATH` environment variable which is a PATH-like list of directories separated by the double colon `:`
68
+ on *NIX systems or the semicolon `;` on Windows system.
69
+
70
+ If the `/data` is a regular directory, it won't be picked up by the Mclone automatically, so it needs to be put into
71
+ the environment for later reuse
72
+
73
+ ```shell
74
+ export MCLONE_PATH=/data
75
+ ```
76
+
77
+ On the other hand, if the `/mnt/backup` is a mount point for a portable storage, it will be autodetected,
78
+ therefore there is no need to put it there.
79
+
80
+ Both source and destination endpoints have to "formatted" in order to be recognized as the Mclone volumes
81
+
82
+ ```shell
83
+ $ mclone volume create /data
84
+ $ mclone volume create /mnt/backup
85
+ ```
86
+
87
+ After that, `mclone info` can be used to review the recognized volumes
88
+
89
+ ```shell
90
+ $ mclone info
91
+
92
+ # Mclone version 0.1.0
93
+
94
+ ## Volumes
95
+
96
+ * [6bfa4a2d] :: (/data)
97
+ * [7443e311] :: (/mnt/backup)
98
+ ```
99
+
100
+ Each volume is identified by the randomly generated tag shown within the square brackets `[...]`.
101
+ _Obviously, the tags will be different in your case._
102
+
103
+ ### 2. Create the Mclone task
104
+
105
+ A Mclone task corresponds to a single Rclone command. It contains the source and destination volume identifiers,
106
+ the source and destination subdirectories _relative to the respective volumes_,
107
+ as well as additional Rclone command line arguments to be used.
108
+
109
+ _There can be multiple tasks linking different source and destination volumes as well as their respective subdirectores._
110
+
111
+ A task with all defaults is created with
112
+
113
+ ```shell
114
+ $ mclone task create /data/files /mnt/backup/files
115
+ ```
116
+
117
+ Note that at with point there is no need to use the above volume tags as they will be auto-determined during task creation.
118
+
119
+ Again, use the `mclone info` to review the changes
120
+
121
+ ```shell
122
+ # Mclone version 0.1.0
123
+
124
+ ## Volumes
125
+
126
+ * [6bfa4a2d] :: (/data)
127
+ * [7443e311] :: (/mnt/backup)
128
+
129
+ ## Intact tasks
130
+
131
+ * [cef63f5e] :: update [6bfa4a2d](files) -> [7443e311](files) :: include **
132
+ ```
133
+
134
+ The output literally means: ready to process (intact) update `cef63f5e` task from the `files` source subdirectory of the
135
+ `6bfa4a2d` volume to the `files` destination subdirectory of the `7443e311` volume
136
+ including `**` all files and subdirectories.
137
+
138
+ Again, the task's tag is randomly generated and will be different in your case.
139
+
140
+ There are two kinds of tasks to encounter - intact and stale.
141
+
142
+ An intact task is a task which is fully ready for processing with the Rclone.
143
+ As with the volumes, its tag is shown in the square brackets `[...]`
144
+
145
+ Conversely, a stale task is not ready for processing due to currently missing source or destination volume.
146
+ A stale task's tag is shown in the angle brackets `<...>`. Also, a missing stale task's volume tag will also be shown in
147
+ the angle brackets.
148
+
149
+ Thank to the indirection in the source and destination directories, **this task will be handled properly regardless of the
150
+ portable storage directory it will be mounted in next time provided that it will be detectable by the Mclone**.
151
+
152
+ The same applies to the Windows system where the portable storage can be appear as different disk drives and yet
153
+ be detectable by the Mclone.
154
+
155
+ ### 3. Modify the Mclone task
156
+
157
+ Once a task is created, its source and destination volumes and directories get fixed and can not be changed.
158
+ Therefore the only way to modify it is to start from scratch preceded by the task deletion with the `mclone task delete` command.
159
+
160
+ A task's optional parameters however can be modified afterwards with the `mclone task modify` command.
161
+
162
+ Suppose you'd want to change the operation mode from default updating to synchronization and exclude `.bak` files.
163
+
164
+ ```shell
165
+ $ mclone task modify -m sync -x '*.bak' cef
166
+ ```
167
+
168
+ This time the task is identified by its tag instead of a directory.
169
+
170
+ Note the mode and task's tag abbreviations: `synchronize` is reduced to `sync` (or it can be cut down further to `sy`)
171
+ and the tag is reduced from full `cef63f5e` to `cef` for convenience and type saving.
172
+ Any part of the full word can be used as an abbreviation provided it is unique among all other full words of the same kind
173
+ otherwise the Mclone will bail out with error.
174
+
175
+ The abbreviations are supported for operation mode, volume and task tags.
176
+
177
+ Behold the changes
178
+
179
+ ```shell
180
+ $ mclone info
181
+
182
+ # Mclone version 0.1.0
183
+
184
+ ## Volumes
185
+
186
+ * [6bfa4a2d] :: (/data)
187
+ * [7443e311] :: (/mnt/backup)
188
+
189
+ ## Intact tasks
190
+
191
+ * [cef63f5e] :: synchronize [6bfa4a2d](files) -> [7443e311](files) :: include ** :: exclude *.bak
192
+ ```
193
+
194
+ ### 4. Process the tasks
195
+
196
+ Once created all intact tasks can be (sequentially) processed with the `mclone task process` command.
197
+
198
+ ```shell
199
+ $ mclone task process
200
+ ```
201
+
202
+ If specific tasks need to be processed, their (possibly abbreviated) tags are specified as command line arguments
203
+
204
+
205
+ ```shell
206
+ $ mclone task process cef
207
+ ```
208
+
209
+ Technically, for a task to be processed the Mclone renders the full source and destination path names from the respective
210
+ volume locations and relative paths and passes them along with other options to the Rclone to do the actual processing.
211
+
212
+ Thats it. No more need to determine (and type in) current locations of the backup directory and retype all those Rclone arguments
213
+ for every occasion.
214
+
215
+ ## Advanced use case
216
+
217
+ Now back to the triple storage scenario outlined above.
218
+
219
+ Let **S** be a source storage from where the data needs to be backed up, **D** be a destination storage where the data is to
220
+ be mirrored and **P** be a portable storage which serves as both an intermediate storage and a means of the **S->D** data propagation.
221
+
222
+ In this case the full data propagation graph is **S->P->D**.
223
+
224
+ ### 1. Set up the S->P route
225
+
226
+ 1.1. Plug in the **P** portable storage to the **S**'s computer and mount it.
227
+
228
+ 1.2. As shown in the basic use case, create **S**'s and **P**'s volumes, then create a **S->P** task.
229
+
230
+ 1.3. Unplug **P**.
231
+
232
+ At this point **S** and **P** are now separated and each carry its own copy of the **S->P** task.
233
+
234
+ ### 2. Set up the P->D route
235
+
236
+ 2.1. Plug in the **P** portable storage to the **D**'s computer and mount it.
237
+
238
+ Note that at this point the **S->P** is a stale task as **D**'s computer knows nothing about **S** storage.
239
+
240
+ 2.2. Create the **D**'s volume, then create a **P->D** task.
241
+ Note that **P** at this point already contains a volume and therefore must not be formatted.
242
+
243
+ 2.3. Unplug **P**.
244
+
245
+ Now **S** and **D** are formatted and carry the respective tasks.
246
+ **P** contains its own copies of both **S->P** and **P->D** tasks.
247
+
248
+ ### 3. Process the **S->P->D** route
249
+
250
+ 3.1. Plug in **P** to the **S**'s computer and mount it.
251
+
252
+ 3.2. Process the intact tasks. In this case it is the **S->P** task (**P->D** is stale at this point).
253
+
254
+ 3.3. Unplug **P**.
255
+
256
+ **P** now carries its own copy of the **S**'s data.
257
+
258
+ 3.4. Plug in **P** to the **D**'s computer and mount it.
259
+
260
+ 3.5. Process the intact tasks. In this case it is the **P->D** task (**S->P** is stale at this point).
261
+
262
+ 3.6. Unplug **P**.
263
+
264
+ _Voilà!_
265
+ Both **P** and **D** now carry a copy of the **S**'s data.
266
+
267
+ There may be more complex data propagation scenarios with multiple source and destination storages utilizing the portable
268
+ storage in the above way.
269
+
270
+ Consider a two-way synchronization between two storages with a portable ferry which carries and propagates data in both directions.
271
+
272
+ ## Whats next
273
+
274
+ ### On-screen help
275
+
276
+ Every `mclone` (sub)command has its own help page which can be shown with `--help` option
277
+
278
+ ```shell
279
+ $ mclone task create --help
280
+
281
+ Usage:
282
+ mclone task create [OPTIONS] SOURCE DESTINATION
283
+
284
+ Parameters:
285
+ SOURCE Source path
286
+ DESTINATION Destination path
287
+
288
+ Options:
289
+ -m, --mode MODE Operation mode (update | synchronize | copy | move) (default: "update")
290
+ -i, --include PATTERN Include paths pattern (default: "**")
291
+ -x, --exclude PATTERN Exclude paths pattern
292
+ -f, --force Insist on potentially dangerous activities (default: false)
293
+ -n, --dry-run Simulation mode with no on-disk modifications (default: false)
294
+ -v, --verbose Verbose operation (default: false)
295
+ -h, --help print help
296
+ ```
297
+
298
+ ### File filtering
299
+
300
+ The Mclone passes its include and exclude options to the Rclone.
301
+ The pattern format is an extended glob (`*.dat`) format described in detail in the corresponding Rclone
302
+ documentation [section](https://rclone.org/filtering).
303
+
304
+ ### Dry run
305
+
306
+ The Mclone respects the Rclone's dry run mode activated with `--dry-run` command line option in which case
307
+ no volume (`.mclone`) files are ever touched (created, overwritten) during any operation.
308
+ The Rclone is run during task processing but in turn is supplied with this option.
309
+
310
+ ### Force mode
311
+
312
+ The Mclone will refuse to automatically perform certain actions which are considered dangerous, such as deleting a volume
313
+ or overwriting existing task.
314
+ In this case a `--force` command line option should be used to pass through.
315
+
316
+ ### Task operation modes
317
+
318
+ #### Update
319
+
320
+ * Copy source files which are newer than the destination's or have different size or checksum.
321
+
322
+ * Do not delete destination files which are nonexistent in the source.
323
+
324
+ * Do not copy source files which are older than the destination's.
325
+
326
+ A default refreshing mode which is considered to be least harmful with respect to the unintentional data override.
327
+
328
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy --update`.
329
+
330
+ #### Synchronize
331
+
332
+ * Copy source files which are newer than the destination's or have different size or checksum.
333
+
334
+ * Delete destination files which are nonexistent in the source.
335
+
336
+ * Copy source files which are older than the destination's.
337
+
338
+ This is the mirroring mode which makes destination completely identical to the source.
339
+
340
+ Rclone [command](https://rclone.org/commands/rclone_sync): `sync`.
341
+
342
+ #### Copy
343
+
344
+ * Copy source files which are newer than the destination's or have different size or checksum.
345
+
346
+ * Do not delete destination files which are nonexistent in the source.
347
+
348
+ * Do not copy source files which are older than the destination's.
349
+
350
+ This mode is much like synchronize with only difference that it does not delete files.
351
+
352
+ Rclone [command](https://rclone.org/commands/rclone_copy): `copy`.
353
+
354
+ #### Move
355
+
356
+ * Copy source files which are newer than the destination's or have different size or checksum.
357
+
358
+ * Do not delete destination files which are nonexistent in the source.
359
+
360
+ * Do not copy source files which are older than the destination's.
361
+
362
+ * Delete source files after successful copy to the destination.
363
+
364
+ Rclone [command](https://rclone.org/commands/rclone_move): `move`.
365
+
366
+ ## The end
367
+
368
+ Cheers,
369
+
370
+ Oleg A. Khlybov <fougas@mail.ru>
data/bin/mclone ADDED
@@ -0,0 +1,144 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ # frozen_string_literal: true
4
+
5
+ require 'clamp'
6
+ require 'mclone'
7
+
8
+ include Mclone
9
+
10
+ begin
11
+
12
+ Clamp do
13
+
14
+ using Refinements
15
+
16
+ option ['-f', '--force'], :flag, 'Insist on potentially dangerous activities', default: false
17
+ option ['-n', '--dry-run'], :flag, 'Simulation mode with no on-disk modifications', default: false
18
+ option ['-v', '--verbose'], :flag, 'Verbose operation', default: false
19
+
20
+ option ['-V', '--version'], :flag, 'Show version' do
21
+ puts VERSION
22
+ exit(true)
23
+ end
24
+
25
+ def session
26
+ session = Mclone::Session.new
27
+ session.force = force?
28
+ session.simulate = dry_run?
29
+ session.verbose = verbose?
30
+ session.restore_volumes!
31
+ session
32
+ end
33
+
34
+ def resolve_mode(mode)
35
+ case (m = Task::MODES.resolve(mode)).size
36
+ when 0 then raise(Task::Error, %(no modes matching pattern "#{mode}"))
37
+ when 1 then m.first
38
+ else raise(Task::Error, %(ambiguous mode pattern "#{mode}"))
39
+ end
40
+ end
41
+
42
+ subcommand 'info', 'Output information on volumes & tasks' do
43
+ def execute
44
+ s = session
45
+ $stdout.puts "# Mclone version #{Mclone::VERSION}"
46
+ $stdout.puts
47
+ $stdout.puts '## Volumes'
48
+ $stdout.puts
49
+ s.volumes.each do |volume|
50
+ $stdout.puts "* [#{volume.id}] :: (#{volume.root})"
51
+ end
52
+ stales = []
53
+ intacts = []
54
+ intact_tasks = s.intact_tasks
55
+ s.tasks.each do |task|
56
+ ts = (t = intact_tasks[task]).nil? ? "<#{task.id}>" : "[#{task.id}]"
57
+ svs = s.volumes.volume(task.source_id).nil? ? "<#{task.source_id}>" : "[#{task.source_id}]"
58
+ dvs = s.volumes.volume(task.destination_id).nil? ? "<#{task.destination_id}>" : "[#{task.destination_id}]"
59
+ xs = ["* #{ts} :: #{task.mode} #{svs}(#{task.source_root}) -> #{dvs}(#{task.destination_root})"]
60
+ xs << "include #{task.include}" unless task.include.nil? || task.include.empty?
61
+ xs << "exclude #{task.exclude}" unless task.exclude.nil? || task.exclude.empty?
62
+ (t.nil? ? stales : intacts) << xs.join(' :: ')
63
+ end
64
+ unless intacts.empty?
65
+ $stdout.puts
66
+ $stdout.puts '## Intact tasks'
67
+ $stdout.puts
68
+ intacts.each { |x| $stdout.puts x }
69
+ end
70
+ unless stales.empty?
71
+ $stdout.puts
72
+ $stdout.puts '## Stale tasks'
73
+ $stdout.puts
74
+ stales.each { |x| $stdout.puts x }
75
+ end
76
+ end
77
+ end
78
+
79
+ subcommand 'volume', 'Volume operations' do
80
+
81
+ subcommand ['new', 'create'], 'Create new volume' do
82
+ parameter 'DIRECTORY', 'Directory to become a Mclone volume'
83
+ def execute
84
+ session.format_volume!(directory)
85
+ end
86
+ end
87
+
88
+ subcommand 'delete', 'Delete existing volume' do
89
+ parameter 'VOLUME', 'Volume ID pattern'
90
+ def execute
91
+ session.delete_volume!(volume).commit!
92
+ end
93
+ end
94
+
95
+ end
96
+
97
+ subcommand ['task'], 'Task operations' do
98
+
99
+ def self.set_task_opts
100
+ modes = Task::MODES.collect(&:to_s).join(' | ')
101
+ option ['-m', '--mode'], 'MODE', "Operation mode (#{modes})", default: Task::MODES.first.to_s
102
+ option ['-i', '--include'], 'PATTERN', 'Include paths pattern', default: '**'
103
+ option ['-x', '--exclude'], 'PATTERN', 'Exclude paths pattern'
104
+ end
105
+
106
+ subcommand ['new', 'create'], 'Create new SOURCE -> DESTINATION task' do
107
+ set_task_opts
108
+ parameter 'SOURCE', 'Source path'
109
+ parameter 'DESTINATION', 'Destination path'
110
+ def execute
111
+ session.create_task!(source, destination, mode: resolve_mode(mode), include: include, exclude: exclude).commit!
112
+ end
113
+ end
114
+
115
+ subcommand 'modify', 'Modify existing task' do
116
+ set_task_opts
117
+ parameter 'TASK', 'Task ID pattern'
118
+ def execute
119
+ session.modify_task!(task, mode: resolve_mode(mode), include: include, exclude: exclude).commit!
120
+ end
121
+ end
122
+
123
+ subcommand 'delete', 'Delete existing task' do
124
+ parameter 'TASK', 'Task ID pattern'
125
+ def execute
126
+ session.delete_task!(task).commit!
127
+ end
128
+ end
129
+
130
+ subcommand 'process', 'Process specified tasks' do
131
+ parameter '[TASK] ...', 'Task ID pattern(s)', attribute_name: :tasks
132
+ def execute
133
+ session.process_tasks!(*tasks)
134
+ end
135
+ end
136
+
137
+ end
138
+
139
+ end
140
+
141
+ rescue Mclone::Error
142
+ $stderr.puts "ERROR: #{$!.message}"
143
+ exit(false)
144
+ end
data/lib/mclone.rb ADDED
@@ -0,0 +1,647 @@
1
+ # frozen_string_literal: true
2
+
3
+
4
+ require 'date'
5
+ require 'json'
6
+ require 'fileutils'
7
+ require 'securerandom'
8
+
9
+
10
+ #
11
+ module Mclone
12
+
13
+
14
+ VERSION = '0.1.0'
15
+
16
+
17
+ #
18
+ class Error < StandardError
19
+
20
+ end
21
+
22
+ #
23
+ module Refinements
24
+
25
+ refine ::Hash do
26
+ # Same as #dig but raises KeyError exception on any non-existent key
27
+ def extract(*args)
28
+ case args.size
29
+ when 0 then raise(KeyError, 'non-empty key sequence expected')
30
+ when 1 then fetch(args.first)
31
+ else fetch(args.shift).extract(*args)
32
+ end
33
+ end
34
+ end
35
+
36
+ refine ::Array do
37
+ # Return a list of items which fully or partially match the specified pattern
38
+ def resolve(partial)
39
+ rx = Regexp.new(partial)
40
+ collect { |item| rx.match?(item.to_s) ? item : nil }.compact
41
+ end
42
+ end
43
+
44
+ end
45
+
46
+
47
+ using Refinements
48
+
49
+
50
+ # Two-way mapping between an object and its ID
51
+ class ObjectSet
52
+
53
+ include Enumerable
54
+
55
+ #
56
+ def each_id(&code)
57
+ @ids.each_key(&code)
58
+ end
59
+
60
+ #
61
+ def each(&code)
62
+ @objects.each_value(&code)
63
+ end
64
+
65
+ #
66
+ def empty?
67
+ @objects.empty?
68
+ end
69
+
70
+ def initialize
71
+ @ids = {} # { id => object }
72
+ @objects = {} # { object => object }
73
+ @modified = false
74
+ end
75
+
76
+ # Return ID of the object considered equal to the specified obj or nil
77
+ def id(obj)
78
+ @objects[obj]&.id
79
+ end
80
+
81
+ # Return object with specified ID or nil
82
+ def object(id)
83
+ @ids[id]
84
+ end
85
+
86
+ # Return object considered equal to obj or nil
87
+ def [](obj)
88
+ @objects[obj]
89
+ end
90
+
91
+ #
92
+ def modified?
93
+ @modified
94
+ end
95
+
96
+ def commit!
97
+ @modified = false
98
+ self
99
+ end
100
+
101
+ # Unregister an object considered equal to the specified obj and return true if object has been actually removed
102
+ private def forget(obj)
103
+ !@ids.delete(@objects.delete(obj)&.id).nil?
104
+ end
105
+
106
+ # Return a list of registered IDs (fully or partially) matching the specified pattern
107
+ def resolve(pattern)
108
+ each_id.to_a.resolve(pattern)
109
+ end
110
+
111
+ # Either add brand new object or replace existing one equal to the specified object
112
+ def <<(obj)
113
+ forget(obj)
114
+ @objects[obj] = @ids[obj.id] = obj
115
+ @modified = true
116
+ obj
117
+ end
118
+
119
+ # Remove object considered equal to the specified obj
120
+ def >>(obj)
121
+ @modified = true if (status = forget(obj))
122
+ status
123
+ end
124
+
125
+ # Add all tasks from enumerable
126
+ def merge!(objs)
127
+ objs.each { |x| self << x }
128
+ self
129
+ end
130
+
131
+ end
132
+
133
+
134
+ #
135
+ class Task
136
+
137
+ #
138
+ class Error < Mclone::Error
139
+ end
140
+
141
+ #
142
+ attr_reader :id
143
+
144
+ #
145
+ attr_reader :source_id, :destination_id
146
+
147
+ #
148
+ attr_reader :source_root, :destination_root
149
+
150
+ #
151
+ attr_reader :mtime
152
+
153
+ #
154
+ attr_reader :mode
155
+
156
+ #
157
+ attr_reader :include, :exclude
158
+
159
+ def hash
160
+ @hash ||= source_id.hash ^ destination_id.hash ^ source_root.hash ^ destination_root.hash
161
+ end
162
+
163
+ def eql?(other)
164
+ equal?(other) || (
165
+ source_id == other.source_id &&
166
+ destination_id == other.destination_id &&
167
+ source_root == other.source_root &&
168
+ destination_root == other.destination_root
169
+ )
170
+ end
171
+
172
+ alias == eql?
173
+
174
+ #
175
+ def initialize(source_id, source_root, destination_id, destination_root)
176
+ @touch = false # Indicates that the time stamp should be updated whenever state of self is altered
177
+ @id = SecureRandom.hex(4)
178
+ @source_id = source_id
179
+ @destination_id = destination_id
180
+ @source_root = source_root
181
+ @destination_root = destination_root
182
+ self.mode = :update
183
+ self.include = '**'
184
+ self.exclude = ''
185
+ ensure
186
+ @touch = true
187
+ touch!
188
+ end
189
+
190
+ #
191
+ MODES = [:update, :synchronize, :copy, :move].freeze
192
+
193
+ #
194
+ def mode=(mode)
195
+ unless mode.nil?
196
+ @mode = MODES.include?(mode = mode.intern) ? mode : raise(Task::Error, %(unknown mode "#{mode}"))
197
+ touch!
198
+ end
199
+ end
200
+
201
+ #
202
+ def include=(mask)
203
+ unless mask.nil?
204
+ @include = mask # TODO extensive verification
205
+ touch!
206
+ end
207
+ end
208
+
209
+ #
210
+ def exclude=(mask)
211
+ unless mask.nil?
212
+ @exclude = mask # TODO extensive verification
213
+ touch!
214
+ end
215
+ end
216
+
217
+ #
218
+ def self.restore(hash)
219
+ obj = allocate
220
+ obj.send(:from_h, hash)
221
+ obj
222
+ end
223
+
224
+ #
225
+ private def from_h(hash)
226
+ initialize(hash.extract(:source, :volume), hash.extract(:source, :root), hash.extract(:destination, :volume), hash.extract(:destination, :root))
227
+ @touch = false
228
+ @id = hash.extract(:task)
229
+ self.mode = hash.extract(:mode)
230
+ # Deleting the mtime key from json
231
+ @mtime = DateTime.parse(hash.extract(:mtime)) rescue @mtime
232
+ self.include = hash.extract(:include) rescue nil
233
+ self.exclude = hash.extract(:exclude) rescue nil
234
+ ensure
235
+ @touch = true
236
+ end
237
+
238
+ #
239
+ def to_h
240
+ {
241
+ mode: mode,
242
+ include: include,
243
+ exclude: exclude,
244
+ task: id,
245
+ source: {volume: source_id, root: source_root},
246
+ destination: {volume: destination_id, root: destination_root},
247
+ mtime: mtime
248
+ }
249
+ end
250
+
251
+ #
252
+ def touch!
253
+ @mtime = DateTime.now if @touch
254
+ end
255
+ end
256
+
257
+
258
+ #
259
+ class TaskSet < ObjectSet
260
+
261
+ alias task object
262
+
263
+ # Add new task or replace existing one with outdated timestamp
264
+ def <<(task)
265
+ t = self[task]
266
+ super if t.nil? || (!t.nil? && t.mtime < task.mtime)
267
+ task
268
+ end
269
+
270
+ #
271
+ def resolve(id)
272
+ case (ids = super).size
273
+ when 0
274
+ raise(Task::Error, %(no task matching "#{id}" pattern found))
275
+ when 1
276
+ ids.first
277
+ else
278
+ raise(Task::Error, %(ambiguous "#{id}" pattern: two or more tasks match))
279
+ end
280
+ end
281
+
282
+ end
283
+
284
+
285
+ #
286
+ class Volume
287
+
288
+ #
289
+ class Error < Mclone::Error
290
+
291
+ end
292
+
293
+ #
294
+ VERSION = 0
295
+
296
+ #
297
+ FILE = '.mclone'
298
+
299
+ #
300
+ attr_reader :id
301
+
302
+ #
303
+ attr_reader :file
304
+
305
+ #
306
+ attr_reader :tasks
307
+
308
+ #
309
+ def root
310
+ @root ||= File.realpath(File.dirname(file))
311
+ end
312
+
313
+ #
314
+ def initialize(file)
315
+ @file = file
316
+ @id = SecureRandom.hex(4)
317
+ @tasks = VolumeTaskSet.new(self)
318
+ end
319
+
320
+ #
321
+ def self.restore(file)
322
+ obj = allocate
323
+ obj.send(:from_file, file)
324
+ obj
325
+ end
326
+
327
+ #
328
+ private def from_file(file)
329
+ initialize(file)
330
+ hash = JSON.parse(IO.read(file), symbolize_names: true)
331
+ raise(Volume::Error, %(unsupported Mclone volume format version "#{version}")) unless hash.extract(:mclone) == VERSION
332
+ @id = hash.extract(:volume).to_s
333
+ hash.extract(:tasks).each { |t| @tasks << Task.restore(t) }
334
+ end
335
+
336
+ #
337
+ def dirty?
338
+ tasks.modified?
339
+ end
340
+
341
+ #
342
+ def hash
343
+ id.hash
344
+ end
345
+
346
+ #
347
+ def eql?(other)
348
+ equal?(other) || id == other.id
349
+ end
350
+
351
+ #
352
+ def commit!(force = false)
353
+ if force || dirty?
354
+ open(file, 'w') do |stream|
355
+ stream << JSON.pretty_generate(to_h)
356
+ tasks.commit!
357
+ end
358
+ end
359
+ end
360
+
361
+ #
362
+ def to_h
363
+ {mclone: VERSION, volume: id, tasks: tasks.collect(&:to_h)}
364
+ end
365
+
366
+ # Volume-bound set of tasks belonging to the specific volume
367
+ class VolumeTaskSet < Mclone::TaskSet
368
+
369
+ def initialize(volume)
370
+ @volume = volume
371
+ super()
372
+ end
373
+
374
+ # Accept only the tasks referencing the volume as either source or destination
375
+ def <<(task)
376
+ task.source_id == @volume.id || task.destination_id == @volume.id ? super : task
377
+ end
378
+
379
+ end
380
+ end
381
+
382
+
383
+ #
384
+ class VolumeSet < ObjectSet
385
+
386
+ alias volume object
387
+
388
+ #
389
+ def resolve(id)
390
+ case (ids = super).size
391
+ when 0
392
+ raise(Volume::Error, %(no volume matching "#{id}" pattern found))
393
+ when 1
394
+ ids.first
395
+ else
396
+ raise(Volume::Error, %(ambiguous "#{id}" pattern: two or more volumes match))
397
+ end
398
+ end
399
+
400
+ end
401
+
402
+
403
+ #
404
+ class Session
405
+
406
+ #
407
+ class Error < Mclone::Error
408
+
409
+ end
410
+
411
+ #
412
+ attr_reader :volumes
413
+
414
+ #
415
+ def simulate?
416
+ @simulate == true
417
+ end
418
+
419
+ #
420
+ def verbose?
421
+ @verbose == true
422
+ end
423
+
424
+ #
425
+ def force?
426
+ @force == true
427
+ end
428
+
429
+ #
430
+ attr_writer :simulate, :verbose, :force
431
+
432
+ #
433
+ def initialize
434
+ @volumes = VolumeSet.new
435
+ end
436
+
437
+ #
438
+ def format_volume!(dir)
439
+ mclone = File.join(dir, Volume::FILE)
440
+ raise(Session::Error, %(refuse to overwrite existing Mclone volume file "#{mclone}")) if File.exist?(mclone) && !force?
441
+ volume = Volume.new(mclone)
442
+ @volumes << volume
443
+ volume.commit!(true) unless simulate? # Force creating a new (empty) volume
444
+ self
445
+ end
446
+
447
+ #
448
+ def restore_volume!(dir)
449
+ volume = Volume.restore(File.join(dir, Volume::FILE))
450
+ @volumes << volume
451
+ self
452
+ end
453
+
454
+ #
455
+ def restore_volumes!
456
+ (Mclone.environment_mounts + Mclone.system_mounts).each { |dir| restore_volume!(dir) rescue Errno::ENOENT }
457
+ self
458
+ end
459
+
460
+ #
461
+ def delete_volume!(id)
462
+ volume = volumes.volume(id = volumes.resolve(id))
463
+ raise(Session::Error, %(refuse to delete non-empty Mclone volume file "#{volume.file}")) unless volume.tasks.empty? || force?
464
+ volumes >> volume
465
+ FileUtils.rm_f(volume.file) unless simulate?
466
+ self
467
+ end
468
+
469
+ #
470
+ def create_task!(source, destination, mode: :update, include: '**', exclude: '')
471
+ task = Task.new(*locate(source), *locate(destination))
472
+ task.mode = mode
473
+ task.include = include
474
+ task.exclude = exclude
475
+ volumes.each do |volume|
476
+ t = volume.tasks[task]
477
+ raise(Session::Error, %(refuse to overwrite existing task "#{t.id}")) unless t.nil? || force?
478
+ volume.tasks << task # It is a volume's responsibility to collect appropriate tasks, see Volume::TaskSet#<<
479
+ end
480
+ self
481
+ end
482
+
483
+ #
484
+ def modify_task!(id, mode:, include:, exclude:)
485
+ ts = tasks
486
+ task = ts.task(ts.resolve(id)).clone
487
+ task.mode = mode
488
+ task.include = include
489
+ task.exclude = exclude
490
+ volumes.each { |volume| volume.tasks << task }
491
+ self
492
+ end
493
+
494
+ #
495
+ def delete_task!(id)
496
+ ts = tasks
497
+ task = ts.task(ts.resolve(id))
498
+ volumes.each { |volume| volume.tasks >> task }
499
+ self
500
+ end
501
+
502
+ #
503
+ def process_tasks!(*ids)
504
+ ts = intact_tasks
505
+ ids = ts.collect(&:id) if ids.empty?
506
+ rclone = 'rclone' if (rclone = ENV['RCLONE']).nil?
507
+ ids.collect { |id| ts.task(ts.resolve(id)) }.each do |task|
508
+ args = [rclone]
509
+ opts = [
510
+ simulate? ? '--dry-run' : nil,
511
+ verbose? ? '--verbose' : nil
512
+ ].compact
513
+ case task.mode
514
+ when :update
515
+ args << 'copy'
516
+ opts << '--update'
517
+ when :synchronize
518
+ args << 'sync'
519
+ when :copy
520
+ args << 'copy'
521
+ when :move
522
+ args << 'move'
523
+ end
524
+ opts.append('--filter', "- /#{Volume::FILE}")
525
+ opts.append('--filter', "- #{task.exclude}") unless task.exclude.nil? || task.exclude.empty?
526
+ opts.append('--filter', "+ #{task.include}") unless task.include.nil? || task.include.empty?
527
+ args.concat(opts)
528
+ args.append(File.join(volumes.volume(task.source_id).root, task.source_root), File.join(volumes.volume(task.destination_id).root, task.destination_root))
529
+ case system(*args)
530
+ when nil then raise(Session::Error, %(failed to execute "#{args.first}"))
531
+ when false then exit($?)
532
+ end
533
+ end
534
+ end
535
+
536
+ # Collect all tasks from all loaded volumes
537
+ def tasks
538
+ tasks = SessionTaskSet.new(self)
539
+ volumes.each { |volume| tasks.merge!(volume.tasks) }
540
+ tasks
541
+ end
542
+
543
+ # Collect all tasks from all loaded volumes which are ready to be executed
544
+ def intact_tasks
545
+ tasks = IntactTaskSet.new(self)
546
+ volumes.each { |volume| tasks.merge!(volume.tasks) }
547
+ tasks
548
+ end
549
+
550
+ #
551
+ private def locate(path)
552
+ path = File.realpath(path)
553
+ x = volumes.each.collect { |v| Regexp.new(%!^#{v.root}/?(.*)!, Mclone.windows? ? Regexp::IGNORECASE : nil) =~ path ? [v.root, v.id, $1] : nil }.compact
554
+ if x.empty?
555
+ raise(Session::Error, %(path "#{path}" does not belong to a loaded Mclone volume))
556
+ else
557
+ root, volume, path = x.sort { |a,b| a.first.size <=> b.first.size}.last
558
+ [volume, path]
559
+ end
560
+ end
561
+
562
+ #
563
+ def commit!
564
+ volumes.each { |v| v.commit!(force?) } unless simulate?
565
+ self
566
+ end
567
+
568
+ #
569
+ class SessionTaskSet < Mclone::TaskSet
570
+
571
+ def initialize(session)
572
+ @session = session
573
+ super()
574
+ end
575
+
576
+ end
577
+
578
+ # Session-bound set of intact tasks for which both source and destination volumes are loaded
579
+ class IntactTaskSet < SessionTaskSet
580
+
581
+ # Accept only intact tasks for which both source and destination volumes are loaded
582
+ def <<(task)
583
+ @session.volumes.volume(task.source_id).nil? || @session.volumes.volume(task.destination_id).nil? ? task : super
584
+ end
585
+
586
+ end
587
+
588
+ end
589
+
590
+ # Return true if run in the real Windows environment (e.g. not in real *NIX or various emulation layers such as MSYS, Cygwin etc.)
591
+ def self.windows?
592
+ @@windows ||= /^(mingw)/.match?(RbConfig::CONFIG['target_os']) # RubyInstaller's MRI, other MinGW-build MRI
593
+ end
594
+
595
+ # Match OS-specific system mount points (/dev /proc etc.) which normally should be omitted when scanning for Mclone voulmes
596
+ UNIX_SYSTEM_MOUNTS = %r!^/(dev|sys|proc|run)!
597
+
598
+ # TODO handle Windows variants
599
+ # Specify OS-specific path name list separator (such as in the $PATH environment variable)
600
+ PATH_LIST_SEPARATOR = windows? ? ';' : ':'
601
+
602
+ # Return list of live user-provided mounts (mount points on *NIX and disk drives on Windows) which may contain Mclone volumes
603
+ # Look for the $mclone_PATH environment variable
604
+ def self.environment_mounts
605
+ ENV['MCLONE_PATH'].split(PATH_LIST_SEPARATOR).collect { |path| File.directory?(path) ? path : nil }.compact rescue []
606
+ end
607
+
608
+ # Return list of live system-managed mounts (mount points on *NIX and disk drives on Windows) which may contain Mclone volumes
609
+ case RbConfig::CONFIG['target_os']
610
+ when 'linux'
611
+ # Linux OS
612
+ def self.system_mounts
613
+ # Query on /proc for currently mounted file systems
614
+ IO.readlines('/proc/self/mountstats').collect do |line|
615
+ mount = line.split[4]
616
+ UNIX_SYSTEM_MOUNTS.match?(mount) || !File.directory?(mount) ? nil : mount
617
+ end.compact
618
+ end
619
+ # TODO handle Windows variants
620
+ when /^mingw/ # RubyInstaller's MRI
621
+ require 'ffi'
622
+ module FileAPI
623
+ extend FFI::Library
624
+ ffi_lib 'kernel32'
625
+ ffi_convention :stdcall
626
+ attach_function :disks_mask, :GetLogicalDrives, [], :ulong
627
+ end
628
+ def self.system_mounts
629
+ mask = FileAPI.disks_mask
630
+ mounts = []
631
+ ('A'..'Z').each do |x|
632
+ mounts << "#{x}:" if mask & 1 == 1
633
+ mask >>= 1
634
+ end
635
+ mounts
636
+ end
637
+ else
638
+ # Generic *NIX-like OS, including Cygwin & MSYS(2)
639
+ def self.system_mounts
640
+ # Use $(mount) system utility to obtain currently mounted file systems
641
+ %x(mount).split("\n").collect do |line|
642
+ mount = line.split[2]
643
+ UNIX_SYSTEM_MOUNTS.match?(mount) || !File.directory?(mount) ? nil : mount
644
+ end.compact
645
+ end
646
+ end
647
+ end
metadata ADDED
@@ -0,0 +1,76 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: mclone
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Oleg A. Khlybov
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2021-06-22 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: clamp
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.3'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.3'
27
+ - !ruby/object:Gem::Dependency
28
+ name: ffi
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '1.15'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '1.15'
41
+ description:
42
+ email:
43
+ - fougas@mail.ru
44
+ executables:
45
+ - mclone
46
+ extensions: []
47
+ extra_rdoc_files:
48
+ - README.md
49
+ files:
50
+ - README.md
51
+ - bin/mclone
52
+ - lib/mclone.rb
53
+ homepage: https://github.com/okhlybov/mclone
54
+ licenses:
55
+ - BSD-3-Clause
56
+ metadata: {}
57
+ post_install_message:
58
+ rdoc_options: []
59
+ require_paths:
60
+ - lib
61
+ required_ruby_version: !ruby/object:Gem::Requirement
62
+ requirements:
63
+ - - ">="
64
+ - !ruby/object:Gem::Version
65
+ version: 2.5.0
66
+ required_rubygems_version: !ruby/object:Gem::Requirement
67
+ requirements:
68
+ - - ">="
69
+ - !ruby/object:Gem::Version
70
+ version: '0'
71
+ requirements: []
72
+ rubygems_version: 3.0.9
73
+ signing_key:
74
+ specification_version: 4
75
+ summary: Rclone frontend for offline synchronization
76
+ test_files: []