rcloner 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: ceb2a6a4709db5bbf91211533d0ab6d7d036f396d49fa4fe282f0418b191eca3
4
+ data.tar.gz: fc7b5aa17b7533b475d926f21cd4f66eccec39a5adc11687c78f2de8d39bc85e
5
+ SHA512:
6
+ metadata.gz: a977239d78a49ac3c6d82127ad3951d5af983de183e111916a587a18141e90e1e219eb9ea6ca783d68d664d622678a52e6892145cdfb104a43c4ea6603777806
7
+ data.tar.gz: 8c1ccbcb343be1f852b574af0c59c08e0b6863f07b6567c4787bb46f3315ea32b8852b6f1d86ff8cf64f225fff910eade0a37513a21c9af8c44ca032993e08df
data/.gitignore ADDED
@@ -0,0 +1,9 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /_yardoc/
4
+ /coverage/
5
+ /doc/
6
+ /pkg/
7
+ /spec/reports/
8
+ /tmp/
9
+ Gemfile.lock
data/Gemfile ADDED
@@ -0,0 +1,7 @@
1
+ source "https://rubygems.org"
2
+
3
+ # Specify your gem's dependencies in rcloner.gemspec
4
+ gemspec
5
+
6
+ gem "rake", "~> 12.0"
7
+ gem "minitest", "~> 5.0"
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2020 Victor Afanasev
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,117 @@
1
+ # Rcloner
2
+
3
+ > README in progress. Project is on the early stage, use it at your own risk.
4
+
5
+ Simple wrapper for Rclone (with optional Duplicity backend for Rclone) which allows to sync/restore your application files/database.
6
+
7
+ ## Installation
8
+
9
+ **1)** [Install](https://rclone.org/install/) **rclone**
10
+
11
+ **2)** [Install](http://duplicity.nongnu.org/) **duplicity**, minimal supported version is `0.8.09`. If you're using Ubuntu, the most simple way to install latest version is via snap: `$ sudo snap install duplicity --classic`
12
+
13
+ **3)** Install gem **rcloner**:
14
+
15
+ `$ gem install rcloner`
16
+
17
+ Or you can install gem directly from github using [specific_install](https://github.com/rdp/specific_install):
18
+
19
+ ```bash
20
+ $ gem install specific_install
21
+ $ gem specific_install https://github.com/vifreefly/rcloner
22
+ ```
23
+
24
+ Another option is to add gem to your application Gemfile:
25
+
26
+ ```ruby
27
+ gem 'rcloner', git: 'https://github.com/vifreefly/rcloner', require: false
28
+ ```
29
+
30
+ ## Configuration
31
+
32
+ First you will need to configure your Rclone `remote` storage.
33
+
34
+ 1. Install rclone
35
+ 2. Configure rclone remote storage (run `$ rclone config`), name it `remote`.
36
+
37
+ ## Usage
38
+
39
+ Write a config file, example:
40
+
41
+ ```yml
42
+ # rcloner.yml
43
+
44
+ name: my_app
45
+ root_path: /home/deploy/my_app
46
+ items:
47
+ - type: pgdatabase
48
+ db_url: "postgres://my_app_username:my_app_userpass@localhost/my_app_database"
49
+
50
+ - type: folder
51
+ duplicity: false
52
+ path: public/images
53
+
54
+ - type: file
55
+ path: config/master.key
56
+ ```
57
+
58
+ At the moment 3 types of items are supported:
59
+
60
+ * `file` - sync a single file
61
+ * `folder` - sync a directory. For `folder` type there is optional `duplicity` flag. If `duplicity: true`, folder will be synced using duplicity with compression. It's a good option if folder contains a lot of files which syncing each by each will take A LOT of time. Duplicity puts all the files in archive file before copying it to a remote storage.
62
+ * `pgdatabase` - sunc application database (postgres only). [Postgressor](https://github.com/vifreefly/postgressor) gem is used under the hood.
63
+
64
+ ### backup
65
+
66
+ To sync all items from local to remote rclone storage use `backup` command:
67
+
68
+ ```
69
+ deploy@server:~/my_app$ rcloner backup --config rcloner.yml
70
+
71
+ Dumped database my_app_database to /home/deploy/my_app/tmp/my_app_database.dump file.
72
+ Synced file `my_app_database.dump` from `/home/deploy/my_app/tmp/my_app_database.dump` to `remote:my_app_backup/my_app_database.dump`
73
+
74
+ Synced folder `images` from `/home/deploy/my_app/public/images` to `remote:my_app_backup/images`
75
+
76
+ Synced file `master.key` from `/home/deploy/my_app/config/master.key` to `remote:my_app_backup/master.key`
77
+ ```
78
+
79
+ > Note: to backup database, Rcloner use gem [postgressor](https://github.com/vifreefly/postgressor) under the hood.
80
+
81
+ ### restore
82
+
83
+ To sync all items from remote rclone storage to local server use `restore` command:
84
+
85
+ ```
86
+ deploy@server:~/my_app$ rcloner restore --config rcloner.yml
87
+
88
+ Synced file `my_app_database.dump` from `remote:my_app_backup/my_app_database.dump` to `/home/deploy/my_app/tmp/my_app_database.dump`
89
+
90
+ Synced folder `images` from `remote:my_app_backup/images` to `/home/deploy/my_app/public/images`
91
+
92
+ Synced file `master.key` from `remote:my_app_backup/master.key` to `/home/deploy/my_app/config/master.key`
93
+ ```
94
+
95
+ ### Information about pgdatabase type restore
96
+
97
+ If you want to automatically restore application database from a synced backup file, you need to provide additional `RESTORE_PGDATABASE=true` env variable.
98
+
99
+ Also you can provide `SWITCH_TO_SUPERUSER=true` env variable to temporary switch postgres user to superuser while importing a database (sometimes it's required for a successful import). For more info see documentation for https://github.com/vifreefly/postgressor .
100
+
101
+ Example:
102
+
103
+ ```
104
+ deploy@server:~/my_app$ SWITCH_TO_SUPERUSER=true RESTORE_PGDATABASE=true rcloner restore --config rcloner.yml
105
+ ```
106
+
107
+ ## How to run backup with a cron
108
+
109
+ Use Whenewer gem.
110
+
111
+ ## Notes
112
+
113
+ * Rclone/duplicity integration https://github.com/GilGalaad/duplicity-rclone
114
+
115
+ ## License
116
+
117
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
data/Rakefile ADDED
@@ -0,0 +1,10 @@
1
+ require "bundler/gem_tasks"
2
+ require "rake/testtask"
3
+
4
+ Rake::TestTask.new(:test) do |t|
5
+ t.libs << "test"
6
+ t.libs << "lib"
7
+ t.test_files = FileList["test/**/*_test.rb"]
8
+ end
9
+
10
+ task :default => :test
data/bin/console ADDED
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "rcloner"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ # (If you use this, don't forget to add pry to your Gemfile!)
10
+ # require "pry"
11
+ # Pry.start
12
+
13
+ require "irb"
14
+ IRB.start(__FILE__)
data/bin/setup ADDED
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ IFS=$'\n\t'
4
+ set -vx
5
+
6
+ bundle install
7
+
8
+ # Do any other automated setup that you need to do here
data/exe/rcloner ADDED
@@ -0,0 +1,6 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'rcloner'
4
+ require 'rcloner/cli'
5
+
6
+ Rcloner::CLI.start(ARGV)
data/lib/rcloner.rb ADDED
@@ -0,0 +1,6 @@
1
+ require "rcloner/version"
2
+
3
+ module Rcloner
4
+ class Error < StandardError; end
5
+ # Your code goes here...
6
+ end
@@ -0,0 +1,144 @@
1
+ require 'uri'
2
+ require 'dotenv'
3
+
4
+ module Rcloner
5
+ class Backuper
6
+ def initialize(config)
7
+ @config = config
8
+ @project_folder = @config['name'] + '_backup'
9
+
10
+ # Create tmp/ folder insite root_path:
11
+ Dir.chdir(@config['root_path']) do
12
+ FileUtils.mkdir_p(File.join @config['root_path'], 'tmp')
13
+ end
14
+
15
+ if @config['items'].any? { |item| item['duplicity'] }
16
+ if ENV['PASSPHRASE'].nil? || ENV['PASSPHRASE'].empty?
17
+ raise "One of your items has duplicity backend but `PASSPHRASE` env variable is not set"
18
+ end
19
+ end
20
+ end
21
+
22
+ def backup!
23
+ sync_items(to: :remote)
24
+ end
25
+
26
+ def restore!
27
+ sync_items(to: :local)
28
+ end
29
+
30
+ private
31
+
32
+ def sync_items(to:)
33
+ @config['items'].each do |item|
34
+ case item['type']
35
+ when 'folder'
36
+ sync_folder(item, to: to)
37
+ when 'file'
38
+ sync_file(item, to: to)
39
+ when 'pgdatabase'
40
+ sync_pgdatabase(item, to: to)
41
+ end
42
+ end
43
+ end
44
+
45
+ def sync_pgdatabase(item, to:)
46
+ db_url =
47
+ if item['read_db_url_from_env']
48
+ Dir.chdir(@config['root_path']) do
49
+ `bundle exec postgressor print_db_url`.strip
50
+ end
51
+ else
52
+ item['db_url']
53
+ end
54
+
55
+ raise 'Cant read pgdatabase item db_url' unless db_url
56
+
57
+ db_backup_filename = URI.parse(db_url).path.sub('/', '') + '.dump'
58
+ relative_db_backup_filepath = 'tmp/' + db_backup_filename
59
+ local_db_backup_file_path = File.join(@config['root_path'], relative_db_backup_filepath)
60
+ item = { 'path' => relative_db_backup_filepath }
61
+
62
+ case to
63
+ when :remote
64
+ env = { 'DATABASE_URL' => db_url }
65
+ execute %W(bundle exec postgressor dumpdb #{local_db_backup_file_path}), env: env
66
+ sync_file(item, to: :remote)
67
+ when :local
68
+ sync_file(item, to: :local)
69
+
70
+ if ENV['RESTORE_PGDATABASE'] == 'true'
71
+ command = %W(bundle exec postgressor restoredb #{local_db_backup_file_path})
72
+ command << '--switch_to_superuser' if ENV['SWITCH_TO_SUPERUSER'] == 'true'
73
+
74
+ execute command, { 'DATABASE_URL' => db_url }
75
+ end
76
+ end
77
+ end
78
+
79
+ def sync_file(item, to:)
80
+ local_file_path = File.join(@config['root_path'], item['path'])
81
+ file_name = Pathname.new(local_file_path).basename.to_s
82
+ remote_file_path = File.join(@project_folder, file_name)
83
+
84
+ case to
85
+ when :remote
86
+ to_path = "remote:#{remote_file_path}"
87
+ from_path = local_file_path
88
+ when :local
89
+ to_path = local_file_path
90
+ from_path = "remote:#{remote_file_path}"
91
+ end
92
+
93
+ execute %W(rclone copyto #{from_path} #{to_path})
94
+ puts "Synced file `#{file_name}` from `#{from_path}` to `#{to_path}`"
95
+ end
96
+
97
+ def sync_folder(item, to:)
98
+ local_folder_path = File.join(@config['root_path'], item['path'])
99
+ folder_name = Pathname.new(local_folder_path).basename.to_s
100
+ remote_folder_path = File.join(@project_folder, folder_name)
101
+
102
+ case to
103
+ when :remote
104
+ to_path = "remote:#{remote_folder_path}"
105
+ from_path = local_folder_path
106
+ when :local
107
+ to_path = local_folder_path
108
+ from_path = "remote:#{remote_folder_path}"
109
+ end
110
+
111
+ execute %W(rclone mkdir #{to_path})
112
+
113
+ if item['duplicity']
114
+ case to
115
+ when :remote
116
+ dup_to_path = "rclone://remote:#{remote_folder_path}"
117
+ dup_from_path = local_folder_path
118
+ dup_command = %W(duplicity #{dup_from_path} #{dup_to_path})
119
+ when :local
120
+ dup_to_path = local_folder_path
121
+ dup_from_path = "rclone://remote:#{remote_folder_path}"
122
+ dup_command = %W(duplicity restore #{dup_from_path} #{dup_to_path} --force)
123
+ end
124
+
125
+ puts "Start syncing folder `#{folder_name}` from `#{from_path}` to `#{to_path}` using duplicity backend..."
126
+ execute dup_command
127
+ else
128
+ execute %W(rclone sync #{from_path} #{to_path})
129
+ end
130
+
131
+ puts "Synced folder `#{folder_name}` from `#{from_path}` to `#{to_path}`"
132
+ end
133
+
134
+ ###
135
+
136
+ def execute(command, env: {}, path: nil)
137
+ if path
138
+ system env, *command, chdir: path
139
+ else
140
+ system env, *command
141
+ end
142
+ end
143
+ end
144
+ end
@@ -0,0 +1,40 @@
1
+ require 'thor'
2
+ require 'yaml'
3
+ require_relative 'backuper'
4
+
5
+ module Rcloner
6
+ class CLI < Thor
7
+ map %w[--version -v] => :__print_version
8
+
9
+ desc '--version, -v', 'Print the version'
10
+ def __print_version
11
+ puts VERSION
12
+ end
13
+
14
+ desc 'backup', 'Backup all items to a remote storage'
15
+ option :config, type: :string, required: true, banner: 'Path to a config file'
16
+ def backup
17
+ backuper = create_backuper
18
+ backuper.backup!
19
+ end
20
+
21
+ desc 'restore', 'Restore all items from a remote storage'
22
+ option :config, type: :string, required: true, banner: 'Path to a config file'
23
+ def restore
24
+ backuper = create_backuper
25
+ backuper.restore!
26
+ end
27
+
28
+ private
29
+
30
+ def create_backuper
31
+ config_path = options['config']
32
+ unless File.exists?(config_path)
33
+ raise "Config file `#{config_path}` does not exists"
34
+ end
35
+
36
+ config = YAML.load(File.read(config_path))
37
+ Backuper.new(config)
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,3 @@
1
+ module Rcloner
2
+ VERSION = "0.1.0"
3
+ end
data/rcloner.gemspec ADDED
@@ -0,0 +1,32 @@
1
+ require_relative 'lib/rcloner/version'
2
+
3
+ Gem::Specification.new do |spec|
4
+ spec.name = "rcloner"
5
+ spec.version = Rcloner::VERSION
6
+ spec.authors = ["Victor Afanasev"]
7
+ spec.email = ["vicfreefly@gmail.com"]
8
+
9
+ spec.summary = "Simple backup tool based on Rclone"
10
+ spec.homepage = "https://github.com/vifreefly/rcloner"
11
+ spec.license = "MIT"
12
+
13
+ spec.required_ruby_version = Gem::Requirement.new(">= 2.4.0")
14
+
15
+ spec.metadata["homepage_uri"] = spec.homepage
16
+ spec.metadata["source_code_uri"] = spec.homepage
17
+ # spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here."
18
+
19
+ # Specify which files should be added to the gem when it is released.
20
+ # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
21
+ spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
22
+ `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
23
+ end
24
+
25
+ spec.add_dependency "thor"
26
+ spec.add_dependency "postgressor", "~> 0.3.1"
27
+ spec.add_dependency "dotenv"
28
+
29
+ spec.bindir = "exe"
30
+ spec.executables = "rcloner"
31
+ spec.require_paths = ["lib"]
32
+ end
metadata ADDED
@@ -0,0 +1,101 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: rcloner
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Victor Afanasev
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2020-05-11 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: thor
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: postgressor
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: 0.3.1
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: 0.3.1
41
+ - !ruby/object:Gem::Dependency
42
+ name: dotenv
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ description:
56
+ email:
57
+ - vicfreefly@gmail.com
58
+ executables:
59
+ - rcloner
60
+ extensions: []
61
+ extra_rdoc_files: []
62
+ files:
63
+ - ".gitignore"
64
+ - Gemfile
65
+ - LICENSE.txt
66
+ - README.md
67
+ - Rakefile
68
+ - bin/console
69
+ - bin/setup
70
+ - exe/rcloner
71
+ - lib/rcloner.rb
72
+ - lib/rcloner/backuper.rb
73
+ - lib/rcloner/cli.rb
74
+ - lib/rcloner/version.rb
75
+ - rcloner.gemspec
76
+ homepage: https://github.com/vifreefly/rcloner
77
+ licenses:
78
+ - MIT
79
+ metadata:
80
+ homepage_uri: https://github.com/vifreefly/rcloner
81
+ source_code_uri: https://github.com/vifreefly/rcloner
82
+ post_install_message:
83
+ rdoc_options: []
84
+ require_paths:
85
+ - lib
86
+ required_ruby_version: !ruby/object:Gem::Requirement
87
+ requirements:
88
+ - - ">="
89
+ - !ruby/object:Gem::Version
90
+ version: 2.4.0
91
+ required_rubygems_version: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - ">="
94
+ - !ruby/object:Gem::Version
95
+ version: '0'
96
+ requirements: []
97
+ rubygems_version: 3.0.3
98
+ signing_key:
99
+ specification_version: 4
100
+ summary: Simple backup tool based on Rclone
101
+ test_files: []