rcloner 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ceb2a6a4709db5bbf91211533d0ab6d7d036f396d49fa4fe282f0418b191eca3
+  data.tar.gz: fc7b5aa17b7533b475d926f21cd4f66eccec39a5adc11687c78f2de8d39bc85e
+SHA512:
+  metadata.gz: a977239d78a49ac3c6d82127ad3951d5af983de183e111916a587a18141e90e1e219eb9ea6ca783d68d664d622678a52e6892145cdfb104a43c4ea6603777806
+  data.tar.gz: 8c1ccbcb343be1f852b574af0c59c08e0b6863f07b6567c4787bb46f3315ea32b8852b6f1d86ff8cf64f225fff910eade0a37513a21c9af8c44ca032993e08df

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+Gemfile.lock

data/Gemfile

@@ -0,0 +1,7 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in rcloner.gemspec
+gemspec
+
+gem "rake", "~> 12.0"
+gem "minitest", "~> 5.0"

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Victor Afanasev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,117 @@
+# Rcloner
+
+> README in progress. Project is on the early stage, use it at your own risk.
+
+Simple wrapper for Rclone (with optional Duplicity backend for Rclone) which allows to sync/restore your application files/database.
+
+## Installation
+
+**1)** [Install](https://rclone.org/install/) **rclone**
+
+**2)** [Install](http://duplicity.nongnu.org/) **duplicity**, minimal supported version is `0.8.09`. If you're using Ubuntu, the most simple way to install latest version is via snap: `$ sudo snap install duplicity --classic`
+
+**3)** Install gem **rcloner**:
+
+`$ gem install rcloner`
+
+Or you can install gem directly from github using [specific_install](https://github.com/rdp/specific_install):
+
+```bash
+$ gem install specific_install
+$ gem specific_install https://github.com/vifreefly/rcloner
+```
+
+Another option is to add gem to your application Gemfile:
+
+```ruby
+gem 'rcloner', git: 'https://github.com/vifreefly/rcloner', require: false
+```
+
+## Configuration
+
+First you will need to configure your Rclone `remote` storage.
+
+1. Install rclone
+2. Configure rclone remote storage (run `$ rclone config`), name it `remote`.
+
+## Usage
+
+Write a config file, example:
+
+```yml
+# rcloner.yml
+
+name: my_app
+root_path: /home/deploy/my_app
+items:
+  - type: pgdatabase
+    db_url: "postgres://my_app_username:my_app_userpass@localhost/my_app_database"
+
+  - type: folder
+    duplicity: false
+    path: public/images
+
+  - type: file
+    path: config/master.key
+```
+
+At the moment 3 types of items are supported:
+
+* `file` - sync a single file
+* `folder` - sync a directory. For `folder` type there is optional `duplicity` flag. If `duplicity: true`, folder will be synced using duplicity with compression. It's a good option if folder contains a lot of files which syncing each by each will take A LOT of time. Duplicity puts all the files in archive file before copying it to a remote storage.
+* `pgdatabase` - sunc application database (postgres only). [Postgressor](https://github.com/vifreefly/postgressor) gem is used under the hood.
+
+### backup
+
+To sync all items from local to remote rclone storage use `backup` command:
+
+```
+deploy@server:~/my_app$ rcloner backup --config rcloner.yml
+
+Dumped database my_app_database to /home/deploy/my_app/tmp/my_app_database.dump file.
+Synced file `my_app_database.dump` from `/home/deploy/my_app/tmp/my_app_database.dump` to `remote:my_app_backup/my_app_database.dump`
+
+Synced folder `images` from `/home/deploy/my_app/public/images` to `remote:my_app_backup/images`
+
+Synced file `master.key` from `/home/deploy/my_app/config/master.key` to `remote:my_app_backup/master.key`
+```
+
+> Note: to backup database, Rcloner use gem [postgressor](https://github.com/vifreefly/postgressor) under the hood.
+
+### restore
+
+To sync all items from remote rclone storage to local server use `restore` command:
+
+```
+deploy@server:~/my_app$ rcloner restore --config rcloner.yml
+
+Synced file `my_app_database.dump` from `remote:my_app_backup/my_app_database.dump` to `/home/deploy/my_app/tmp/my_app_database.dump`
+
+Synced folder `images` from `remote:my_app_backup/images` to `/home/deploy/my_app/public/images`
+
+Synced file `master.key` from `remote:my_app_backup/master.key` to `/home/deploy/my_app/config/master.key`
+```
+
+### Information about pgdatabase type restore
+
+If you want to automatically restore application database from a synced backup file, you need to provide additional `RESTORE_PGDATABASE=true` env variable.
+
+Also you can provide `SWITCH_TO_SUPERUSER=true` env variable to temporary switch postgres user to superuser while importing a database (sometimes it's required for a successful import). For more info see documentation for https://github.com/vifreefly/postgressor .
+
+Example:
+
+```
+deploy@server:~/my_app$ SWITCH_TO_SUPERUSER=true RESTORE_PGDATABASE=true rcloner restore --config rcloner.yml
+```
+
+## How to run backup with a cron
+
+Use Whenewer gem.
+
+## Notes
+
+* Rclone/duplicity integration https://github.com/GilGalaad/duplicity-rclone
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "rcloner"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/rcloner

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'rcloner'
+require 'rcloner/cli'
+
+Rcloner::CLI.start(ARGV)

data/lib/rcloner.rb

@@ -0,0 +1,6 @@
+require "rcloner/version"
+
+module Rcloner
+  class Error < StandardError; end
+  # Your code goes here...
+end

data/lib/rcloner/backuper.rb

@@ -0,0 +1,144 @@
+require 'uri'
+require 'dotenv'
+
+module Rcloner
+  class Backuper
+    def initialize(config)
+      @config = config
+      @project_folder = @config['name'] + '_backup'
+
+      # Create tmp/ folder insite root_path:
+      Dir.chdir(@config['root_path']) do
+        FileUtils.mkdir_p(File.join @config['root_path'], 'tmp')
+      end
+
+      if @config['items'].any? { |item| item['duplicity'] }
+        if ENV['PASSPHRASE'].nil? || ENV['PASSPHRASE'].empty?
+          raise "One of your items has duplicity backend but `PASSPHRASE` env variable is not set"
+        end
+      end
+    end
+
+    def backup!
+      sync_items(to: :remote)
+    end
+
+    def restore!
+      sync_items(to: :local)
+    end
+
+    private
+
+    def sync_items(to:)
+      @config['items'].each do |item|
+        case item['type']
+        when 'folder'
+          sync_folder(item, to: to)
+        when 'file'
+          sync_file(item, to: to)
+        when 'pgdatabase'
+          sync_pgdatabase(item, to: to)
+        end
+      end
+    end
+
+    def sync_pgdatabase(item, to:)
+      db_url =
+        if item['read_db_url_from_env']
+          Dir.chdir(@config['root_path']) do
+            `bundle exec postgressor print_db_url`.strip
+          end
+        else
+          item['db_url']
+        end
+
+      raise 'Cant read pgdatabase item db_url' unless db_url
+
+      db_backup_filename = URI.parse(db_url).path.sub('/', '') + '.dump'
+      relative_db_backup_filepath = 'tmp/' + db_backup_filename
+      local_db_backup_file_path = File.join(@config['root_path'], relative_db_backup_filepath)
+      item = { 'path' => relative_db_backup_filepath }
+
+      case to
+      when :remote
+        env = { 'DATABASE_URL' => db_url }
+        execute %W(bundle exec postgressor dumpdb #{local_db_backup_file_path}), env: env
+        sync_file(item, to: :remote)
+      when :local
+        sync_file(item, to: :local)
+
+        if ENV['RESTORE_PGDATABASE'] == 'true'
+          command = %W(bundle exec postgressor restoredb #{local_db_backup_file_path})
+          command << '--switch_to_superuser' if ENV['SWITCH_TO_SUPERUSER'] == 'true'
+
+          execute command, { 'DATABASE_URL' => db_url }
+        end
+      end
+    end
+
+    def sync_file(item, to:)
+      local_file_path = File.join(@config['root_path'], item['path'])
+      file_name = Pathname.new(local_file_path).basename.to_s
+      remote_file_path = File.join(@project_folder, file_name)
+
+      case to
+      when :remote
+        to_path = "remote:#{remote_file_path}"
+        from_path = local_file_path
+      when :local
+        to_path = local_file_path
+        from_path = "remote:#{remote_file_path}"
+      end
+
+      execute %W(rclone copyto #{from_path} #{to_path})
+      puts "Synced file `#{file_name}` from `#{from_path}` to `#{to_path}`"
+    end
+
+    def sync_folder(item, to:)
+      local_folder_path = File.join(@config['root_path'], item['path'])
+      folder_name = Pathname.new(local_folder_path).basename.to_s
+      remote_folder_path = File.join(@project_folder, folder_name)
+
+      case to
+      when :remote
+        to_path = "remote:#{remote_folder_path}"
+        from_path = local_folder_path
+      when :local
+        to_path = local_folder_path
+        from_path = "remote:#{remote_folder_path}"
+      end
+
+      execute %W(rclone mkdir #{to_path})
+
+      if item['duplicity']
+        case to
+        when :remote
+          dup_to_path = "rclone://remote:#{remote_folder_path}"
+          dup_from_path = local_folder_path
+          dup_command = %W(duplicity #{dup_from_path} #{dup_to_path})
+        when :local
+          dup_to_path = local_folder_path
+          dup_from_path = "rclone://remote:#{remote_folder_path}"
+          dup_command = %W(duplicity restore #{dup_from_path} #{dup_to_path} --force)
+        end
+
+        puts "Start syncing folder `#{folder_name}` from `#{from_path}` to `#{to_path}` using duplicity backend..."
+        execute dup_command
+      else
+        execute %W(rclone sync #{from_path} #{to_path})
+      end
+
+      puts "Synced folder `#{folder_name}` from `#{from_path}` to `#{to_path}`"
+    end
+
+    ###
+
+    def execute(command, env: {}, path: nil)
+      if path
+        system env, *command, chdir: path
+      else
+        system env, *command
+      end
+    end
+  end
+end

data/lib/rcloner/cli.rb

@@ -0,0 +1,40 @@
+require 'thor'
+require 'yaml'
+require_relative 'backuper'
+
+module Rcloner
+  class CLI < Thor
+    map %w[--version -v] => :__print_version
+
+    desc '--version, -v', 'Print the version'
+    def __print_version
+      puts VERSION
+    end
+
+    desc 'backup', 'Backup all items to a remote storage'
+    option :config, type: :string, required: true, banner: 'Path to a config file'
+    def backup
+      backuper = create_backuper
+      backuper.backup!
+    end
+
+    desc 'restore', 'Restore all items from a remote storage'
+    option :config, type: :string, required: true, banner: 'Path to a config file'
+    def restore
+      backuper = create_backuper
+      backuper.restore!
+    end
+
+    private
+
+    def create_backuper
+      config_path = options['config']
+      unless File.exists?(config_path)
+        raise "Config file `#{config_path}` does not exists"
+      end
+
+      config = YAML.load(File.read(config_path))
+      Backuper.new(config)
+    end
+  end
+end

data/lib/rcloner/version.rb

@@ -0,0 +1,3 @@
+module Rcloner
+  VERSION = "0.1.0"
+end

data/rcloner.gemspec

@@ -0,0 +1,32 @@
+require_relative 'lib/rcloner/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "rcloner"
+  spec.version       = Rcloner::VERSION
+  spec.authors       = ["Victor Afanasev"]
+  spec.email         = ["vicfreefly@gmail.com"]
+
+  spec.summary       = "Simple backup tool based on Rclone"
+  spec.homepage      = "https://github.com/vifreefly/rcloner"
+  spec.license       = "MIT"
+
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.4.0")
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  # spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here."
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+
+  spec.add_dependency "thor"
+  spec.add_dependency "postgressor", "~> 0.3.1"
+  spec.add_dependency "dotenv"
+
+  spec.bindir        = "exe"
+  spec.executables   = "rcloner"
+  spec.require_paths = ["lib"]
+end

metadata

@@ -0,0 +1,101 @@
+--- !ruby/object:Gem::Specification
+name: rcloner
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Victor Afanasev
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2020-05-11 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: postgressor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+- !ruby/object:Gem::Dependency
+  name: dotenv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- vicfreefly@gmail.com
+executables:
+- rcloner
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- exe/rcloner
+- lib/rcloner.rb
+- lib/rcloner/backuper.rb
+- lib/rcloner/cli.rb
+- lib/rcloner/version.rb
+- rcloner.gemspec
+homepage: https://github.com/vifreefly/rcloner
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/vifreefly/rcloner
+  source_code_uri: https://github.com/vifreefly/rcloner
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.4.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3
+signing_key: 
+specification_version: 4
+summary: Simple backup tool based on Rclone
+test_files: []