rcloner 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ceb2a6a4709db5bbf91211533d0ab6d7d036f396d49fa4fe282f0418b191eca3
-  data.tar.gz: fc7b5aa17b7533b475d926f21cd4f66eccec39a5adc11687c78f2de8d39bc85e
+  metadata.gz: 6c1f7494bc312030d905a1cbec89e8ffd1dd89434a6ad10401d8c3c23e695354
+  data.tar.gz: b8b88229daf6d237b5a69024bcc18180f2032096c45f5d7351513acd219a9f05
 SHA512:
-  metadata.gz: a977239d78a49ac3c6d82127ad3951d5af983de183e111916a587a18141e90e1e219eb9ea6ca783d68d664d622678a52e6892145cdfb104a43c4ea6603777806
-  data.tar.gz: 8c1ccbcb343be1f852b574af0c59c08e0b6863f07b6567c4787bb46f3315ea32b8852b6f1d86ff8cf64f225fff910eade0a37513a21c9af8c44ca032993e08df
+  metadata.gz: 8fdadf6c7f5a78dce6f9bf256512d1c85893f773c68f5616b7c922c8d1f219423248eb748b6b2b6d8c87e6823cac4b90c5ba152988c8b8bef077da0ca10fea1c
+  data.tar.gz: 06e8093496e5e0a01df644cd25c33b6eae661da09b42540ee49d91f68fe2f071ffba228b146ab5bf58748f2b4828d75e7c792f1956a74c91cc7e2ed97ff15569

data/README.md

@@ -2,17 +2,50 @@
 
 > README in progress. Project is on the early stage, use it at your own risk.
 
-Simple wrapper for Rclone (with optional Duplicity backend for Rclone) which allows to sync/restore your application files/database.
+Simple wrapper for Duplicity (with optional Rclone backend) which allows to sync/restore your application files/database from a remote storage. All files stored on a remote storage are encrypted and backups are incremental, thanks to Duplicity.
+
+Example config (`rcloner.yml`)
+
+```yml
+origin: /home/deploy/myapp
+destination: webdavs://login:pass@webdav.yandex.ru/myapp_backup
+include:
+  - 'public/images'
+  - 'config/master.key'
+  - '.env'
+  - 'tmp/db.dump'
+on_backup:
+  before: 'postgressor dumpdb tmp/db.dump'
+  after: 'rm tmp/db.dump'
+```
+
+Where:
+  * `origin` is the root folder path if your application. If not provided - current directory will be taken. Can be also provided as `RCLONER_ORIGIN` env variable.
+  * `destination` - remote path in dyplicity format. Can be also provided as env variable `RCLONER_DESTINATION`.
+  * `include` if provided - only this paths will be taken for backup, not the whole `origin`. Urls should be provided in relative paths to `origin` folder.
+  * `on_backup` (`before`/`after`) custom optional commands/scripts to execute before/after backup. Commands will be executed into `origin` directory.
+
+Note that rcloner by default loads env variables from .env file (if exists) in current category.
 
 ## Installation
 
-**1)** [Install](https://rclone.org/install/) **rclone**
+**1)** [Install](https://rclone.org/install/) **rclone**:
 
-**2)** [Install](http://duplicity.nongnu.org/) **duplicity**, minimal supported version is `0.8.09`. If you're using Ubuntu, the most simple way to install latest version is via snap: `$ sudo snap install duplicity --classic`
+```bash
+$ curl https://rclone.org/install.sh | sudo bash
+```
+
+**2)** [Install](http://duplicity.nongnu.org/) **duplicity**, minimal supported version is `0.8.09`. If you're using Ubuntu, the most simple way to install latest version is via snap:
+
+```bash
+$ sudo snap install duplicity --classic
+```
 
 **3)** Install gem **rcloner**:
 
-`$ gem install rcloner`
+```bash
+$ gem install rcloner
+```
 
 Or you can install gem directly from github using [specific_install](https://github.com/rdp/specific_install):
 
@@ -27,16 +60,24 @@ Another option is to add gem to your application Gemfile:
 gem 'rcloner', git: 'https://github.com/vifreefly/rcloner', require: false
 ```
 
+<!-- **4) Install gem postgressor (optional for postgres database)**:
+
+```bash
+$ gem install postgressor
+```
+
 ## Configuration
 
 First you will need to configure your Rclone `remote` storage.
 
 1. Install rclone
-2. Configure rclone remote storage (run `$ rclone config`), name it `remote`.
+2. Configure rclone remote storage (run `$ rclone config`), name it `remote`. -->
 
 ## Usage
 
-Write a config file, example:
+Readme in progress...
+
+<!-- Write a config file, example:
 
 ```yml
 # rcloner.yml
@@ -59,7 +100,7 @@ At the moment 3 types of items are supported:
 
 * `file` - sync a single file
 * `folder` - sync a directory. For `folder` type there is optional `duplicity` flag. If `duplicity: true`, folder will be synced using duplicity with compression. It's a good option if folder contains a lot of files which syncing each by each will take A LOT of time. Duplicity puts all the files in archive file before copying it to a remote storage.
-* `pgdatabase` - sunc application database (postgres only). [Postgressor](https://github.com/vifreefly/postgressor) gem is used under the hood.
+* `pgdatabase` - sunc application database (postgres only). [Postgressor](https://github.com/vifreefly/postgressor) gem is used under the hood. -->
 
 ### backup
 
@@ -67,7 +108,8 @@ To sync all items from local to remote rclone storage use `backup` command:
 
 ```
 deploy@server:~/my_app$ rcloner backup --config rcloner.yml
-
+```
+<!--
 Dumped database my_app_database to /home/deploy/my_app/tmp/my_app_database.dump file.
 Synced file `my_app_database.dump` from `/home/deploy/my_app/tmp/my_app_database.dump` to `remote:my_app_backup/my_app_database.dump`
 
@@ -76,7 +118,7 @@ Synced folder `images` from `/home/deploy/my_app/public/images` to `remote:my_ap
 Synced file `master.key` from `/home/deploy/my_app/config/master.key` to `remote:my_app_backup/master.key`
 ```
 
-> Note: to backup database, Rcloner use gem [postgressor](https://github.com/vifreefly/postgressor) under the hood.
+> Note: to backup database, Rcloner use gem [postgressor](https://github.com/vifreefly/postgressor) under the hood. -->
 
 ### restore
 
@@ -84,7 +126,9 @@ To sync all items from remote rclone storage to local server use `restore` comma
 
 ```
 deploy@server:~/my_app$ rcloner restore --config rcloner.yml
+```
 
+<!--
 Synced file `my_app_database.dump` from `remote:my_app_backup/my_app_database.dump` to `/home/deploy/my_app/tmp/my_app_database.dump`
 
 Synced folder `images` from `remote:my_app_backup/images` to `/home/deploy/my_app/public/images`
@@ -102,7 +146,7 @@ Example:
 
 ```
 deploy@server:~/my_app$ SWITCH_TO_SUPERUSER=true RESTORE_PGDATABASE=true rcloner restore --config rcloner.yml
-```
+``` -->
 
 ## How to run backup with a cron
 

data/lib/rcloner/backuper.rb

@@ -1,144 +1,121 @@
 require 'uri'
-require 'dotenv'
+require 'dotenv/load'
+require 'pathname'
 
 module Rcloner
   class Backuper
+    class ConfigError < StandardError; end
+
     def initialize(config)
       @config = config
-      @project_folder = @config['name'] + '_backup'
 
-      # Create tmp/ folder insite root_path:
-      Dir.chdir(@config['root_path']) do
-        FileUtils.mkdir_p(File.join @config['root_path'], 'tmp')
+      @config['origin'] ||= ENV['RCLONER_ORIGIN'] || ENV['PWD']
+      @config['destination'] ||= ENV['RCLONER_DESTINATION']
+
+      # Provide absolute paths for files in include list
+      if @config['include']
+        @config['include'] = @config['include'].map { |path| full_local_path(path) }
       end
 
-      if @config['items'].any? { |item| item['duplicity'] }
-        if ENV['PASSPHRASE'].nil? || ENV['PASSPHRASE'].empty?
-          raise "One of your items has duplicity backend but `PASSPHRASE` env variable is not set"
-        end
+      # If no entry provided for exclude list and include list exists,
+      # that probably means we should only backup these derectories,
+      # and skip all the rest - so for that we should provide '**' operator:
+      unless @config['exclude']
+        @config['exclude'] = ['**']
       end
-    end
 
-    def backup!
-      sync_items(to: :remote)
-    end
+      # Create tmp/ folder inside root_path:
+      Dir.chdir(@config['origin']) do
+        FileUtils.mkdir_p(File.join @config['origin'], 'tmp')
+      end
 
-    def restore!
-      sync_items(to: :local)
+      validate_config!
     end
 
-    private
+    def backup!
+      if before_command = @config.dig('on_backup', 'before')
+        puts "Perform before command: #{before_command}"
 
-    def sync_items(to:)
-      @config['items'].each do |item|
-        case item['type']
-        when 'folder'
-          sync_folder(item, to: to)
-        when 'file'
-          sync_file(item, to: to)
-        when 'pgdatabase'
-          sync_pgdatabase(item, to: to)
+        Dir.chdir(@config['origin']) do
+          execute before_command
         end
       end
-    end
 
-    def sync_pgdatabase(item, to:)
-      db_url =
-        if item['read_db_url_from_env']
-          Dir.chdir(@config['root_path']) do
-            `bundle exec postgressor print_db_url`.strip
-          end
-        else
-          item['db_url']
-        end
-
-      raise 'Cant read pgdatabase item db_url' unless db_url
+      @command = %W(duplicity)
+      @config['include'].each { |i| @command.push('--include', i) }
+      @config['exclude'].each { |i| @command.push('--exclude', i) }
 
-      db_backup_filename = URI.parse(db_url).path.sub('/', '') + '.dump'
-      relative_db_backup_filepath = 'tmp/' + db_backup_filename
-      local_db_backup_file_path = File.join(@config['root_path'], relative_db_backup_filepath)
-      item = { 'path' => relative_db_backup_filepath }
+      @command.push(@config['origin'], @config['destination'])
+      execute @command
 
-      case to
-      when :remote
-        env = { 'DATABASE_URL' => db_url }
-        execute %W(bundle exec postgressor dumpdb #{local_db_backup_file_path}), env: env
-        sync_file(item, to: :remote)
-      when :local
-        sync_file(item, to: :local)
+      if after_command = @config.dig('on_backup', 'after')
+        puts "Perform after command: #{after_command}"
 
-        if ENV['RESTORE_PGDATABASE'] == 'true'
-          command = %W(bundle exec postgressor restoredb #{local_db_backup_file_path})
-          command << '--switch_to_superuser' if ENV['SWITCH_TO_SUPERUSER'] == 'true'
-
-          execute command, { 'DATABASE_URL' => db_url }
+        Dir.chdir(@config['origin']) do
+          execute after_command
         end
       end
     end
 
-    def sync_file(item, to:)
-      local_file_path = File.join(@config['root_path'], item['path'])
-      file_name = Pathname.new(local_file_path).basename.to_s
-      remote_file_path = File.join(@project_folder, file_name)
-
-      case to
-      when :remote
-        to_path = "remote:#{remote_file_path}"
-        from_path = local_file_path
-      when :local
-        to_path = local_file_path
-        from_path = "remote:#{remote_file_path}"
-      end
-
-      execute %W(rclone copyto #{from_path} #{to_path})
-      puts "Synced file `#{file_name}` from `#{from_path}` to `#{to_path}`"
-    end
-
-    def sync_folder(item, to:)
-      local_folder_path = File.join(@config['root_path'], item['path'])
-      folder_name = Pathname.new(local_folder_path).basename.to_s
-      remote_folder_path = File.join(@project_folder, folder_name)
-
-      case to
-      when :remote
-        to_path = "remote:#{remote_folder_path}"
-        from_path = local_folder_path
-      when :local
-        to_path = local_folder_path
-        from_path = "remote:#{remote_folder_path}"
-      end
+    def restore!(to = nil, force = false)
+      @command = %W(duplicity)
 
-      execute %W(rclone mkdir #{to_path})
-
-      if item['duplicity']
-        case to
-        when :remote
-          dup_to_path = "rclone://remote:#{remote_folder_path}"
-          dup_from_path = local_folder_path
-          dup_command = %W(duplicity #{dup_from_path} #{dup_to_path})
-        when :local
-          dup_to_path = local_folder_path
-          dup_from_path = "rclone://remote:#{remote_folder_path}"
-          dup_command = %W(duplicity restore #{dup_from_path} #{dup_to_path} --force)
-        end
-
-        puts "Start syncing folder `#{folder_name}` from `#{from_path}` to `#{to_path}` using duplicity backend..."
-        execute dup_command
+      if to
+        to = File.expand_path(to)
       else
-        execute %W(rclone sync #{from_path} #{to_path})
+        to = @config['origin']
       end
 
-      puts "Synced folder `#{folder_name}` from `#{from_path}` to `#{to_path}`"
+      @command.push('restore', @config['destination'], to)
+      @command.push('--force') if force
+      execute @command
     end
 
+    private
+
     ###
 
     def execute(command, env: {}, path: nil)
+      if ENV['VERBOSE'] == 'true'
+        print_command =
+          if command.class == Array
+            command.join(' ')
+          else
+            command
+          end
+
+        puts "Execute: `#{print_command}`\n\n"
+      end
+
       if path
         system env, *command, chdir: path
       else
         system env, *command
       end
     end
+
+    ###
+
+    def full_local_path(relative_path)
+      File.join(@config['origin'], relative_path)
+    end
+
+    # def create_symlink!(from_path, to_path)
+    #   if execute %W(ln -s #{from_path} #{to_path})
+    #     puts "Created symlink `#{from_path}` -> `#{to_path}`"
+    #   end
+    # end
+
+    ###
+
+    def validate_config!
+      if ENV['PASSPHRASE'].nil? || ENV['PASSPHRASE'].empty?
+        raise ConfigError, '`PASSPHRASE` env variable is not set'
+      end
+
+      unless @config['destination']
+        raise ConfigError, 'Please provide a destination option'
+      end
+    end
   end
 end

data/lib/rcloner/cli.rb

@@ -20,9 +20,10 @@ module Rcloner
 
     desc 'restore', 'Restore all items from a remote storage'
     option :config, type: :string, required: true, banner: 'Path to a config file'
-    def restore
+    option :force, type: :boolean, default: false, banner: 'Allow to overwrite existing directory'
+    def restore(to = nil)
       backuper = create_backuper
-      backuper.restore!
+      backuper.restore!(to, options['force'])
     end
 
     private

data/lib/rcloner/version.rb

@@ -1,3 +1,3 @@
 module Rcloner
-  VERSION = "0.1.0"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rcloner
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Victor Afanasev
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-05-11 00:00:00.000000000 Z
+date: 2021-02-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor