synco 1.3.0 → 1.3.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/lib/synco/command.rb +1 -1
- data/lib/synco/version.rb +1 -1
- data/readme.md +6 -215
- data.tar.gz.sig +0 -0
- metadata +2 -1
- metadata.gz.sig +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 42343f27c1a3ff659177999629a756000e4c5ced7b6474872ca5a0917d854f8e
|
4
|
+
data.tar.gz: dca509a5620feadc9479dcb0ae2d6e30eae6b4eb13bb093b7d986731d760a9f0
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 36215207716af1b46eebb94f2a6515d0d67fa5ecec8d85d8106eef5664ba74a3a294fcab277becf56a30d73d180d63e1b9a528f881774bb0c53ddd66408aa595
|
7
|
+
data.tar.gz: bcbdd5a44feaa92e3898b167718b4f798abbc614579ff94897b91dfebaec472ee1f99e49dedb42704da3b296b57fc82ca3a2051429ee57fc151220cd0119935b
|
checksums.yaml.gz.sig
CHANGED
Binary file
|
data/lib/synco/command.rb
CHANGED
data/lib/synco/version.rb
CHANGED
data/readme.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
|
-
# Synco
|
1
|
+
# Synco
|
2
2
|
|
3
|
-
Synco is a tool for scripted synchronization and backups. It provides a custom Ruby DSL for describing backup and synchronization tasks involving one more more system and disk. It is designed to provide flexibility while reducing the complexity multi-server backups.
|
3
|
+
Synco is a tool for scripted synchronization and backups. It provides a custom Ruby domain specific language (DSL) for describing backup and synchronization tasks involving one more more system and disk. It is designed to provide flexibility while reducing the complexity multi-server backups.
|
4
4
|
|
5
5
|
- Single and multi-server data synchronization.
|
6
6
|
- Incremental backups both locally and remotely.
|
@@ -10,224 +10,15 @@ Synco is a tool for scripted synchronization and backups. It provides a custom R
|
|
10
10
|
|
11
11
|
[![Development Status](https://github.com/ioquatix/synco/workflows/Test/badge.svg)](https://github.com/ioquatix/synco/actions?workflow=Test)
|
12
12
|
|
13
|
-
## Installation
|
14
|
-
|
15
|
-
Add this line to your application's Gemfile:
|
16
|
-
|
17
|
-
gem 'synco'
|
18
|
-
|
19
|
-
And then execute:
|
20
|
-
|
21
|
-
$ bundle
|
22
|
-
|
23
|
-
Or install it yourself as:
|
24
|
-
|
25
|
-
$ gem install synco
|
26
|
-
|
27
13
|
## Usage
|
28
14
|
|
29
|
-
|
30
|
-
|
31
|
-
A simple backup script might look something like this:
|
32
|
-
|
33
|
-
``` ruby
|
34
|
-
#!/usr/bin/env ruby
|
35
|
-
|
36
|
-
require 'synco'
|
37
|
-
require 'synco/methods/rsync'
|
38
|
-
|
39
|
-
Synco::run_script do |script|
|
40
|
-
script.method = Synco::Methods::RSync.new(archive: true)
|
41
|
-
|
42
|
-
server(:master) do |server|
|
43
|
-
server.host = "server.example.com"
|
44
|
-
server.root = "/"
|
45
|
-
end
|
46
|
-
|
47
|
-
server(:backup) do |server|
|
48
|
-
server.host = "backup.example.com"
|
49
|
-
server.root = "/tank/backups/server.example.com"
|
50
|
-
end
|
51
|
-
|
52
|
-
backup('etc', 'var', 'srv', 'home',
|
53
|
-
arguments: %W{--exclude cache/ --exclude tmp/}
|
54
|
-
)
|
55
|
-
end
|
56
|
-
```
|
57
|
-
|
58
|
-
This will produce an identical copy using rsync of the specified directories. As an example `server.example.com:/etc` would be copied to `backup.example.com:/tank/backups/server.example.com/etc`.
|
59
|
-
|
60
|
-
### RSync Snapshots
|
61
|
-
|
62
|
-
Building on the above backup, you can use `Synco::Methods::RSyncSnapshot` which supports snapshot based backups. It creates a snapshot into a sub-directory called `latest.snapshot` and uses RSync's `--link-dest` to hard-link files when unchanged. Synco provides scripts to rotate and prune these backups as required, but you must invoke them as part of the script:
|
63
|
-
|
64
|
-
``` ruby
|
65
|
-
server(:backup) do |server|
|
66
|
-
server.host = "backup.example.com"
|
67
|
-
server.root = "/"
|
68
|
-
|
69
|
-
server.on(:success) do
|
70
|
-
run "synco", "rotate", chdir: target_server.root
|
71
|
-
run "synco", "prune", chdir: target_server.root
|
72
|
-
end
|
73
|
-
end
|
74
|
-
```
|
75
|
-
|
76
|
-
These commands can also be run from the command line.
|
77
|
-
|
78
|
-
```
|
79
|
-
rotate [--format <name>] [--latest <name>] [--snapshot <name>]
|
80
|
-
Rotate a backup snapshot into a timestamped directory.
|
81
|
-
|
82
|
-
[--format <name>] Set the name of the backup rotations, including strftime expansions. Default: %Y.%m.%d-%H.%M.%S
|
83
|
-
[--latest <name>] The name of the latest backup symlink. Default: latest
|
84
|
-
[--snapshot <name>] The name of the in-progress backup snapshot. Default: latest.snapshot
|
85
|
-
|
86
|
-
prune [--hourly <count>] [--daily <count>] [--weekly <count>] [--monthly <count>] [--quarterly <count>] [--yearly <count>] [--format <name>] [--latest <name>] [--keep <new|old>] [--dry]
|
87
|
-
Prune old backups to reduce disk usage according to a given policy.
|
88
|
-
|
89
|
-
[--hourly <count>] Set the number of hourly backups to keep. Default: 24
|
90
|
-
[--daily <count>] Set the number of daily backups to keep. Default: 28
|
91
|
-
[--weekly <count>] Set the number of weekly backups to keep. Default: 52
|
92
|
-
[--monthly <count>] Set the number of monthly backups to keep. Default: 36
|
93
|
-
[--quarterly <count>] Set the number of quaterly backups to keep. Default: 40
|
94
|
-
[--yearly <count>] Set the number of yearly backups to keep. Default: 20
|
95
|
-
[--format <name>] Set the name of the backup rotations, including strftime expansions. Default: %Y.%m.%d-%H.%M.%S
|
96
|
-
[--latest <name>] The name of the latest backup symlink. Default: latest
|
97
|
-
[--keep <new|old>] Keep the younger or older backups within the same period division Default: old
|
98
|
-
[--dry] Print out what would be done rather than doing it.
|
99
|
-
```
|
100
|
-
|
101
|
-
### Mounting Disks
|
102
|
-
|
103
|
-
Synco supports mounting disks before the backup begins and unmounting them after done. The specifics of this process may require some adjustment based on your OS. For example on linux, `sudo` is used to invoke `mount` and `umount`.
|
104
|
-
|
105
|
-
``` ruby
|
106
|
-
server(:destination) do |server|
|
107
|
-
self.mountpoint = '/mnt/backups'
|
108
|
-
self.root = File.join(server.mountpoint, 'laptop')
|
109
|
-
|
110
|
-
server.on(:prepare) do
|
111
|
-
# synco mount uses labels, e.g. the disk partition has LABEL=backups
|
112
|
-
target_server.run "synco", "mount", target_server.mountpoint, 'backups'
|
113
|
-
end
|
114
|
-
|
115
|
-
server.on(:finish) do
|
116
|
-
target_server.run "synco", "unmount", target_server.mountpoint
|
117
|
-
end
|
118
|
-
end
|
119
|
-
```
|
120
|
-
|
121
|
-
On Linux, you might want to create the file `/etc/sudoers.d/synco` with the following contents:
|
122
|
-
|
123
|
-
%wheel ALL=(root) NOPASSWD: /bin/mount
|
124
|
-
%wheel ALL=(root) NOPASSWD: /bin/umount
|
125
|
-
|
126
|
-
Please make sure you take the time to educate yourself on the security of such a setup.
|
127
|
-
|
128
|
-
### Database Backups
|
129
|
-
|
130
|
-
If you'd like to dump data before running the backup, it's possible using the event handling mechanisms:
|
131
|
-
|
132
|
-
``` ruby
|
133
|
-
server(:master) do |server|
|
134
|
-
server.host = "server.example.com"
|
135
|
-
server.root = "/"
|
136
|
-
|
137
|
-
server.on(:prepare) do
|
138
|
-
# Dump MySQL to /srv/mysql
|
139
|
-
run '/etc/lsync/mysql-backup.sh'
|
140
|
-
end
|
141
|
-
end
|
142
|
-
```
|
143
|
-
|
144
|
-
The exact contents of `mysql-backup.sh` will depend on your requirements, but here is an example:
|
145
|
-
|
146
|
-
``` bash
|
147
|
-
#!/usr/bin/env bash
|
148
|
-
|
149
|
-
BACKUP_DIR=/srv/mysql
|
150
|
-
MYSQL=/usr/bin/mysql
|
151
|
-
MYSQLDUMP=/usr/bin/mysqldump
|
152
|
-
|
153
|
-
databases=`mysql --user=backup -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema|performance_schema|_test|_restore)"`
|
154
|
-
|
155
|
-
# http://stackoverflow.com/questions/451404/how-to-obtain-a-correct-dump-using-mysqldump-and-single-transaction-when-ddl-is
|
156
|
-
MYSQLDUMP_OPTIONS="--force --skip-opt --single-transaction --add-drop-table --create-options --quick --extended-insert --set-charset --disable-keys"
|
157
|
-
|
158
|
-
for db in $databases; do
|
159
|
-
echo "Dumping database $db to $BACKUP_DIR/$db.sql.xz..."
|
160
|
-
mysqldump --user=backup $MYSQLDUMP_OPTIONS --databases $db | xz > "$BACKUP_DIR/$db.sql.xz"
|
161
|
-
done
|
162
|
-
```
|
163
|
-
|
164
|
-
### Fingerprint Integration
|
165
|
-
|
166
|
-
It is possible to make a [cryptographic checksum of the data](https://github.com/ioquatix/fingerprint). On a filesystem that support immutable snapshots, you can do this before the data is copied. For traditional filesystems, you generally need to do this afterwards.
|
167
|
-
|
168
|
-
``` ruby
|
169
|
-
server(:master) do |server|
|
170
|
-
server.host = "server.example.com"
|
171
|
-
server.root = "/"
|
172
|
-
|
173
|
-
server.on(:success) do
|
174
|
-
# Run fingerprint on the backup data:
|
175
|
-
run 'fingerprint', '--root', target_server.root, 'analyze'
|
176
|
-
end
|
177
|
-
end
|
178
|
-
```
|
179
|
-
|
180
|
-
Fingerprint is used in many of the specs to verify file copies.
|
181
|
-
|
182
|
-
### ZFS Snapshots
|
183
|
-
|
184
|
-
*This part of Synco is still under heavy development*
|
185
|
-
|
186
|
-
Synco can manage synchronization and backups of ZFS partitions. However, to use the standard tools, it is necessary to enable `zfs_admin_snapshot`, in `/etc/modprobe.d/zfs.conf`:
|
187
|
-
|
188
|
-
options zfs zfs_admin_snapshot=1
|
189
|
-
|
190
|
-
Propagate user permissions for the ZFS partition:
|
191
|
-
|
192
|
-
``` bash
|
193
|
-
sudo zfs allow -ld -u `whoami` create,mount,send,receive,snapshot tank/test
|
194
|
-
```
|
195
|
-
|
196
|
-
### Backup staging
|
197
|
-
|
198
|
-
Synco in a previous life supported backup staging. However, at this time it's not available except in a very limited form: backup scripts which use `Synco.run_script` use explicit locking so that it's not possible to run the same backup at the same time. In the future, staging sequential and parallel backups will be added.
|
199
|
-
|
200
|
-
### DNS Failover
|
201
|
-
|
202
|
-
**This behaviour is not well tested**
|
203
|
-
|
204
|
-
Synco uses DNS to resolve the master server. This allows for bi-directional synchronization and other interesting setups.
|
205
|
-
|
206
|
-
Firstly, a backup script defaults to the server with the name `:master` as the master, where data is replicated FROM.
|
15
|
+
Please see the [project documentation](https://ioquatix.github.io/synco/) for more details.
|
207
16
|
|
208
|
-
|
17
|
+
- [Getting Started](https://ioquatix.github.io/synco/guides/getting-started/index) - This guide gives an overview of Synco, how to install it how to use it to backup and replicate data.
|
209
18
|
|
210
|
-
|
211
|
-
Synco::run_script do |script|
|
212
|
-
script.method = Synco::Methods::RSync.new(archive: true)
|
213
|
-
|
214
|
-
script.master = "primary.example.com"
|
215
|
-
|
216
|
-
server("s01.example.com") do |server|
|
217
|
-
server.root = "/"
|
218
|
-
end
|
219
|
-
|
220
|
-
server("s02.example.com") do |server|
|
221
|
-
server.root = "/"
|
222
|
-
end
|
223
|
-
|
224
|
-
backup('srv/http',
|
225
|
-
arguments: %W{--exclude cache/ --exclude tmp/}
|
226
|
-
)
|
227
|
-
end
|
228
|
-
```
|
19
|
+
- [Backup Policy](https://ioquatix.github.io/synco/guides/backup-policy/index) - This guide provides an overview of a Digital Information Backup Policy, including the main concerns affecting data retention and backup, the specific details that need to be considered, and the hardware and software solutions available to match your exact requirements.
|
229
20
|
|
230
|
-
|
21
|
+
- [Backup Script](https://ioquatix.github.io/synco/guides/backup-script/index) - This guide explains how to create a backup script and the various options available to you.
|
231
22
|
|
232
23
|
## Contributing
|
233
24
|
|
data.tar.gz.sig
CHANGED
Binary file
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: synco
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.3.
|
4
|
+
version: 1.3.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Samuel Williams
|
@@ -170,6 +170,7 @@ homepage: https://github.com/ioquatix/synco
|
|
170
170
|
licenses:
|
171
171
|
- MIT
|
172
172
|
metadata:
|
173
|
+
documentation_uri: https://ioquatix.github.io/synco/
|
173
174
|
funding_uri: https://github.com/sponsors/ioquatix/
|
174
175
|
source_code_uri: https://github.com/ioquatix/synco.git
|
175
176
|
post_install_message:
|
metadata.gz.sig
CHANGED
Binary file
|