travis-backup 0.0.1 → 0.1.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 7b88fafbc38de9343ca5ead6600c182858c39b14210a60f05081543bf74c8964
4
- data.tar.gz: d45a8734295f8dfb01675e333d99c6e19b441fd2290be042ced8b0e2d3e161c8
3
+ metadata.gz: f8b5825713610ce11c333d20573340404da43e46afadad8a8149b35194aa371e
4
+ data.tar.gz: acc50d6c047c57af32368a5fcd634e130a9ccab611222f1bf8b235be26e474fb
5
5
  SHA512:
6
- metadata.gz: 879009e7cbffc3be645bc12746ce25598b9d013ca81e0aa18370644c6c5bfba3d300d67fcd297e36fc0f941c41a0250cbd50fb3426211ae655e16c0f006b2a56
7
- data.tar.gz: 316429ee0dd693f3ba737abbc341f0afef919c4ff85d677fc2652dc71f42b6ba6a76580526f0f42f8bff8ed9fc28fbd0743f3157f559209f19844899ee13b421
6
+ metadata.gz: 14ac9ae0fdc86e32e68dc396a8ad02cc0adf988a6614146ba46c2353a290d4e57d33e5ead26a5cb7f8f8d6c4654c1833fb71af0ce9725d0431e5698d9c984adc
7
+ data.tar.gz: b07f0529a93a964f3280daaad33ceaff21c547438d93709c256b905cc8325e5cdfaf5d74204a0a31cd0fa497e43958f789dd66b600d1c5e131a9c74f50003222
data/.travis.yml CHANGED
@@ -30,4 +30,7 @@ before_script:
30
30
  - psql --version
31
31
  - psql -c 'CREATE DATABASE travis_test;' -U postgres
32
32
  - psql -t -c "SELECT 1 FROM pg_roles WHERE rolname='travis'" -U postgres | grep 1 || psql -c 'CREATE ROLE travis SUPERUSER LOGIN CREATEDB;' -U postgres
33
- - psql -f db/schema.sql -v ON_ERROR_STOP=1 travis_test
33
+ - psql -f db/schema.sql -v ON_ERROR_STOP=1 travis_test
34
+ - psql -c 'CREATE DATABASE travis_test_destination;' -U postgres
35
+ - psql -t -c "SELECT 1 FROM pg_roles WHERE rolname='travis'" -U postgres | grep 1 || psql -c 'CREATE ROLE travis SUPERUSER LOGIN CREATEDB;' -U postgres
36
+ - psql -f db/schema.sql -v ON_ERROR_STOP=1 travis_test_destination
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- travis-backup (0.0.1)
4
+ travis-backup (0.1.1)
5
5
  activerecord
6
6
  bootsnap
7
7
  pg
data/README.md CHANGED
@@ -1,7 +1,6 @@
1
1
  # README
2
2
 
3
- *travis-backup* is an application that removes builds and their corresponding jobs
4
- and exports them (optionally) to json files.
3
+ *travis-backup* is an application that helps with housekeeping and backup for Travis CI database v2.2 and with migration to v3.0 database. By default it removes requests and builds with their corresponding jobs and logs, as long as they are older than given threshold says (and backups them in files, if this option is active). Although it can be also run with special modes: `move_logs`, for moving logs from one database to another, and `remove_orphans`, for deleting all orphaned data.
5
4
 
6
5
  ### Installation and run
7
6
 
@@ -9,15 +8,33 @@ You can install the gem using
9
8
 
10
9
  `gem install travis-backup`
11
10
 
12
- Next you can run it in your app like
11
+ Next you can run it like:
13
12
 
14
13
  ```
15
- require 'travis-backup'
14
+ travis_backup 'postgres://user:pass@localhost:5432/my_db' --threshold 6
15
+ ```
16
16
 
17
- backup = Backup.new
18
- backup.run
17
+ All arguments:
18
+
19
+ ```
20
+ first argument, no flag # database url
21
+ -b, --backup # when not present, removes data without saving it to file
22
+ -d, --dry_run # only prints in console what data will be backuped and deleted
23
+ -l, --limit LIMIT # builds limit for one backup file
24
+ -t, --threshold MONTHS # number of months from now - data younger than this time won't be backuped
25
+ -f, --files_location PATH # path of the folder in which backup files will be placed
26
+ -u, --user_id ID # run only for given user
27
+ -o, --org_id ID # run only for given organization
28
+ -r, --repo_id ID # run only for given repository
29
+ --move_logs # run in move logs mode - move all logs to database at destination_db_url URL
30
+ --destination_db_url URL # URL for moving logs to
31
+ --remove_orphans # run in remove orphans mode
32
+ ```
19
33
 
20
- # or with configs:
34
+ Or inside your app:
35
+
36
+ ```
37
+ require 'travis-backup'
21
38
 
22
39
  backup = Backup.new(
23
40
  if_backup: true,
@@ -32,7 +49,6 @@ backup.run
32
49
  You can also run backup only for given user, organisation or repository:
33
50
 
34
51
  ```
35
- backup = Backup.new
36
52
  backup.run(user_id: 1)
37
53
  # or
38
54
  backup.run(org_id: 1)
@@ -40,9 +56,17 @@ backup.run(org_id: 1)
40
56
  backup.run(repo_id: 1)
41
57
  ```
42
58
 
43
- ### Configuration
59
+ #### Special modes
60
+
61
+ Using `--move_logs` flag you can move all logs to database at `destination_db_url` URL (which is required in this case). When you run gem in this mode no files are created and no other tables are being touched.
62
+
63
+ Using `--remove_orphans` flag you can remove all orphaned data from tables. When you run gem in this mode no files are created.
64
+
65
+ Using `--dry_run` flag you can check which data would be removed by gem, but without removing them actually. Instead of that reports will be printed on standard output. This flag can be also combined with `--move_logs` or `--remove_orphans`.
66
+
67
+ ### Configuration options
44
68
 
45
- One of the ways you can configure your export is a file `config/settinigs.yml` that you should place in your app's main directory. The gem uses the properties in following format:
69
+ Despite of command line arguments, one of the ways you can configure your export is a file `config/settings.yml` that you can place in your app's main directory. The gem expects properties in the following format:
46
70
 
47
71
  ```
48
72
  backup:
@@ -51,15 +75,19 @@ backup:
51
75
  limit: 1000 # builds limit for one backup file
52
76
  threshold: 6 # number of months from now - data younger than this time won't be backuped
53
77
  files_location: './dump' # path of the folder in which backup files will be placed
54
- user_id # run only for given user
55
- org_id # run only for given organization
56
- repo_id # run only for given repository
78
+ user_id: 1 # run only for given user
79
+ org_id: 1 # run only for given organization
80
+ repo_id: 1 # run only for given repository
81
+ move_logs: false # run in move logs mode - move all logs to database at destination_db_url URL
82
+ remove_orphans: false # run in remove orphans mode
57
83
  ```
58
84
 
59
- You can also set these properties as hash arguments while creating `Backup` instance or use env vars corresponding to them: `IF_BACKUP`, `BACKUP_DRY_RUN`, `BACKUP_LIMIT`, `BACKUP_THRESHOLD`, `BACKUP_FILES_LOCATION`, `USER_ID`, `ORG_ID`, `REPO_ID`.
85
+ You can also set these properties using env vars corresponding to them: `IF_BACKUP`, `BACKUP_DRY_RUN`, `BACKUP_LIMIT`, `BACKUP_THRESHOLD`, `BACKUP_FILES_LOCATION`, `BACKUP_USER_ID`, `BACKUP_ORG_ID`, `BACKUP_REPO_ID`, `BACKUP_MOVE_LOGS`, `BACKUP_REMOVE_ORPHANS`.
60
86
 
61
87
  You should also specify your database url. You can do this the standard way in `config/database.yml` file, setting the `database_url` hash argument while creating `Backup` instance or using the `DATABASE_URL` env var. Your database should be consistent with the Travis 2.2 database schema.
62
88
 
89
+ For `move_logs` mode you need also to specify a destination database. You can set it also in `config/database.yml` file, in `destination` subsection, setting the `destination_db_url` hash argument while creating `Backup` instance or using the `BACKUP_DESTINATION_DB_URL` env var. Your destination database should be consistent with the Travis 3.0 database schema.
90
+
63
91
  ### How to run the test suite
64
92
 
65
93
  You can run the test after cloning this repository. Next you should call
@@ -74,35 +102,9 @@ and
74
102
  bundle exec rspec
75
103
  ```
76
104
 
77
- To make tests working properly you should also ensure the database connection string for an empty test database. You can set it as `DATABASE_URL` environment variable or in `config/database.yml`.
78
-
79
- **Warning: this database will be cleaned during tests, so ensure that it includes no important data.**
80
-
81
- #### Using as standalone application
105
+ To make tests working properly you should also ensure database connection strings for empty test databases. You can set them as `DATABASE_URL` and `BACKUP_DESTINATION_DB_URL` environment variables or in `config/database.yml`.
82
106
 
83
- After cloning this repo you can also run it as a standalone app using
84
-
85
- ```
86
- bundle exec bin/travis_backup
87
- ```
88
-
89
- You can also pass arguments:
90
-
91
- ```
92
- first argument, no flag # database url
93
- -b, --backup # when not present, removes data without saving it to file
94
- -d, --dry_run # only prints in console what data will be backuped and deleted
95
- -l, --limit LIMIT # builds limit for one backup file
96
- -t, --threshold MONTHS # number of months from now - data younger than this time won't be backuped
97
- -f, --files_location PATH # path of the folder in which backup files will be placed
98
- -u, --user_id ID # run only for given user
99
- -o, --org_id ID # run only for given organization
100
- -r, --repo_id ID # run only for given repository
101
-
102
- # example:
103
-
104
- bundle exec bin/travis_backup 'postgres://user:pass@localhost:5432/my_db' -b
105
- ```
107
+ **Warning: these databases will be cleaned during tests, so ensure that they include no important data.**
106
108
 
107
109
  ### Ruby version
108
110
 
data/config/database.yml CHANGED
@@ -18,3 +18,6 @@ test:
18
18
  <<: *default
19
19
  url: 'postgresql://localhost/travis_test'
20
20
  eager_load: false
21
+ destination:
22
+ url: 'postgresql://localhost/travis_test_destination'
23
+ eager_load: false
data/lib/config.rb CHANGED
@@ -2,7 +2,18 @@
2
2
  require 'optparse'
3
3
 
4
4
  class Config
5
- attr_reader :if_backup, :dry_run, :limit, :threshold, :files_location, :database_url, :user_id, :repo_id, :org_id
5
+ attr_reader :if_backup,
6
+ :dry_run,
7
+ :limit,
8
+ :threshold,
9
+ :files_location,
10
+ :database_url,
11
+ :user_id,
12
+ :repo_id,
13
+ :org_id,
14
+ :move_logs,
15
+ :remove_orphans,
16
+ :destination_db_url
6
17
 
7
18
  def initialize(args={}) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity, Metrics/MethodLength
8
19
  set_values(args)
@@ -43,7 +54,7 @@ class Config
43
54
  @files_location = first_not_nil(
44
55
  args[:files_location],
45
56
  argv_opts[:files_location],
46
- ENV['BACKUP_BACKUP_FILES_LOCATION'],
57
+ ENV['BACKUP_FILES_LOCATION'],
47
58
  config.dig('backup', 'files_location'),
48
59
  './dump'
49
60
  )
@@ -56,25 +67,45 @@ class Config
56
67
  @user_id = first_not_nil(
57
68
  args[:user_id],
58
69
  argv_opts[:user_id],
59
- ENV['USER_ID'],
70
+ ENV['BACKUP_USER_ID'],
60
71
  config.dig('backup', 'user_id')
61
72
  )
62
73
  @repo_id = first_not_nil(
63
74
  args[:repo_id],
64
75
  argv_opts[:repo_id],
65
- ENV['REPO_ID'],
76
+ ENV['BACKUP_REPO_ID'],
66
77
  config.dig('backup', 'repo_id')
67
78
  )
68
79
  @org_id = first_not_nil(
69
80
  args[:org_id],
70
81
  argv_opts[:org_id],
71
- ENV['ORG_ID'],
82
+ ENV['BACKUP_ORG_ID'],
72
83
  config.dig('backup', 'org_id')
73
84
  )
85
+ @move_logs = first_not_nil(
86
+ args[:move_logs],
87
+ argv_opts[:move_logs],
88
+ ENV['BACKUP_MOVE_LOGS'],
89
+ config.dig('backup', 'move_logs'),
90
+ false
91
+ )
92
+ @remove_orphans = first_not_nil(
93
+ args[:remove_orphans],
94
+ argv_opts[:remove_orphans],
95
+ ENV['BACKUP_REMOVE_ORPHANS'],
96
+ config.dig('backup', 'remove_orphans'),
97
+ false
98
+ )
99
+ @destination_db_url = first_not_nil(
100
+ args[:destination_db_url],
101
+ argv_opts[:destination_db_url],
102
+ ENV['BACKUP_DESTINATION_DB_URL'],
103
+ connection_details.dig(ENV['RAILS_ENV'], 'destination')
104
+ )
74
105
  end
75
106
 
76
107
  def check_values
77
- if !@threshold
108
+ if !@move_logs && !@remove_orphans && !@threshold
78
109
  message = abort_message("Please provide the threshold argument. Data younger than it will be omitted. " +
79
110
  "Threshold defines number of months from now.")
80
111
  abort message
@@ -84,12 +115,22 @@ class Config
84
115
  message = abort_message("Please provide proper database URL.")
85
116
  abort message
86
117
  end
118
+
119
+ if (@move_logs && !@destination_db_url)
120
+ abort "\nFor moving logs you need to specify your destination database. Example usage:\n" +
121
+ "\n $ bin/travis_backup 'postgres://source_url' --move_logs --destination_db_url 'postgres://destination_url'\n" +
122
+ "\nor using in code:\n" +
123
+ "\n Backup.new(database_url: 'postgres://source_url', destination_db_url: 'postgres://destination_url', move_logs: true)\n" +
124
+ "\nYou can also set it using environment variables or configuration files.\n"
125
+ end
87
126
  end
88
127
 
89
128
  def abort_message(intro)
90
129
  "\n#{intro} Example usage:\n"+
91
130
  "\n $ bin/travis_backup 'postgres://my_database_url' --threshold 6\n" +
92
- "\nYou can also set it using environment variables or config/database.yml file.\n"
131
+ "\nor using in code:\n" +
132
+ "\n Backup.new(database_url: 'postgres://my_database_url', threshold: 6)\n" +
133
+ "\nYou can also set it using environment variables or configuration files.\n"
93
134
  end
94
135
 
95
136
  def argv_options
@@ -104,6 +145,9 @@ class Config
104
145
  opt.on('-u', '--user_id X') { |o| options[:user_id] = o.to_i }
105
146
  opt.on('-r', '--repo_id X') { |o| options[:repo_id] = o.to_i }
106
147
  opt.on('-o', '--org_id X') { |o| options[:org_id] = o.to_i }
148
+ opt.on('--move_logs') { |o| options[:move_logs] = o }
149
+ opt.on('--remove_orphans') { |o| options[:remove_orphans] = o }
150
+ opt.on('--destination_db_url X') { |o| options[:destination_db_url] = o }
107
151
  end.parse!
108
152
 
109
153
  options[:database_url] = ARGV.shift if ARGV[0]
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Branch < Model
6
+ self.table_name = 'branches'
7
+ end
data/lib/models/build.rb CHANGED
@@ -7,7 +7,7 @@ require 'models/repository'
7
7
  # Build model
8
8
  class Build < Model
9
9
  belongs_to :repository
10
- has_many :jobs, -> { order('id') }, foreign_key: :source_id, dependent: :delete_all, class_name: 'Job'
10
+ has_many :jobs, -> { order('id') }, foreign_key: :source_id, dependent: :destroy, class_name: 'Job'
11
11
 
12
12
  self.table_name = 'builds'
13
13
  end
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Commit < Model
6
+ self.table_name = 'commits'
7
+ end
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Cron < Model
6
+ self.table_name = 'crons'
7
+ end
data/lib/models/job.rb CHANGED
@@ -2,12 +2,14 @@
2
2
 
3
3
  require 'models/model'
4
4
  require 'models/repository'
5
+ require 'models/log'
5
6
 
6
7
  # Job model
7
8
  class Job < Model
8
9
  self.inheritance_column = :_type_disabled
9
10
 
10
11
  belongs_to :repository
12
+ has_many :logs, -> { order('id') }, foreign_key: :job_id, dependent: :destroy, class_name: 'Log'
11
13
 
12
14
  self.table_name = 'jobs'
13
15
  end
data/lib/models/log.rb ADDED
@@ -0,0 +1,9 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Log < Model
6
+ belongs_to :job
7
+
8
+ self.table_name = 'logs'
9
+ end
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class PullRequest < Model
6
+ self.table_name = 'pull_requests'
7
+ end
@@ -2,10 +2,12 @@
2
2
 
3
3
  require 'models/model'
4
4
  require 'models/build'
5
+ require 'models/request'
5
6
 
6
7
  # Repository model
7
8
  class Repository < Model
8
9
  has_many :builds, -> { order('id') }, foreign_key: :repository_id, dependent: :destroy, class_name: 'Build'
10
+ has_many :requests, -> { order('id') }, foreign_key: :repository_id, dependent: :destroy, class_name: 'Request'
9
11
 
10
12
  self.table_name = 'repositories'
11
13
  end
@@ -0,0 +1,10 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+ require 'models/repository'
5
+
6
+ class Request < Model
7
+ belongs_to :repository
8
+
9
+ self.table_name = 'requests'
10
+ end
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class SslKey < Model
6
+ self.table_name = 'ssl_keys'
7
+ end
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Stage < Model
6
+ self.table_name = 'stages'
7
+ end
data/lib/models/tag.rb ADDED
@@ -0,0 +1,7 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'models/model'
4
+
5
+ class Tag < Model
6
+ self.table_name = 'tags'
7
+ end
data/lib/travis-backup.rb CHANGED
@@ -4,21 +4,33 @@ require 'active_support/core_ext/array'
4
4
  require 'active_support/time'
5
5
  require 'config'
6
6
  require 'models/repository'
7
+ require 'models/log'
8
+ require 'models/branch'
9
+ require 'models/tag'
10
+ require 'models/commit'
11
+ require 'models/cron'
12
+ require 'models/pull_request'
13
+ require 'models/ssl_key'
14
+ require 'models/request'
15
+ require 'models/stage'
7
16
 
8
17
  # main travis-backup class
9
18
  class Backup
19
+ attr_accessor :config
20
+ attr_reader :dry_run_report
21
+
10
22
  def initialize(config_args={})
11
23
  @config = Config.new(config_args)
12
24
 
13
25
  if @config.dry_run
14
- @dry_run_removed = {builds: [], jobs: []}
26
+ @dry_run_report = {builds: [], jobs: [], logs: [], requests: []}
15
27
  end
16
28
 
17
29
  connect_db
18
30
  end
19
31
 
20
- def connect_db
21
- ActiveRecord::Base.establish_connection(@config.database_url)
32
+ def connect_db(url=@config.database_url)
33
+ ActiveRecord::Base.establish_connection(url)
22
34
  end
23
35
 
24
36
  def run(args={})
@@ -32,63 +44,204 @@ class Backup
32
44
  elsif org_id
33
45
  owner_id = org_id
34
46
  owner_type = 'Organization'
35
- elsif repo_id
36
- repo_id = repo_id
37
47
  end
38
48
 
39
- if owner_id
40
- Repository.where('owner_id = ? and owner_type = ?', owner_id, owner_type).order(:id).each do |repository|
41
- process_repo(repository)
42
- end
49
+ if @config.move_logs
50
+ move_logs
51
+ elsif @config.remove_orphans
52
+ remove_orphans
53
+ elsif owner_id
54
+ process_repos_for_owner(owner_id, owner_type)
43
55
  elsif repo_id
44
- repository = Repository.find(repo_id)
56
+ process_repo_with_id(repo_id)
57
+ else
58
+ process_all_repos
59
+ end
60
+
61
+ print_dry_run_report if @config.dry_run
62
+ end
63
+
64
+ def process_repos_for_owner(owner_id, owner_type)
65
+ Repository.where('owner_id = ? and owner_type = ?', owner_id, owner_type).order(:id).each do |repository|
45
66
  process_repo(repository)
67
+ end
68
+ end
69
+
70
+ def process_repo_with_id(repo_id)
71
+ process_repo(Repository.find(repo_id))
72
+ end
73
+
74
+ def process_all_repos
75
+ Repository.order(:id).each do |repository|
76
+ process_repo(repository)
77
+ end
78
+ end
79
+
80
+ def print_dry_run_report
81
+ if @dry_run_report.to_a.map(&:second).flatten.empty?
82
+ puts 'Dry run active. No data would be removed in normal run.'
46
83
  else
47
- Repository.order(:id).each do |repository|
48
- process_repo(repository)
84
+ puts 'Dry run active. The following data would be removed in normal run:'
85
+
86
+ @dry_run_report.to_a.map(&:first).each do |symbol|
87
+ print_dry_run_report_line(symbol)
49
88
  end
50
89
  end
90
+ end
51
91
 
52
- if @config.dry_run
53
- puts 'Dry run active. The following data would be removed in normal mode:'
54
- puts " - builds: #{@dry_run_removed[:builds].to_json}"
55
- puts " - jobs: #{@dry_run_removed[:jobs].to_json}"
56
- end
92
+ def print_dry_run_report_line(symbol)
93
+ puts " - #{symbol}: #{@dry_run_report[symbol].to_json}" if @dry_run_report[symbol].any?
94
+ end
95
+
96
+ def process_repo(repository)
97
+ process_repo_builds(repository)
98
+ process_repo_requests(repository)
57
99
  end
58
100
 
59
- def process_repo(repository) # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
101
+ def process_repo_builds(repository) # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
60
102
  threshold = @config.threshold.to_i.months.ago.to_datetime
61
103
  current_build_id = repository.current_build_id || -1
62
104
  repository.builds.where('created_at < ? and id != ?', threshold, current_build_id)
63
105
  .in_groups_of(@config.limit.to_i, false).map do |builds_batch|
64
- @config.if_backup ? save_and_destroy_batch(builds_batch, repository) : destroy_batch(builds_batch)
106
+ @config.if_backup ? save_and_destroy_builds_batch(builds_batch, repository) : destroy_builds_batch(builds_batch)
107
+ end.compact
108
+ end
109
+
110
+ def process_repo_requests(repository)
111
+ threshold = @config.threshold.to_i.months.ago.to_datetime
112
+ repository.requests.where('created_at < ?', threshold)
113
+ .in_groups_of(@config.limit.to_i, false).map do |requests_batch|
114
+ @config.if_backup ? save_and_destroy_requests_batch(requests_batch, repository) : destroy_requests_batch(requests_batch)
65
115
  end.compact
66
116
  end
67
117
 
118
+ def move_logs
119
+ return move_logs_dry if config.dry_run
120
+
121
+ connect_db(@config.database_url)
122
+ Log.order(:id).in_groups_of(@config.limit.to_i, false).map do |logs_batch|
123
+ log_hashes = logs_batch.as_json
124
+ connect_db(@config.destination_db_url)
125
+
126
+ log_hashes.each do |log_hash|
127
+ new_log = Log.new(log_hash)
128
+ new_log.save!
129
+ end
130
+
131
+ connect_db(@config.database_url)
132
+
133
+ logs_batch.each(&:destroy)
134
+ end
135
+ end
136
+
137
+ def move_logs_dry
138
+ dry_run_report[:logs].concat(Log.order(:id).map(&:id))
139
+ end
140
+
141
+ def remove_orphans
142
+ remove_orphans_for_table(Repository, 'repositories', 'builds', 'current_build_id')
143
+ remove_orphans_for_table(Repository, 'repositories', 'builds', 'last_build_id')
144
+ remove_orphans_for_table(Build, 'builds', 'repositories', 'repository_id')
145
+ remove_orphans_for_table(Build, 'builds', 'commits', 'commit_id')
146
+ remove_orphans_for_table(Build, 'builds', 'requests', 'request_id')
147
+ remove_orphans_for_table(Build, 'builds', 'pull_requests', 'pull_request_id')
148
+ remove_orphans_for_table(Build, 'builds', 'branches', 'branch_id')
149
+ remove_orphans_for_table(Build, 'builds', 'tags', 'tag_id')
150
+ remove_orphans_for_table(Job, 'jobs', 'repositories', 'repository_id')
151
+ remove_orphans_for_table(Job, 'jobs', 'commits', 'commit_id')
152
+ remove_orphans_for_table(Job, 'jobs', 'stages', 'stage_id')
153
+ remove_orphans_for_table(Branch, 'branches', 'repositories', 'repository_id')
154
+ remove_orphans_for_table(Branch, 'branches', 'builds', 'last_build_id')
155
+ remove_orphans_for_table(Tag, 'tags', 'repositories', 'repository_id')
156
+ remove_orphans_for_table(Tag, 'tags', 'builds', 'last_build_id')
157
+ remove_orphans_for_table(Commit, 'commits', 'repositories', 'repository_id')
158
+ remove_orphans_for_table(Commit, 'commits', 'branches', 'branch_id')
159
+ remove_orphans_for_table(Commit, 'commits', 'tags', 'tag_id')
160
+ remove_orphans_for_table(Cron, 'crons', 'branches', 'branch_id')
161
+ remove_orphans_for_table(PullRequest, 'pull_requests', 'repositories', 'repository_id')
162
+ remove_orphans_for_table(SslKey, 'ssl_keys', 'repositories', 'repository_id')
163
+ remove_orphans_for_table(Request, 'requests', 'commits', 'commit_id')
164
+ remove_orphans_for_table(Request, 'requests', 'pull_requests', 'pull_request_id')
165
+ remove_orphans_for_table(Request, 'requests', 'branches', 'branch_id')
166
+ remove_orphans_for_table(Request, 'requests', 'tags', 'tag_id')
167
+ remove_orphans_for_table(Stage, 'stages', 'builds', 'build_id')
168
+ end
169
+
170
+ def remove_orphans_for_table(model_class, table_a_name, table_b_name, fk_name)
171
+ for_delete = model_class.find_by_sql(%{
172
+ select a.*
173
+ from #{table_a_name} a
174
+ left join #{table_b_name} b
175
+ on a.#{fk_name} = b.id
176
+ where
177
+ a.#{fk_name} is not null
178
+ and b.id is null;
179
+ })
180
+
181
+ if config.dry_run
182
+ key = table_a_name.to_sym
183
+ dry_run_report[key] = [] if dry_run_report[key].nil?
184
+ dry_run_report[key].concat(for_delete.map(&:id))
185
+ dry_run_report[key].uniq!
186
+ else
187
+ model_class.where(id: for_delete.map(&:id)).delete_all
188
+ end
189
+ end
190
+
68
191
  private
69
192
 
70
- def save_and_destroy_batch(builds_batch, repository)
193
+ def save_and_destroy_builds_batch(builds_batch, repository)
71
194
  builds_export = export_builds(builds_batch)
72
195
  file_name = "repository_#{repository.id}_builds_#{builds_batch.first.id}-#{builds_batch.last.id}.json"
73
196
  pretty_json = JSON.pretty_generate(builds_export)
74
197
  if save_file(file_name, pretty_json)
75
- destroy_batch(builds_batch)
198
+ destroy_builds_batch(builds_batch)
76
199
  end
77
200
  builds_export
78
201
  end
79
202
 
80
- def destroy_batch(builds_batch)
81
- return destroy_batch_dry(builds_batch) if @config.dry_run
203
+ def destroy_builds_batch(builds_batch)
204
+ return destroy_builds_batch_dry(builds_batch) if @config.dry_run
82
205
 
83
206
  builds_batch.each(&:destroy)
84
207
  end
85
208
 
86
- def destroy_batch_dry(builds_batch)
87
- @dry_run_removed[:builds].concat(builds_batch.map(&:id))
209
+ def destroy_builds_batch_dry(builds_batch)
210
+ @dry_run_report[:builds].concat(builds_batch.map(&:id))
211
+
88
212
  jobs = builds_batch.map do |build|
89
213
  build.jobs.map(&:id) || []
90
214
  end.flatten
91
- @dry_run_removed[:jobs].concat(jobs)
215
+
216
+ @dry_run_report[:jobs].concat(jobs)
217
+
218
+ logs = builds_batch.map do |build|
219
+ build.jobs.map do |job|
220
+ job.logs.map(&:id) || []
221
+ end.flatten || []
222
+ end.flatten
223
+
224
+ @dry_run_report[:logs].concat(logs)
225
+ end
226
+
227
+ def save_and_destroy_requests_batch(requests_batch, repository)
228
+ requests_export = export_requests(requests_batch)
229
+ file_name = "repository_#{repository.id}_requests_#{requests_batch.first.id}-#{requests_batch.last.id}.json"
230
+ pretty_json = JSON.pretty_generate(requests_export)
231
+ if save_file(file_name, pretty_json)
232
+ destroy_requests_batch(requests_batch)
233
+ end
234
+ requests_export
235
+ end
236
+
237
+ def destroy_requests_batch(requests_batch)
238
+ return destroy_requests_batch_dry(requests_batch) if @config.dry_run
239
+
240
+ requests_batch.each(&:destroy)
241
+ end
242
+
243
+ def destroy_requests_batch_dry(requests_batch)
244
+ @dry_run_report[:requests].concat(requests_batch.map(&:id))
92
245
  end
93
246
 
94
247
  def save_file(file_name, content) # rubocop:disable Metrics/MethodLength
@@ -127,8 +280,21 @@ class Backup
127
280
  def export_jobs(jobs)
128
281
  jobs.map do |job|
129
282
  job_export = job.attributes
283
+ job_export[:logs] = export_logs(job.logs)
130
284
 
131
285
  job_export
132
286
  end
133
287
  end
288
+
289
+ def export_logs(logs)
290
+ logs.map do |log|
291
+ log.attributes
292
+ end
293
+ end
294
+
295
+ def export_requests(requests)
296
+ requests.map do |request|
297
+ request.attributes
298
+ end
299
+ end
134
300
  end
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'travis-backup'
3
- s.version = '0.0.1'
3
+ s.version = '0.1.1'
4
4
  s.summary = 'Travis CI backup tool'
5
5
  s.authors = ['Karol Selak']
6
6
  s.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: travis-backup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Karol Selak
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-08-24 00:00:00.000000000 Z
11
+ date: 2021-09-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord
@@ -248,11 +248,20 @@ files:
248
248
  - db/schema.sql
249
249
  - dump/.keep
250
250
  - lib/config.rb
251
+ - lib/models/branch.rb
251
252
  - lib/models/build.rb
253
+ - lib/models/commit.rb
254
+ - lib/models/cron.rb
252
255
  - lib/models/job.rb
256
+ - lib/models/log.rb
253
257
  - lib/models/model.rb
254
258
  - lib/models/organization.rb
259
+ - lib/models/pull_request.rb
255
260
  - lib/models/repository.rb
261
+ - lib/models/request.rb
262
+ - lib/models/ssl_key.rb
263
+ - lib/models/stage.rb
264
+ - lib/models/tag.rb
256
265
  - lib/models/user.rb
257
266
  - lib/travis-backup.rb
258
267
  - log/.keep