lokalise_manager 2.0.0 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +45 -0
- data/README.md +32 -3
- data/lib/lokalise_manager/global_config.rb +7 -1
- data/lib/lokalise_manager/task_definitions/exporter.rb +11 -6
- data/lib/lokalise_manager/version.rb +1 -1
- data/spec/lib/lokalise_manager/global_config_spec.rb +5 -0
- data/spec/lib/lokalise_manager/task_definitions/exporter_spec.rb +31 -4
- metadata +1 -1
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f3a7da094cb7939fd6826c89afbbbc525dff59b553a23fba7b49ed38046e68b9
|
4
|
+
data.tar.gz: c0a323464c864d12042f126d765a483adc128a4397604d012cdeadee31003fcf
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 79c0b5a93767b5ffa8cd55dbf6591a90cafbb586043e8e75ca040761b8fa882882971783358e6d342afab96e298d34c67ca5d414d376c909fa564e8ad03d2466
|
7
|
+
data.tar.gz: ed2ad8e59fb6e15db8ba7c60de48e808c2eab0ff94b8f287c54bb2e618e1c930d5d3244a2483a6e61e69b7094943a05db59ea54ef32fc3596ef344f5956553ee
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,50 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## 2.1.0 (27-Jan-22)
|
4
|
+
|
5
|
+
* **Breaking change**: `export!` will now return an array of objects responding to the following methods:
|
6
|
+
+ `success` — usually returns `true` (to learn more, check documentation for the `:raise_on_export_fail` option below)
|
7
|
+
+ `process` — returns an object (an instance of the `Lokalise::Resources::QueuedProcess`) representing a [queued background process](https://lokalise.github.io/ruby-lokalise-api/api/queued-processes) as uploading is done in the background on Lokalise. You can use this object to check the process status (whether the uploading is completed or not).
|
8
|
+
+ `path` — returns an instance of the `Pathname` class which represent the file being uploaded.
|
9
|
+
* Here's an example:
|
10
|
+
|
11
|
+
```ruby
|
12
|
+
def uploaded?(process)
|
13
|
+
5.times do # try to check the status 5 times
|
14
|
+
process = process.reload_data # load new data
|
15
|
+
return(true) if process.status == 'finished' # return true is the upload has finished
|
16
|
+
sleep 1 # wait for 1 second, adjust this number with regards to the upload size
|
17
|
+
end
|
18
|
+
|
19
|
+
false # if all 5 checks failed, return false (probably something is wrong)
|
20
|
+
end
|
21
|
+
|
22
|
+
processes = exporter.export!
|
23
|
+
puts "Checking status for the #{processes[0].path} file"
|
24
|
+
uploaded? processes[0].process
|
25
|
+
```
|
26
|
+
|
27
|
+
* Introduced a new option `raise_on_export_fail` (`boolean`) which is `true` by default. When this option is enabled, LokaliseManager will re-raise any exceptions that happened during the file uploading. When this option is disabled, the exporting process will continue even if something goes wrong. In this case you'll probably need to check the result yourself and make the necessary actions. For example:
|
28
|
+
|
29
|
+
```ruby
|
30
|
+
processes = exporter.export!
|
31
|
+
|
32
|
+
processes.each do |proc_data|
|
33
|
+
if proc_data.success
|
34
|
+
# Everything is good, the uploading is queued
|
35
|
+
puts "#{proc_data.path} is sent to Lokalise!"
|
36
|
+
process = proc_data.process
|
37
|
+
puts "Current process status is #{process.status}"
|
38
|
+
else
|
39
|
+
# Something bad has happened
|
40
|
+
puts "Could not send #{proc_data.path} to Lokalise"
|
41
|
+
puts "Error #{proc_data.error.class}: #{proc_data.error.message}"
|
42
|
+
# Or you could re-raise this exception:
|
43
|
+
# raise proc_data.error.class
|
44
|
+
end
|
45
|
+
end
|
46
|
+
```
|
47
|
+
|
3
48
|
## 2.0.0 (27-Jan-22)
|
4
49
|
|
5
50
|
* `export!` method is now taking advantage of multi-threading (as Lokalise API allows to send requests in parallel since January 2022)
|
data/README.md
CHANGED
@@ -65,7 +65,13 @@ To upload your translation files from a local directory (defaults to `locales/`)
|
|
65
65
|
processes = exporter.export!
|
66
66
|
```
|
67
67
|
|
68
|
-
`processes` will contain an array of
|
68
|
+
`processes` will contain an array of objects responding to the following methods:
|
69
|
+
|
70
|
+
* `success` — usually returns `true` (to learn more, check documentation for the `:raise_on_export_fail` option below)
|
71
|
+
* `process` — returns an object (an instance of the `Lokalise::Resources::QueuedProcess`) representing a [queued background process](https://lokalise.github.io/ruby-lokalise-api/api/queued-processes) as uploading is done in the background on Lokalise.
|
72
|
+
* `path` — returns an instance of the `Pathname` class which represent the file being uploaded.
|
73
|
+
|
74
|
+
You can perform periodic checks to read the status of the process. Here's a very simple example:
|
69
75
|
|
70
76
|
```ruby
|
71
77
|
def uploaded?(process)
|
@@ -79,7 +85,8 @@ def uploaded?(process)
|
|
79
85
|
end
|
80
86
|
|
81
87
|
processes = exporter.export!
|
82
|
-
|
88
|
+
puts "Checking status for the #{processes[0].path} file"
|
89
|
+
uploaded? processes[0].process
|
83
90
|
```
|
84
91
|
|
85
92
|
Please don't forget that Lokalise API has rate limiting and you cannot send more than six requests per second.
|
@@ -162,7 +169,29 @@ In this case the `export_opts` will have `detect_icu_plurals` set to `true` and
|
|
162
169
|
c.skip_file_export = ->(file) { f.split[1].to_s.include?('fr') }
|
163
170
|
```
|
164
171
|
|
165
|
-
* `max_retries_export` (`integer`) — this option is introduced to properly handle Lokalise API rate limiting. If the HTTP status code 429 (too many requests) has been received, LokaliseRails will apply an exponential backoff mechanism with a very simple formula: `2 ** retries` (initially `retries` is `0`). If the maximum number of retries has been reached, a `Lokalise::Error::TooManyRequests` exception will be raised and the export operation will be halted. By default,
|
172
|
+
* `max_retries_export` (`integer`) — this option is introduced to properly handle Lokalise API rate limiting. If the HTTP status code 429 (too many requests) has been received, LokaliseRails will apply an exponential backoff mechanism with a very simple formula: `2 ** retries` (initially `retries` is `0`). If the maximum number of retries has been reached, a `Lokalise::Error::TooManyRequests` exception will be raised and the export operation will be halted. By default, LokaliseManager will make up to `5` retries which potentially means `1 + 2 + 4 + 8 + 16 + 32 = 63` seconds of waiting time. If the `max_retries_export` is less than `1`, LokaliseRails will not perform any retries and give up immediately after receiving error 429.
|
173
|
+
* `raise_on_export_fail` (`boolean`) — default is `true`. When this option is enabled, LokaliseManager will re-raise any exceptions that happened during the file uploading. In other words, if any uploading thread raised an exception, your exporting process will exit with an exception. Suppose, you are uploading 12 translation files; these files will be split in 2 groups with 6 files each, and each group will be uploaded in parallel (using threads). However, suppose some exception happens when uploading the first group. By default this exception will be re-raised for the whole process and the script will never try to upload the second group. If you would like to continue uploading even if an exception happened, set the `raise_on_export_fail` to `false`. In this case the `export!` method will return an array with scheduled processes and with information about processes that were not successfully scheduled. This information is represented as an object with three methods: `path` (contains an instance of the `Pathname` class which says which file could not be uploaded), `error` (the actual exception), and `success` (returns `false`). So, you can use the following snippet to check your processes:
|
174
|
+
|
175
|
+
```ruby
|
176
|
+
processes = exporter.export!
|
177
|
+
|
178
|
+
processes.each do |proc_data|
|
179
|
+
if proc_data.success
|
180
|
+
# Everything is good, the uploading is queued
|
181
|
+
puts "#{proc_data.path} is sent to Lokalise!"
|
182
|
+
process = proc_data.process
|
183
|
+
puts "Current process status is #{process.status}"
|
184
|
+
else
|
185
|
+
# Something bad has happened
|
186
|
+
puts "Could not send #{proc_data.path} to Lokalise"
|
187
|
+
puts "Error #{proc_data.error.class}: #{proc_data.error.message}"
|
188
|
+
# Or you could re-raise this exception:
|
189
|
+
# raise proc_data.error.class
|
190
|
+
end
|
191
|
+
end
|
192
|
+
```
|
193
|
+
|
194
|
+
* For example, you could collect all the files that were uploaded successfully, re-create the exporter object with the `skip_file_export` option (skipping all files that were successfully imported), and re-run the whole exporting process once again.
|
166
195
|
|
167
196
|
### Config to work with formats other than YAML
|
168
197
|
|
@@ -7,13 +7,19 @@ module LokaliseManager
|
|
7
7
|
attr_writer :import_opts, :import_safe_mode, :export_opts, :locales_path,
|
8
8
|
:file_ext_regexp, :skip_file_export, :branch, :timeouts,
|
9
9
|
:translations_loader, :translations_converter, :lang_iso_inferer,
|
10
|
-
:max_retries_export, :max_retries_import, :use_oauth2_token, :silent_mode
|
10
|
+
:max_retries_export, :max_retries_import, :use_oauth2_token, :silent_mode,
|
11
|
+
:raise_on_export_fail
|
11
12
|
|
12
13
|
# Main interface to provide configuration options
|
13
14
|
def config
|
14
15
|
yield self
|
15
16
|
end
|
16
17
|
|
18
|
+
# When enabled, will re-raise any exception that happens during file exporting
|
19
|
+
def raise_on_export_fail
|
20
|
+
@raise_on_export_fail || true
|
21
|
+
end
|
22
|
+
|
17
23
|
# When enabled, won't print any debugging info to $stdout
|
18
24
|
def silent_mode
|
19
25
|
@silent_mode || false
|
@@ -6,6 +6,9 @@ module LokaliseManager
|
|
6
6
|
module TaskDefinitions
|
7
7
|
class Exporter < Base
|
8
8
|
using LokaliseManager::Utils::ArrayUtils
|
9
|
+
|
10
|
+
MAX_THREADS = 6
|
11
|
+
|
9
12
|
# Performs translation file export to Lokalise and returns an array of queued processes
|
10
13
|
#
|
11
14
|
# @return [Array]
|
@@ -14,11 +17,11 @@ module LokaliseManager
|
|
14
17
|
|
15
18
|
queued_processes = []
|
16
19
|
|
17
|
-
all_files.in_groups_of(
|
20
|
+
all_files.in_groups_of(MAX_THREADS) do |files_group|
|
18
21
|
parallel_upload(files_group).each do |thr|
|
19
|
-
raise_on_fail
|
22
|
+
raise_on_fail(thr) if config.raise_on_export_fail
|
20
23
|
|
21
|
-
queued_processes.push thr
|
24
|
+
queued_processes.push thr
|
22
25
|
end
|
23
26
|
end
|
24
27
|
|
@@ -36,19 +39,21 @@ module LokaliseManager
|
|
36
39
|
end
|
37
40
|
|
38
41
|
def raise_on_fail(thread)
|
39
|
-
raise
|
42
|
+
raise(thread.error.class, "Error while trying to upload #{thread.path}: #{thread.error.message}") unless thread.success
|
40
43
|
end
|
41
44
|
|
42
45
|
# Performs the actual file uploading to Lokalise. If the API rate limit is exceeed,
|
43
46
|
# applies exponential backoff
|
44
47
|
def do_upload(f_path, r_path)
|
48
|
+
proc_klass = Struct.new(:success, :process, :path, :error, keyword_init: true)
|
49
|
+
|
45
50
|
Thread.new do
|
46
51
|
process = with_exp_backoff(config.max_retries_export) do
|
47
52
|
api_client.upload_file project_id_with_branch, opts(f_path, r_path)
|
48
53
|
end
|
49
|
-
|
54
|
+
proc_klass.new success: true, process: process, path: f_path
|
50
55
|
rescue StandardError => e
|
51
|
-
|
56
|
+
proc_klass.new success: false, path: f_path, error: e
|
52
57
|
end
|
53
58
|
end
|
54
59
|
|
@@ -14,6 +14,11 @@ describe LokaliseManager::GlobalConfig do
|
|
14
14
|
fake_class.project_id = '123.abc'
|
15
15
|
end
|
16
16
|
|
17
|
+
it 'is possible to set raise_on_export_fail' do
|
18
|
+
allow(fake_class).to receive(:raise_on_export_fail=).with(false)
|
19
|
+
fake_class.raise_on_export_fail = false
|
20
|
+
end
|
21
|
+
|
17
22
|
it 'is possible to set silent_mode' do
|
18
23
|
allow(fake_class).to receive(:silent_mode=).with(true)
|
19
24
|
fake_class.silent_mode = true
|
@@ -27,7 +27,7 @@ describe LokaliseManager::TaskDefinitions::Exporter do
|
|
27
27
|
process = nil
|
28
28
|
|
29
29
|
VCR.use_cassette('upload_files_multiple') do
|
30
|
-
expect(-> { process = described_object.export!.first }).to output(/complete!/).to_stdout
|
30
|
+
expect(-> { process = described_object.export!.first.process }).to output(/complete!/).to_stdout
|
31
31
|
end
|
32
32
|
|
33
33
|
expect(process.project_id).to eq(project_id)
|
@@ -49,6 +49,28 @@ describe LokaliseManager::TaskDefinitions::Exporter do
|
|
49
49
|
expect(described_object).to have_received(:api_client).at_least(12).times
|
50
50
|
expect(fake_client).to have_received(:upload_file).exactly(12).times
|
51
51
|
end
|
52
|
+
|
53
|
+
it 'handles too many requests but does not re-raise anything when raise_on_export_fail is false' do
|
54
|
+
allow(described_object.config).to receive(:max_retries_export).and_return(1)
|
55
|
+
allow(described_object.config).to receive(:raise_on_export_fail).and_return(false)
|
56
|
+
allow(described_object).to receive(:sleep).and_return(0)
|
57
|
+
|
58
|
+
fake_client = instance_double('Lokalise::Client')
|
59
|
+
allow(fake_client).to receive(:token).with(any_args).and_return('fake_token')
|
60
|
+
allow(fake_client).to receive(:upload_file).with(any_args).and_raise(Lokalise::Error::TooManyRequests)
|
61
|
+
allow(described_object).to receive(:api_client).and_return(fake_client)
|
62
|
+
processes = []
|
63
|
+
expect(-> { processes = described_object.export! }).not_to raise_error
|
64
|
+
|
65
|
+
expect(processes[0].path.to_s).to include('en_0')
|
66
|
+
expect(processes[0].success).to be false
|
67
|
+
expect(processes[1].error.class).to eq(Lokalise::Error::TooManyRequests)
|
68
|
+
expect(processes.count).to eq(7)
|
69
|
+
|
70
|
+
expect(described_object).to have_received(:sleep).exactly(7).times
|
71
|
+
expect(described_object).to have_received(:api_client).at_least(14).times
|
72
|
+
expect(fake_client).to have_received(:upload_file).exactly(14).times
|
73
|
+
end
|
52
74
|
end
|
53
75
|
end
|
54
76
|
|
@@ -68,7 +90,7 @@ describe LokaliseManager::TaskDefinitions::Exporter do
|
|
68
90
|
process = nil
|
69
91
|
|
70
92
|
VCR.use_cassette('upload_files') do
|
71
|
-
expect(-> { process = described_object.export!.first }).not_to output(/complete!/).to_stdout
|
93
|
+
expect(-> { process = described_object.export!.first.process }).not_to output(/complete!/).to_stdout
|
72
94
|
end
|
73
95
|
|
74
96
|
expect(process.status).to eq('queued')
|
@@ -78,7 +100,7 @@ describe LokaliseManager::TaskDefinitions::Exporter do
|
|
78
100
|
it 'sends a proper API request' do
|
79
101
|
process = VCR.use_cassette('upload_files') do
|
80
102
|
described_object.export!
|
81
|
-
end.first
|
103
|
+
end.first.process
|
82
104
|
|
83
105
|
expect(process.project_id).to eq(project_id)
|
84
106
|
expect(process.status).to eq('queued')
|
@@ -87,11 +109,16 @@ describe LokaliseManager::TaskDefinitions::Exporter do
|
|
87
109
|
it 'sends a proper API request when a different branch is provided' do
|
88
110
|
allow(described_object.config).to receive(:branch).and_return('develop')
|
89
111
|
|
90
|
-
|
112
|
+
process_data = VCR.use_cassette('upload_files_branch') do
|
91
113
|
described_object.export!
|
92
114
|
end.first
|
93
115
|
|
94
116
|
expect(described_object.config).to have_received(:branch).at_most(2).times
|
117
|
+
expect(process_data.success).to be true
|
118
|
+
expect(process_data.path.to_s).to include('en.yml')
|
119
|
+
|
120
|
+
process = process_data.process
|
121
|
+
expect(process).to be_an_instance_of(Lokalise::Resources::QueuedProcess)
|
95
122
|
expect(process.project_id).to eq(project_id)
|
96
123
|
expect(process.status).to eq('queued')
|
97
124
|
end
|