consul-templaterb 1.28.0 → 1.28.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c15ee78520048d7d091882b9d3b1d43b4cc5f5658cf39d782f06e1dc617d8cd9
4
- data.tar.gz: 6f77a9daab557420f2640a314a97f78af68e26fec91c1a66f3ee575d9bf150f8
3
+ metadata.gz: ca62994383929bd3874db5527f85efeaaf184fdb83b8f3b17e05fbadbfbbc90a
4
+ data.tar.gz: e878ff880557ffdfd8db6e5cb900a19e8191d4db11ba28fa0b4e7bbe9b1b9240
5
5
  SHA512:
6
- metadata.gz: 8c2b374a9400061f3ffd673e418771bb5685ec3d13a631c3839660121d86575be1fbbd1cd99f7ee1a4ddedf850ec72fa984ca084575da587b7a239d2e4654b2e
7
- data.tar.gz: '0933c0a2f1f58c617c184679ab68c1e90d55f1d9ab796e45336ddbbf49ae2fd73cf1aafbfea6e921254a520c8be3e5f0f380cfbdacd694ba2de6c5f8e4f5a074'
6
+ metadata.gz: 62afc852ef61329a30bba3a67f43ed372130d9a87cd1afce559b5f88f9aa0a6db336c84c63b173eaa96896e8c9088686cfa37f10d540cd91c80ca2fb976b00b3
7
+ data.tar.gz: 4a02aa15f2d4f95e5d144e916bf6a51190a90756e134456aece5b4afbf7ba106e150faeb8cf7b0e09ff27f1f61b237c47dd11f236299d922a519e91b019e3705
@@ -2,6 +2,12 @@
2
2
 
3
3
  ## (UNRELEASED)
4
4
 
5
+ ## 1.28.1 (Sept 30,, 2020)
6
+
7
+ IMPROVEMENTS:
8
+
9
+ * Reduced size of GEM from 400k o 100k by removing not needed files
10
+
5
11
  ## 1.28.0 (Sept 25, 2020)
6
12
 
7
13
  NEW FEATURES:
@@ -10,7 +16,7 @@ NEW FEATURES:
10
16
  to a process executed. This avoids for instance reloading too much a HAProxy configuration
11
17
  without having to play with `-w` as described in [#69](https://github.com/criteo/consul-templaterb/issues/69)
12
18
 
13
- BUGFIX:
19
+ BUG FIXES:
14
20
 
15
21
  * Removed warnings at runtime with Ruby 2.7+
16
22
  * Minor JS fix in Consul-UI (Added missing unused parameter to function `serviceTitleGenerator`)
@@ -25,7 +31,7 @@ IMPROVEMENTS:
25
31
 
26
32
  BUGIX:
27
33
 
28
- * Fix collision in JSON queries when using payload in requests #68
34
+ * Fix collision in JSON queries when using payload in requests [#68](https://github.com/criteo/consul-templaterb/pull/68)
29
35
 
30
36
  ## 1.27.0 (June 5, 2020)
31
37
 
@@ -38,15 +44,15 @@ NEW FEATURES:
38
44
 
39
45
  ## 1.26.3 (April 15, 2020)
40
46
 
41
- BUGFIX:
47
+ BUG FIXES:
42
48
 
43
49
  * Removed all Criteo specific decorators from Consul-UI
44
50
 
45
51
  ## 1.26.2 (April 15, 2020)
46
52
 
47
- BUGFIX:
53
+ BUG FIXES:
48
54
 
49
- * Fixed broken Dockerfile (was missing the new `decorator.js.erb` file). Fixes #61 (Thanks to @ simongareste)
55
+ * Fixed broken Dockerfile (was missing the new `decorator.js.erb` file). Fixes #61 (Thanks to [@simongareste](https://github.com/simongareste))
50
56
 
51
57
  NEW FEATURES:
52
58
 
@@ -54,7 +60,7 @@ NEW FEATURES:
54
60
 
55
61
  ## 1.26.1 (March 27, 2020)
56
62
 
57
- BUGFIX:
63
+ BUG FIXES:
58
64
 
59
65
  * Using `agent: http://vault_or_consul_agent:port>` was not properly taken into account in some endpoints
60
66
 
@@ -72,7 +78,7 @@ NEW FEATURES:
72
78
 
73
79
  ## 1.25.2 (February 29, 2020)
74
80
 
75
- BUGFIX:
81
+ BUG FIXES:
76
82
 
77
83
  * Update rake to 12.3.3 to fix [CVE-2020-8130](https://github.com/advisories/GHSA-jppv-gw3r-w3q8)
78
84
 
@@ -91,7 +97,7 @@ NEW FEATURES:
91
97
 
92
98
  ## 1.24.1 (February 19, 2020)
93
99
 
94
- BUGFIX:
100
+ BUG FIXES:
95
101
 
96
102
  * Properly set service meta in node meta decorator
97
103
 
@@ -108,7 +114,7 @@ NEW FEATURES:
108
114
  * Implementation of #59 - implementation of `--retry` and `--vault-retry` new flags
109
115
  Those flags work in a similar way as in consul-template: stop program after X failures
110
116
  of consul or vault endpoints
111
- * Added --fail-fast that stop the programs immediately if vault or consul are not available
117
+ * Added `--fail-fast` that stop the programs immediately if vault or consul are not available
112
118
  at startup (also works with `--once`)
113
119
 
114
120
  ## 1.22.0 (January 17, 2020)
@@ -119,7 +125,7 @@ NEW FEATURES:
119
125
 
120
126
  ## 1.21.8 (January 2, 2020)
121
127
 
122
- BUGFIX:
128
+ BUG FIXES:
123
129
 
124
130
  * Escape properly metadata containing double quotes in prometheus exporter
125
131
 
@@ -133,13 +139,13 @@ IMPROVEMENTS:
133
139
 
134
140
  IMPROVEMENTS:
135
141
 
136
- * Added node_meta_info for serviceInstanceDecorator and serviceMetaDecorator
142
+ * Added `node_meta_info` for `serviceInstanceDecorator` and `serviceMetaDecorator` in Consul-UI
137
143
 
138
144
  ## 1.21.5 (December 6, 2019)
139
145
 
140
146
  NEW FEATURES:
141
147
 
142
- * Added clean() method in nodes.js to allow nexw behaviors
148
+ * Added `clean()` method in nodes.js to allow new behaviors
143
149
 
144
150
  ## 1.21.4 (November 28, 2019)
145
151
 
@@ -170,7 +176,7 @@ IMPROVEMENTS:
170
176
  ## 1.21.0 (November 21, 2019)
171
177
 
172
178
  * added function `templates` to list all templates being rendered
173
- * added support for JS decorators in consul-ui (thanks to @Thib17)
179
+ * added support for JS decorators in consul-ui (thanks to [@Thib17](https://github.com/Thib17))
174
180
 
175
181
  ## 1.20.0 (October 16, 2019)
176
182
 
@@ -189,7 +195,7 @@ NEW FEATURES:
189
195
 
190
196
  * Added new function `checks_for_node`
191
197
 
192
- BUGFIXs:
198
+ BUG FIXES:
193
199
 
194
200
  * Avoid try publishing several times Gem on rubygems.org
195
201
 
@@ -211,13 +217,13 @@ IMPROVEMENTS:
211
217
 
212
218
  ## 1.18.3 (September 2, 2019)
213
219
 
214
- BUGFIX:
220
+ BUG FIXES:
215
221
 
216
222
  * When vault receives at timeout, correctly reschedule it
217
223
 
218
224
  ## 1.18.2 (August 28, 2019)
219
225
 
220
- BUGFIX:
226
+ BUG FIXES:
221
227
 
222
228
  * In Consul UI, showing data from KV with markup was not properly handled
223
229
 
@@ -233,7 +239,7 @@ Support any request method for remote_resource.as_json (#41)
233
239
 
234
240
  ## 1.18.1 (July 27, 2019)
235
241
 
236
- BUGFIX:
242
+ BUG FIXES:
237
243
 
238
244
  Fixed wrong lazy initialization in `remote_resource.as_json` that
239
245
  cause too many connections to be opened.
@@ -242,13 +248,13 @@ cause too many connections to be opened.
242
248
 
243
249
  NEW FEATURES:
244
250
 
245
- * Support for `remote_resource` provided by @kamaradclimber
251
+ * Support for `remote_resource` provided by [@kamaradclimber](https://github.com/kamaradclimber)
246
252
  * Added support for `remote_resource.as_json` to fetch JSON remote resource from a web server
247
253
  * Added `samples/list_ruby_versions_from_rubygems.txt.erb` to demonstrate usage
248
254
 
249
255
  ## 1.17.3 (July 18, 2019)
250
256
 
251
- BUGFIX:
257
+ BUG FIXES:
252
258
 
253
259
  * Added gem parallel as a dependency to allow `samples/prometheus_consul_coordinates.erb`
254
260
  to work properly
@@ -373,7 +379,7 @@ NEW FEATURES:
373
379
 
374
380
  ## 1.10.1 (February 28, 2019)
375
381
 
376
- BUGFIX:
382
+ BUG FIXES:
377
383
 
378
384
  * Ensure that timeline sort properly events when healthchecks are removed (eg: maintenance)
379
385
 
@@ -396,7 +402,7 @@ IMPROVEMENTS:
396
402
 
397
403
  ## 1.9.8 (January 16, 2019)
398
404
 
399
- BUGFIX:
405
+ BUG FIXES:
400
406
 
401
407
  * When default value was the same as real value, endpoints were always marked as
402
408
  dirty, thus rendering of templates did never succeed.
@@ -412,7 +418,7 @@ IMPROVEMENTS:
412
418
 
413
419
  ## 1.9.6 (January 15, 2019)
414
420
 
415
- BUGFIX:
421
+ BUG FIXES:
416
422
 
417
423
  * Keep connections open properly as it increase timeouts on Consul servers on
418
424
  very large templates
@@ -423,7 +429,7 @@ IMPROVEMENTS:
423
429
 
424
430
  ## 1.9.5 (January 14, 2019)
425
431
 
426
- BUGFIX:
432
+ BUG FIXES:
427
433
 
428
434
  * Ensure to always re-open Connection to Consul agent in case of network error
429
435
 
@@ -473,11 +479,11 @@ IMPROVEMENTS:
473
479
 
474
480
  OPTIMIZATIONS:
475
481
 
476
- * Better network issue because of X-Consul-Index parsing bug
482
+ * Better network usage because of X-Consul-Index parsing bug
477
483
 
478
- NEW FEATURES:
484
+ BUG FIXES:
479
485
 
480
- * value.endpoint.x_consul_index now works as expected
486
+ * `value.endpoint.x_consul_index` now works as expected
481
487
 
482
488
  IMPROVEMENTS:
483
489
 
@@ -517,7 +523,7 @@ NEW FEATURES:
517
523
 
518
524
  ## 1.8.1 (December 12, 2018)
519
525
 
520
- BUGFIX:
526
+ BUG FIXES:
521
527
 
522
528
  * Properly fill `template_info` strtucture when hot reload is performed so templates using
523
529
  `template_info()` new function can behave nicely.
@@ -626,8 +632,8 @@ IMPROVEMENTS:
626
632
  * [Prometheus template](samples/metrics.erb) to export easily Consul
627
633
  informations about nodes, datacenters and all services states
628
634
  * Code style cleanup + travis now enforces Rubocop
629
- * Remove criteo references in spec files thanks to @pierrecdn
630
- * Bitrate display more consistent thanks to @pierrecdn
635
+ * Remove criteo references in spec files thanks to [@pierrecdn](https://github.com/pierrecdn)
636
+ * Bitrate display more consistent thanks to [@pierrecdn](https://github.com/pierrecdn)
631
637
 
632
638
  ## 1.5.3 (September 24, 2018)
633
639
 
@@ -388,7 +388,7 @@ name or its ID. If DC is specified, will lookup for given node in another datace
388
388
 
389
389
  [Find all the checks](https://www.consul.io/api/health.html#list-checks-for-service) of a given service.
390
390
 
391
- ## def checks_in_state(check_state, dc: nil, [agent: consul_agent_address])
391
+ ## checks_in_state(check_state, dc: nil, [agent: consul_agent_address])
392
392
 
393
393
  [Find all the checks in a given state](https://www.consul.io/api-docs/health#list-checks-in-state) in the whole cluster.
394
394
 
@@ -1,5 +1,5 @@
1
1
  module Consul
2
2
  module Async
3
- VERSION = '1.28.0'.freeze
3
+ VERSION = '1.28.1'.freeze
4
4
  end
5
5
  end
metadata CHANGED
@@ -1,57 +1,57 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: consul-templaterb
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.28.0
4
+ version: 1.28.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - SRE Core Services
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-09-25 00:00:00.000000000 Z
11
+ date: 2020-09-29 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: em-http-request
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - ">="
17
+ - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: 1.1.5
19
+ version: '1.1'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - ">="
24
+ - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: 1.1.5
26
+ version: '1.1'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: eventmachine
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - ">="
31
+ - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: 1.2.7
33
+ version: '1.2'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - ">="
38
+ - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: 1.2.7
40
+ version: '1.2'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: parallel
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
- - - ">="
45
+ - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: 1.2.7
47
+ version: '1.2'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
- - - ">="
52
+ - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: 1.2.7
54
+ version: '1.2'
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: bundler
57
57
  requirement: !ruby/object:Gem::Requirement
@@ -72,14 +72,14 @@ dependencies:
72
72
  requirements:
73
73
  - - "~>"
74
74
  - !ruby/object:Gem::Version
75
- version: 12.3.3
75
+ version: '12.3'
76
76
  type: :development
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
80
  - - "~>"
81
81
  - !ruby/object:Gem::Version
82
- version: 12.3.3
82
+ version: '12.3'
83
83
  - !ruby/object:Gem::Dependency
84
84
  name: rspec
85
85
  requirement: !ruby/object:Gem::Requirement
@@ -98,58 +98,58 @@ dependencies:
98
98
  name: rspec_junit_formatter
99
99
  requirement: !ruby/object:Gem::Requirement
100
100
  requirements:
101
- - - ">="
101
+ - - "~>"
102
102
  - !ruby/object:Gem::Version
103
- version: '0'
103
+ version: 0.4.1
104
104
  type: :development
105
105
  prerelease: false
106
106
  version_requirements: !ruby/object:Gem::Requirement
107
107
  requirements:
108
- - - ">="
108
+ - - "~>"
109
109
  - !ruby/object:Gem::Version
110
- version: '0'
110
+ version: 0.4.1
111
111
  - !ruby/object:Gem::Dependency
112
112
  name: rubocop
113
113
  requirement: !ruby/object:Gem::Requirement
114
114
  requirements:
115
- - - '='
115
+ - - "~>"
116
116
  - !ruby/object:Gem::Version
117
117
  version: 0.80.0
118
118
  type: :development
119
119
  prerelease: false
120
120
  version_requirements: !ruby/object:Gem::Requirement
121
121
  requirements:
122
- - - '='
122
+ - - "~>"
123
123
  - !ruby/object:Gem::Version
124
124
  version: 0.80.0
125
125
  - !ruby/object:Gem::Dependency
126
126
  name: rubocop-junit-formatter
127
127
  requirement: !ruby/object:Gem::Requirement
128
128
  requirements:
129
- - - ">="
129
+ - - '='
130
130
  - !ruby/object:Gem::Version
131
- version: '0'
131
+ version: 0.1.4
132
132
  type: :development
133
133
  prerelease: false
134
134
  version_requirements: !ruby/object:Gem::Requirement
135
135
  requirements:
136
- - - ">="
136
+ - - '='
137
137
  - !ruby/object:Gem::Version
138
- version: '0'
138
+ version: 0.1.4
139
139
  - !ruby/object:Gem::Dependency
140
140
  name: webmock
141
141
  requirement: !ruby/object:Gem::Requirement
142
142
  requirements:
143
- - - ">="
143
+ - - "~>"
144
144
  - !ruby/object:Gem::Version
145
- version: '0'
145
+ version: '2.1'
146
146
  type: :development
147
147
  prerelease: false
148
148
  version_requirements: !ruby/object:Gem::Requirement
149
149
  requirements:
150
- - - ">="
150
+ - - "~>"
151
151
  - !ruby/object:Gem::Version
152
- version: '0'
152
+ version: '2.1'
153
153
  description: A ruby implementation of Consul Template with support of erb templating
154
154
  with hi-performance on large clusters and advanced process management features.
155
155
  email:
@@ -162,28 +162,10 @@ extra_rdoc_files:
162
162
  - CHANGELOG.md
163
163
  - TemplateAPI.md
164
164
  files:
165
- - ".gitignore"
166
- - ".rspec"
167
- - ".rubocop.yml"
168
- - ".ruby_app"
169
- - ".travis.yml"
170
165
  - CHANGELOG.md
171
- - CODE_OF_CONDUCT.md
172
- - CONTRIBUTING.md
173
- - Dockerfile
174
- - Gemfile
175
- - INTERNALS.md
176
- - LICENSE.txt
177
166
  - README.md
178
- - Rakefile
179
167
  - TemplateAPI.md
180
168
  - bin/consul-templaterb
181
- - consul-templaterb.gemspec
182
- - docker-nginx-conf/nginx.conf
183
- - docs/article-06_Template-based_discovery_with_consul-templaterb.md
184
- - docs/images/consul-templaterb.png
185
- - docs/images/consul-templaterb.svg
186
- - docs/images/consul-ui_001.png
187
169
  - lib/consul/async/consul_endpoint.rb
188
170
  - lib/consul/async/consul_template.rb
189
171
  - lib/consul/async/consul_template_engine.rb
@@ -252,7 +234,7 @@ files:
252
234
  - samples/vault-ldap.txt.erb
253
235
  homepage: https://rubygems.org/gems/consul-templaterb
254
236
  licenses:
255
- - Apache v2
237
+ - Apache-2.0
256
238
  metadata:
257
239
  bug_tracker_uri: https://github.com/criteo/consul-templaterb/issues
258
240
  changelog_uri: https://github.com/criteo/consul-templaterb/blob/master/CHANGELOG.md
data/.gitignore DELETED
@@ -1,41 +0,0 @@
1
- /.bundle/
2
- /.yardoc
3
- /Gemfile.lock
4
- /_yardoc/
5
- /coverage/
6
- /doc/
7
- /pkg/
8
- /spec/reports/
9
- /tmp/
10
- /samples/**/*.json
11
- /samples/**/*.xml
12
- /samples/**/*.html
13
- /samples/**/*.txt
14
- /samples/**/*.cfg
15
- samples/consul-ui/decorators.js
16
- /samples/hosts
17
- /samples/hosts_per_services
18
- /samples/prometheus_consul_coordinates
19
- /samples/ready
20
- /samples/render_template_from_kv
21
- /samples/consul_template
22
- /samples/metrics
23
- /samples/service_checks_metrics
24
- /samples/find_blocked_choregraphies
25
- /spec/consul/async/resources/templates/*.txt
26
- /spec/consul/async/resources/templates/*.result
27
- /out.txt
28
-
29
- # rspec failure tracking
30
- .rspec_status
31
- # Mac OS
32
- .DS_Store
33
-
34
- # IntelliJ project files
35
- *.iml
36
- *.iws
37
- *.ipr
38
- .idea/
39
-
40
- # Misc
41
- *.swp
data/.rspec DELETED
@@ -1,2 +0,0 @@
1
- --format documentation
2
- --color
@@ -1,57 +0,0 @@
1
- AllCops:
2
- Exclude:
3
- - bundle/**/*
4
- TargetRubyVersion: 2.4
5
-
6
- Layout/LineLength:
7
- Max: 175
8
-
9
- Metrics/AbcSize:
10
- Max: 87
11
-
12
- Metrics/BlockLength:
13
- Max: 188
14
-
15
- Metrics/BlockNesting:
16
- Max: 4
17
-
18
- Metrics/ClassLength:
19
- Max: 285
20
-
21
- Metrics/CyclomaticComplexity:
22
- Max: 21
23
-
24
- Metrics/MethodLength:
25
- Max: 68
26
-
27
- Metrics/ParameterLists:
28
- Max: 18
29
-
30
- Metrics/PerceivedComplexity:
31
- Max: 24
32
-
33
- # We use `dc` as a parameter in many methods
34
- Naming/MethodParameterName:
35
- Enabled: false
36
-
37
- Naming/VariableNumber:
38
- Enabled: false
39
-
40
- Style/Documentation:
41
- Enabled: true
42
-
43
- Style/FrozenStringLiteralComment:
44
- Enabled: false
45
-
46
- Style/HashEachMethods:
47
- Enabled: true
48
-
49
- Style/HashTransformKeys:
50
- Enabled: true
51
-
52
- Style/HashTransformValues:
53
- Enabled: true
54
-
55
- Style/MultilineBlockChain:
56
- Enabled: false
57
-
data/.ruby_app DELETED
File without changes
@@ -1,19 +0,0 @@
1
- language: ruby
2
- rvm:
3
- - 2.4.10
4
- - 2.5.8
5
- - 2.6.6
6
- - 2.7.1
7
- jobs:
8
- include:
9
- - stage: Gem release
10
- rvm: 2.5.8
11
- script: echo "Publishing consul-templaterb on rubygems.org ..."
12
- deploy:
13
- provider: rubygems
14
- api_key:
15
- secure: l78H6Ow9HCeEz0hcAP2XCXaGGBNRXZ/QLFPNzwyNmHToko9WqZj6mbiNIcVwZ8aCcZEzFmtcB7m1lC1l+mvot3Snfh4OUErK7EXMLiNxUWybtA4qJqmsb2J8JGS69pTohPKagi1YyRbyVs2tJpM5OVazcNxEQDAVkWVxJuPVz6tsn4rXZsL77nTSVyB6YGKgguRGhtS/3M0ufk1ymOpuk8pxXKaotlYGRXmht2OHW+FDQczTMgqu6jiZubjWlNHVxtYn+CsLYNj1BswzD6fF1uous6nchV0if2QW5N4OJVyZCx3yEQ0RswfI8G1kG/Jm8wd2lSknY2wpnd1qkRnAcFlKBpUv8aQd6SdbSO84GTAOvucgK7Mcf0LRvYSE2FyagaAPFSfMybWd+GS26g5AuRPpkhtsS9ijfdIzpudiF18pvnJ+wQj+hWM/THWowYNyKv9NeckxdrULaHPisA346r5eM/EADOhyEUtPm11PFgcheGjff1H4qXGc9gcm50u6mNNHkWv38oGhAPpRk/bU7qybTKqRKNGWGP0MiT4WVIFXp62/JMY7AmufOdJx5Ca5JVsaWoN4fsS9r3+R+FxvRtV9NOrRJVKKaQhzi4UHEUDQ2hQjpLDijD5L0vUsl6arl+nufTt5G5jp9QvdY7fhYT1mpGgWj0hTBd8Pxsl3N8U=
16
- gem: consul-templaterb
17
- on:
18
- tags: true
19
- repo: criteo/consul-templaterb
@@ -1,76 +0,0 @@
1
- # Contributor Covenant Code of Conduct
2
-
3
- ## Our Pledge
4
-
5
- In the interest of fostering an open and welcoming environment, we as
6
- contributors and maintainers pledge to making participation in our project and
7
- our community a harassment-free experience for everyone, regardless of age, body
8
- size, disability, ethnicity, sex characteristics, gender identity and expression,
9
- level of experience, education, socio-economic status, nationality, personal
10
- appearance, race, religion, or sexual identity and orientation.
11
-
12
- ## Our Standards
13
-
14
- Examples of behavior that contributes to creating a positive environment
15
- include:
16
-
17
- * Using welcoming and inclusive language
18
- * Being respectful of differing viewpoints and experiences
19
- * Gracefully accepting constructive criticism
20
- * Focusing on what is best for the community
21
- * Showing empathy towards other community members
22
-
23
- Examples of unacceptable behavior by participants include:
24
-
25
- * The use of sexualized language or imagery and unwelcome sexual attention or
26
- advances
27
- * Trolling, insulting/derogatory comments, and personal or political attacks
28
- * Public or private harassment
29
- * Publishing others' private information, such as a physical or electronic
30
- address, without explicit permission
31
- * Other conduct which could reasonably be considered inappropriate in a
32
- professional setting
33
-
34
- ## Our Responsibilities
35
-
36
- Project maintainers are responsible for clarifying the standards of acceptable
37
- behavior and are expected to take appropriate and fair corrective action in
38
- response to any instances of unacceptable behavior.
39
-
40
- Project maintainers have the right and responsibility to remove, edit, or
41
- reject comments, commits, code, wiki edits, issues, and other contributions
42
- that are not aligned to this Code of Conduct, or to ban temporarily or
43
- permanently any contributor for other behaviors that they deem inappropriate,
44
- threatening, offensive, or harmful.
45
-
46
- ## Scope
47
-
48
- This Code of Conduct applies both within project spaces and in public spaces
49
- when an individual is representing the project or its community. Examples of
50
- representing a project or community include using an official project e-mail
51
- address, posting via an official social media account, or acting as an appointed
52
- representative at an online or offline event. Representation of a project may be
53
- further defined and clarified by project maintainers.
54
-
55
- ## Enforcement
56
-
57
- Instances of abusive, harassing, or otherwise unacceptable behavior may be
58
- reported by contacting the project team at p.souchay@criteo.com. All
59
- complaints will be reviewed and investigated and will result in a response that
60
- is deemed necessary and appropriate to the circumstances. The project team is
61
- obligated to maintain confidentiality with regard to the reporter of an incident.
62
- Further details of specific enforcement policies may be posted separately.
63
-
64
- Project maintainers who do not follow or enforce the Code of Conduct in good
65
- faith may face temporary or permanent repercussions as determined by other
66
- members of the project's leadership.
67
-
68
- ## Attribution
69
-
70
- This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
71
- available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
72
-
73
- [homepage]: https://www.contributor-covenant.org
74
-
75
- For answers to common questions about this code of conduct, see
76
- https://www.contributor-covenant.org/faq
@@ -1,25 +0,0 @@
1
- # How to contribute to Consul-Templaterb ?
2
-
3
- ## Architecxture Overview
4
-
5
- Please read [INTERNALS.md](INTERNALS.md) to understand how it works.
6
-
7
- ## Samples
8
-
9
- The [samples/](samples/) directory contains lots of useable templates that might help others using your new features.
10
- Keep them less specific as possible (for instance use environment variables to allow others to use it directly
11
- without relying on a specic path in KV).
12
-
13
- Read first [TemplateAPI.md](TemplateAPI.md) to ensure you are using the best functions for the job, the API is kept
14
- compact and powerful with many examples.
15
-
16
- All the samples are unit-tested, so launch rspec before commiting, a template not converging will not be merged.
17
-
18
- ## Want new features/Found a bug?
19
-
20
- [Report it](https://github.com/criteo/consul-templaterb/issues), but read carefully [TemplateAPI.md](TemplateAPI.md) and
21
- [README.md](README.md)
22
-
23
- ## Want to add features/fix a bug?
24
-
25
- Contributions in code are welcomed, create a fork and then a Pull Request, we will review it quickly.
data/Dockerfile DELETED
@@ -1,15 +0,0 @@
1
- FROM ruby:2.5
2
-
3
- WORKDIR /usr/src/app
4
- COPY . .
5
-
6
- RUN apt-get update -qq && apt-get install -y build-essential nginx-light
7
-
8
- RUN bundle install
9
-
10
- EXPOSE 80
11
- ENV LANG C.UTF-8
12
- ENV CONSUL_HTTP_ADDR http://consul-relay.service.consul.preprod.crto.in:8500
13
-
14
- ENTRYPOINT ["/usr/local/bin/bundle", "exec", "consul-templaterb"]
15
- CMD ["--template", "samples/consul-ui/consul-keys-ui.html.erb", "--template", "samples/consul-ui/decorators.js.erb", "--template", "samples/consul-ui/consul-nodes-ui.html.erb", "--template", "samples/consul-ui/consul-services-ui.html.erb", "--template", "samples/consul-ui/consul-timeline-ui.html.erb", "--template", "samples/consul-ui/consul_keys.json.erb", "--template", "samples/consul-ui/consul_nodes.json.erb", "--template", "samples/consul-ui/consul_services.json.erb", "--template", "samples/consul-ui/timeline.json.erb", "--template", "samples/consul-ui/consul-services-ui.html.erb:samples/consul-ui/index.html:touch samples/consul-ui/ready", "--sig-reload=NONE", "--exec=nginx -c /usr/src/app/docker-nginx-conf/nginx.conf"]
data/Gemfile DELETED
@@ -1,5 +0,0 @@
1
- source 'https://rubygems.org'
2
-
3
- # Specify your gem's dependencies in consul-templaterb.gemspec
4
- gemspec
5
- gem 'bootstrap', '~> 4.1', '>= 4.1.2'
@@ -1,49 +0,0 @@
1
- # consul-templaterb Internal Architecture Overview
2
-
3
- ## Consul basics
4
-
5
- `consul-templaterb` is a ruby library (or `GEM`) with a single binary that does render `.erb` files based on Consul exposed endpoints.
6
-
7
- Consul is a kind of distributed K/V tree store that exposes most of his APIs using HTTP endpoints with a way to watch for changes based on an Index. Each part of the Consul Tree has its own Index reflecting the last id of transaction for this
8
- part of the tree. It means that Consul has a kind of transaction ID for each subtree of the data.
9
-
10
- Consul also exposes some specific endpoints to get some discovery specific endpoints (eg: `/v1/catalog/services` to get
11
- the list of services registered in the cluster).
12
-
13
- For each endpoint, `consul-templaterb` performs a watch by getting information about the last transaction on this endpoints, storing it locally and watching at Consul to get updates about the data within this endpoint.
14
-
15
- ![Main Loop](docs/images/consul-templaterb.svg)
16
-
17
- ## How it works
18
-
19
- There is a loop in the engine, every second, all templates are rendered using the `ERB` template engine.
20
-
21
- Each template is assigned to a `ConsulTemplateRender` object that keeps tracks of changes on disk of template (in order
22
- to be able to hot-reload templates) as well as monitoring if the template is completely rendered, aka all data from
23
- Consul is properly retrieved and consistent.
24
-
25
- At each rendering, each template using a Consul Endpoint (for instance, calling `datacenters()`) does register itself
26
- in `EndPointsManager`. If the endpoint is already registered, fetch the result if available and return the value, if not,
27
- the endpoint is tagged as dirty. When the rendering of a template is completed, a check is performed to see if some endpoints used by the template are still marked as dirty. Being dirty means that some data from Consul is missing, so
28
- that the result is not the definitive one. If no endpoint is dirty, then the result of template can be rendered on disk
29
- and template is complete.
30
-
31
- At startup, in order to converge faster, templates are rendered more quickly that every 1s to speed up startup time for first convergence, meaning that the delay between each rendering of template might be very fast, especially at startup (so, DO NOT USE Ruby I/O in templates unless you really know what you are doing)
32
-
33
- ### EndPointsManager
34
-
35
- `EndPointsManager` is the object responsible of keeping track the endpoints (basically the I/O) of the whole library.
36
- When a new Consul endpoint is added by a template, it is registered in that object. Each time a template is rendered,
37
- all the endpoints are marked as used. When some endpoints are not used for a while (a few rendering loops), the
38
- `EndPointsManager` will garbage collect those endpoints.
39
-
40
- ### ConsulTemplateEngine
41
-
42
- When `consul-templaterb` starts, it creates an [ConsulTemplateEngine](lib/consul/async/consul_template_engine.rb) that
43
- aggregates all options, template files and parameters. The engine then starts a loop running every second (this can be changed) that:
44
- * perform a rendering of each of the `.erb` template files using a `ConsulTemplateRender` object
45
- * when all template are rendered the first time, will start some program(s) if specified (the `--exec` parameters)
46
-
47
- ### ConsulTemplateRender
48
-
49
- This object keeps track of template, its last state (rendered or not), and whether the file template needs to be reloaded (aka if file has been modified on filesystem).
@@ -1,201 +0,0 @@
1
- Apache License
2
- Version 2.0, January 2004
3
- http://www.apache.org/licenses/
4
-
5
- TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
-
7
- 1. Definitions.
8
-
9
- "License" shall mean the terms and conditions for use, reproduction,
10
- and distribution as defined by Sections 1 through 9 of this document.
11
-
12
- "Licensor" shall mean the copyright owner or entity authorized by
13
- the copyright owner that is granting the License.
14
-
15
- "Legal Entity" shall mean the union of the acting entity and all
16
- other entities that control, are controlled by, or are under common
17
- control with that entity. For the purposes of this definition,
18
- "control" means (i) the power, direct or indirect, to cause the
19
- direction or management of such entity, whether by contract or
20
- otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
- outstanding shares, or (iii) beneficial ownership of such entity.
22
-
23
- "You" (or "Your") shall mean an individual or Legal Entity
24
- exercising permissions granted by this License.
25
-
26
- "Source" form shall mean the preferred form for making modifications,
27
- including but not limited to software source code, documentation
28
- source, and configuration files.
29
-
30
- "Object" form shall mean any form resulting from mechanical
31
- transformation or translation of a Source form, including but
32
- not limited to compiled object code, generated documentation,
33
- and conversions to other media types.
34
-
35
- "Work" shall mean the work of authorship, whether in Source or
36
- Object form, made available under the License, as indicated by a
37
- copyright notice that is included in or attached to the work
38
- (an example is provided in the Appendix below).
39
-
40
- "Derivative Works" shall mean any work, whether in Source or Object
41
- form, that is based on (or derived from) the Work and for which the
42
- editorial revisions, annotations, elaborations, or other modifications
43
- represent, as a whole, an original work of authorship. For the purposes
44
- of this License, Derivative Works shall not include works that remain
45
- separable from, or merely link (or bind by name) to the interfaces of,
46
- the Work and Derivative Works thereof.
47
-
48
- "Contribution" shall mean any work of authorship, including
49
- the original version of the Work and any modifications or additions
50
- to that Work or Derivative Works thereof, that is intentionally
51
- submitted to Licensor for inclusion in the Work by the copyright owner
52
- or by an individual or Legal Entity authorized to submit on behalf of
53
- the copyright owner. For the purposes of this definition, "submitted"
54
- means any form of electronic, verbal, or written communication sent
55
- to the Licensor or its representatives, including but not limited to
56
- communication on electronic mailing lists, source code control systems,
57
- and issue tracking systems that are managed by, or on behalf of, the
58
- Licensor for the purpose of discussing and improving the Work, but
59
- excluding communication that is conspicuously marked or otherwise
60
- designated in writing by the copyright owner as "Not a Contribution."
61
-
62
- "Contributor" shall mean Licensor and any individual or Legal Entity
63
- on behalf of whom a Contribution has been received by Licensor and
64
- subsequently incorporated within the Work.
65
-
66
- 2. Grant of Copyright License. Subject to the terms and conditions of
67
- this License, each Contributor hereby grants to You a perpetual,
68
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
- copyright license to reproduce, prepare Derivative Works of,
70
- publicly display, publicly perform, sublicense, and distribute the
71
- Work and such Derivative Works in Source or Object form.
72
-
73
- 3. Grant of Patent License. Subject to the terms and conditions of
74
- this License, each Contributor hereby grants to You a perpetual,
75
- worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
- (except as stated in this section) patent license to make, have made,
77
- use, offer to sell, sell, import, and otherwise transfer the Work,
78
- where such license applies only to those patent claims licensable
79
- by such Contributor that are necessarily infringed by their
80
- Contribution(s) alone or by combination of their Contribution(s)
81
- with the Work to which such Contribution(s) was submitted. If You
82
- institute patent litigation against any entity (including a
83
- cross-claim or counterclaim in a lawsuit) alleging that the Work
84
- or a Contribution incorporated within the Work constitutes direct
85
- or contributory patent infringement, then any patent licenses
86
- granted to You under this License for that Work shall terminate
87
- as of the date such litigation is filed.
88
-
89
- 4. Redistribution. You may reproduce and distribute copies of the
90
- Work or Derivative Works thereof in any medium, with or without
91
- modifications, and in Source or Object form, provided that You
92
- meet the following conditions:
93
-
94
- (a) You must give any other recipients of the Work or
95
- Derivative Works a copy of this License; and
96
-
97
- (b) You must cause any modified files to carry prominent notices
98
- stating that You changed the files; and
99
-
100
- (c) You must retain, in the Source form of any Derivative Works
101
- that You distribute, all copyright, patent, trademark, and
102
- attribution notices from the Source form of the Work,
103
- excluding those notices that do not pertain to any part of
104
- the Derivative Works; and
105
-
106
- (d) If the Work includes a "NOTICE" text file as part of its
107
- distribution, then any Derivative Works that You distribute must
108
- include a readable copy of the attribution notices contained
109
- within such NOTICE file, excluding those notices that do not
110
- pertain to any part of the Derivative Works, in at least one
111
- of the following places: within a NOTICE text file distributed
112
- as part of the Derivative Works; within the Source form or
113
- documentation, if provided along with the Derivative Works; or,
114
- within a display generated by the Derivative Works, if and
115
- wherever such third-party notices normally appear. The contents
116
- of the NOTICE file are for informational purposes only and
117
- do not modify the License. You may add Your own attribution
118
- notices within Derivative Works that You distribute, alongside
119
- or as an addendum to the NOTICE text from the Work, provided
120
- that such additional attribution notices cannot be construed
121
- as modifying the License.
122
-
123
- You may add Your own copyright statement to Your modifications and
124
- may provide additional or different license terms and conditions
125
- for use, reproduction, or distribution of Your modifications, or
126
- for any such Derivative Works as a whole, provided Your use,
127
- reproduction, and distribution of the Work otherwise complies with
128
- the conditions stated in this License.
129
-
130
- 5. Submission of Contributions. Unless You explicitly state otherwise,
131
- any Contribution intentionally submitted for inclusion in the Work
132
- by You to the Licensor shall be under the terms and conditions of
133
- this License, without any additional terms or conditions.
134
- Notwithstanding the above, nothing herein shall supersede or modify
135
- the terms of any separate license agreement you may have executed
136
- with Licensor regarding such Contributions.
137
-
138
- 6. Trademarks. This License does not grant permission to use the trade
139
- names, trademarks, service marks, or product names of the Licensor,
140
- except as required for reasonable and customary use in describing the
141
- origin of the Work and reproducing the content of the NOTICE file.
142
-
143
- 7. Disclaimer of Warranty. Unless required by applicable law or
144
- agreed to in writing, Licensor provides the Work (and each
145
- Contributor provides its Contributions) on an "AS IS" BASIS,
146
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
- implied, including, without limitation, any warranties or conditions
148
- of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
- PARTICULAR PURPOSE. You are solely responsible for determining the
150
- appropriateness of using or redistributing the Work and assume any
151
- risks associated with Your exercise of permissions under this License.
152
-
153
- 8. Limitation of Liability. In no event and under no legal theory,
154
- whether in tort (including negligence), contract, or otherwise,
155
- unless required by applicable law (such as deliberate and grossly
156
- negligent acts) or agreed to in writing, shall any Contributor be
157
- liable to You for damages, including any direct, indirect, special,
158
- incidental, or consequential damages of any character arising as a
159
- result of this License or out of the use or inability to use the
160
- Work (including but not limited to damages for loss of goodwill,
161
- work stoppage, computer failure or malfunction, or any and all
162
- other commercial damages or losses), even if such Contributor
163
- has been advised of the possibility of such damages.
164
-
165
- 9. Accepting Warranty or Additional Liability. While redistributing
166
- the Work or Derivative Works thereof, You may choose to offer,
167
- and charge a fee for, acceptance of support, warranty, indemnity,
168
- or other liability obligations and/or rights consistent with this
169
- License. However, in accepting such obligations, You may act only
170
- on Your own behalf and on Your sole responsibility, not on behalf
171
- of any other Contributor, and only if You agree to indemnify,
172
- defend, and hold each Contributor harmless for any liability
173
- incurred by, or claims asserted against, such Contributor by reason
174
- of your accepting any such warranty or additional liability.
175
-
176
- END OF TERMS AND CONDITIONS
177
-
178
- APPENDIX: How to apply the Apache License to your work.
179
-
180
- To apply the Apache License to your work, attach the following
181
- boilerplate notice, with the fields enclosed by brackets "[]"
182
- replaced with your own identifying information. (Don't include
183
- the brackets!) The text should be enclosed in the appropriate
184
- comment syntax for the file format. We also recommend that a
185
- file or class name and description of purpose be included on the
186
- same "printed page" as the copyright notice for easier
187
- identification within third-party archives.
188
-
189
- Copyright [yyyy] [name of copyright owner]
190
-
191
- Licensed under the Apache License, Version 2.0 (the "License");
192
- you may not use this file except in compliance with the License.
193
- You may obtain a copy of the License at
194
-
195
- http://www.apache.org/licenses/LICENSE-2.0
196
-
197
- Unless required by applicable law or agreed to in writing, software
198
- distributed under the License is distributed on an "AS IS" BASIS,
199
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
- See the License for the specific language governing permissions and
201
- limitations under the License.
data/Rakefile DELETED
@@ -1,8 +0,0 @@
1
- require 'bundler/gem_tasks'
2
- require 'rspec/core/rake_task'
3
- require 'rubocop/rake_task'
4
-
5
- RuboCop::RakeTask.new
6
- RSpec::Core::RakeTask.new(:spec)
7
-
8
- task default: %i[rubocop spec]
@@ -1,42 +0,0 @@
1
- lib = File.expand_path('lib', __dir__)
2
- $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
3
- require 'consul/async/version'
4
-
5
- Gem::Specification.new do |spec|
6
- spec.name = 'consul-templaterb'
7
-
8
- spec.version = Consul::Async::VERSION
9
- spec.authors = ['SRE Core Services']
10
- spec.email = ['sre-core-services@criteo.com']
11
-
12
- spec.summary = 'Implementation of Consul template using Ruby and .erb templating language'
13
- spec.homepage = 'https://rubygems.org/gems/consul-templaterb'
14
- spec.description = 'A ruby implementation of Consul Template with support of erb templating ' \
15
- 'with hi-performance on large clusters and advanced process management features.'
16
- spec.metadata = { 'bug_tracker_uri' => 'https://github.com/criteo/consul-templaterb/issues',
17
- 'changelog_uri' => 'https://github.com/criteo/consul-templaterb/blob/master/CHANGELOG.md',
18
- 'homepage_uri' => 'https://github.com/criteo/consul-templaterb',
19
- 'source_code_uri' => 'https://github.com/criteo/consul-templaterb' }
20
- spec.license = 'Apache v2'
21
-
22
- spec.files = `git ls-files -z`.split("\x0").reject do |f|
23
- f.match(%r{^(test|spec|features)/})
24
- end
25
- spec.bindir = 'bin'
26
- spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
27
- spec.extra_rdoc_files = ['README.md', 'CHANGELOG.md', 'TemplateAPI.md']
28
- spec.require_paths = ['lib']
29
- spec.required_ruby_version = '>= 2.4.0'
30
-
31
- spec.add_runtime_dependency 'em-http-request', '>= 1.1.5'
32
- spec.add_runtime_dependency 'eventmachine', '>= 1.2.7'
33
- spec.add_runtime_dependency 'parallel', '>= 1.2.7'
34
-
35
- spec.add_development_dependency 'bundler', '>= 1.14'
36
- spec.add_development_dependency 'rake', '~> 12.3.3'
37
- spec.add_development_dependency 'rspec', '~> 3.0'
38
- spec.add_development_dependency 'rspec_junit_formatter'
39
- spec.add_development_dependency 'rubocop', '0.80.0'
40
- spec.add_development_dependency 'rubocop-junit-formatter'
41
- spec.add_development_dependency 'webmock'
42
- end
@@ -1,27 +0,0 @@
1
- daemon off;
2
- worker_processes 2;
3
-
4
- error_log /usr/src/app/error.log info;
5
-
6
- events {
7
- worker_connections 128;
8
- }
9
-
10
- http {
11
- server_tokens off;
12
- include /etc/nginx/mime.types;
13
- charset utf-8;
14
-
15
- access_log /usr/src/app/access.log combined;
16
-
17
- server {
18
- server_name localhost;
19
- listen 0.0.0.0:80;
20
- root /usr/src/app/samples/consul-ui;
21
- sendfile on;
22
-
23
- error_page 500 502 503 504 /50x.html;
24
-
25
- }
26
-
27
- }
@@ -1,124 +0,0 @@
1
- # Template-based discovery with consul-templaterb
2
-
3
- Source of article https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
4
-
5
- # How and why Criteo built a template-based system for Consul
6
-
7
- [consul-templaterb](https://github.com/criteo/consul-templaterb/) is an OpenSource Software (OSS) written in Ruby that allows you to very quickly create UIs for Consul or configuration files for your beloved software to interact with Consul. This article describes why we wrote it and why you might be interested in using it.
8
-
9
- # The context
10
-
11
- At the end of 2017, Consul started becoming a very important piece in Criteo’s infrastructure as it was responsible for dealing with the discovery of all major business applications at Criteo. At that time, Criteo had more than 25,000 servers on production and some datacenters had more than 5,000 agents per datacenter. At the same time, we started building our Load-Balancer stack based on the Consul state. So it became more and more important to make it efficient, stable and expand it with new features.
12
-
13
- Criteo wanted to move towards HAProxy for its load-balancing stack, thus we started generating some configuration for HAProxy with all HTTP services with the template-based configuration generator consul-template. However, things were a bit more complicated than expected because the program consul-template was using almost all bandwidth available just to discover all services. Moreover, the system was constantly updating HAProxy configuration.
14
-
15
- As explained in our first article “[Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)”, Consul includes a notification system to ensure your systems are always up-to-date with the content of the discovery. When this system starts to push too many notifications, on large services, the amount of data sent to all agents can be quite impressive (several megabytes/s for a single service having hundreds of instances).
16
-
17
- Thus, we started investigating the issue and figured out that it was linked to a race condition in large clusters that caused all listeners of Consul to be notified continuously. This issue was eventually solved by our [Pull-Request #3899](https://github.com/hashicorp/consul/pull/3899) (and quite a few others after that), but the investigation showed that it was quite hard to investigate the root cause of such issues.
18
-
19
- Therefore, I started a simple tool to watch all the services matching some patterns and report the bandwidth usage, but I also wanted to prove it would be possible to generate all configurations for HAProxy with an optimized tool, thus having a templating mechanism. The [Pull Request #1066](https://github.com/hashicorp/consul-template/pull/1066) to fix the issue on the consul-template itself was not yet considered (and it was hard to have something better), so I decided to implement it in Ruby with asynchronous I/Os (Input/Outputs) because I was very frustrated by the Go templating language used in consul-template and was in love with ERB templates.
20
-
21
- # The goals
22
-
23
- From scratch, I wanted to take the various shortcoming of consul-template into account:
24
-
25
- * The ability to generate high-level code to generate various formats (YAML, JSON, XML): it is dangerous, error-prone and painful to generate JSON or XML content using text templating. People fighting with YAML indentation will tell you that.
26
- * The ability to use a Turing complete and effective language to perform simple transformation such as sorting (Consul by default has some rules to sort the output, but it might not be predictable, for instance, based on the round-trip between servers).
27
- * Being able to push real-life optimizations (for instance, we don’t care about being notified immediately when a change occurs, we want to be notified once before dumping the configuration file, not 100 times without taking it into account), based on the type of query performed.
28
- * Being able to scale nicely with Consul and avoid overloading the cluster for nothing in case of bug (for instance, we use a penalty for services changing too much, we rate-limit the notifications).
29
- * Being very simple to use, hiding all the complexity to the template creator and using optimizations to reduce the number of calls to Consul, meaning being as purely functional as possible by hiding all I/O operations and taking decisions based on our experience with Consul internals.
30
- * Be evolutive: When some functions or fields are missing in a consul-template, you have to do a pull request on consul-template to add your new fields/methods, then you can work. It would be nice if any field of endpoints would be supported natively (so new versions of Consul can be used with an old version of our tool).
31
- * Support for hot-reload of templates, and nice error messages with the precise line when an error occurs, so writing templates should be very fast and efficient.
32
- * Have most of the consul-template features including babysitting of processes, spawn commands when files do change…
33
-
34
- # Internals: make your template engine in Ruby
35
-
36
- ## At startup
37
-
38
- The engine collects all template sources and computes the destinations. Each template and its destination also register some commands to run when the destination does change. Those objects save the last time the template has been read (to allow hot-reload) and the last binary content associated with them (to detect binary changes).
39
-
40
- ## The main loop
41
-
42
- First, the main loop is working on pending events. Those events are in fact the processing of the I/O operations querying the Consul agent. But every second by default, the main loop also evaluates templates which are Ruby code. When this is just regular code, this is evaluated normally, but when the code called is part of the [I/O API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md), the code does the following:
43
-
44
- 1. Checks in a registry if this method has been called with the same parameters, if not, create it, mark it as dirty (so the registry knows we are waiting for data from Consul), launch the request asynchronously to Consul, store it in the registry. This will be translated into events that will be run as events in the main loop.
45
- 2. Return the object stored within the registry. By default, this object returns the empty object or collection according to its type. When the request finally gets a response from Consul, the object is marked as “non-dirty” and the result is put within the registry, so the next evaluation returns the content of the requests.
46
-
47
- Once all code has been evaluated and the result has been stored in memory, the main loop checks whether the template is using “dirty” objects in the registry. Having at least one dirty object means that some requests are still in flight and we don’t have the result yet, so basically means the template, while evaluated, has been evaluated with fake (empty) data and is not yet ready.
48
-
49
- However, if the template is not using any dirty object it means we did receive all the data and we can render the template on disk. We first check if our last rendering was different. If the new rendering phase was not different, nothing did change, however, if the data rendered is different, the file is stored to disk and commands might be evaluated (to notify a program to reload for instance).
50
-
51
- All the I/O are done asynchronously (using [eventmachine](https://github.com/eventmachine/eventmachine)) but performed within the main loop, the one also performing the rendering, so, there is no possibility of threading issues.
52
-
53
- ![Main loop behavior, in purple, the async fetch, in yellow, the rendering](images/consul-templaterb.png)
54
-
55
- ## Handling of I/O
56
-
57
- The Input/Outputs are handled very carefully with regards to what we explained in our previous article “[Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)” and adds a few tricks:
58
-
59
- * When a given endpoint is changing too fast with any real change (the data is the same as last call), a penalty is applied, so the endpoint won’t be queried before a few seconds (this allows to work well even with old Consul versions with patch [#3899](https://github.com/hashicorp/consul/pull/3899)).
60
- * Some endpoints [are configured specifically](https://github.com/criteo/consul-templaterb/blob/master/bin/consul-templaterb#L61) to avoid too many calls (for instance, we don’t consider we add a new DC every 30 seconds).
61
- * Some specific behavior exists for non-existing service (was causing lots of issues with Prometheus for instance, see [Prometheus PR #3814](https://github.com/prometheus/prometheus/pull/3814)), so we will work on the old Consul version with our [PR #4810](https://github.com/hashicorp/consul/pull/4810).
62
- * By default, when some errors occur, try to limit the number of calls to avoid increasing the pressure on the cluster.
63
- * Collect statistics that can be used in your templates. For instance, the [Prometheus exporter uses it to know whether a service is unstable or not](https://github.com/criteo/consul-templaterb/blob/master/samples/metrics.erb#L88).
64
-
65
- ## The hard part about async I/Os with Ruby
66
-
67
- This program was my first project dealing with complex code with important performance issues. I first discovered that Ruby async I/O code is really hard and most libraries are not dealing with it properly. Many libraries just lie (by creating threads and pretending to be async). The only library I found handling it correctly was [eventmachine](https://github.com/eventmachine/eventmachine/).
68
-
69
- While the library is petty mature (the project is quite old, 9 years old), I had lots of issues very quickly. I spent quite some time making it work on my targeted operating systems (Linux, Mac OS, Windows).
70
-
71
- * On Windows, installation is painful (see https://github.com/criteo/consul-templaterb/#quick-install-on-windows) and due to limitation into the Ruby runtime, it does not support more than 2048 file descriptors with native implementation (while this works with Linux Subsystem on Windows 10).
72
- * On all Operating Systems, opening lots of HTTP connections concurrently led to crashes (see https://github.com/igrigorik/em-http-request/issues/315, [#604](https://github.com/eventmachine/eventmachine/issues/604), [#770](https://github.com/eventmachine/eventmachine/issues/770), and [#824](https://github.com/eventmachine/eventmachine/issues/824)) that took me lots of time to debug (including native C code debugging), quite frustrating for such mature library (the good news is that eventmachine is now fixed thanks to [consul-templaterb](https://github.com/criteo/consul-templaterb/)).
73
-
74
- While those problems are now solved and you can use it properly, it took me weeks to solve.
75
-
76
- Furthermore, the lack of HTTP/2 is a bit worrisome today as more and more systems allow it (it would avoid the lack of file descriptors on very large clusters).
77
-
78
- ## Some fun features
79
-
80
- * All templates in the repository are unit tested for each release of consul-templaterb, so all samples are working on your local installation.
81
- * You can generate lots of formats: [Services in XML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.xml.erb), [JSON](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb), [YAML](https://github.com/criteo/consul-templaterb/blob/master/samples/consul_template.json.erb#L58) using Hash and just dumping in the right format!
82
- * You can load templates from Consul Key/Value store, so you can change templates dynamically from Consul K/V Store (we use this feature in for our Prometheus configurations for instance as described in “[Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)”).
83
- * You can also query Vault / some JSON APIs very easily.
84
- * You can have a very high-performance UI (see [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)) easily for your own use-cases. This UI is generated in real-time as static files and can be served by your favorite webserver (we do use nginx on our side), so it scales indefinitely if you have lots of users.
85
- * Huge Performance gap with consul-template: in version 1.0 of Consul, consul-template was using 800Mb/s to scrape all services in one of large DCs, consul-templaterb is using less than 100kb/s (800x reduction!) to generate the full UI of Consul and fetch all services in our largest DC now.
86
- * Good abstraction with impressive performance compared to other projects such as [consult](https://github.com/veracross/consult) (no need to deal with kind of “magic TTL”) or consul-template itself: the APIs are simpler, you have to deal less with Consul specific optimizations and the performance is better.
87
-
88
- # Usages at Criteo (non-exhaustive)
89
-
90
- We use this executable for many things at Criteo (see “Inversion of Control with Consul”):
91
-
92
- * Automatic Alerting / Prometheus configuration
93
- * [Consul-UI](https://github.com/criteo/consul-templaterb/tree/master/samples/consul-ui)s and its Consul Timeline (keep track of the history of all changes for all services)
94
- * List of assets per DC/Racks
95
- * Monitoring of services/racks
96
- * Ownership enforcement
97
- * Live generation of configuration for several programs/services
98
-
99
- # Try it quickly
100
-
101
- If you don’t want to bother configuring and tuning it, you can try it very quickly with our docker images: In 2 minutes (download included), you have a full scalable UI with Consul, serving static files with nginx and keeping a history of all changes on your services live.
102
-
103
- ![Consul UI in action, scalable UI with excellent performance](images/consul-ui_001.png)
104
-
105
- # More on the subject
106
-
107
- * The [consul-templaterb API](https://github.com/criteo/consul-templaterb/blob/master/TemplateAPI.md) to write templates, with links to real-world examples in the samples directory.
108
- * [INTERNALS.md](../INTERNALS.md) on Github (describes the objects in the code)
109
- * Slides at HashiConf ’19: “[Consul Templates on Steroids](https://fr.slideshare.net/PierreSouchay/2019-hashiconf-consultemplaterb)”
110
- * A video introduction of consul-templaterb:
111
-
112
- [![Quick presentation of consul-templaterb](images/consul-ui_001.png)](https://youtu.be/zLzrLGLLl4Q)
113
-
114
- # Other articles from the Discovery team
115
-
116
- * [Discovery with Consul at scale](https://medium.com/criteo-labs/discovery-with-consul-at-scale-1d6808202d86)
117
- * [Be a good Consul client](https://medium.com/criteo-labs/be-a-good-consul-client-5b55160cff7d)
118
- * [Anatomy of a bug: When Consul has too much to deliver for the big day](https://medium.com/criteo-labs/anatomy-of-a-bug-when-consul-has-too-much-to-deliver-for-the-big-day-4904d19a46a4)
119
- * [Inversion of Control for the Infrastructure with Consul](https://medium.com/criteo-labs/inversion-of-control-for-the-infrastructure-with-consul-b894877b33a4)
120
- * [Mixing Observability with Service Discovery](https://medium.com/criteo-labs/mixing-observability-with-service-discovery-2bb8909e8530)
121
-
122
- Pierre Souchay, 2020-03-03
123
-
124
- Also published on https://medium.com/criteo-labs/template-based-discovery-with-consul-templaterb-8ff88434c457
@@ -1,3 +0,0 @@
1
- <?xml version="1.0" encoding="UTF-8"?>
2
- <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
3
- <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="896px" height="455px" viewBox="-0.5 -0.5 896 455" content="&lt;mxfile host=&quot;www.draw.io&quot; modified=&quot;2020-03-02T13:27:25.463Z&quot; agent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:73.0) Gecko/20100101 Firefox/73.0&quot; etag=&quot;Qxj7PV3jLHrlLdg27esA&quot; version=&quot;12.7.9&quot; type=&quot;device&quot;&gt;&lt;diagram name=&quot;Page-1&quot; id=&quot;58cdce13-f638-feb5-8d6f-7d28b1aa9fa0&quot;&gt;7VzZkps4FP0aP3YXIDY/9pqpTDI1k04myVNKBtlmgpEL5HY7Xz8SSCwSm21wO4n7IUELAt977rmLZE/A3erlTQzXy/fYR+HE0PyXCbifGIauG4D+x3p2WY9jT7OORRz4fFLR8RT8QLxT472bwEdJZSLBOCTButrp4ShCHqn0wTjG2+q0OQ6rT13DBVI6njwYqr2fA58sxefStGLgDxQslvzRrsUHZtD7vojxJuLPmxhgnv5lwyso1uLzkyX08bbUBR4m4C7GmGRXq5c7FDLZCrFl9z02jObvHaOI9LkBuMZsqs8sS9dnvjZ3rvgKzzDcIPER0hclOyEcugLVA23cLskqpH06vaQfZM3GEwJj8kQgYePzIAzvcIjj9EagpX9sMonxd1Qamc/5SCo65BeLZtJhQx5eBR6/DuEMhbe5rMVKEY7Sx+KIPMJVEDL0/YtiH0aQd3Oo6QZv170dDINFRPs8KkNEB2/9IKYgCzDrTPCGqfBWFbSQHIoJeil1ccG/QXiFSLyjU8Qo4CDYibaVtbcF5MSUZQlsog9ykC/ylQtF0wuu6556t2v0bock/fTP9HLBLj8gqpyYTvuIVuuQ6ZjPoY8rTau5cxvQT0T5ABLIwBR7iZg0i+Xb5MUk+FUhsl0GBD2tocdGt5SNqrCEscdVbpgqIJlheu2APBMIDgA3w5TgJjioBLe8r4w3ewi8eT6yfei7tgl1B8IrU6UVn1Iwb+KYLPECRzB8KHrLem2XPoXpDfMCbKU1irIeLne3Vd099Sb0E6I5wyh6CcgXhotri7e+cpSw6/uXcmPHG40apQwTe1wm2nL+58dvH9/Opt/evFt9fvjw9p/vVzr3pZRpF0hwQoM9M6G2giNG1I6D56rfq9N0eiuVKtyVJqxxEJGktPLfrKPA3NSoYs6RvJE0HWht0+lF9vyGZ+kSvnUJt5lg+V0tb2FKL21a0kKZ4JWFUhvIRXsYDU8VGi6oVoswSWMe2oEIZcALMw7DjI72msyoRGBaGwaCJDVa6O8u6h9I/aZjVdRvuK/pGK1Wx4jCGd7+HD4xopL4Ijwfa3wtPCRrFm4xbQm/KHypXvakuV89wpc2cW3Zjbb624F9qeJ38iRS+J2p1cuBHeB3DAhmnmFp0Jtrtu74Qha/Rjh2GBx0FQ7jR1V9aUJRmPASv4bG2sjiMG2eR/xrypGF1R4Ad8zviIBNp/7uEQJXFY/gODzmKZRK+wIbuoqNM0Rxi1sb0nkJ++/0XuZpnBewq9gDZr/saxDsmUrE/BceCI6VjF6vUFWBTpHtnxkcGyHXBtRecKyPk3o4UAHbseFoTSUqtKVYakwqbA/he2OvJgJucZMX7HVhb5wwft+QwJaR2RESdMyvhgQHwLdNnO27Ae+DJAmiBZ2HY1Gd6rkdMEfEW7IKlrZD/e+KUYYDKunHvpsFS7yabZLuesikvmhQg8aWOkJVU666naPX7efoQ2zo1KrRUNT4FSWKjOgnJlJxSK6yMLkEHgxv+MAq8P2UtqhKgh9wli7FLJijn65r3U6se7bWhuCEE8Uwcral+mzdvplZI2djLDGDPtbyyCA/ybfAYLKLvN7In8EktRYc0X9CmJD0lV7OYQ8N6b6FnDoXMrUdAO3J2KXChpLgkaZcLQmCaY0tj1USbAvjSxj7HAdpRRhvyHpDxlTxz1YNPkr18japccrNgFrVO2OkMnnwVZPKnGF96IhwUor3DosvzR7xpT1OJmNKuxNGv6pw90LT0apD9TGl28dNflr7+5wrybxpz8mtPndvExMnnmBmCLfeJn7O+ba45R3Ga975HyJkx1HOwiLJTgtr1J2qPV5rAHRgnSsPxQHVE4oZ1bPYnN7cwQHtlpUvOLxJuScwqaN4V8XrfRa75dnLJaray7W6r7jPWqvhI3e9fvea9SA0AFQaqM9lnVHcqymBEshZYl/3CqTsX1mowb3uWz+SC+0mjwca36t9/kj1I/Uc043PEthVXjMqijmXzGUgepU0fdJjTPU2qx5lymAQoW1RCqGBEPep5xWFKds7Q8df+qk4tu7EaL3CQCvCtGvNAKACsitBe0fysCOl3QC49GGHUbFr2dVAw6QvXv4DvZi5WFdMxPN5gkZJjgy14n6zXoepg6P5UBisglFLPL9DvGlar13EM4zjAs7zCh37E1Q374yzC6xLJY++hxKUhQy9I0ocKLhTXrjjwHzH/HGCO0Pd73hi3/oSXjyL7zIXf5Io79ckL/lAzatvQRhHfmfnki4PEMoZffPlhkhu4HTZBAdWo0/FqM6e6XLH/JEY9cgTQ2dmIwec4G/bijrRFk/vYGUcy9ItySDkw7t9LUteyOx59G1fy7KA9Bz3FJbiKLHHA8V2nJ4IkiOMvmd/9g8nBvDvhvRtD1BzXMitce/uaO7dHci9N+8zX9x7Jwk5PUnIPW/3DqRyjrLQQCRkSKcjgX6KBEitbtfsNrMj6L22msc9f9dgYMUPk7SwGf95Fb7+JD8KU/l9kWYuaeS+K+1a10XQcyhkxy/KiXOZIyY82rVz7t8i+GkocaxvtmgNifi5pjy6dHjxJCkMUAvYpz9r/CpcB7TJWXMdbRa/J5VNL360Czz8Dw==&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g><ellipse cx="60" cy="89" rx="11" ry="11" fill="#000000" stroke="#ff0000" transform="rotate(90,60,89)" pointer-events="all"/><rect x="155" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Render Template</div><div>with data srcs<br /></div></div></div></div></foreignObject><text x="215" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Render Template...</text></switch></g><path d="M 835 59 L 835 19 Q 835 9 825 9 L 225 9 Q 215 9 215 19 L 215 56.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 210.5 48.88 L 215 57.88 L 219.5 48.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="615" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 616px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template not completed</div></div></div></foreignObject><text x="675" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template not complet...</text></switch></g><rect x="390" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Template is ready</div></div></div></foreignObject><text x="450" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Template is ready</text></switch></g><path d="M 735 89 L 745 89 Q 755 89 763.88 89 L 772.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 764.88 93.5 L 773.88 89 L 764.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 75 89 L 152.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 144.88 93.5 L 153.88 89 L 144.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 275 89 L 305 89 Q 315 89 325 89 L 382.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 374.88 93.5 L 383.88 89 L 374.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 390 249 L 342.5 249 Q 332.5 249 332.5 239 L 332.5 199 Q 332.5 189 322.5 189 L 277.24 189" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 285.12 184.5 L 276.12 189 L 285.12 193.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 154 L 450 216.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 445.5 208.88 L 450 217.88 L 454.5 208.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe flex-start; width: 1px; height: 1px; padding-top: 187px; margin-left: 452px;"><div style="box-sizing: border-box; font-size: 0; text-align: left; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">No</div></div></div></foreignObject><text x="452" y="191" fill="#000000" font-family="Verdana" font-size="12px">No</text></switch></g><path d="M 515 89 L 595 89 Q 605 89 608.88 89 L 612.76 89" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 604.88 93.5 L 613.88 89 L 604.88 84.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 450 24 L 515 89 L 450 154 L 385 89 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 128px; height: 1px; padding-top: 89px; margin-left: 386px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Missing or not</div><div>fetched yet</div><div>resources?</div></div></div></div></foreignObject><text x="450" y="93" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Missing or not...</text></switch></g><rect x="535" y="74" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 84px; margin-left: 555px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="555" y="88" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g><rect x="390" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 391px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div>Fetch data async</div><div>based on last idx<br /></div></div></div></div></foreignObject><text x="450" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Fetch data async...</text></switch></g><rect x="155" y="159" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 189px; margin-left: 156px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Write output</div></div></div></foreignObject><text x="215" y="193" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Write output</text></switch></g><path d="M 215 159 L 215 121.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 219.5 129.12 L 215 120.12 L 210.5 129.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 63.23 222.82 L 56.77 215.18 L 149.86 136.52 L 143.4 128.89 L 176 120.98 L 162.77 151.8 L 156.31 144.16 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 169px; margin-left: 117px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>Update</div><div>data</div><div>async</div></div></div></div></foreignObject><text x="117" y="172" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Update...</text></switch></g><rect x="0" y="219" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 249px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Data fetched</div></div></div></foreignObject><text x="60" y="253" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Data fetched</text></switch></g><path d="M 390 364 L 285 364 Q 275 364 265 364 L 247.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 255.12 359.5 L 246.12 364 L 255.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="59" width="120" height="60" rx="14.4" ry="14.4" fill="#ffffc0" stroke="#ff0000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 89px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Add missing resources</div></div></div></foreignObject><text x="835" y="93" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Add missing resources</text></switch></g><path d="M 830 119 L 840 119 L 840 304 L 850 304 L 835 334 L 820 304 L 830 304 Z" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 229px; margin-left: 765px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Add new data to fetch</div></div></div></foreignObject><text x="765" y="232" fill="#000000" font-family="Helvetica" font-size="11px" text-anchor="middle">Add new data to fetch</text></switch></g><rect x="0" y="394" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 424px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Apply ratelimit</div></div></div></foreignObject><text x="60" y="428" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Apply ratelimit</text></switch></g><path d="M 60 279 L 60 299 Q 60 309 60 319 L 60 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 55.5 383.88 L 60 392.88 L 64.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><rect x="775" y="334" width="120" height="60" rx="14.4" ry="14.4" fill="#e1d5e7" stroke="#9673a6" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 364px; margin-left: 776px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Verdana; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Start fetching new resources</div></div></div></foreignObject><text x="835" y="368" fill="#000000" font-family="Verdana" font-size="12px" text-anchor="middle">Start fetching new r...</text></switch></g><path d="M 775 364 L 685 364 Q 675 364 665 364 L 512.24 364" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 520.12 359.5 L 511.12 364 L 520.12 368.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 120 424 L 440 424 Q 450 424 450 414 L 450 396.24" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 454.5 404.12 L 450 395.12 L 445.5 404.12" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 205 324 L 245 364 L 205 404 L 165 364 Z" fill="#e1d5e7" stroke="#9673a6" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 364px; margin-left: 166px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Error?</div></div></div></foreignObject><text x="205" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Error?</text></switch></g><path d="M 205 324 L 205 259 Q 205 249 195 249 L 122.24 249" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 130.12 244.5 L 121.12 249 L 130.12 253.5" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 254px; margin-left: 206px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; "><div>No</div></div></div></div></foreignObject><text x="206" y="257" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">No</text></switch></g><path d="M 165 364 L 100 364 Q 90 364 90 374 L 90 391.76" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 85.5 383.88 L 90 392.88 L 94.5 383.88" fill="none" stroke="#ff0000" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 365px; margin-left: 119px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: #ffffff; white-space: nowrap; ">Yes</div></div></div></foreignObject><text x="119" y="368" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Yes</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>