gemstash 2.8.1 → 2.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/lib/gemstash/authorization.rb +0 -2
  3. data/lib/gemstash/cache.rb +5 -4
  4. data/lib/gemstash/cli/base.rb +9 -2
  5. data/lib/gemstash/cli/setup.rb +1 -1
  6. data/lib/gemstash/cli/start.rb +4 -0
  7. data/lib/gemstash/configuration.rb +71 -1
  8. data/lib/gemstash/db/upstream.rb +30 -2
  9. data/lib/gemstash/dependencies.rb +28 -5
  10. data/lib/gemstash/env.rb +4 -2
  11. data/lib/gemstash/gem_fetcher.rb +1 -2
  12. data/lib/gemstash/gem_pusher.rb +24 -20
  13. data/lib/gemstash/gem_source/private_source.rb +22 -6
  14. data/lib/gemstash/gem_source/upstream_source.rb +129 -34
  15. data/lib/gemstash/http_client.rb +167 -8
  16. data/lib/gemstash/logging.rb +3 -1
  17. data/lib/gemstash/man/gemstash-authorize.1 +29 -48
  18. data/lib/gemstash/man/gemstash-authorize.1.txt +14 -14
  19. data/lib/gemstash/man/gemstash-configuration.5 +231 -154
  20. data/lib/gemstash/man/gemstash-configuration.5.txt +208 -93
  21. data/lib/gemstash/man/gemstash-customize.7 +150 -168
  22. data/lib/gemstash/man/gemstash-customize.7.txt +80 -86
  23. data/lib/gemstash/man/gemstash-debugging.7 +20 -33
  24. data/lib/gemstash/man/gemstash-debugging.7.txt +5 -7
  25. data/lib/gemstash/man/gemstash-deploy.7 +31 -53
  26. data/lib/gemstash/man/gemstash-deploy.7.txt +17 -18
  27. data/lib/gemstash/man/gemstash-mirror.7 +24 -37
  28. data/lib/gemstash/man/gemstash-mirror.7.txt +14 -9
  29. data/lib/gemstash/man/gemstash-multiple-sources.7 +105 -110
  30. data/lib/gemstash/man/gemstash-multiple-sources.7.txt +62 -41
  31. data/lib/gemstash/man/gemstash-private-gems.7 +87 -136
  32. data/lib/gemstash/man/gemstash-private-gems.7.txt +30 -30
  33. data/lib/gemstash/man/gemstash-readme.7 +105 -127
  34. data/lib/gemstash/man/gemstash-readme.7.txt +43 -49
  35. data/lib/gemstash/man/gemstash-setup.1 +22 -42
  36. data/lib/gemstash/man/gemstash-setup.1.txt +12 -12
  37. data/lib/gemstash/man/gemstash-start.1 +9 -27
  38. data/lib/gemstash/man/gemstash-start.1.txt +5 -5
  39. data/lib/gemstash/man/gemstash-status.1 +9 -27
  40. data/lib/gemstash/man/gemstash-status.1.txt +5 -5
  41. data/lib/gemstash/man/gemstash-stop.1 +9 -27
  42. data/lib/gemstash/man/gemstash-stop.1.txt +5 -5
  43. data/lib/gemstash/man/gemstash-version.1 +8 -28
  44. data/lib/gemstash/man/gemstash-version.1.txt +2 -2
  45. data/lib/gemstash/puma.rb +1 -0
  46. data/lib/gemstash/rack_env_rewriter.rb +3 -0
  47. data/lib/gemstash/upstream.rb +83 -6
  48. data/lib/gemstash/version.rb +1 -1
  49. data/lib/gemstash/web.rb +12 -0
  50. metadata +7 -7
@@ -1,221 +1,336 @@
1
- 4mgemstash-configuration24m(5) 4mgemstash-configuration24m(5)
1
+ 4mgemstash-configuration24m(5) File Formats Manual 4mgemstash-configuration24m(5)
2
2
 
3
3
  1mName0m
4
4
  gemstash-configuration
5
5
 
6
6
  1mSynopsis0m
7
- # ~/.gemstash/config.yml
7
+ 4m#24m 4m~/.gemstash/config.yml0m
8
8
  ---
9
- :base_path: "/var/gemstash"
10
- :cache_type: memcached
11
- :memcached_servers: localhost:11211
12
- :db_adapter: postgres
13
- :db_url: postgres:///gemstash
14
- :db_connection_options:
15
- :test: true
16
- :pool_timeout: 2
17
- :rubygems_url: https://my.gem-source.local
18
- :ignore_gemfile_source: false
19
- :puma_threads: 32
20
- :bind: tcp://0.0.0.0:4242
21
- :protected_fetch: true
22
- :fetch_timeout: 10
23
- :open_timeout: 2
24
- :log_file: gemstash.log
9
+ :base_path1m: 22m"/var/gemstash"
10
+ :cache_type1m: 22mmemcached
11
+ :memcached_servers1m: 22mlocalhost:11211
12
+ :db_adapter1m: 22mpostgres
13
+ :db_url1m: 22mpostgres:///gemstash
14
+ :db_connection_options1m:0m
15
+ :test1m: 22mtrue
16
+ :pool_timeout1m: 22m2
17
+ :rubygems_url1m: 22mhttps://my.gem-source.local
18
+ :ignore_gemfile_source1m: 22mfalse
19
+ :allowed_upstreams1m:0m
20
+ 1m- 22mhttps://my-other.gem-source.local
21
+ :allowed_redirects1m:0m
22
+ 1m- 22mhttps://cdn.my-other.gem-source.local
23
+ :puma_threads1m: 22m32
24
+ :bind1m: 22mtcp://0.0.0.0:4242
25
+ :protected_fetch1m: 22mtrue
26
+ :fetch_timeout1m: 22m10
27
+ :max_upstream_duration1m: 22m300
28
+ :open_timeout1m: 22m2
29
+ :max_upstream_bytes1m: 22m67108864
30
+ :log_file1m: 22mgemstash.log
25
31
 
26
32
  1mBase Path0m
27
- 1m:base_path0m
33
+ :base_path
28
34
 
29
35
  Specifies where to store local files like the server log, cached gem
30
36
  files, and the database (when using SQLite). If the default is being
31
37
  used, the directory will be created if it does not exist. Any other
32
38
  directory needs to be created ahead of time and be writable to the Gem-
33
- stash server process. Specifying the 1m:base_path 22mvia 1mgemstash setup0m
39
+ stash server process. Specifying the :base_path via gemstash setup
34
40
  will create the directory for you.
35
41
 
36
42
  1mDefault value0m
37
- 1m~/.gemstash0m
43
+ ~/.gemstash
38
44
 
39
45
  1mValid values0m
40
46
  Any valid path
41
47
 
42
48
  1mCache Type0m
43
- 1m:cache_type0m
49
+ :cache_type
44
50
 
45
51
  Specifies how to cache values other than gem files (such as gem depen-
46
- dencies). 1mmemory 22mwill use an in memory cache while 1mmemcached 22mwill
47
- point to 1 or more Memcached servers. Use the 1m:memcached_servers 22mcon-
52
+ dencies). memory will use an in memory cache while memcached will
53
+ point to 1 or more Memcached servers. Use the :memcached_servers con-
48
54
  figuration key for specifying where the Memcached server(s) are.
49
55
 
50
56
  1mDefault value0m
51
- 1mmemory0m
57
+ memory
52
58
 
53
59
  1mValid values0m
54
- 1mmemory22m, 1mmemcached0m
60
+ memory, memcached
55
61
 
56
62
  1mMemcached Servers0m
57
- 1m:memcached_servers0m
63
+ :memcached_servers
58
64
 
59
- Specifies the Memcached servers to connect to when using 1mmemcached 22mfor
60
- the 1m:cache_type22m. Only used when 1mmemcached 22mis used for 1m:cache_type22m.
65
+ Specifies the Memcached servers to connect to when using memcached for
66
+ the :cache_type. Only used when memcached is used for :cache_type.
61
67
 
62
68
  1mDefault value0m
63
- 1mlocalhost:112110m
69
+ localhost:11211
64
70
 
65
71
  1mValid values0m
66
72
  A comma delimited list of Memcached servers
67
73
 
68
74
  1mDB Adapter0m
69
- 1m:db_adapter0m
75
+ :db_adapter
70
76
 
71
- Specifies what database adapter to use. When 1msqlite3 22mis used, the
72
- database will be located at 1mgemstash.db 22mwithin the directory specified
73
- by 1m:base_path22m. The database will automatically be created when using
74
- 1msqlite322m. When 1mpostgres22m, 1mmysql22m, or 1mmysql2 22mis used, the database to con-
75
- nect to must be specified in the 1m:db_url 22mconfiguration key. The data-
76
- base must already be created when using anything other than 1msqlite322m.
77
+ Specifies what database adapter to use. When sqlite3 is used, the
78
+ database will be located at gemstash.db within the directory specified
79
+ by :base_path. The database will automatically be created when using
80
+ sqlite3. When postgres, mysql, or mysql2 is used, the database to con-
81
+ nect to must be specified in the :db_url configuration key. The data-
82
+ base must already be created when using anything other than sqlite3.
77
83
 
78
84
  1mDefault value0m
79
- 1msqlite30m
85
+ sqlite3
80
86
 
81
87
  1mValid values0m
82
- 1msqlite322m, 1mpostgres22m, 1mmysql22m, 1mmysql20m
88
+ sqlite3, postgres, mysql, mysql2
83
89
 
84
90
  1mDB URL0m
85
- 1m:db_url0m
91
+ :db_url
86
92
 
87
- Specifies the database to connect to when using 1mpostgres22m, 1mmysql22m, or
88
- 1mmysql2 22mfor the 1m:db_adapter22m. Only used when the 1m:db_adapter 22mis not
89
- 1msqlite322m.
93
+ Specifies the database to connect to when using postgres, mysql, or
94
+ mysql2 for the :db_adapter. Only used when the :db_adapter is not
95
+ sqlite3.
90
96
 
91
97
  1mDefault value0m
92
98
  None
93
99
 
94
100
  1mValid values0m
95
- A valid database URL for the Sequel gem (http://sequel.jeremye-
96
- vans.net/)
101
+ A valid database URL for the 8;;http://sequel.jeremyevans.net/Sequel gem8;;
97
102
 
98
103
  1mDB Connection Options0m
99
- 1m:db_connection_options0m
104
+ :db_connection_options
100
105
 
101
- Specifies additional 1mSequel.connect 22moptions to use. Note that any op-
102
- tions here are merged in with the default options, so you need not
103
- specify the 1mmax_connections 22mif you customize this option.
106
+ Specifies additional Sequel.connect options to use. Note that any op-
107
+ tions here are merged in with the default options, so you need not
108
+ specify the max_connections if you customize this option.
104
109
 
105
110
  1mDefault value0m
106
- 1m{ max_connections: 1 } 22mfor 1msqlite3 22madapter, 1m{ max_connections: con-0m
107
- 1mfig[:puma_threads] + 1 } 22mfor any other adapter.
111
+ { max_connections: 1 } for sqlite3 adapter, { max_connections: con-
112
+ fig[:puma_threads] + 1 } for any other adapter.
108
113
 
109
114
  1mValid values0m
110
- A valid connection options Hash for the Sequel.connect (http://se-
111
- quel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-
112
- General+connection+options) method.
115
+ A valid connection options Hash for the 8;;http://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-General+connection+optionsSequel.connect8;; method.
113
116
 
114
117
  1mRubygems URL0m
115
- 1m:rubygems_url0m
118
+ :rubygems_url
116
119
 
117
120
  Specifies the default gem source URL. When any API endpoint is called
118
- without a 1m/private 22mor 1m/upstream/<url> 22mprefix, this URL will be used to
121
+ without a /private or /upstream/<url> prefix, this URL will be used to
119
122
  fetch the result. This value can be safely changed even if there are
120
123
  already gems stashed for the previous value.
121
124
 
122
125
  1mDefault value0m
123
- 1mhttps://rubygems.org0m
126
+ https://rubygems.org
124
127
 
125
128
  1mValid values0m
126
129
  A valid gem source URL
127
130
 
128
131
  1mIgnore Gemfile source0m
129
- 1m:ignore_gemfile_source0m
132
+ :ignore_gemfile_source
130
133
 
131
- Ignore the source specified in Gemfile and always use 1m:rubygems_url 22mas
134
+ Ignore the source specified in Gemfile and always use :rubygems_url as
132
135
  gems upstream.
133
136
 
134
137
  1mDefault value0m
135
- 1mfalse0m
138
+ false
136
139
 
137
140
  1mValid values0m
138
- Boolean: 1mtrue 22mor 1mfalse0m
141
+ Boolean: true or false
142
+
143
+ 1mAllowed Upstreams0m
144
+ :allowed_upstreams
145
+
146
+ Specifies the gem sources that clients may ask Gemstash to fetch from,
147
+ whether they use an /upstream/<url> or /redirect/<url> prefix, or the
148
+ header Bundler sends when Gemstash is used as a mirror. The gem source
149
+ specified by :rubygems_url is always allowed. A request for any other
150
+ gem source is answered with a 403 status, and that gem source is not
151
+ contacted.
152
+
153
+ Bundler reports a 403 as Access token could not be authenticated, so
154
+ that is the error someone sees when they bundle with a gem source that
155
+ is not allowed. The actual reason is in the body of the response and
156
+ in the Gemstash log, as a Refusing upstream warning.
157
+
158
+ A gem source is allowed when its scheme, host, port, and path are the
159
+ same as those of an entry. Credentials and a trailing slash are ig-
160
+ nored, so an entry of https://my.gem-source.local allows
161
+ https://user:password@my.gem-source.local/, but does not allow
162
+ http://my.gem-source.local or https://my.gem-source.local/private.
163
+
164
+ Credentials in an entry are ignored only for that comparison. When a
165
+ client asks for the gem source without credentials of its own, Gemstash
166
+ fetches from it with the credentials of the entry, so anyone who can
167
+ reach Gemstash can read that gem source. Gemstash does not pass them
168
+ on to clients: the requests it answers with a redirect, such as /ver-
169
+ sions and /info/*, reach the gem source without credentials. This also
170
+ applies to credentials in :rubygems_url.
171
+
172
+ By default only :rubygems_url is allowed. Before upgrading, list any
173
+ other Gemfile and mirror sources you use, or Bundler requests for them
174
+ will fail with 403. Set "*" only if you deliberately accept arbitrary
175
+ client-selected sources. Redirects from an allowed source to an un-
176
+ listed source are refused. If an allowed source redirects gem down-
177
+ loads to a CDN, list the CDN in :allowed_redirects. Redirect targets
178
+ containing credentials are refused.
179
+
180
+ 1mDefault value0m
181
+ [] (only :rubygems_url)
182
+
183
+ 1mValid values0m
184
+ "*" to allow any gem source, or a list of gem source URLs. An empty
185
+ list allows only the gem source specified by :rubygems_url.
186
+
187
+ Each URL needs a host, and its scheme, if it has one, must be http://
188
+ or https://. gemstash start refuses a configuration with any other en-
189
+ try. The error gives the position of the entry in the list and not its
190
+ value, because the value may contain credentials.
191
+
192
+ 1mAllowed Redirects0m
193
+ :allowed_redirects
194
+
195
+ Specifies further URLs that a fetch from an allowed gem source may be
196
+ redirected to, such as the CDN a gem server sends its downloads to.
197
+ Clients cannot ask Gemstash to use these as gem sources. :rubygems_url
198
+ and the entries of :allowed_upstreams are always valid redirect tar-
199
+ gets.
200
+
201
+ A redirect is followed when its scheme, host, and port are those of an
202
+ entry and its path is the path of the entry or lies below it. An http
203
+ entry also accepts the same host over https, and a redirect from https
204
+ to http is never followed. Paths are compared segment by segment, and
205
+ a redirect below an entry with a path is refused when a segment could
206
+ be resolved to another path, such as ..; or a doubly encoded ... Do
207
+ not rely on the path of an entry alone to keep gem sources on one host
208
+ apart: the access control of that host is what separates them.
209
+
210
+ 1mDefault value0m
211
+ []
212
+
213
+ 1mValid values0m
214
+ A list of URLs, checked like the entries of :allowed_upstreams. "*" is
215
+ not supported.
139
216
 
140
217
  1mPuma Threads0m
141
- 1m:puma_threads0m
218
+ :puma_threads
142
219
 
143
220
  Specifies the number of threads used for the Gemstash server.
144
221
 
145
222
  1mDefault value0m
146
- 1m160m
223
+ 16
147
224
 
148
225
  1mValid values0m
149
- Integer value with a minimum of 1m10m
226
+ Integer value with a minimum of 1
150
227
 
151
228
  1mBind Address0m
152
- 1m:bind0m
229
+ :bind
153
230
 
154
- Specifies the binding used to start the Gemstash server. Keep in mind
231
+ Specifies the binding used to start the Gemstash server. Keep in mind
155
232
  the user starting Gemstash needs to have access to bind in this manner.
156
- For example, if you use a port below 1024, you will need to run Gem-
233
+ For example, if you use a port below 1024, you will need to run Gem-
157
234
  stash as the root user.
158
235
 
159
236
  1mDefault value0m
160
- 1mtcp://0.0.0.0:92920m
237
+ tcp://0.0.0.0:9292
161
238
 
162
239
  1mValid values0m
163
- Any valid binding that is supported by Puma
164
- (https://github.com/puma/puma#binding-tcp--sockets)
240
+ Any valid binding that 8;;https://github.com/puma/puma#binding-tcp--socketsis supported by Puma8;;
165
241
 
166
242
  1mProtected Fetch0m
167
- 1m:protected_fetch0m
243
+ :protected_fetch
168
244
 
169
245
  Tells Gemstash to authenticate via an API key before allowing the
170
246
  fetching of private gems and specs. The default behavior is to allow
171
247
  unauthenticated download of private gems and specs.
172
248
 
173
249
  1mDefault value0m
174
- 1mfalse0m
250
+ false
175
251
 
176
252
  1mValid values0m
177
- Boolean values 1mtrue 22mor 1mfalse0m
253
+ Boolean values true or false
178
254
 
179
255
  1mFetch Timeout0m
180
- 1m:fetch_timeout0m
256
+ :fetch_timeout
257
+
258
+ This is the number of seconds to wait for an upstream read, not the to-
259
+ tal download time. A connection that keeps delivering data can take
260
+ longer; :max_upstream_duration bounds the entire request. After
261
+ :fetch_timeout seconds, the upstream connection must also average at
262
+ least 1 KiB/s across all response bytes, including headers, redirects,
263
+ chunk metadata and trailers, to prevent a connection that drips one
264
+ byte at a time from holding a worker. Fetching gems over a very slow
265
+ connection may cause timeout errors. The default is 20 seconds.
181
266
 
182
- This is the number of seconds to allow for fetching a gem from up-
183
- stream. It covers establishing the connection and receiving the re-
184
- sponse. Fetching gems over a slow connection may cause timeout errors.
185
- If you experience timeout errors, you may want to increase this value.
186
- The default is 1m20 22mseconds.
267
+ 1mDefault value0m
268
+ 20
269
+
270
+ 1mValid values0m
271
+ Positive finite number of seconds (including fractions); nil is not
272
+ supported
273
+
274
+ 1mMax Upstream Duration0m
275
+ :max_upstream_duration
276
+
277
+ Maximum seconds for an entire upstream request, including redirects and
278
+ streaming the body. Slow but steady downloads may exceed :fetch_time-
279
+ out without exceeding this limit. Increase it if legitimate downloads
280
+ need longer than five minutes.
187
281
 
188
282
  1mDefault value0m
189
- 1m200m
283
+ 300 seconds
190
284
 
191
285
  1mValid values0m
192
- Integer value with a minimum of 1m10m
286
+ Positive finite number of seconds (including fractions); nil is not
287
+ supported
193
288
 
194
289
  1mOpen Timeout0m
195
- 1m:open_timeout0m
290
+ :open_timeout
196
291
 
197
- The timeout setting for opening the connection to an upstream gem serv-
198
- er. On high-latency networks, even establishing the connection to an
199
- upstream gem server can take a while. If you experience connection
200
- failures instead of timeout errors, you may want to increase this val-
201
- ue. The default is 1m2 22mseconds.
292
+ The timeout setting for opening the connection to an upstream gem
293
+ server. On high-latency networks, even establishing the connection to
294
+ an upstream gem server can take a while. If you experience connection
295
+ failures instead of timeout errors, you may want to increase this
296
+ value. The default is 2 seconds.
297
+
298
+ 1mDefault value0m
299
+ 2
300
+
301
+ 1mValid values0m
302
+ Integer value with a minimum of 1
303
+
304
+ 1mMax Upstream Bytes0m
305
+ :max_upstream_bytes
306
+
307
+ The size in bytes of the largest response to accept from an upstream
308
+ gem server. This applies to every gem, gemspec, index of specs, and
309
+ list of dependencies that is fetched. A larger response is abandoned
310
+ while it is being downloaded and is not stashed, and the request for it
311
+ fails. The size is that of the response after any compression used to
312
+ transfer it has been undone. Make sure the value is larger than the
313
+ largest gem you need. Response headers are limited to 64 KiB total,
314
+ and each chunk metadata line to 64 KiB. A response can use about twice
315
+ the byte cap while buffers grow (over 2 GiB with 16 default Puma
316
+ threads). Increase this limit if your upstream has larger gems.
202
317
 
203
318
  1mDefault value0m
204
- 1m20m
319
+ 67108864 (64 MiB)
205
320
 
206
321
  1mValid values0m
207
- Integer value with a minimum of 1m10m
322
+ Positive integer, or nil to disable the body cap (not recommended)
208
323
 
209
324
  1mLog File0m
210
- 1m:log_file0m
325
+ :log_file
211
326
 
212
- Indicates the name of the file to use for logging. The file will be
327
+ Indicates the name of the file to use for logging. The file will be
213
328
  placed in the base path.
214
329
 
215
330
  1mDefault value0m
216
- 1mserver.log0m
331
+ server.log
217
332
 
218
333
  1mValid values0m
219
- Any valid file name, or 1m:stdout 22mto log to 1m$stdout0m
334
+ Any valid file name, or :stdout to log to $stdout
220
335
 
221
336
  October 13, 2015 4mgemstash-configuration24m(5)