gemstash 1.0.0.pre.1-java

Sign up to get free protection for your applications and to get access to all the features.
Files changed (59) hide show
  1. checksums.yaml +7 -0
  2. data/CODE_OF_CONDUCT.md +13 -0
  3. data/Gemfile +4 -0
  4. data/LICENSE.txt +21 -0
  5. data/README.md +139 -0
  6. data/Rakefile +35 -0
  7. data/bin/console +14 -0
  8. data/bin/gemstash +3 -0
  9. data/bin/setup +5 -0
  10. data/docs/config.md +136 -0
  11. data/docs/debug.md +24 -0
  12. data/docs/deploy.md +30 -0
  13. data/docs/mirror.md +30 -0
  14. data/docs/multiple_sources.md +68 -0
  15. data/docs/private_gems.md +140 -0
  16. data/docs/reference.md +308 -0
  17. data/exe/gemstash +3 -0
  18. data/gemstash.gemspec +47 -0
  19. data/gemstash.png +0 -0
  20. data/lib/gemstash.rb +26 -0
  21. data/lib/gemstash/authorization.rb +87 -0
  22. data/lib/gemstash/cache.rb +79 -0
  23. data/lib/gemstash/cli.rb +71 -0
  24. data/lib/gemstash/cli/authorize.rb +69 -0
  25. data/lib/gemstash/cli/base.rb +46 -0
  26. data/lib/gemstash/cli/setup.rb +173 -0
  27. data/lib/gemstash/cli/start.rb +52 -0
  28. data/lib/gemstash/cli/status.rb +21 -0
  29. data/lib/gemstash/cli/stop.rb +21 -0
  30. data/lib/gemstash/config.ru +13 -0
  31. data/lib/gemstash/configuration.rb +41 -0
  32. data/lib/gemstash/db.rb +15 -0
  33. data/lib/gemstash/db/authorization.rb +20 -0
  34. data/lib/gemstash/db/dependency.rb +50 -0
  35. data/lib/gemstash/db/rubygem.rb +14 -0
  36. data/lib/gemstash/db/version.rb +51 -0
  37. data/lib/gemstash/dependencies.rb +93 -0
  38. data/lib/gemstash/env.rb +150 -0
  39. data/lib/gemstash/gem_fetcher.rb +50 -0
  40. data/lib/gemstash/gem_pusher.rb +125 -0
  41. data/lib/gemstash/gem_source.rb +37 -0
  42. data/lib/gemstash/gem_source/dependency_caching.rb +40 -0
  43. data/lib/gemstash/gem_source/private_source.rb +139 -0
  44. data/lib/gemstash/gem_source/rack_middleware.rb +22 -0
  45. data/lib/gemstash/gem_source/upstream_source.rb +183 -0
  46. data/lib/gemstash/gem_unyanker.rb +61 -0
  47. data/lib/gemstash/gem_yanker.rb +61 -0
  48. data/lib/gemstash/http_client.rb +77 -0
  49. data/lib/gemstash/logging.rb +93 -0
  50. data/lib/gemstash/migrations/01_gem_dependencies.rb +41 -0
  51. data/lib/gemstash/migrations/02_authorizations.rb +12 -0
  52. data/lib/gemstash/puma.rb +6 -0
  53. data/lib/gemstash/rack_env_rewriter.rb +66 -0
  54. data/lib/gemstash/specs_builder.rb +93 -0
  55. data/lib/gemstash/storage.rb +207 -0
  56. data/lib/gemstash/upstream.rb +65 -0
  57. data/lib/gemstash/version.rb +4 -0
  58. data/lib/gemstash/web.rb +97 -0
  59. metadata +304 -0
data/docs/mirror.md ADDED
@@ -0,0 +1,30 @@
1
+ # Using Gemstash as a Mirror
2
+
3
+ If you don't have control over your `Gemfile`, or you don't want to force
4
+ everyone on your team to go through the Gemstash server, you can use Bundler
5
+ mirroring to bundle against your Gemstash server.
6
+
7
+ For each source in your `Gemfile`, add a mirror pointing to your Gemstash
8
+ server:
9
+ ```
10
+ $ bundle config mirror.http://rubygems.org http://localhost:9292
11
+ $ bundle config mirror.https://my.gem-source.local http://localhost:9292/upstream/$(ruby -rcgi -e 'puts CGI.escape("https://my.gem-source.local")')
12
+ ```
13
+
14
+ From now on, bundler will fetch gems from those sources via your Gemstash
15
+ server.
16
+
17
+ # Simpler Gemstash Mirrors
18
+
19
+ **WARNING: This feature is not yet available.**
20
+
21
+ **This feature requires Bundler to be at least version `1.11.0`.**
22
+
23
+ If you are using Bundler version `1.11.0` or greater, the mirroring becomes a
24
+ bit easier:
25
+ ```
26
+ $ bundle config mirror.http://rubygems.org http://localhost:9292
27
+ $ bundle config mirror.https://my.gem-source.local http://localhost:9292
28
+ ```
29
+
30
+ Bundler will then send headers to Gemstash to indicate the correct upstream.
@@ -0,0 +1,68 @@
1
+ # Multiple Gem Sources
2
+
3
+ Gemstash will stash from any amount of gem sources. By the end of this guide,
4
+ you will be able to bundle using multiple gem sources, all stashed within your
5
+ Gemstash server.
6
+
7
+ ## Default Source
8
+
9
+ When you don't provide an explicit source (as with the [Quickstart
10
+ Guide](../README.md#quickstart-guide)), your gems will be fetched from
11
+ https://www.rubygems.org. This default source is not set in stone. To change it,
12
+ you need only edit the Gemstash configuration found at `~/.gemstash/config.yml`:
13
+ ```yaml
14
+ # ~/.gemstash/config.yml
15
+ ---
16
+ :rubygems_url: https://my.gem-source.local
17
+ ```
18
+
19
+ Make sure to restart your Gemstash server after changing the config:
20
+ ```
21
+ $ gemstash stop
22
+ $ gemstash start
23
+ ```
24
+
25
+ Once restarted, bundling against `http://localhost:9292` will fetch gems from
26
+ `https://my.gem-source.local`. If you had bundled before making these changes,
27
+ fear not; bundling with a different default gem source will store gems in a
28
+ separate location, ensuring different sources won't leak between each other.
29
+
30
+ ## Bundling with Multiple Sources
31
+
32
+ Changing the default source won't help you if you need to bundle against
33
+ https://www.rubygems.org along with additional sources. If you need to bundle
34
+ with multiple gem sources, Gemstash doesn't need to be specially configured.
35
+ Your Gemstash server will honor any gem source specified via a specialized URL.
36
+ Consider the following `Gemfile`:
37
+ ```ruby
38
+ # ./Gemfile
39
+ require "cgi"
40
+ source "http://localhost:9292"
41
+ gem "rubywarrior"
42
+
43
+ source "http://localhost:9292/upstream/#{CGI.escape("https://my.gem-source.local")}" do
44
+ gem "my-gem"
45
+ end
46
+ ```
47
+
48
+ Notice the `CGI.escape` call in the second source. This is important, as it
49
+ properly URL escapes the source URL so Gemstash knows what gem source you want.
50
+ The `/upstream` prefix tells Gemstash to use a gem source other than the default
51
+ source. You can now bundle with the additional source.
52
+
53
+ ## Redirecting
54
+
55
+ Gemstash supports an alternate mode of specifying your gem sources. If you want
56
+ Gemstash to redirect Bundler to your given gem sources, then you can specify
57
+ your `Gemfile` like so:
58
+ ```ruby
59
+ # ./Gemfile
60
+ require "cgi"
61
+ source "http://localhost:9292/redirect/#{CGI.escape("https://www.rubygems.org")}"
62
+ gem "rubywarrior"
63
+ ```
64
+
65
+ Notice the `/redirect` prefix. This prefix tells Gemstash to redirect API calls
66
+ to the provided URL. Redirected calls like this will not be cached by Gemstash,
67
+ and gem files will not be stashed, even if they were previously cached or
68
+ stashed from the same gem source.
@@ -0,0 +1,140 @@
1
+ # Private Gems
2
+
3
+ Stashing private gems in your Gemstash server requires a bit of additional
4
+ setup. If you haven't read through the [Quickstart
5
+ Guide](../README.md#quickstart-guide), you should do that first. By the end of
6
+ this guide, you will be able to interact with your Gemstash server to store and
7
+ retrieve your private gems.
8
+
9
+ ## Authorizing
10
+
11
+ **IMPORTANT NOTE:** Do not use the actual key value in this document, otherwise
12
+ your Gemstash server will be vulnerable to anyone who wants to try to use the
13
+ key against your server. Instead of the key value here, use whatever key is
14
+ generated from running the commands.
15
+
16
+ In order to push a gem to your Gemstash server, you need to first create an API
17
+ key. Utilize the `gemstash` command to create the API key:
18
+ ```
19
+ $ gemstash authorize
20
+ Your new key is: e374e237fdf5fa5718d2a21bd63dc911
21
+ ```
22
+
23
+ This new key can `push`, `yank`, and `unyank` gems from your Gemstash server.
24
+ Run `gemstash authorize` with just the permissions you want to limit what the
25
+ key will be allowed to do. You can similarly update a specific key by providing
26
+ it via the `--key` option:
27
+ ```
28
+ $ gemstash authorize push yank --key e374e237fdf5fa5718d2a21bd63dc911
29
+ ```
30
+
31
+ When no permissions are provided (like the first example), the key will be
32
+ authorized for all permissions. Leave the key authorized with everything if you
33
+ want to use it to try all private gem interactions:
34
+ ```
35
+ $ gemstash authorize --key e374e237fdf5fa5718d2a21bd63dc911
36
+ ```
37
+
38
+ With the key generated, you'll need to tell Rubygems about your new key. If
39
+ you've pushed a gem to https://www.rubygems.org, then you will already have a
40
+ credentials file to add the key to. If not, run the following commands before
41
+ modifying the credentials file:
42
+ ```
43
+ $ mkdir -p ~/.gem
44
+ $ touch ~/.gem/credentials
45
+ $ chmod 0600 ~/.gem/credentials
46
+ ```
47
+
48
+ Add your new key to credentials such that it looks something like this (but make
49
+ sure not to remove any existing keys):
50
+ ```yaml
51
+ # ~/.gem/credentials
52
+ ---
53
+ :test_key: e374e237fdf5fa5718d2a21bd63dc911
54
+ ```
55
+
56
+ The name `test_key` can be anything you want, but you will need to remember it
57
+ and use it again later in this guide for the `--key` option.
58
+
59
+ ## Creating a Test Gem
60
+
61
+ You'll need a test gem before you can play with private gems on your Gemstash
62
+ server. If you have a gem you can use, move along to the next section. You can
63
+ start by instantiating a test gem via Bundler:
64
+ ```
65
+ $ bundle gem private-example
66
+ ```
67
+
68
+ You'll need to add a summary and description to the new gem's gemspec file in
69
+ order to successfully build it. Once you've built the gem, you will be ready to
70
+ push the new gem.
71
+ ```
72
+ $ cd private-example
73
+ $ rake build
74
+ ```
75
+
76
+ You will now have a gem at `private-example/pkg/private-example-0.1.0.gem`.
77
+
78
+ ## Pushing
79
+
80
+ If your Gemstash server isn't running, go ahead and start it:
81
+ ```
82
+ $ gemstash start
83
+ ```
84
+
85
+ Push your test gem using Rubygems:
86
+ ```
87
+ $ gem push --key test_key --host http://localhost:9292/private pkg/private-example-0.1.0.gem
88
+ ```
89
+
90
+ The `/private` portion of the `--host` option tells Gemstash you are interacting
91
+ with the private gems. Gemstash will not let you push, yank, or unyank from
92
+ anything except `/private`.
93
+
94
+ ## Bundling
95
+
96
+ Once your gem is pushed to your Gemstash server, you are ready to bundle it.
97
+ Create a `Gemfile` and specify the gem. You will probably want to wrap the
98
+ private gem in a source block, and let the rest of Gemstash handle all other
99
+ gems:
100
+ ```ruby
101
+ # ./Gemfile
102
+ source "http://localhost:9292"
103
+ gem "rubywarrior"
104
+
105
+ source "http://localhost:9292/private" do
106
+ gem "private-example"
107
+ end
108
+ ```
109
+
110
+ Notice that the Gemstash server points to `/private` again when installing your
111
+ private gem. Go ahead and bundle to install your new private gem:
112
+ ```
113
+ $ bundle
114
+ ```
115
+
116
+ ## Yanking
117
+
118
+ If you push a private gem by accident, you can yank the gem with Rubygems:
119
+ ```
120
+ $ RUBYGEMS_HOST=http://localhost:9292/private gem yank --key test_key private-example --version 0.1.0
121
+ ```
122
+
123
+ Like with pushing, the `/private` portion of the host option tells Gemstash you
124
+ are interacting with private gems. Gemstash will only let you yank from
125
+ `/private`. Unlike pushing, Rubygems doesn't support `--host` for yank and
126
+ unyank (yet), so you need to specify the host via the `RUBYGEMS_HOST`
127
+ environment variable.
128
+
129
+ ## Unyanking
130
+
131
+ If you yank a private gem by accident, you can unyank the gem with Rubygems:
132
+ ```
133
+ $ RUBYGEMS_HOST=http://localhost:9292/private gem yank --key test_key private-example --version 0.1.0 --undo
134
+ ```
135
+
136
+ Like with pushing and yanking, the `/private` portion of the host option tells
137
+ Gemstash you are interacting with private gems. Gemstash will only let you
138
+ unyank from `/private`. Unlike pushing, Rubygems doesn't support `--host` for
139
+ unyank and yank (yet), so you need to specify the host via the `RUBYGEMS_HOST`
140
+ environment variable.
data/docs/reference.md ADDED
@@ -0,0 +1,308 @@
1
+
2
+ Table of Contents
3
+ =================
4
+
5
+ * [Reference](#reference)
6
+ * [Configuration](#configuration)
7
+ * [:base_path](#base_path)
8
+ * [:cache_type](#cache_type)
9
+ * [:memcached_servers](#memcached_servers)
10
+ * [:db_adapter](#db_adapter)
11
+ * [:db_url](#db_url)
12
+ * [:rubygems_url](#rubygems_url)
13
+ * [:bind](#bind)
14
+ * [Authorize](#authorize)
15
+ * [Usage](#usage)
16
+ * [Arguments](#arguments)
17
+ * [Options](#options)
18
+ * [--config-file](#--config-file)
19
+ * [--key](#--key)
20
+ * [--remove](#--remove)
21
+ * [Start](#start)
22
+ * [Usage](#usage-1)
23
+ * [Options](#options-1)
24
+ * [--config-file](#--config-file-1)
25
+ * [--no-daemonize](#--no-daemonize)
26
+ * [Stop](#stop)
27
+ * [Usage](#usage-2)
28
+ * [Options](#options-2)
29
+ * [--config-file](#--config-file-2)
30
+ * [Status](#status)
31
+ * [Usage](#usage-3)
32
+ * [Options](#options-3)
33
+ * [--config-file](#--config-file-3)
34
+ * [Setup](#setup)
35
+ * [Usage](#usage-4)
36
+ * [Options](#options-4)
37
+ * [--redo](#--redo)
38
+ * [--debug](#--debug)
39
+ * [--config-file](#--config-file-4)
40
+
41
+
42
+
43
+ ---
44
+
45
+ # Reference
46
+
47
+ ## Configuration
48
+
49
+ **Example:**
50
+ ```yaml
51
+ # ~/.gemstash/config.yml
52
+ ---
53
+ :base_path: "/var/gemstash"
54
+ :cache_type: memcached
55
+ :memcached_servers: localhost:11211
56
+ :db_adapter: postgres
57
+ :db_url: postgres:///gemstash
58
+ :rubygems_url: https://my.gem-source.local
59
+ :bind: tcp://0.0.0.0:4242
60
+ ```
61
+
62
+ ### :base_path
63
+
64
+ **Default value:** `~/.gemstash`
65
+
66
+ **Valid values:** Any valid path
67
+
68
+ **Description**<br />
69
+ Specifies where to store local files like the server log, cached gem files, and
70
+ the database (when using SQLite). If the default is being used, the directory
71
+ will be created if it does not exist. Any other directory needs to be created
72
+ ahead of time and be writable to the Gemstash server process. Specifying the
73
+ `:base_path` via [`gemstash setup`](reference.md#setup) will create the
74
+ directory for you.
75
+
76
+ ### :cache_type
77
+
78
+ **Default value:** `memory`
79
+
80
+ **Valid values:** `memory`, `memcached`
81
+
82
+ **Description**<br />
83
+ Specifies how to cache values other than gem files (such as gem dependencies).
84
+ `memory` will use an in memory cache while `memcached` will point to 1 or more
85
+ Memcached servers. Use the `:memcached_servers` configuration key for specifying
86
+ where the Memcached server(s) are.
87
+
88
+ ### :memcached_servers
89
+
90
+ **Default value:** `localhost:11211`
91
+
92
+ **Valid values:** A comma delimited list of Memcached servers
93
+
94
+ **Description**<br />
95
+ Specifies the Memcached servers to connect to when using `memcached` for the
96
+ `:cache_type`. Only used when `memcached` is used for `:cache_type`.
97
+
98
+ ### :db_adapter
99
+
100
+ **Default value:** `sqlite3`
101
+
102
+ **Valid values:** `sqlite3`, `postgres`
103
+
104
+ **Description**<br />
105
+ Specifies what database adapter to use. When `sqlite3` is used, the database
106
+ will be located at `gemstash.db` within the directory specified by `:base_path`.
107
+ The database will automatically be created when using `sqlite3`. When `postgres`
108
+ is used, the database to connect to must be specified in the `:db_url`
109
+ configuration key. The database must already be created when using `postgres`.
110
+
111
+ ### :db_url
112
+
113
+ **Default value:** None
114
+
115
+ **Valid values:** A valid database URL for the [Sequel
116
+ gem](http://sequel.jeremyevans.net/)
117
+
118
+ **Description**<br />
119
+ Specifies the database to connect to when using `postgres` for the
120
+ `:db_adapter`. Only used when `postgres` is used for `:db_adapter`.
121
+
122
+ ### :rubygems_url
123
+
124
+ **Default value:** `https://www.rubygems.org`
125
+
126
+ **Valid values:** A valid gem source URL
127
+
128
+ **Description**<br />
129
+ Specifies the default gem source URL. When any API endpoint is called without a
130
+ `/private` or `/upstream/<url>` prefix, this URL will be used to fetch the
131
+ result. This value can be safely changed even if there are already gems stashed
132
+ for the previous value.
133
+
134
+ ### :bind
135
+
136
+ **Default value:** `tcp://0.0.0.0:9292`
137
+
138
+ **Valid values:** Any valid binding that [is supported by
139
+ Puma](https://github.com/puma/puma#binding-tcp--sockets)
140
+
141
+ **Description**<br />
142
+ Specifies the binding used to start the Gemstash server. Keep in mind the user
143
+ starting Gemstash needs to have access to bind in this manner. For example, if
144
+ you use a port below 1024, you will need to run Gemstash as the root user.
145
+
146
+ ## Authorize
147
+
148
+ Adds or removes authorization to interact with privately stored gems.
149
+
150
+ ### Usage
151
+
152
+ ```
153
+ gemstash authorize
154
+ gemstash authorize push yank
155
+ gemstash authorize yank unyank --key <secure-key>
156
+ gemstash authorize --remove --key <secure-key>
157
+ ```
158
+
159
+ ### Arguments
160
+
161
+ Any arguments will be used as specific permissions. Valid permissions include
162
+ `push`, `yank`, and `unyank`. If no permissions are provided, then all
163
+ permissions will be granted (including any that may be added in future versions
164
+ of Gemstash).
165
+
166
+ ### Options
167
+
168
+ #### --config-file
169
+
170
+ **Usage:** `--config-file <file>`
171
+
172
+ **Description**<br />
173
+ Specify the config file to use. If you aren't using the default config file at
174
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
175
+
176
+ #### --key
177
+
178
+ **Usage:** `--key <secure-key>`
179
+
180
+ **Description**<br />
181
+ Specify the API key to affect. This should be the actual key value, not a name.
182
+ This option is required when using `--remove` but is optional otherwise. If
183
+ adding an authorization, using this will either create or update the permissions
184
+ for the specified API key. If missing, a new API key will always be generated.
185
+
186
+ #### --remove
187
+
188
+ **Usage:** `--remove`
189
+
190
+ **Description**<br />
191
+ Remove an authorization rather than add or update one. When removing, permission
192
+ values are not allowed. The `--key <secure-key>` option is required.
193
+
194
+ ## Start
195
+
196
+ Starts the Gemstash server.
197
+
198
+ ### Usage
199
+
200
+ ```
201
+ gemstash start
202
+ gemstash start --no-daemonize
203
+ ```
204
+
205
+ ### Options
206
+
207
+ #### --config-file
208
+
209
+ **Usage:** `--config-file <file>`
210
+
211
+ **Description**<br />
212
+ Specify the config file to use. If you aren't using the default config file at
213
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
214
+
215
+ #### --no-daemonize
216
+
217
+ **Usage:** `--no-daemonize`
218
+
219
+ **Description**<br />
220
+ The Gemstash server daemonizes itself by default. Provide this option to instead
221
+ run the server until `Ctrl-C` is typed. When not daemonized, the log will be
222
+ output to standard out.
223
+
224
+ ## Stop
225
+
226
+ Stops the Gemstash server.
227
+
228
+ ### Usage
229
+
230
+ ```
231
+ gemstash stop
232
+ ```
233
+
234
+ ### Options
235
+
236
+ #### --config-file
237
+
238
+ **Usage:** `--config-file <file>`
239
+
240
+ **Description**<br />
241
+ Specify the config file to use. If you aren't using the default config file at
242
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
243
+
244
+ ## Status
245
+
246
+ Checks status of the Gemstash server.
247
+
248
+ ### Usage
249
+
250
+ ```
251
+ gemstash status
252
+ ```
253
+
254
+ ### Options
255
+
256
+ #### --config-file
257
+
258
+ **Usage:** `--config-file <file>`
259
+
260
+ **Description**<br />
261
+ Specify the config file to use. If you aren't using the default config file at
262
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
263
+
264
+ ## Setup
265
+
266
+ Customize your Gemstash configuration interactively. This will save your config
267
+ file, but only if a few checks pass after you've provided your answers.
268
+
269
+ ### Usage
270
+
271
+ ```
272
+ gemstash setup
273
+ gemstash setup --redo
274
+ gemstash setup --config-file <file>
275
+ ```
276
+
277
+ ### Options
278
+
279
+ #### --redo
280
+
281
+ **Usage:** `--redo`
282
+
283
+ **Description**<br />
284
+ Redo the configuration. This does nothing the first time `gemstash setup` is
285
+ run. If you want to change your configuration using `gemstash setup` after
286
+ you've run it before, you must provide this option, otherwise Gemstash will
287
+ simply indicate your setup is complete.
288
+
289
+ #### --debug
290
+
291
+ **Usage:** `--debug`
292
+
293
+ **Description**<br />
294
+ Output additional information if one of the checks at the end of setup fails.
295
+ This will do nothing if all checks pass.
296
+
297
+ #### --config-file
298
+
299
+ **Usage:** `--config-file <file>`
300
+
301
+ **Description**<br />
302
+ Specify the config file to write to. Without this option, your configuration
303
+ will be written to `~/.gemstash/config.yml`. If you write to a custom location,
304
+ you will need to pass the `--config-file` option to all Gemstash commands.
305
+
306
+ ---
307
+
308
+ Table of contents thanks to [gh-md-toc](https://github.com/ekalinin/github-markdown-toc).