gemstash 1.0.0.pre.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. checksums.yaml +7 -0
  2. data/CODE_OF_CONDUCT.md +13 -0
  3. data/Gemfile +4 -0
  4. data/LICENSE.txt +21 -0
  5. data/README.md +139 -0
  6. data/Rakefile +35 -0
  7. data/bin/console +14 -0
  8. data/bin/gemstash +3 -0
  9. data/bin/setup +5 -0
  10. data/docs/config.md +136 -0
  11. data/docs/debug.md +24 -0
  12. data/docs/deploy.md +30 -0
  13. data/docs/mirror.md +30 -0
  14. data/docs/multiple_sources.md +68 -0
  15. data/docs/private_gems.md +140 -0
  16. data/docs/reference.md +308 -0
  17. data/exe/gemstash +3 -0
  18. data/gemstash.gemspec +47 -0
  19. data/gemstash.png +0 -0
  20. data/lib/gemstash.rb +26 -0
  21. data/lib/gemstash/authorization.rb +87 -0
  22. data/lib/gemstash/cache.rb +79 -0
  23. data/lib/gemstash/cli.rb +71 -0
  24. data/lib/gemstash/cli/authorize.rb +69 -0
  25. data/lib/gemstash/cli/base.rb +46 -0
  26. data/lib/gemstash/cli/setup.rb +173 -0
  27. data/lib/gemstash/cli/start.rb +52 -0
  28. data/lib/gemstash/cli/status.rb +21 -0
  29. data/lib/gemstash/cli/stop.rb +21 -0
  30. data/lib/gemstash/config.ru +13 -0
  31. data/lib/gemstash/configuration.rb +41 -0
  32. data/lib/gemstash/db.rb +15 -0
  33. data/lib/gemstash/db/authorization.rb +20 -0
  34. data/lib/gemstash/db/dependency.rb +50 -0
  35. data/lib/gemstash/db/rubygem.rb +14 -0
  36. data/lib/gemstash/db/version.rb +51 -0
  37. data/lib/gemstash/dependencies.rb +93 -0
  38. data/lib/gemstash/env.rb +150 -0
  39. data/lib/gemstash/gem_fetcher.rb +50 -0
  40. data/lib/gemstash/gem_pusher.rb +125 -0
  41. data/lib/gemstash/gem_source.rb +37 -0
  42. data/lib/gemstash/gem_source/dependency_caching.rb +40 -0
  43. data/lib/gemstash/gem_source/private_source.rb +139 -0
  44. data/lib/gemstash/gem_source/rack_middleware.rb +22 -0
  45. data/lib/gemstash/gem_source/upstream_source.rb +183 -0
  46. data/lib/gemstash/gem_unyanker.rb +61 -0
  47. data/lib/gemstash/gem_yanker.rb +61 -0
  48. data/lib/gemstash/http_client.rb +77 -0
  49. data/lib/gemstash/logging.rb +93 -0
  50. data/lib/gemstash/migrations/01_gem_dependencies.rb +41 -0
  51. data/lib/gemstash/migrations/02_authorizations.rb +12 -0
  52. data/lib/gemstash/puma.rb +6 -0
  53. data/lib/gemstash/rack_env_rewriter.rb +66 -0
  54. data/lib/gemstash/specs_builder.rb +93 -0
  55. data/lib/gemstash/storage.rb +207 -0
  56. data/lib/gemstash/upstream.rb +65 -0
  57. data/lib/gemstash/version.rb +4 -0
  58. data/lib/gemstash/web.rb +97 -0
  59. metadata +306 -0
data/docs/mirror.md ADDED
@@ -0,0 +1,30 @@
1
+ # Using Gemstash as a Mirror
2
+
3
+ If you don't have control over your `Gemfile`, or you don't want to force
4
+ everyone on your team to go through the Gemstash server, you can use Bundler
5
+ mirroring to bundle against your Gemstash server.
6
+
7
+ For each source in your `Gemfile`, add a mirror pointing to your Gemstash
8
+ server:
9
+ ```
10
+ $ bundle config mirror.http://rubygems.org http://localhost:9292
11
+ $ bundle config mirror.https://my.gem-source.local http://localhost:9292/upstream/$(ruby -rcgi -e 'puts CGI.escape("https://my.gem-source.local")')
12
+ ```
13
+
14
+ From now on, bundler will fetch gems from those sources via your Gemstash
15
+ server.
16
+
17
+ # Simpler Gemstash Mirrors
18
+
19
+ **WARNING: This feature is not yet available.**
20
+
21
+ **This feature requires Bundler to be at least version `1.11.0`.**
22
+
23
+ If you are using Bundler version `1.11.0` or greater, the mirroring becomes a
24
+ bit easier:
25
+ ```
26
+ $ bundle config mirror.http://rubygems.org http://localhost:9292
27
+ $ bundle config mirror.https://my.gem-source.local http://localhost:9292
28
+ ```
29
+
30
+ Bundler will then send headers to Gemstash to indicate the correct upstream.
@@ -0,0 +1,68 @@
1
+ # Multiple Gem Sources
2
+
3
+ Gemstash will stash from any amount of gem sources. By the end of this guide,
4
+ you will be able to bundle using multiple gem sources, all stashed within your
5
+ Gemstash server.
6
+
7
+ ## Default Source
8
+
9
+ When you don't provide an explicit source (as with the [Quickstart
10
+ Guide](../README.md#quickstart-guide)), your gems will be fetched from
11
+ https://www.rubygems.org. This default source is not set in stone. To change it,
12
+ you need only edit the Gemstash configuration found at `~/.gemstash/config.yml`:
13
+ ```yaml
14
+ # ~/.gemstash/config.yml
15
+ ---
16
+ :rubygems_url: https://my.gem-source.local
17
+ ```
18
+
19
+ Make sure to restart your Gemstash server after changing the config:
20
+ ```
21
+ $ gemstash stop
22
+ $ gemstash start
23
+ ```
24
+
25
+ Once restarted, bundling against `http://localhost:9292` will fetch gems from
26
+ `https://my.gem-source.local`. If you had bundled before making these changes,
27
+ fear not; bundling with a different default gem source will store gems in a
28
+ separate location, ensuring different sources won't leak between each other.
29
+
30
+ ## Bundling with Multiple Sources
31
+
32
+ Changing the default source won't help you if you need to bundle against
33
+ https://www.rubygems.org along with additional sources. If you need to bundle
34
+ with multiple gem sources, Gemstash doesn't need to be specially configured.
35
+ Your Gemstash server will honor any gem source specified via a specialized URL.
36
+ Consider the following `Gemfile`:
37
+ ```ruby
38
+ # ./Gemfile
39
+ require "cgi"
40
+ source "http://localhost:9292"
41
+ gem "rubywarrior"
42
+
43
+ source "http://localhost:9292/upstream/#{CGI.escape("https://my.gem-source.local")}" do
44
+ gem "my-gem"
45
+ end
46
+ ```
47
+
48
+ Notice the `CGI.escape` call in the second source. This is important, as it
49
+ properly URL escapes the source URL so Gemstash knows what gem source you want.
50
+ The `/upstream` prefix tells Gemstash to use a gem source other than the default
51
+ source. You can now bundle with the additional source.
52
+
53
+ ## Redirecting
54
+
55
+ Gemstash supports an alternate mode of specifying your gem sources. If you want
56
+ Gemstash to redirect Bundler to your given gem sources, then you can specify
57
+ your `Gemfile` like so:
58
+ ```ruby
59
+ # ./Gemfile
60
+ require "cgi"
61
+ source "http://localhost:9292/redirect/#{CGI.escape("https://www.rubygems.org")}"
62
+ gem "rubywarrior"
63
+ ```
64
+
65
+ Notice the `/redirect` prefix. This prefix tells Gemstash to redirect API calls
66
+ to the provided URL. Redirected calls like this will not be cached by Gemstash,
67
+ and gem files will not be stashed, even if they were previously cached or
68
+ stashed from the same gem source.
@@ -0,0 +1,140 @@
1
+ # Private Gems
2
+
3
+ Stashing private gems in your Gemstash server requires a bit of additional
4
+ setup. If you haven't read through the [Quickstart
5
+ Guide](../README.md#quickstart-guide), you should do that first. By the end of
6
+ this guide, you will be able to interact with your Gemstash server to store and
7
+ retrieve your private gems.
8
+
9
+ ## Authorizing
10
+
11
+ **IMPORTANT NOTE:** Do not use the actual key value in this document, otherwise
12
+ your Gemstash server will be vulnerable to anyone who wants to try to use the
13
+ key against your server. Instead of the key value here, use whatever key is
14
+ generated from running the commands.
15
+
16
+ In order to push a gem to your Gemstash server, you need to first create an API
17
+ key. Utilize the `gemstash` command to create the API key:
18
+ ```
19
+ $ gemstash authorize
20
+ Your new key is: e374e237fdf5fa5718d2a21bd63dc911
21
+ ```
22
+
23
+ This new key can `push`, `yank`, and `unyank` gems from your Gemstash server.
24
+ Run `gemstash authorize` with just the permissions you want to limit what the
25
+ key will be allowed to do. You can similarly update a specific key by providing
26
+ it via the `--key` option:
27
+ ```
28
+ $ gemstash authorize push yank --key e374e237fdf5fa5718d2a21bd63dc911
29
+ ```
30
+
31
+ When no permissions are provided (like the first example), the key will be
32
+ authorized for all permissions. Leave the key authorized with everything if you
33
+ want to use it to try all private gem interactions:
34
+ ```
35
+ $ gemstash authorize --key e374e237fdf5fa5718d2a21bd63dc911
36
+ ```
37
+
38
+ With the key generated, you'll need to tell Rubygems about your new key. If
39
+ you've pushed a gem to https://www.rubygems.org, then you will already have a
40
+ credentials file to add the key to. If not, run the following commands before
41
+ modifying the credentials file:
42
+ ```
43
+ $ mkdir -p ~/.gem
44
+ $ touch ~/.gem/credentials
45
+ $ chmod 0600 ~/.gem/credentials
46
+ ```
47
+
48
+ Add your new key to credentials such that it looks something like this (but make
49
+ sure not to remove any existing keys):
50
+ ```yaml
51
+ # ~/.gem/credentials
52
+ ---
53
+ :test_key: e374e237fdf5fa5718d2a21bd63dc911
54
+ ```
55
+
56
+ The name `test_key` can be anything you want, but you will need to remember it
57
+ and use it again later in this guide for the `--key` option.
58
+
59
+ ## Creating a Test Gem
60
+
61
+ You'll need a test gem before you can play with private gems on your Gemstash
62
+ server. If you have a gem you can use, move along to the next section. You can
63
+ start by instantiating a test gem via Bundler:
64
+ ```
65
+ $ bundle gem private-example
66
+ ```
67
+
68
+ You'll need to add a summary and description to the new gem's gemspec file in
69
+ order to successfully build it. Once you've built the gem, you will be ready to
70
+ push the new gem.
71
+ ```
72
+ $ cd private-example
73
+ $ rake build
74
+ ```
75
+
76
+ You will now have a gem at `private-example/pkg/private-example-0.1.0.gem`.
77
+
78
+ ## Pushing
79
+
80
+ If your Gemstash server isn't running, go ahead and start it:
81
+ ```
82
+ $ gemstash start
83
+ ```
84
+
85
+ Push your test gem using Rubygems:
86
+ ```
87
+ $ gem push --key test_key --host http://localhost:9292/private pkg/private-example-0.1.0.gem
88
+ ```
89
+
90
+ The `/private` portion of the `--host` option tells Gemstash you are interacting
91
+ with the private gems. Gemstash will not let you push, yank, or unyank from
92
+ anything except `/private`.
93
+
94
+ ## Bundling
95
+
96
+ Once your gem is pushed to your Gemstash server, you are ready to bundle it.
97
+ Create a `Gemfile` and specify the gem. You will probably want to wrap the
98
+ private gem in a source block, and let the rest of Gemstash handle all other
99
+ gems:
100
+ ```ruby
101
+ # ./Gemfile
102
+ source "http://localhost:9292"
103
+ gem "rubywarrior"
104
+
105
+ source "http://localhost:9292/private" do
106
+ gem "private-example"
107
+ end
108
+ ```
109
+
110
+ Notice that the Gemstash server points to `/private` again when installing your
111
+ private gem. Go ahead and bundle to install your new private gem:
112
+ ```
113
+ $ bundle
114
+ ```
115
+
116
+ ## Yanking
117
+
118
+ If you push a private gem by accident, you can yank the gem with Rubygems:
119
+ ```
120
+ $ RUBYGEMS_HOST=http://localhost:9292/private gem yank --key test_key private-example --version 0.1.0
121
+ ```
122
+
123
+ Like with pushing, the `/private` portion of the host option tells Gemstash you
124
+ are interacting with private gems. Gemstash will only let you yank from
125
+ `/private`. Unlike pushing, Rubygems doesn't support `--host` for yank and
126
+ unyank (yet), so you need to specify the host via the `RUBYGEMS_HOST`
127
+ environment variable.
128
+
129
+ ## Unyanking
130
+
131
+ If you yank a private gem by accident, you can unyank the gem with Rubygems:
132
+ ```
133
+ $ RUBYGEMS_HOST=http://localhost:9292/private gem yank --key test_key private-example --version 0.1.0 --undo
134
+ ```
135
+
136
+ Like with pushing and yanking, the `/private` portion of the host option tells
137
+ Gemstash you are interacting with private gems. Gemstash will only let you
138
+ unyank from `/private`. Unlike pushing, Rubygems doesn't support `--host` for
139
+ unyank and yank (yet), so you need to specify the host via the `RUBYGEMS_HOST`
140
+ environment variable.
data/docs/reference.md ADDED
@@ -0,0 +1,308 @@
1
+
2
+ Table of Contents
3
+ =================
4
+
5
+ * [Reference](#reference)
6
+ * [Configuration](#configuration)
7
+ * [:base_path](#base_path)
8
+ * [:cache_type](#cache_type)
9
+ * [:memcached_servers](#memcached_servers)
10
+ * [:db_adapter](#db_adapter)
11
+ * [:db_url](#db_url)
12
+ * [:rubygems_url](#rubygems_url)
13
+ * [:bind](#bind)
14
+ * [Authorize](#authorize)
15
+ * [Usage](#usage)
16
+ * [Arguments](#arguments)
17
+ * [Options](#options)
18
+ * [--config-file](#--config-file)
19
+ * [--key](#--key)
20
+ * [--remove](#--remove)
21
+ * [Start](#start)
22
+ * [Usage](#usage-1)
23
+ * [Options](#options-1)
24
+ * [--config-file](#--config-file-1)
25
+ * [--no-daemonize](#--no-daemonize)
26
+ * [Stop](#stop)
27
+ * [Usage](#usage-2)
28
+ * [Options](#options-2)
29
+ * [--config-file](#--config-file-2)
30
+ * [Status](#status)
31
+ * [Usage](#usage-3)
32
+ * [Options](#options-3)
33
+ * [--config-file](#--config-file-3)
34
+ * [Setup](#setup)
35
+ * [Usage](#usage-4)
36
+ * [Options](#options-4)
37
+ * [--redo](#--redo)
38
+ * [--debug](#--debug)
39
+ * [--config-file](#--config-file-4)
40
+
41
+
42
+
43
+ ---
44
+
45
+ # Reference
46
+
47
+ ## Configuration
48
+
49
+ **Example:**
50
+ ```yaml
51
+ # ~/.gemstash/config.yml
52
+ ---
53
+ :base_path: "/var/gemstash"
54
+ :cache_type: memcached
55
+ :memcached_servers: localhost:11211
56
+ :db_adapter: postgres
57
+ :db_url: postgres:///gemstash
58
+ :rubygems_url: https://my.gem-source.local
59
+ :bind: tcp://0.0.0.0:4242
60
+ ```
61
+
62
+ ### :base_path
63
+
64
+ **Default value:** `~/.gemstash`
65
+
66
+ **Valid values:** Any valid path
67
+
68
+ **Description**<br />
69
+ Specifies where to store local files like the server log, cached gem files, and
70
+ the database (when using SQLite). If the default is being used, the directory
71
+ will be created if it does not exist. Any other directory needs to be created
72
+ ahead of time and be writable to the Gemstash server process. Specifying the
73
+ `:base_path` via [`gemstash setup`](reference.md#setup) will create the
74
+ directory for you.
75
+
76
+ ### :cache_type
77
+
78
+ **Default value:** `memory`
79
+
80
+ **Valid values:** `memory`, `memcached`
81
+
82
+ **Description**<br />
83
+ Specifies how to cache values other than gem files (such as gem dependencies).
84
+ `memory` will use an in memory cache while `memcached` will point to 1 or more
85
+ Memcached servers. Use the `:memcached_servers` configuration key for specifying
86
+ where the Memcached server(s) are.
87
+
88
+ ### :memcached_servers
89
+
90
+ **Default value:** `localhost:11211`
91
+
92
+ **Valid values:** A comma delimited list of Memcached servers
93
+
94
+ **Description**<br />
95
+ Specifies the Memcached servers to connect to when using `memcached` for the
96
+ `:cache_type`. Only used when `memcached` is used for `:cache_type`.
97
+
98
+ ### :db_adapter
99
+
100
+ **Default value:** `sqlite3`
101
+
102
+ **Valid values:** `sqlite3`, `postgres`
103
+
104
+ **Description**<br />
105
+ Specifies what database adapter to use. When `sqlite3` is used, the database
106
+ will be located at `gemstash.db` within the directory specified by `:base_path`.
107
+ The database will automatically be created when using `sqlite3`. When `postgres`
108
+ is used, the database to connect to must be specified in the `:db_url`
109
+ configuration key. The database must already be created when using `postgres`.
110
+
111
+ ### :db_url
112
+
113
+ **Default value:** None
114
+
115
+ **Valid values:** A valid database URL for the [Sequel
116
+ gem](http://sequel.jeremyevans.net/)
117
+
118
+ **Description**<br />
119
+ Specifies the database to connect to when using `postgres` for the
120
+ `:db_adapter`. Only used when `postgres` is used for `:db_adapter`.
121
+
122
+ ### :rubygems_url
123
+
124
+ **Default value:** `https://www.rubygems.org`
125
+
126
+ **Valid values:** A valid gem source URL
127
+
128
+ **Description**<br />
129
+ Specifies the default gem source URL. When any API endpoint is called without a
130
+ `/private` or `/upstream/<url>` prefix, this URL will be used to fetch the
131
+ result. This value can be safely changed even if there are already gems stashed
132
+ for the previous value.
133
+
134
+ ### :bind
135
+
136
+ **Default value:** `tcp://0.0.0.0:9292`
137
+
138
+ **Valid values:** Any valid binding that [is supported by
139
+ Puma](https://github.com/puma/puma#binding-tcp--sockets)
140
+
141
+ **Description**<br />
142
+ Specifies the binding used to start the Gemstash server. Keep in mind the user
143
+ starting Gemstash needs to have access to bind in this manner. For example, if
144
+ you use a port below 1024, you will need to run Gemstash as the root user.
145
+
146
+ ## Authorize
147
+
148
+ Adds or removes authorization to interact with privately stored gems.
149
+
150
+ ### Usage
151
+
152
+ ```
153
+ gemstash authorize
154
+ gemstash authorize push yank
155
+ gemstash authorize yank unyank --key <secure-key>
156
+ gemstash authorize --remove --key <secure-key>
157
+ ```
158
+
159
+ ### Arguments
160
+
161
+ Any arguments will be used as specific permissions. Valid permissions include
162
+ `push`, `yank`, and `unyank`. If no permissions are provided, then all
163
+ permissions will be granted (including any that may be added in future versions
164
+ of Gemstash).
165
+
166
+ ### Options
167
+
168
+ #### --config-file
169
+
170
+ **Usage:** `--config-file <file>`
171
+
172
+ **Description**<br />
173
+ Specify the config file to use. If you aren't using the default config file at
174
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
175
+
176
+ #### --key
177
+
178
+ **Usage:** `--key <secure-key>`
179
+
180
+ **Description**<br />
181
+ Specify the API key to affect. This should be the actual key value, not a name.
182
+ This option is required when using `--remove` but is optional otherwise. If
183
+ adding an authorization, using this will either create or update the permissions
184
+ for the specified API key. If missing, a new API key will always be generated.
185
+
186
+ #### --remove
187
+
188
+ **Usage:** `--remove`
189
+
190
+ **Description**<br />
191
+ Remove an authorization rather than add or update one. When removing, permission
192
+ values are not allowed. The `--key <secure-key>` option is required.
193
+
194
+ ## Start
195
+
196
+ Starts the Gemstash server.
197
+
198
+ ### Usage
199
+
200
+ ```
201
+ gemstash start
202
+ gemstash start --no-daemonize
203
+ ```
204
+
205
+ ### Options
206
+
207
+ #### --config-file
208
+
209
+ **Usage:** `--config-file <file>`
210
+
211
+ **Description**<br />
212
+ Specify the config file to use. If you aren't using the default config file at
213
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
214
+
215
+ #### --no-daemonize
216
+
217
+ **Usage:** `--no-daemonize`
218
+
219
+ **Description**<br />
220
+ The Gemstash server daemonizes itself by default. Provide this option to instead
221
+ run the server until `Ctrl-C` is typed. When not daemonized, the log will be
222
+ output to standard out.
223
+
224
+ ## Stop
225
+
226
+ Stops the Gemstash server.
227
+
228
+ ### Usage
229
+
230
+ ```
231
+ gemstash stop
232
+ ```
233
+
234
+ ### Options
235
+
236
+ #### --config-file
237
+
238
+ **Usage:** `--config-file <file>`
239
+
240
+ **Description**<br />
241
+ Specify the config file to use. If you aren't using the default config file at
242
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
243
+
244
+ ## Status
245
+
246
+ Checks status of the Gemstash server.
247
+
248
+ ### Usage
249
+
250
+ ```
251
+ gemstash status
252
+ ```
253
+
254
+ ### Options
255
+
256
+ #### --config-file
257
+
258
+ **Usage:** `--config-file <file>`
259
+
260
+ **Description**<br />
261
+ Specify the config file to use. If you aren't using the default config file at
262
+ `~/.gemstash/config.yml`, then you must specify the config file via this option.
263
+
264
+ ## Setup
265
+
266
+ Customize your Gemstash configuration interactively. This will save your config
267
+ file, but only if a few checks pass after you've provided your answers.
268
+
269
+ ### Usage
270
+
271
+ ```
272
+ gemstash setup
273
+ gemstash setup --redo
274
+ gemstash setup --config-file <file>
275
+ ```
276
+
277
+ ### Options
278
+
279
+ #### --redo
280
+
281
+ **Usage:** `--redo`
282
+
283
+ **Description**<br />
284
+ Redo the configuration. This does nothing the first time `gemstash setup` is
285
+ run. If you want to change your configuration using `gemstash setup` after
286
+ you've run it before, you must provide this option, otherwise Gemstash will
287
+ simply indicate your setup is complete.
288
+
289
+ #### --debug
290
+
291
+ **Usage:** `--debug`
292
+
293
+ **Description**<br />
294
+ Output additional information if one of the checks at the end of setup fails.
295
+ This will do nothing if all checks pass.
296
+
297
+ #### --config-file
298
+
299
+ **Usage:** `--config-file <file>`
300
+
301
+ **Description**<br />
302
+ Specify the config file to write to. Without this option, your configuration
303
+ will be written to `~/.gemstash/config.yml`. If you write to a custom location,
304
+ you will need to pass the `--config-file` option to all Gemstash commands.
305
+
306
+ ---
307
+
308
+ Table of contents thanks to [gh-md-toc](https://github.com/ekalinin/github-markdown-toc).