gemstash 2.0.0 → 2.1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (75) hide show
  1. checksums.yaml +4 -4
  2. data/exe/gemstash +3 -0
  3. data/lib/gemstash.rb +2 -0
  4. data/lib/gemstash/api_key_authorization.rb +3 -0
  5. data/lib/gemstash/authorization.rb +7 -3
  6. data/lib/gemstash/cache.rb +5 -2
  7. data/lib/gemstash/cli.rb +2 -0
  8. data/lib/gemstash/cli/authorize.rb +3 -3
  9. data/lib/gemstash/cli/base.rb +4 -1
  10. data/lib/gemstash/cli/setup.rb +8 -1
  11. data/lib/gemstash/cli/start.rb +4 -1
  12. data/lib/gemstash/cli/status.rb +2 -0
  13. data/lib/gemstash/cli/stop.rb +2 -0
  14. data/lib/gemstash/config.ru +3 -3
  15. data/lib/gemstash/configuration.rb +7 -1
  16. data/lib/gemstash/db.rb +3 -0
  17. data/lib/gemstash/db/authorization.rb +2 -0
  18. data/lib/gemstash/db/cached_rubygem.rb +3 -0
  19. data/lib/gemstash/db/dependency.rb +2 -0
  20. data/lib/gemstash/db/rubygem.rb +3 -0
  21. data/lib/gemstash/db/upstream.rb +3 -0
  22. data/lib/gemstash/db/version.rb +3 -0
  23. data/lib/gemstash/dependencies.rb +4 -0
  24. data/lib/gemstash/env.rb +22 -4
  25. data/lib/gemstash/gem_fetcher.rb +2 -0
  26. data/lib/gemstash/gem_pusher.rb +5 -3
  27. data/lib/gemstash/gem_source.rb +2 -0
  28. data/lib/gemstash/gem_source/dependency_caching.rb +4 -4
  29. data/lib/gemstash/gem_source/private_source.rb +3 -0
  30. data/lib/gemstash/gem_source/rack_middleware.rb +3 -0
  31. data/lib/gemstash/gem_source/upstream_source.rb +9 -3
  32. data/lib/gemstash/gem_yanker.rb +4 -0
  33. data/lib/gemstash/health.rb +2 -0
  34. data/lib/gemstash/http_client.rb +4 -1
  35. data/lib/gemstash/logging.rb +2 -0
  36. data/lib/gemstash/man/gemstash-authorize.1 +51 -0
  37. data/lib/gemstash/man/gemstash-authorize.1.txt +33 -53
  38. data/lib/gemstash/man/gemstash-configuration.5 +229 -0
  39. data/lib/gemstash/man/gemstash-configuration.5.txt +150 -6
  40. data/lib/gemstash/man/gemstash-customize.7 +301 -0
  41. data/lib/gemstash/man/gemstash-customize.7.txt +183 -62
  42. data/lib/gemstash/man/gemstash-debugging.7 +34 -0
  43. data/lib/gemstash/man/gemstash-debugging.7.txt +17 -54
  44. data/lib/gemstash/man/gemstash-deploy.7 +72 -0
  45. data/lib/gemstash/man/gemstash-deploy.7.txt +44 -51
  46. data/lib/gemstash/man/gemstash-mirror.7 +40 -0
  47. data/lib/gemstash/man/gemstash-mirror.7.txt +20 -53
  48. data/lib/gemstash/man/gemstash-multiple-sources.7 +89 -0
  49. data/lib/gemstash/man/gemstash-multiple-sources.7.txt +53 -48
  50. data/lib/gemstash/man/gemstash-private-gems.7 +227 -0
  51. data/lib/gemstash/man/gemstash-private-gems.7.txt +108 -18
  52. data/lib/gemstash/man/gemstash-readme.7 +233 -0
  53. data/lib/gemstash/man/gemstash-readme.7.txt +126 -10
  54. data/lib/gemstash/man/gemstash-setup.1 +43 -0
  55. data/lib/gemstash/man/gemstash-setup.1.txt +28 -54
  56. data/lib/gemstash/man/gemstash-start.1 +26 -0
  57. data/lib/gemstash/man/gemstash-start.1.txt +16 -56
  58. data/lib/gemstash/man/gemstash-status.1 +20 -0
  59. data/lib/gemstash/man/gemstash-status.1.txt +13 -57
  60. data/lib/gemstash/man/gemstash-stop.1 +20 -0
  61. data/lib/gemstash/man/gemstash-stop.1.txt +13 -57
  62. data/lib/gemstash/man/gemstash-version.1 +22 -0
  63. data/lib/gemstash/man/gemstash-version.1.txt +12 -57
  64. data/lib/gemstash/migrations/01_gem_dependencies.rb +2 -0
  65. data/lib/gemstash/migrations/02_authorizations.rb +2 -0
  66. data/lib/gemstash/migrations/03_cached_gems.rb +2 -0
  67. data/lib/gemstash/migrations/04_health_tests.rb +2 -0
  68. data/lib/gemstash/puma.rb +2 -0
  69. data/lib/gemstash/rack_env_rewriter.rb +11 -2
  70. data/lib/gemstash/specs_builder.rb +5 -1
  71. data/lib/gemstash/storage.rb +12 -3
  72. data/lib/gemstash/upstream.rb +2 -0
  73. data/lib/gemstash/version.rb +3 -1
  74. data/lib/gemstash/web.rb +2 -0
  75. metadata +69 -22
@@ -1,66 +1,187 @@
1
+ gemstash-customize(7) gemstash-customize(7)
2
+
3
+
4
+
5
+ 1mCUSTOMIZING THE SERVER0m
6
+ Although Gemstash is designed for as minimal setup as possible, there
7
+ may be times you will want to change some of the default configuration.
8
+ By the end of this guide, you will be able to customize some of the
9
+ Gemstash behavior, including where files are stored, what database Gem-
10
+ stash uses, and how Gemstash caches certain requests.
11
+
12
+ 1mSETUP0m
13
+ Several customizable options are available via an interactive Gemstash
14
+ command. Run gemstash setup and answer the questions it provides (a
15
+ blank answer will use the default value):
16
+
17
+ $ gemstash setup
18
+ Where should files go? [~/.gemstash]
19
+ Cache with what? [MEMORY, memcached] 1mmemcached0m
20
+ What is the comma separated Memcached servers? [local-
21
+ host:11211]
22
+ What database adapter? [SQLITE3, postgres, mysql, mysql2] 1mpost-0m
23
+ 1mgres0m
24
+ Where is the database? [postgres:///gemstash]
25
+ Checking that the cache is available
26
+ Checking that the database is available
27
+ The database is not available
28
+ Once you've answered the questions, some checks will be made to ensure
29
+ the configuration will work. For example, the database didn't exist in
30
+ the previous example, so the command failed and the configuration
31
+ wasn't saved. If the command passes, you may provide the --redo option
32
+ to force configuration to be redone:
33
+
34
+ $ gemstash setup --redo
35
+ Where should files go? [~/.gemstash]
36
+ Cache with what? [MEMORY, memcached] 1mmemcached0m
37
+ What is the comma separated Memcached servers? [local-
38
+ host:11211]
39
+ What database adapter? [SQLITE3, postgres, mysql, mysql2]
40
+ Checking that the cache is available
41
+ Checking that the database is available
42
+ You are all setup!
43
+ Once all checks have passed, Gemstash will store your answers in the
44
+ configuration file located at ~/.gemstash/config.yml.
45
+
46
+ 1mFILES0m
47
+ Storage in Gemstash defaults to ~/.gemstash unless otherwise specified.
48
+ You can change this in your config file via the :base_path key:
49
+
50
+ # ~/.gemstash/config.yml
51
+ ---
52
+ :base_path: "/var/gemstash"
53
+
54
+ When customizing the base_path, the directory must exist, otherwise
55
+ Gemstash will fail to run. Thus, if you want to use /var/gemstash like
56
+ in the previous example, make sure to mkdir /var/gemstash and grant ac-
57
+ cess to the directory for the user you run Gemstash with.
58
+
59
+ 1mDATABASE0m
60
+ The :db_adapter configuration key specifies what database you will be
61
+ using. The default :db_adapter is sqlite3 (https://www.sqlite.org/),
62
+ which will use a database file located within your :base_path. The
63
+ database file will always be named gemstash.db.
64
+
65
+ You may also use postgres (http://www.postgresql.org/), mysql
66
+ (http://www.mysql.com/), or mysql2 (http://sequel.jeremye-
67
+ vans.net/rdoc/files/doc/opening_databases_rdoc.html#label-mysql2) for
68
+ your :db_adapter. When using any of these options, you need to specify
69
+ the :db_url to point to an existing database. Here is an example con-
70
+ figuration to use the postgres adapter:
71
+
72
+ # ~/.gemstash/config.yml
73
+ ---
74
+ :db_adapter: postgres
75
+ :db_url: postgres:///gemstash
76
+ :db_connection_options: # Sequel.connect options
77
+ :connect_timeout: 10
78
+ :read_timeout: 5
79
+ :timeout: 30
80
+
81
+ Regardless of the adapter you choose, the database will automatically
82
+ migrate to your version of Gemstash whenever the database is needed.
83
+ You only need to ensure the database exists and Gemstash will do the
84
+ rest, except for sqlite3 (for which Gemstash will also create the data-
85
+ base for you).
86
+
87
+ 1mCACHE0m
88
+ Certain things (like dependencies) are cached in memory. This avoids
89
+ web calls to the gem source, and database calls for private gems.
90
+
91
+ # ~/.gemstash/config.yml
92
+ ---
93
+ :cache_type: memory
94
+ :cache_max_size: 2000
95
+
96
+ This configuration uses the default memory cache, and has increased the
97
+ cache_max_size setting from its default of 500 items.
98
+
99
+ The memory cache can optionally be swapped out with a Memcached
100
+ (http://memcached.org/) server (or cluster of servers).
101
+
102
+ To use Memcached, use the memcached :cache_type configuration.
103
+
104
+ Provide the servers as a comma-separated list to the :memcached_servers
105
+ configuration key:
106
+
107
+ # ~/.gemstash/config.yml
108
+ ---
109
+ :cache_type: memcached
110
+ :memcached_servers: memcached1.local:11211,memcached2.local:11211
111
+ :cache_expiration: 1800
112
+
113
+ All caching expires in cache_expiration number of seconds. Default is
114
+ 1800 seconds (30 minutes). This option applies to all caching.
115
+
116
+ 1mSERVER0m
117
+ Gemstash uses Puma (http://puma.io/) and Rack (http://rack.github.io/)
118
+ as the server. Alternate server configurations are not currently sup-
119
+ ported, but you can take a look at the Puma configuration
120
+ (https://github.com/rubygems/gemstash/blob/master/lib/gemstash/puma.rb)
121
+ and the rackup file (https://github.com/rubygems/gemstash/blob/mas-
122
+ ter/lib/gemstash/config.ru) for inspiration.
123
+
124
+ While the server is not customizable, the way Gemstash binds the port
125
+ can be changed. To change the binding, update the :bind configuration
126
+ key:
127
+
128
+ # ~/.gemstash/config.yml
129
+ ---
130
+ :bind: tcp://0.0.0.0:4242
131
+
132
+ This maps directly to the Puma bind flag
133
+ (https://github.com/puma/puma#binding-tcp--sockets), and will support
134
+ anything valid for that flag.
135
+
136
+ The number of threads Puma uses is also customizable via the
137
+ :puma_threads configuration key. The default is 16.
138
+
139
+ 1mPROTECTED FETCH0m
140
+ Gemstash by default allows unauthenticated access for private gems.
141
+ Authenticated access is available via the :protected_fetch configura-
142
+ tion key.
143
+
144
+ # ~/.gemstash/config.yml
145
+ ---
146
+ :protected_fetch: true
147
+
148
+ More details on protected_fetch are here (gemstash help private-
149
+ gems.7).
150
+
151
+ 1mFETCH TIMEOUT0m
152
+ The default fetch timeout is 20 seconds. Use the :fetch_timeout con-
153
+ figuration key to change it.
154
+
155
+ ---
156
+ :fetch_timeout: 20
157
+
158
+ 1mCONFIG FILE LOCATION0m
159
+ By default, configuration for Gemstash will be at ~/.gemstash/con-
160
+ fig.yml. This can be changed by providing the --config-file option to
161
+ the various Gemstash commands:
162
+
163
+ $ gemstash setup --config-file ./gemstash-config.yml
164
+ $ gemstash authorize --config-file ./gemstash-config.yml
165
+ $ gemstash start --config-file ./gemstash-config.yml
166
+ $ gemstash stop --config-file ./gemstash-config.yml
167
+ $ gemstash status --config-file ./gemstash-config.yml
168
+
169
+ When providing --config-file to gemstash setup, the provided file will
170
+ be output to with the provided configuration. 1mThis will overwrite 22many
171
+ existing configuration. If the file doesn't exist when providing
172
+ --config-file to gemstash start, gemstash stop, gemstash status, and
173
+ gemstash authorize, the default configuration will be used.
174
+
175
+ 1mERB PARSED CONFIG0m
176
+ You may also create a ~/.gemstash/config.yml.erb file. If present,
177
+ this will be used instead of ~/.gemstash/config.yml. For example, with
178
+ this you can use environment variables in the config:
1
179
 
180
+ # ~/.gemstash/config.yml.erb
181
+ ---
182
+ :db_adapter: postgres
183
+ :db_url: <%= ENV["DATABASE_URL"] %>
2
184
 
3
185
 
4
186
 
5
-
6
-
7
-
8
-
9
-
10
-
11
-
12
-
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
187
+ October 28, 2015 gemstash-customize(7)
@@ -0,0 +1,34 @@
1
+ .\" Automatically generated by Pandoc 2.5
2
+ .\"
3
+ .TH "gemstash\-debugging" "7" "October 28, 2015" "" ""
4
+ .hy
5
+ .SH DEBUGGING GEMSTASH
6
+ .PP
7
+ If you are finding Gemstash isn\[cq]t behaving as you would expect, you
8
+ might want to start by looking at the server log.
9
+ You can find the log at \f[C]server.log\f[R] within your base directory.
10
+ By default, this will be at \f[C]\[ti]/.gemstash/server.log\f[R].
11
+ .PP
12
+ You might find it easier to view the log directly in your terminal.
13
+ If you run Gemstash in non\-daemonized form (gemstash help start.1), the
14
+ log will be output directly to standard out:
15
+ .IP
16
+ .nf
17
+ \f[C]
18
+ $ gemstash start \-\-no\-daemonize
19
+ \f[R]
20
+ .fi
21
+ .PP
22
+ You can also check the status (gemstash help status.1) of the server:
23
+ .IP
24
+ .nf
25
+ \f[C]
26
+ $ gemstash status
27
+ \f[R]
28
+ .fi
29
+ .PP
30
+ The server status is checked by passing through to
31
+ pumactl (https://github.com/puma/puma#pumactl).
32
+ .PP
33
+ If you find a bug, please don\[cq]t hesitate to open a bug
34
+ report (https://github.com/rubygems/gemstash#contributing)!
@@ -1,66 +1,29 @@
1
+ gemstash-debugging(7) gemstash-debugging(7)
1
2
 
2
3
 
3
4
 
5
+ 1mDEBUGGING GEMSTASH0m
6
+ If you are finding Gemstash isn't behaving as you would expect, you
7
+ might want to start by looking at the server log. You can find the log
8
+ at server.log within your base directory. By default, this will be at
9
+ ~/.gemstash/server.log.
4
10
 
11
+ You might find it easier to view the log directly in your terminal. If
12
+ you run Gemstash in non-daemonized form (gemstash help start.1), the
13
+ log will be output directly to standard out:
5
14
 
15
+ $ gemstash start --no-daemonize
6
16
 
17
+ You can also check the status (gemstash help status.1) of the server:
7
18
 
19
+ $ gemstash status
8
20
 
21
+ The server status is checked by passing through to pumactl
22
+ (https://github.com/puma/puma#pumactl).
9
23
 
24
+ If you find a bug, please don't hesitate to open a bug report
25
+ (https://github.com/rubygems/gemstash#contributing)!
10
26
 
11
27
 
12
28
 
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
29
+ October 28, 2015 gemstash-debugging(7)
@@ -0,0 +1,72 @@
1
+ .\" Automatically generated by Pandoc 2.5
2
+ .\"
3
+ .TH "gemstash\-deploy" "7" "October 25, 2015" "" ""
4
+ .hy
5
+ .SH DEPLOYING GEMSTASH
6
+ .PP
7
+ Bundler is here for the rescue to keep Gemstash up to date! Create a
8
+ \f[C]Gemfile\f[R] pointing to Gemstash:
9
+ .IP
10
+ .nf
11
+ \f[C]
12
+ # ./Gemfile
13
+ source \[dq]https://rubygems.org\[dq]
14
+ gem \[dq]gemstash\[dq]
15
+ \f[R]
16
+ .fi
17
+ .PP
18
+ Then \f[C]bundle\f[R] to create your \f[C]Gemfile.lock\f[R].
19
+ When you are ready to upgrade, simply \f[C]bundle update\f[R].
20
+ You may need to run \f[C]gemstash\f[R] via \f[C]bundle exec\f[R].
21
+ Alternatively, you can \f[C]gem uninstall gemstash\f[R] and
22
+ \f[C]gem install gemstash\f[R] when you want to upgrade.
23
+ .PP
24
+ Gemstash will automatically run any necessary migrations, so updating
25
+ the gem is all that needs to be done.
26
+ .PP
27
+ It is probably wise to stop Gemstash before upgrading, then starting
28
+ again once you are done:
29
+ .IP
30
+ .nf
31
+ \f[C]
32
+ $ bundle exec gemstash stop
33
+ $ bundle update
34
+ $ bundle exec gemstash start
35
+ \f[R]
36
+ .fi
37
+ .SS MONITORING
38
+ .PP
39
+ Health monitoring is built in to Gemstash using the
40
+ server_health_check\-rack (https://github.com/on-site/server_health_check-rack)
41
+ gem.
42
+ If you request \f[C]/health\f[R] from your Gemstash instance, you will
43
+ get a JSON response along with an HTTP status code indicating success or
44
+ failure.
45
+ The JSON response will look something like this for a success case:
46
+ .IP
47
+ .nf
48
+ \f[C]
49
+ {
50
+ \[dq]status\[dq]: {
51
+ \[dq]heartbeat\[dq]: \[dq]OK\[dq],
52
+ \[dq]storage_read\[dq]: \[dq]OK\[dq],
53
+ \[dq]storage_write\[dq]: \[dq]OK\[dq],
54
+ \[dq]db_read\[dq]: \[dq]OK\[dq],
55
+ \[dq]db_write\[dq]: \[dq]OK\[dq]
56
+ }
57
+ }
58
+ \f[R]
59
+ .fi
60
+ .PP
61
+ This request will test storage and database access and report on the
62
+ result.
63
+ Each key in the status can be requested alone to just report on that
64
+ status.
65
+ For example, if you would like a health check that doesn\[cq]t interact
66
+ with storage or the database, you can use \f[C]/health/heartbeat\f[R]
67
+ which will always respond with a success while your Gemstash server is
68
+ running.
69
+ .SS DOWNGRADING
70
+ .PP
71
+ It is not recommended to go backwards in Gemstash versions.
72
+ Migrations may have run that could leave the database in a bad state.
@@ -1,66 +1,59 @@
1
+ gemstash-deploy(7) gemstash-deploy(7)
1
2
 
2
3
 
3
4
 
5
+ 1mDEPLOYING GEMSTASH0m
6
+ Bundler is here for the rescue to keep Gemstash up to date! Create a
7
+ Gemfile pointing to Gemstash:
4
8
 
9
+ # ./Gemfile
10
+ source "https://rubygems.org"
11
+ gem "gemstash"
5
12
 
13
+ Then bundle to create your Gemfile.lock. When you are ready to up-
14
+ grade, simply bundle update. You may need to run gemstash via bundle
15
+ exec. Alternatively, you can gem uninstall gemstash and gem install
16
+ gemstash when you want to upgrade.
6
17
 
18
+ Gemstash will automatically run any necessary migrations, so updating
19
+ the gem is all that needs to be done.
7
20
 
21
+ It is probably wise to stop Gemstash before upgrading, then starting
22
+ again once you are done:
8
23
 
24
+ $ bundle exec gemstash stop
25
+ $ bundle update
26
+ $ bundle exec gemstash start
9
27
 
28
+ 1mMONITORING0m
29
+ Health monitoring is built in to Gemstash using the serv-
30
+ er_health_check-rack (https://github.com/on-site/server_health_check-
31
+ rack) gem. If you request /health from your Gemstash instance, you
32
+ will get a JSON response along with an HTTP status code indicating suc-
33
+ cess or failure. The JSON response will look something like this for a
34
+ success case:
10
35
 
36
+ {
37
+ "status": {
38
+ "heartbeat": "OK",
39
+ "storage_read": "OK",
40
+ "storage_write": "OK",
41
+ "db_read": "OK",
42
+ "db_write": "OK"
43
+ }
44
+ }
11
45
 
46
+ This request will test storage and database access and report on the
47
+ result. Each key in the status can be requested alone to just report
48
+ on that status. For example, if you would like a health check that
49
+ doesn't interact with storage or the database, you can use
50
+ /health/heartbeat which will always respond with a success while your
51
+ Gemstash server is running.
12
52
 
53
+ 1mDOWNGRADING0m
54
+ It is not recommended to go backwards in Gemstash versions. Migrations
55
+ may have run that could leave the database in a bad state.
13
56
 
14
57
 
15
58
 
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
59
+ October 25, 2015 gemstash-deploy(7)