gemstash 2.0.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (75) hide show
  1. checksums.yaml +4 -4
  2. data/exe/gemstash +3 -0
  3. data/lib/gemstash.rb +2 -0
  4. data/lib/gemstash/api_key_authorization.rb +3 -0
  5. data/lib/gemstash/authorization.rb +7 -3
  6. data/lib/gemstash/cache.rb +5 -2
  7. data/lib/gemstash/cli.rb +2 -0
  8. data/lib/gemstash/cli/authorize.rb +3 -3
  9. data/lib/gemstash/cli/base.rb +4 -1
  10. data/lib/gemstash/cli/setup.rb +8 -1
  11. data/lib/gemstash/cli/start.rb +4 -1
  12. data/lib/gemstash/cli/status.rb +2 -0
  13. data/lib/gemstash/cli/stop.rb +2 -0
  14. data/lib/gemstash/config.ru +3 -3
  15. data/lib/gemstash/configuration.rb +7 -1
  16. data/lib/gemstash/db.rb +3 -0
  17. data/lib/gemstash/db/authorization.rb +2 -0
  18. data/lib/gemstash/db/cached_rubygem.rb +3 -0
  19. data/lib/gemstash/db/dependency.rb +2 -0
  20. data/lib/gemstash/db/rubygem.rb +3 -0
  21. data/lib/gemstash/db/upstream.rb +3 -0
  22. data/lib/gemstash/db/version.rb +3 -0
  23. data/lib/gemstash/dependencies.rb +4 -0
  24. data/lib/gemstash/env.rb +22 -4
  25. data/lib/gemstash/gem_fetcher.rb +2 -0
  26. data/lib/gemstash/gem_pusher.rb +5 -3
  27. data/lib/gemstash/gem_source.rb +2 -0
  28. data/lib/gemstash/gem_source/dependency_caching.rb +4 -4
  29. data/lib/gemstash/gem_source/private_source.rb +3 -0
  30. data/lib/gemstash/gem_source/rack_middleware.rb +3 -0
  31. data/lib/gemstash/gem_source/upstream_source.rb +9 -3
  32. data/lib/gemstash/gem_yanker.rb +4 -0
  33. data/lib/gemstash/health.rb +2 -0
  34. data/lib/gemstash/http_client.rb +4 -1
  35. data/lib/gemstash/logging.rb +2 -0
  36. data/lib/gemstash/man/gemstash-authorize.1 +51 -0
  37. data/lib/gemstash/man/gemstash-authorize.1.txt +33 -53
  38. data/lib/gemstash/man/gemstash-configuration.5 +229 -0
  39. data/lib/gemstash/man/gemstash-configuration.5.txt +150 -6
  40. data/lib/gemstash/man/gemstash-customize.7 +301 -0
  41. data/lib/gemstash/man/gemstash-customize.7.txt +183 -62
  42. data/lib/gemstash/man/gemstash-debugging.7 +34 -0
  43. data/lib/gemstash/man/gemstash-debugging.7.txt +17 -54
  44. data/lib/gemstash/man/gemstash-deploy.7 +72 -0
  45. data/lib/gemstash/man/gemstash-deploy.7.txt +44 -51
  46. data/lib/gemstash/man/gemstash-mirror.7 +40 -0
  47. data/lib/gemstash/man/gemstash-mirror.7.txt +20 -53
  48. data/lib/gemstash/man/gemstash-multiple-sources.7 +89 -0
  49. data/lib/gemstash/man/gemstash-multiple-sources.7.txt +53 -48
  50. data/lib/gemstash/man/gemstash-private-gems.7 +227 -0
  51. data/lib/gemstash/man/gemstash-private-gems.7.txt +108 -18
  52. data/lib/gemstash/man/gemstash-readme.7 +233 -0
  53. data/lib/gemstash/man/gemstash-readme.7.txt +126 -10
  54. data/lib/gemstash/man/gemstash-setup.1 +43 -0
  55. data/lib/gemstash/man/gemstash-setup.1.txt +28 -54
  56. data/lib/gemstash/man/gemstash-start.1 +26 -0
  57. data/lib/gemstash/man/gemstash-start.1.txt +16 -56
  58. data/lib/gemstash/man/gemstash-status.1 +20 -0
  59. data/lib/gemstash/man/gemstash-status.1.txt +13 -57
  60. data/lib/gemstash/man/gemstash-stop.1 +20 -0
  61. data/lib/gemstash/man/gemstash-stop.1.txt +13 -57
  62. data/lib/gemstash/man/gemstash-version.1 +22 -0
  63. data/lib/gemstash/man/gemstash-version.1.txt +12 -57
  64. data/lib/gemstash/migrations/01_gem_dependencies.rb +2 -0
  65. data/lib/gemstash/migrations/02_authorizations.rb +2 -0
  66. data/lib/gemstash/migrations/03_cached_gems.rb +2 -0
  67. data/lib/gemstash/migrations/04_health_tests.rb +2 -0
  68. data/lib/gemstash/puma.rb +2 -0
  69. data/lib/gemstash/rack_env_rewriter.rb +11 -2
  70. data/lib/gemstash/specs_builder.rb +5 -1
  71. data/lib/gemstash/storage.rb +12 -3
  72. data/lib/gemstash/upstream.rb +2 -0
  73. data/lib/gemstash/version.rb +3 -1
  74. data/lib/gemstash/web.rb +2 -0
  75. metadata +69 -22
@@ -1,66 +1,187 @@
1
+ gemstash-customize(7) gemstash-customize(7)
2
+
3
+
4
+
5
+ 1mCUSTOMIZING THE SERVER0m
6
+ Although Gemstash is designed for as minimal setup as possible, there
7
+ may be times you will want to change some of the default configuration.
8
+ By the end of this guide, you will be able to customize some of the
9
+ Gemstash behavior, including where files are stored, what database Gem-
10
+ stash uses, and how Gemstash caches certain requests.
11
+
12
+ 1mSETUP0m
13
+ Several customizable options are available via an interactive Gemstash
14
+ command. Run gemstash setup and answer the questions it provides (a
15
+ blank answer will use the default value):
16
+
17
+ $ gemstash setup
18
+ Where should files go? [~/.gemstash]
19
+ Cache with what? [MEMORY, memcached] 1mmemcached0m
20
+ What is the comma separated Memcached servers? [local-
21
+ host:11211]
22
+ What database adapter? [SQLITE3, postgres, mysql, mysql2] 1mpost-0m
23
+ 1mgres0m
24
+ Where is the database? [postgres:///gemstash]
25
+ Checking that the cache is available
26
+ Checking that the database is available
27
+ The database is not available
28
+ Once you've answered the questions, some checks will be made to ensure
29
+ the configuration will work. For example, the database didn't exist in
30
+ the previous example, so the command failed and the configuration
31
+ wasn't saved. If the command passes, you may provide the --redo option
32
+ to force configuration to be redone:
33
+
34
+ $ gemstash setup --redo
35
+ Where should files go? [~/.gemstash]
36
+ Cache with what? [MEMORY, memcached] 1mmemcached0m
37
+ What is the comma separated Memcached servers? [local-
38
+ host:11211]
39
+ What database adapter? [SQLITE3, postgres, mysql, mysql2]
40
+ Checking that the cache is available
41
+ Checking that the database is available
42
+ You are all setup!
43
+ Once all checks have passed, Gemstash will store your answers in the
44
+ configuration file located at ~/.gemstash/config.yml.
45
+
46
+ 1mFILES0m
47
+ Storage in Gemstash defaults to ~/.gemstash unless otherwise specified.
48
+ You can change this in your config file via the :base_path key:
49
+
50
+ # ~/.gemstash/config.yml
51
+ ---
52
+ :base_path: "/var/gemstash"
53
+
54
+ When customizing the base_path, the directory must exist, otherwise
55
+ Gemstash will fail to run. Thus, if you want to use /var/gemstash like
56
+ in the previous example, make sure to mkdir /var/gemstash and grant ac-
57
+ cess to the directory for the user you run Gemstash with.
58
+
59
+ 1mDATABASE0m
60
+ The :db_adapter configuration key specifies what database you will be
61
+ using. The default :db_adapter is sqlite3 (https://www.sqlite.org/),
62
+ which will use a database file located within your :base_path. The
63
+ database file will always be named gemstash.db.
64
+
65
+ You may also use postgres (http://www.postgresql.org/), mysql
66
+ (http://www.mysql.com/), or mysql2 (http://sequel.jeremye-
67
+ vans.net/rdoc/files/doc/opening_databases_rdoc.html#label-mysql2) for
68
+ your :db_adapter. When using any of these options, you need to specify
69
+ the :db_url to point to an existing database. Here is an example con-
70
+ figuration to use the postgres adapter:
71
+
72
+ # ~/.gemstash/config.yml
73
+ ---
74
+ :db_adapter: postgres
75
+ :db_url: postgres:///gemstash
76
+ :db_connection_options: # Sequel.connect options
77
+ :connect_timeout: 10
78
+ :read_timeout: 5
79
+ :timeout: 30
80
+
81
+ Regardless of the adapter you choose, the database will automatically
82
+ migrate to your version of Gemstash whenever the database is needed.
83
+ You only need to ensure the database exists and Gemstash will do the
84
+ rest, except for sqlite3 (for which Gemstash will also create the data-
85
+ base for you).
86
+
87
+ 1mCACHE0m
88
+ Certain things (like dependencies) are cached in memory. This avoids
89
+ web calls to the gem source, and database calls for private gems.
90
+
91
+ # ~/.gemstash/config.yml
92
+ ---
93
+ :cache_type: memory
94
+ :cache_max_size: 2000
95
+
96
+ This configuration uses the default memory cache, and has increased the
97
+ cache_max_size setting from its default of 500 items.
98
+
99
+ The memory cache can optionally be swapped out with a Memcached
100
+ (http://memcached.org/) server (or cluster of servers).
101
+
102
+ To use Memcached, use the memcached :cache_type configuration.
103
+
104
+ Provide the servers as a comma-separated list to the :memcached_servers
105
+ configuration key:
106
+
107
+ # ~/.gemstash/config.yml
108
+ ---
109
+ :cache_type: memcached
110
+ :memcached_servers: memcached1.local:11211,memcached2.local:11211
111
+ :cache_expiration: 1800
112
+
113
+ All caching expires in cache_expiration number of seconds. Default is
114
+ 1800 seconds (30 minutes). This option applies to all caching.
115
+
116
+ 1mSERVER0m
117
+ Gemstash uses Puma (http://puma.io/) and Rack (http://rack.github.io/)
118
+ as the server. Alternate server configurations are not currently sup-
119
+ ported, but you can take a look at the Puma configuration
120
+ (https://github.com/rubygems/gemstash/blob/master/lib/gemstash/puma.rb)
121
+ and the rackup file (https://github.com/rubygems/gemstash/blob/mas-
122
+ ter/lib/gemstash/config.ru) for inspiration.
123
+
124
+ While the server is not customizable, the way Gemstash binds the port
125
+ can be changed. To change the binding, update the :bind configuration
126
+ key:
127
+
128
+ # ~/.gemstash/config.yml
129
+ ---
130
+ :bind: tcp://0.0.0.0:4242
131
+
132
+ This maps directly to the Puma bind flag
133
+ (https://github.com/puma/puma#binding-tcp--sockets), and will support
134
+ anything valid for that flag.
135
+
136
+ The number of threads Puma uses is also customizable via the
137
+ :puma_threads configuration key. The default is 16.
138
+
139
+ 1mPROTECTED FETCH0m
140
+ Gemstash by default allows unauthenticated access for private gems.
141
+ Authenticated access is available via the :protected_fetch configura-
142
+ tion key.
143
+
144
+ # ~/.gemstash/config.yml
145
+ ---
146
+ :protected_fetch: true
147
+
148
+ More details on protected_fetch are here (gemstash help private-
149
+ gems.7).
150
+
151
+ 1mFETCH TIMEOUT0m
152
+ The default fetch timeout is 20 seconds. Use the :fetch_timeout con-
153
+ figuration key to change it.
154
+
155
+ ---
156
+ :fetch_timeout: 20
157
+
158
+ 1mCONFIG FILE LOCATION0m
159
+ By default, configuration for Gemstash will be at ~/.gemstash/con-
160
+ fig.yml. This can be changed by providing the --config-file option to
161
+ the various Gemstash commands:
162
+
163
+ $ gemstash setup --config-file ./gemstash-config.yml
164
+ $ gemstash authorize --config-file ./gemstash-config.yml
165
+ $ gemstash start --config-file ./gemstash-config.yml
166
+ $ gemstash stop --config-file ./gemstash-config.yml
167
+ $ gemstash status --config-file ./gemstash-config.yml
168
+
169
+ When providing --config-file to gemstash setup, the provided file will
170
+ be output to with the provided configuration. 1mThis will overwrite 22many
171
+ existing configuration. If the file doesn't exist when providing
172
+ --config-file to gemstash start, gemstash stop, gemstash status, and
173
+ gemstash authorize, the default configuration will be used.
174
+
175
+ 1mERB PARSED CONFIG0m
176
+ You may also create a ~/.gemstash/config.yml.erb file. If present,
177
+ this will be used instead of ~/.gemstash/config.yml. For example, with
178
+ this you can use environment variables in the config:
1
179
 
180
+ # ~/.gemstash/config.yml.erb
181
+ ---
182
+ :db_adapter: postgres
183
+ :db_url: <%= ENV["DATABASE_URL"] %>
2
184
 
3
185
 
4
186
 
5
-
6
-
7
-
8
-
9
-
10
-
11
-
12
-
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
187
+ October 28, 2015 gemstash-customize(7)
@@ -0,0 +1,34 @@
1
+ .\" Automatically generated by Pandoc 2.5
2
+ .\"
3
+ .TH "gemstash\-debugging" "7" "October 28, 2015" "" ""
4
+ .hy
5
+ .SH DEBUGGING GEMSTASH
6
+ .PP
7
+ If you are finding Gemstash isn\[cq]t behaving as you would expect, you
8
+ might want to start by looking at the server log.
9
+ You can find the log at \f[C]server.log\f[R] within your base directory.
10
+ By default, this will be at \f[C]\[ti]/.gemstash/server.log\f[R].
11
+ .PP
12
+ You might find it easier to view the log directly in your terminal.
13
+ If you run Gemstash in non\-daemonized form (gemstash help start.1), the
14
+ log will be output directly to standard out:
15
+ .IP
16
+ .nf
17
+ \f[C]
18
+ $ gemstash start \-\-no\-daemonize
19
+ \f[R]
20
+ .fi
21
+ .PP
22
+ You can also check the status (gemstash help status.1) of the server:
23
+ .IP
24
+ .nf
25
+ \f[C]
26
+ $ gemstash status
27
+ \f[R]
28
+ .fi
29
+ .PP
30
+ The server status is checked by passing through to
31
+ pumactl (https://github.com/puma/puma#pumactl).
32
+ .PP
33
+ If you find a bug, please don\[cq]t hesitate to open a bug
34
+ report (https://github.com/rubygems/gemstash#contributing)!
@@ -1,66 +1,29 @@
1
+ gemstash-debugging(7) gemstash-debugging(7)
1
2
 
2
3
 
3
4
 
5
+ 1mDEBUGGING GEMSTASH0m
6
+ If you are finding Gemstash isn't behaving as you would expect, you
7
+ might want to start by looking at the server log. You can find the log
8
+ at server.log within your base directory. By default, this will be at
9
+ ~/.gemstash/server.log.
4
10
 
11
+ You might find it easier to view the log directly in your terminal. If
12
+ you run Gemstash in non-daemonized form (gemstash help start.1), the
13
+ log will be output directly to standard out:
5
14
 
15
+ $ gemstash start --no-daemonize
6
16
 
17
+ You can also check the status (gemstash help status.1) of the server:
7
18
 
19
+ $ gemstash status
8
20
 
21
+ The server status is checked by passing through to pumactl
22
+ (https://github.com/puma/puma#pumactl).
9
23
 
24
+ If you find a bug, please don't hesitate to open a bug report
25
+ (https://github.com/rubygems/gemstash#contributing)!
10
26
 
11
27
 
12
28
 
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
29
+ October 28, 2015 gemstash-debugging(7)
@@ -0,0 +1,72 @@
1
+ .\" Automatically generated by Pandoc 2.5
2
+ .\"
3
+ .TH "gemstash\-deploy" "7" "October 25, 2015" "" ""
4
+ .hy
5
+ .SH DEPLOYING GEMSTASH
6
+ .PP
7
+ Bundler is here for the rescue to keep Gemstash up to date! Create a
8
+ \f[C]Gemfile\f[R] pointing to Gemstash:
9
+ .IP
10
+ .nf
11
+ \f[C]
12
+ # ./Gemfile
13
+ source \[dq]https://rubygems.org\[dq]
14
+ gem \[dq]gemstash\[dq]
15
+ \f[R]
16
+ .fi
17
+ .PP
18
+ Then \f[C]bundle\f[R] to create your \f[C]Gemfile.lock\f[R].
19
+ When you are ready to upgrade, simply \f[C]bundle update\f[R].
20
+ You may need to run \f[C]gemstash\f[R] via \f[C]bundle exec\f[R].
21
+ Alternatively, you can \f[C]gem uninstall gemstash\f[R] and
22
+ \f[C]gem install gemstash\f[R] when you want to upgrade.
23
+ .PP
24
+ Gemstash will automatically run any necessary migrations, so updating
25
+ the gem is all that needs to be done.
26
+ .PP
27
+ It is probably wise to stop Gemstash before upgrading, then starting
28
+ again once you are done:
29
+ .IP
30
+ .nf
31
+ \f[C]
32
+ $ bundle exec gemstash stop
33
+ $ bundle update
34
+ $ bundle exec gemstash start
35
+ \f[R]
36
+ .fi
37
+ .SS MONITORING
38
+ .PP
39
+ Health monitoring is built in to Gemstash using the
40
+ server_health_check\-rack (https://github.com/on-site/server_health_check-rack)
41
+ gem.
42
+ If you request \f[C]/health\f[R] from your Gemstash instance, you will
43
+ get a JSON response along with an HTTP status code indicating success or
44
+ failure.
45
+ The JSON response will look something like this for a success case:
46
+ .IP
47
+ .nf
48
+ \f[C]
49
+ {
50
+ \[dq]status\[dq]: {
51
+ \[dq]heartbeat\[dq]: \[dq]OK\[dq],
52
+ \[dq]storage_read\[dq]: \[dq]OK\[dq],
53
+ \[dq]storage_write\[dq]: \[dq]OK\[dq],
54
+ \[dq]db_read\[dq]: \[dq]OK\[dq],
55
+ \[dq]db_write\[dq]: \[dq]OK\[dq]
56
+ }
57
+ }
58
+ \f[R]
59
+ .fi
60
+ .PP
61
+ This request will test storage and database access and report on the
62
+ result.
63
+ Each key in the status can be requested alone to just report on that
64
+ status.
65
+ For example, if you would like a health check that doesn\[cq]t interact
66
+ with storage or the database, you can use \f[C]/health/heartbeat\f[R]
67
+ which will always respond with a success while your Gemstash server is
68
+ running.
69
+ .SS DOWNGRADING
70
+ .PP
71
+ It is not recommended to go backwards in Gemstash versions.
72
+ Migrations may have run that could leave the database in a bad state.
@@ -1,66 +1,59 @@
1
+ gemstash-deploy(7) gemstash-deploy(7)
1
2
 
2
3
 
3
4
 
5
+ 1mDEPLOYING GEMSTASH0m
6
+ Bundler is here for the rescue to keep Gemstash up to date! Create a
7
+ Gemfile pointing to Gemstash:
4
8
 
9
+ # ./Gemfile
10
+ source "https://rubygems.org"
11
+ gem "gemstash"
5
12
 
13
+ Then bundle to create your Gemfile.lock. When you are ready to up-
14
+ grade, simply bundle update. You may need to run gemstash via bundle
15
+ exec. Alternatively, you can gem uninstall gemstash and gem install
16
+ gemstash when you want to upgrade.
6
17
 
18
+ Gemstash will automatically run any necessary migrations, so updating
19
+ the gem is all that needs to be done.
7
20
 
21
+ It is probably wise to stop Gemstash before upgrading, then starting
22
+ again once you are done:
8
23
 
24
+ $ bundle exec gemstash stop
25
+ $ bundle update
26
+ $ bundle exec gemstash start
9
27
 
28
+ 1mMONITORING0m
29
+ Health monitoring is built in to Gemstash using the serv-
30
+ er_health_check-rack (https://github.com/on-site/server_health_check-
31
+ rack) gem. If you request /health from your Gemstash instance, you
32
+ will get a JSON response along with an HTTP status code indicating suc-
33
+ cess or failure. The JSON response will look something like this for a
34
+ success case:
10
35
 
36
+ {
37
+ "status": {
38
+ "heartbeat": "OK",
39
+ "storage_read": "OK",
40
+ "storage_write": "OK",
41
+ "db_read": "OK",
42
+ "db_write": "OK"
43
+ }
44
+ }
11
45
 
46
+ This request will test storage and database access and report on the
47
+ result. Each key in the status can be requested alone to just report
48
+ on that status. For example, if you would like a health check that
49
+ doesn't interact with storage or the database, you can use
50
+ /health/heartbeat which will always respond with a success while your
51
+ Gemstash server is running.
12
52
 
53
+ 1mDOWNGRADING0m
54
+ It is not recommended to go backwards in Gemstash versions. Migrations
55
+ may have run that could leave the database in a bad state.
13
56
 
14
57
 
15
58
 
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
59
+ October 25, 2015 gemstash-deploy(7)