chef-expander 0.10.0.beta.0

Sign up to get free protection for your applications and to get access to all the features.
data/LICENSE ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
data/README.rdoc ADDED
@@ -0,0 +1,20 @@
1
+ = Chef Expander
2
+
3
+ == What's This?
4
+ Chef Expander replaces the chef-solr-indexer daemon that was included with Chef 0.8 and 0.9
5
+
6
+ == Dependencies
7
+
8
+ * bunny
9
+ * yajl
10
+ * eventmachine
11
+ * em-http-request
12
+ * amqp
13
+ * highline
14
+
15
+ == Monitoring With Nagios
16
+ A Nagios plugin to monitor queue backlog is included in scripts/ directory as <tt>check_queue_size</tt>
17
+ To run it with the warning threshold at 250 messages and critical at 500 messages:
18
+
19
+ check_queue_size -w 250 -c 500
20
+
data/bin/chef-expander ADDED
@@ -0,0 +1,30 @@
1
+ #!/usr/bin/env ruby
2
+ #
3
+ # Author:: Daniel DeLeo (<dan@opscode.com>)
4
+ # Author:: Seth Falcon (<seth@opscode.com>)
5
+ # Author:: Chris Walters (<cw@opscode.com>)
6
+ # Copyright:: Copyright (c) 2010-2011 Opscode, Inc.
7
+ # License:: Apache License, Version 2.0
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ #
21
+
22
+
23
+ require "rubygems"
24
+
25
+ $:.unshift(File.dirname(__FILE__) + '/../lib/')
26
+
27
+ require 'chef/expander'
28
+ require 'chef/expander/vnode_supervisor'
29
+
30
+ Chef::Expander::VNodeSupervisor.start
@@ -0,0 +1,29 @@
1
+ #!/usr/bin/env ruby
2
+ #
3
+ # Author:: Daniel DeLeo (<dan@opscode.com>)
4
+ # Author:: Seth Falcon (<seth@opscode.com>)
5
+ # Author:: Chris Walters (<cw@opscode.com>)
6
+ # Copyright:: Copyright (c) 2010-2011 Opscode, Inc.
7
+ # License:: Apache License, Version 2.0
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ #
21
+
22
+ require "rubygems"
23
+
24
+ $:.unshift(File.dirname(__FILE__) + '/../lib/')
25
+
26
+ require 'chef/expander'
27
+ require 'chef/expander/cluster_supervisor'
28
+
29
+ Chef::Expander::ClusterSupervisor.new.start
@@ -0,0 +1,30 @@
1
+ #!/usr/bin/env ruby
2
+ #
3
+ # Author:: Daniel DeLeo (<dan@opscode.com>)
4
+ # Author:: Seth Falcon (<seth@opscode.com>)
5
+ # Author:: Chris Walters (<cw@opscode.com>)
6
+ # Copyright:: Copyright (c) 2010-2011 Opscode, Inc.
7
+ # License:: Apache License, Version 2.0
8
+ #
9
+ # Licensed under the Apache License, Version 2.0 (the "License");
10
+ # you may not use this file except in compliance with the License.
11
+ # You may obtain a copy of the License at
12
+ #
13
+ # http://www.apache.org/licenses/LICENSE-2.0
14
+ #
15
+ # Unless required by applicable law or agreed to in writing, software
16
+ # distributed under the License is distributed on an "AS IS" BASIS,
17
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ # See the License for the specific language governing permissions and
19
+ # limitations under the License.
20
+ #
21
+
22
+
23
+ require "rubygems"
24
+
25
+ $:.unshift(File.dirname(__FILE__) + '/../lib/')
26
+
27
+ require 'chef/expander'
28
+ require 'chef/expander/control'
29
+
30
+ Chef::Expander::Control.run(ARGV)
@@ -0,0 +1,9 @@
1
+ # The URL is the only Solr config Chef Expander needs #
2
+ solr_url "http://localhost:8983"
3
+
4
+ # Parameters for connecting to RabbitMQ
5
+ amqp_host 'localhost'
6
+ amqp_port '5672'
7
+ amqp_user 'chef'
8
+ amqp_pass 'testing'
9
+ amqp_vhost '/chef'
@@ -0,0 +1,127 @@
1
+ #
2
+ # Author:: Daniel DeLeo (<dan@opscode.com>)
3
+ # Author:: Seth Falcon (<seth@opscode.com>)
4
+ # Author:: Chris Walters (<cw@opscode.com>)
5
+ # Copyright:: Copyright (c) 2010-2011 Opscode, Inc.
6
+ # License:: Apache License, Version 2.0
7
+ #
8
+ # Licensed under the Apache License, Version 2.0 (the "License");
9
+ # you may not use this file except in compliance with the License.
10
+ # You may obtain a copy of the License at
11
+ #
12
+ # http://www.apache.org/licenses/LICENSE-2.0
13
+ #
14
+ # Unless required by applicable law or agreed to in writing, software
15
+ # distributed under the License is distributed on an "AS IS" BASIS,
16
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17
+ # See the License for the specific language governing permissions and
18
+ # limitations under the License.
19
+ #
20
+
21
+ require 'chef/expander/loggable'
22
+ require 'chef/expander/daemonizable'
23
+ require 'chef/expander/version'
24
+ require 'chef/expander/configuration'
25
+ require 'chef/expander/vnode_supervisor'
26
+
27
+ module Chef
28
+ module Expander
29
+ #==ClusterSupervisor
30
+ # Manages a cluster of chef-expander processes. Usually this class will
31
+ # be instantiated from the chef-expander-cluster executable.
32
+ #
33
+ # ClusterSupervisor works by forking the desired number of processes, then
34
+ # running VNodeSupervisor.start_cluster_worker within the forked process.
35
+ # ClusterSupervisor keeps track of the process ids of its children, and will
36
+ # periodically attempt to reap them in a non-blocking call. If they are
37
+ # reaped, ClusterSupervisor knows they died and need to be respawned.
38
+ #
39
+ # The child processes are responsible for checking on the master process and
40
+ # dying if the master has died (VNodeSupervisor does this when started in
41
+ # with start_cluster_worker).
42
+ #
43
+ #===TODO:
44
+ # * This implementation currently assumes there is only one cluster, so it
45
+ # will claim all of the vnodes. It may be advantageous to allow multiple
46
+ # clusters.
47
+ # * There is no heartbeat implementation at this time, so a zombified child
48
+ # process will not be automatically killed--This behavior is left to the
49
+ # meatcloud for now.
50
+ class ClusterSupervisor
51
+ include Loggable
52
+ include Daemonizable
53
+
54
+ def initialize
55
+ @workers = {}
56
+ @running = true
57
+ @kill = :TERM
58
+ end
59
+
60
+ def start
61
+ trap(:INT) { stop(:INT) }
62
+ trap(:TERM) { stop(:TERM)}
63
+ Expander.init_config(ARGV)
64
+
65
+ log.info("Chef Expander #{Expander.version} starting cluster with #{Expander.config.node_count} nodes")
66
+ configure_process
67
+ start_workers
68
+ maintain_workers
69
+ release_locks
70
+ rescue Exception => e
71
+ raise if SystemExit === e
72
+
73
+ log.fatal {e}
74
+ exit(1)
75
+ end
76
+
77
+ def start_workers
78
+ Expander.config.node_count.times do |i|
79
+ start_worker(i + 1)
80
+ end
81
+ end
82
+
83
+ def start_worker(index)
84
+ log.info { "Starting cluster worker #{index}" }
85
+ worker_params = {:index => index}
86
+ child_pid = fork do
87
+ Expander.config.index = index
88
+ VNodeSupervisor.start_cluster_worker
89
+ end
90
+ @workers[child_pid] = worker_params
91
+ end
92
+
93
+ def stop(signal)
94
+ log.info { "Stopping cluster on signal (#{signal})" }
95
+ @running = false
96
+ @kill = signal
97
+ end
98
+
99
+ def maintain_workers
100
+ while @running
101
+ sleep 1
102
+ workers_to_replace = {}
103
+ @workers.each do |process_id, worker_params|
104
+ if result = Process.waitpid2(process_id, Process::WNOHANG)
105
+ log.error { "worker #{worker_params[:index]} (PID: #{process_id}) died with status #{result[1].exitstatus || '(no status)'}"}
106
+ workers_to_replace[process_id] = worker_params
107
+ end
108
+ end
109
+ workers_to_replace.each do |dead_pid, worker_params|
110
+ @workers.delete(dead_pid)
111
+ start_worker(worker_params[:index])
112
+ end
113
+ end
114
+
115
+ @workers.each do |pid, worker_params|
116
+ log.info { "Stopping worker #{worker_params[:index]} (PID: #{pid})"}
117
+ Process.kill(@kill, pid)
118
+ end
119
+ @workers.each do |pid, worker_params|
120
+ Process.waitpid2(pid)
121
+ end
122
+
123
+ end
124
+
125
+ end
126
+ end
127
+ end