berkshelf-api 1.0.0 → 1.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: f238c1aec4a689f0461aaea5792d0588501e4d8c
4
- data.tar.gz: f99f8dac7ac5aa4133ac4bef0420e74d128cd01f
3
+ metadata.gz: 00fbbdefda724c9caf02b454baeddf35e2da8744
4
+ data.tar.gz: b71657d7e5e315e63c82d543d93eaefae3f6348c
5
5
  SHA512:
6
- metadata.gz: 0910bcf676f10ea3f0c7061a685400ee232350b39fe08866ed89558e6d3dc649cb6a6b0288041a59924c010825a23510fada8adc043466a37b56374b74ad1543
7
- data.tar.gz: de9395b533f039629f2eef14bd583de33839afac67e225844616c5cdc22a6d3038b3852839e2f110b4f19e2e227ec946284d58bfc0f9dfd04792a645b966d1aa
6
+ metadata.gz: c871060bd3057105ddcd5266c29ad8947eb1e42c5b2c91048076da25c564c1bdeefb46270adadec74f0f8d3429974a4d90e89b62bbfe8752735e0bfcf40e6af8
7
+ data.tar.gz: 905361134b015f6c4d8ccd4cbbbdb7d767bd79866ed3a8bcc72cd68bb65e6321332616ce7d93383af1ebf5a84a5baf53a7abef56604e8eeb79acaffa3c7971bc
data/.ruby-version CHANGED
@@ -1 +1 @@
1
- 2.0.0-p247
1
+ 2.0.0-p353
data/.travis.yml CHANGED
@@ -2,7 +2,6 @@ language: ruby
2
2
  rvm:
3
3
  - 1.9.3
4
4
  - 2.0.0
5
- - jruby-19mode
6
5
  notifications:
7
6
  irc:
8
7
  channels:
data/CHANGELOG.md CHANGED
@@ -1,3 +1,12 @@
1
+ # 1.1.0
2
+
3
+ * Enhancements
4
+ * Added Github cache builder and Github endpoint. You can now add Github organization's as endpoints and the cache builder will crawl each repository looking for repositories which contain SemVer tags and raw cookbook metadata.
5
+
6
+ * Bug Fixes
7
+ * Fix cache builder crashes when builder encounters a cookbook with invalid or missing metadata.
8
+ * Invalid cookbooks won't be added to cache.
9
+
1
10
  # 1.0.0
2
11
 
3
12
  * Prioritize Cache Builders so their entries do not clobber one another
data/CONTRIBUTING.md CHANGED
@@ -23,7 +23,7 @@ Install the latest version of [Bundler](http://gembundler.com)
23
23
 
24
24
  Clone the project
25
25
 
26
- $ git clone git://github.com/RiotGames/berkshelf-api.git
26
+ $ git clone git://github.com/berkshelf/berkshelf-api.git
27
27
 
28
28
  and run:
29
29
 
data/README.md CHANGED
@@ -1,8 +1,6 @@
1
1
  # Berkshelf::API
2
2
  [![Gem Version](https://badge.fury.io/rb/berkshelf-api.png)](http://badge.fury.io/rb/berkshelf-api)
3
- [![Build Status](https://secure.travis-ci.org/RiotGames/berkshelf-api.png?branch=master)](http://travis-ci.org/RiotGames/berkshelf-api)
4
- [![Dependency Status](https://gemnasium.com/RiotGames/berkshelf-api.png?travis)](https://gemnasium.com/RiotGames/berkshelf-api)
5
- [![Code Climate](https://codeclimate.com/github/RiotGames/berkshelf-api.png)](https://codeclimate.com/github/RiotGames/berkshelf-api)
3
+ [![Build Status](https://secure.travis-ci.org/berkshelf/berkshelf-api.png?branch=master)](http://travis-ci.org/berkshelf/berkshelf-api)
6
4
 
7
5
  A server which indexes cookbooks from various sources and hosts it over a REST API
8
6
 
@@ -58,6 +56,25 @@ You may configure the endpoints to index by editing the JSON configuration file
58
56
  }
59
57
  ```
60
58
 
59
+ ### Github Organization
60
+
61
+ Github limits the rate of requests to their API if not authenticated. For this reason the access_token option
62
+ is required.
63
+
64
+ ```json
65
+ {
66
+ "endpoints": [
67
+ {
68
+ "type": "github",
69
+ "options": {
70
+ "organization": "opscode-cookbooks",
71
+ "access_token": ""
72
+ }
73
+ }
74
+ ]
75
+ }
76
+ ```
77
+
61
78
  ## Contributing
62
79
 
63
80
  1. Fork it
@@ -71,4 +88,4 @@ You may configure the endpoints to index by editing the JSON configuration file
71
88
  * Jamie Winsor (<jamie@vialstudios.com>)
72
89
  * Andrew Garson (<agarson@riotgames.com>)
73
90
 
74
- Thank you to all of our [Contributors](https://github.com/RiotGames/berkshelf-api/graphs/contributors), testers, and users.
91
+ Thank you to all of our [Contributors](https://github.com/berkshelf/berkshelf-api/graphs/contributors), testers, and users.
@@ -10,7 +10,7 @@ Gem::Specification.new do |spec|
10
10
  spec.email = ["jamie@vialstudios.com", "agarson@riotgames"]
11
11
  spec.description = %q{Berkshelf dependency API server}
12
12
  spec.summary = %q{A server which indexes cookbooks from various sources and hosts it over a REST API}
13
- spec.homepage = "https://github.com/RiotGames/berkshelf-api"
13
+ spec.homepage = "https://github.com/berkshelf/berkshelf-api"
14
14
  spec.license = "Apache 2.0"
15
15
 
16
16
  spec.files = `git ls-files`.split($/)
@@ -19,7 +19,7 @@ Gem::Specification.new do |spec|
19
19
  spec.require_paths = ["lib"]
20
20
  spec.required_ruby_version = ">= 1.9.3"
21
21
 
22
- spec.add_dependency 'ridley', '~> 1.6'
22
+ spec.add_dependency 'ridley', '~> 2.0'
23
23
  spec.add_dependency 'celluloid', '~> 0.15'
24
24
  spec.add_dependency 'reel', '>= 0.4.0'
25
25
  spec.add_dependency 'grape', '~> 0.5.0'
@@ -28,6 +28,8 @@ Gem::Specification.new do |spec|
28
28
  spec.add_dependency 'retryable', '~> 1.3.3'
29
29
  spec.add_dependency 'archive', '= 0.0.2'
30
30
  spec.add_dependency 'buff-config', '~> 0.1'
31
+ spec.add_dependency 'octokit', '~> 2.6'
32
+ spec.add_dependency 'solve'
31
33
 
32
34
  spec.add_development_dependency 'bundler', '~> 1.3'
33
35
  spec.add_development_dependency 'thor', '~> 0.18.0'
@@ -38,7 +38,6 @@ module Berkshelf::API
38
38
  def cookbooks
39
39
  raise RuntimeError, "must be implemented"
40
40
  end
41
-
42
41
  end
43
42
 
44
43
  class << self
@@ -6,6 +6,14 @@ module Berkshelf::API
6
6
 
7
7
  finalizer :finalize_callback
8
8
 
9
+ # @option options [String] :url
10
+ # the URL of the target Chef Server
11
+ # @option options [String] :client_name
12
+ # the name of the client for authenticating to the Chef Server
13
+ # @option options [String] :client_key
14
+ # a client key for authenticating to the Chef Server
15
+ # @option options [Boolean] :ssl_verify
16
+ # turn ssl verification off if you have an unsigned SSL certificate
9
17
  def initialize(options = {})
10
18
  @connection = Ridley::Client.new_link(server_url: options[:url], client_key: options[:client_key],
11
19
  client_name: options[:client_name], ssl: { verify: options[:ssl_verify] })
@@ -0,0 +1,102 @@
1
+ require 'octokit'
2
+ require 'solve'
3
+ require 'net/http'
4
+
5
+ module Berkshelf::API
6
+ class CacheBuilder
7
+ module Worker
8
+ class Github < Worker::Base
9
+ worker_type "github"
10
+
11
+ # @return [String]
12
+ attr_reader :organization
13
+
14
+ # @option options [String] :organization
15
+ # the organization to crawl for cookbooks
16
+ # @option options [String] :access_token
17
+ # authentication token for accessing the Github organization. This is necessary
18
+ # since Github throttles unauthenticated API requests
19
+ def initialize(options = {})
20
+ @connection = Octokit::Client.new(access_token: options[:access_token])
21
+ @organization = options[:organization]
22
+ super(options)
23
+ end
24
+
25
+ # @return [Array<RemoteCookbook>]
26
+ # The list of cookbooks this builder can find
27
+ def cookbooks
28
+ [].tap do |cookbook_versions|
29
+ connection.organization_repositories(organization).each do |repo|
30
+ connection.tags(repo.full_name).each do |tag|
31
+ if match = /^v(?<version>.*)$/.match(tag.name)
32
+ begin
33
+ next unless cookbook_metadata = load_metadata(repo.name, tag.name)
34
+
35
+ if cookbook_metadata.version.to_s == match[:version].to_s
36
+ cookbook_versions << RemoteCookbook.new(repo.name, cookbook_metadata.version,
37
+ self.class.worker_type, repo.full_name, priority)
38
+ else
39
+ log.warn "Version found in metadata for #{repo.name} (#{tag.name}) does not " +
40
+ "match the tag. Got #{cookbook_metadata.version}."
41
+ end
42
+ rescue Solve::Errors::InvalidVersionFormat
43
+ log.debug "Ignoring tag #{tag.name} for: #{repo.name}. Does not conform to semver."
44
+ rescue Octokit::NotFound
45
+ log.debug "Ignoring tag #{tag.name} for: #{repo.name}. No raw metadata found."
46
+ end
47
+ else
48
+ log.debug "Version number cannot be parsed"
49
+ end
50
+ end
51
+ end
52
+ end
53
+ end
54
+
55
+ # Return the metadata of the given RemoteCookbook. If the metadata could not be found or parsed
56
+ # nil is returned.
57
+ #
58
+ # @param [RemoteCookbook] remote
59
+ #
60
+ # @return [Ridley::Chef::Cookbook::Metadata, nil]
61
+ def metadata(remote)
62
+ load_metadata(remote.name, "v#{remote.version}")
63
+ end
64
+
65
+ private
66
+
67
+ attr_reader :connection
68
+
69
+ # Helper function for loading metadata from a particular ref in a Github repository
70
+ #
71
+ # @param [String] repo
72
+ # name of repository to load from
73
+ # @param [String] ref
74
+ # reference, tag, or branch to load from
75
+ #
76
+ # @return [Ridley::Chef::Cookbook::Metadata, nil]
77
+ def load_metadata(repo, ref)
78
+ metadata_content = read_content(repo, ref, Ridley::Chef::Cookbook::Metadata::RAW_FILE_NAME)
79
+ cookbook_metadata = Ridley::Chef::Cookbook::Metadata.new
80
+ cookbook_metadata.instance_eval(metadata_content)
81
+ cookbook_metadata
82
+ rescue => ex
83
+ nil
84
+ end
85
+
86
+ # Read content from a file from a particular ref in a Github repository
87
+ #
88
+ # @param [String] repo
89
+ # name of the repository to load from
90
+ # @param [String] ref
91
+ # reference, tag, or branch to load from
92
+ # @param [String] file
93
+ # file to read content from
94
+ #
95
+ # @return [String]
96
+ def read_content(repo, ref, file)
97
+ Base64.decode64(connection.contents("#{organization}/#{repo}", path: file, ref: ref).content)
98
+ end
99
+ end
100
+ end
101
+ end
102
+ end
@@ -6,6 +6,8 @@ module Berkshelf::API
6
6
 
7
7
  finalizer :finalize_callback
8
8
 
9
+ # @param [Hash] options
10
+ # see {API::SiteConnector::Opscode.new} for options
9
11
  def initialize(options = {})
10
12
  @connection = Berkshelf::API::SiteConnector::Opscode.pool_link(size: 25, args: [ options ])
11
13
  super
@@ -25,13 +27,17 @@ module Berkshelf::API
25
27
  end
26
28
  end
27
29
 
30
+ # Return the metadata of the given RemoteCookbook. If the metadata could not be found or parsed
31
+ # nil is returned.
32
+ #
28
33
  # @param [RemoteCookbook] remote
29
34
  #
30
- # @return [Ridley::Chef::Cookbook::Metadata]
35
+ # @return [Ridley::Chef::Cookbook::Metadata, nil]
31
36
  def metadata(remote)
32
- Dir.mktmpdir do |destination|
33
- connection.download(remote.name, remote.version, destination)
34
- load_metadata(destination, remote.name)
37
+ Dir.mktmpdir('metadata') do |destination|
38
+ if connection.download(remote.name, remote.version, destination)
39
+ load_metadata(destination, remote.name)
40
+ end
35
41
  end
36
42
  end
37
43
 
@@ -46,7 +52,9 @@ module Berkshelf::API
46
52
  def load_metadata(directory, cookbook)
47
53
  # The community site does not enforce the name of the cookbook contained in the archive
48
54
  # downloaded and extracted. This will just find the first metadata.json and load it.
49
- file = Dir["#{directory}/**/*/metadata.json"].first
55
+ file = Dir["#{directory}/**/*/metadata.json"].first
56
+ return nil if file.nil?
57
+
50
58
  metadata = File.read(file)
51
59
  Ridley::Chef::Cookbook::Metadata.from_json(metadata)
52
60
  rescue JSON::ParserError => ex
@@ -97,9 +97,10 @@ module Berkshelf::API
97
97
  work.map! do |remote|
98
98
  [ remote, worker.future(:metadata, remote) ]
99
99
  end.map! do |remote, metadata|
100
- [remote, metadata.value]
100
+ metadata.value ? [remote, metadata.value] : nil
101
101
  end
102
- created_cookbooks_with_metadata += work
102
+
103
+ created_cookbooks_with_metadata += work.compact
103
104
  end
104
105
 
105
106
  log.info "about to merge cookbooks"
@@ -36,9 +36,12 @@ module Berkshelf::API
36
36
  # time to wait between retries
37
37
  attr_reader :retry_interval
38
38
 
39
- # @param [Faraday::Connection] connection
40
- # Optional parameter for setting the connection object
41
- # This should only be set manually for testing
39
+ # @option options [String] :url ({V1_API})
40
+ # url of community site
41
+ # @option options [Integer] :retries (5)
42
+ # how many download retry attempts to make
43
+ # @option options [Float] :retry_interval (0.5)
44
+ # how long to wait before retrying again
42
45
  def initialize(options = {})
43
46
  options = { url: V1_API, retries: 5, retry_interval: 0.5 }.merge(options)
44
47
  @api_uri = options[:url]
@@ -83,6 +86,10 @@ module Berkshelf::API
83
86
  end
84
87
  end
85
88
 
89
+ # Download the cookbook with the given name and version to the destination. The directory
90
+ # containing the extracted contents will be returned on success. On failure, nil will
91
+ # be returned.
92
+ #
86
93
  # @param [String] cookbook
87
94
  # The name of the cookbook to download
88
95
  # @param [String] version
@@ -94,11 +101,17 @@ module Berkshelf::API
94
101
  def download(name, version, destination = Dir.mktmpdir)
95
102
  log.debug "downloading #{name}(#{version})"
96
103
  if uri = download_uri(name, version)
97
- archive = stream(uri)
98
- Archive.extract(archive.path, destination)
104
+ begin
105
+ archive = stream(uri)
106
+ Archive.extract(archive.path, destination)
107
+ destination
108
+ rescue => ex
109
+ log.warn "error downloading/extracting #{name} (#{version}): #{ex}"
110
+ nil
111
+ ensure
112
+ archive.unlink unless archive.nil?
113
+ end
99
114
  end
100
- ensure
101
- archive.unlink unless archive.nil?
102
115
  end
103
116
 
104
117
  # Return the location where a cookbook of the given name and version can be downloaded from
@@ -1,5 +1,5 @@
1
1
  module Berkshelf
2
2
  module API
3
- VERSION = "1.0.0"
3
+ VERSION = "1.1.0"
4
4
  end
5
5
  end
@@ -0,0 +1,71 @@
1
+ require 'spec_helper'
2
+
3
+ describe Berkshelf::API::CacheBuilder::Worker::Github do
4
+ describe "ClassMethods" do
5
+ subject { described_class }
6
+ its(:worker_type) { should eql("github") }
7
+ end
8
+
9
+ let(:connection) do
10
+ double('connection')
11
+ end
12
+
13
+ let(:good_tag) do
14
+ tag = double('good_tag')
15
+ tag.stub(:name) { 'v1.0.0' }
16
+ tag
17
+ end
18
+
19
+ let(:bad_tag) do
20
+ tag = double('good_tag')
21
+ tag.stub(:name) { 'beta2' }
22
+ tag
23
+ end
24
+
25
+ let :contents do
26
+ contents = double('contents')
27
+ contents.stub(:content) { 'dmVyc2lvbiAiMS4wLjAi' }
28
+ contents
29
+ end
30
+
31
+ let(:repo) do
32
+ repo = double('repo')
33
+ repo.stub(:full_name) { 'opscode-cookbooks/apt' }
34
+ repo.stub(:name) { 'apt' }
35
+ repo
36
+ end
37
+
38
+ let(:repos) do
39
+ [repo]
40
+ end
41
+
42
+ subject do
43
+ expect(Octokit::Client).to receive(:new) { connection }
44
+ described_class.new(organization: "opscode-cookbooks", access_token: "asdf")
45
+ end
46
+
47
+ describe "#cookbooks" do
48
+ before do
49
+ expect(connection).to receive(:organization_repositories) { repos }
50
+ expect(connection).to receive(:tags) { [good_tag, bad_tag] }
51
+ expect(connection).to receive(:contents).with("opscode-cookbooks/apt",
52
+ { path: "metadata.rb", ref: "v1.0.0"}) { contents }
53
+ end
54
+
55
+ it "returns an array containing an item for each valid cookbook on the server" do
56
+ expect(subject.cookbooks).to have(1).items
57
+ end
58
+
59
+ it "returns an array of RemoteCookbooks" do
60
+ subject.cookbooks.each do |cookbook|
61
+ expect(cookbook).to be_a(Berkshelf::API::RemoteCookbook)
62
+ end
63
+ end
64
+
65
+ it "each RemoteCookbook is tagged with a location_type matching the worker_type of the builder" do
66
+ subject.cookbooks.each do |cookbook|
67
+ expect(cookbook.location_type).to eql(described_class.worker_type)
68
+ end
69
+ end
70
+ end
71
+ end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: berkshelf-api
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jamie Winsor
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2013-10-15 00:00:00.000000000 Z
12
+ date: 2013-12-26 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: ridley
@@ -17,14 +17,14 @@ dependencies:
17
17
  requirements:
18
18
  - - ~>
19
19
  - !ruby/object:Gem::Version
20
- version: '1.6'
20
+ version: '2.0'
21
21
  type: :runtime
22
22
  prerelease: false
23
23
  version_requirements: !ruby/object:Gem::Requirement
24
24
  requirements:
25
25
  - - ~>
26
26
  - !ruby/object:Gem::Version
27
- version: '1.6'
27
+ version: '2.0'
28
28
  - !ruby/object:Gem::Dependency
29
29
  name: celluloid
30
30
  requirement: !ruby/object:Gem::Requirement
@@ -137,6 +137,34 @@ dependencies:
137
137
  - - ~>
138
138
  - !ruby/object:Gem::Version
139
139
  version: '0.1'
140
+ - !ruby/object:Gem::Dependency
141
+ name: octokit
142
+ requirement: !ruby/object:Gem::Requirement
143
+ requirements:
144
+ - - ~>
145
+ - !ruby/object:Gem::Version
146
+ version: '2.6'
147
+ type: :runtime
148
+ prerelease: false
149
+ version_requirements: !ruby/object:Gem::Requirement
150
+ requirements:
151
+ - - ~>
152
+ - !ruby/object:Gem::Version
153
+ version: '2.6'
154
+ - !ruby/object:Gem::Dependency
155
+ name: solve
156
+ requirement: !ruby/object:Gem::Requirement
157
+ requirements:
158
+ - - '>='
159
+ - !ruby/object:Gem::Version
160
+ version: '0'
161
+ type: :runtime
162
+ prerelease: false
163
+ version_requirements: !ruby/object:Gem::Requirement
164
+ requirements:
165
+ - - '>='
166
+ - !ruby/object:Gem::Version
167
+ version: '0'
140
168
  - !ruby/object:Gem::Dependency
141
169
  name: bundler
142
170
  requirement: !ruby/object:Gem::Requirement
@@ -206,6 +234,7 @@ files:
206
234
  - lib/berkshelf/api/cache_builder.rb
207
235
  - lib/berkshelf/api/cache_builder/worker.rb
208
236
  - lib/berkshelf/api/cache_builder/worker/chef_server.rb
237
+ - lib/berkshelf/api/cache_builder/worker/github.rb
209
238
  - lib/berkshelf/api/cache_builder/worker/opscode.rb
210
239
  - lib/berkshelf/api/cache_manager.rb
211
240
  - lib/berkshelf/api/config.rb
@@ -233,6 +262,7 @@ files:
233
262
  - spec/support/chef_server.rb
234
263
  - spec/unit/berkshelf/api/application_spec.rb
235
264
  - spec/unit/berkshelf/api/cache_builder/worker/chef_server_spec.rb
265
+ - spec/unit/berkshelf/api/cache_builder/worker/github_spec.rb
236
266
  - spec/unit/berkshelf/api/cache_builder/worker/opscode_spec.rb
237
267
  - spec/unit/berkshelf/api/cache_builder/worker_spec.rb
238
268
  - spec/unit/berkshelf/api/cache_builder_spec.rb
@@ -246,7 +276,7 @@ files:
246
276
  - spec/unit/berkshelf/api/rest_gateway_spec.rb
247
277
  - spec/unit/berkshelf/api/site_connector/opscode_spec.rb
248
278
  - spec/unit/berkshelf/api/srv_ctl_spec.rb
249
- homepage: https://github.com/RiotGames/berkshelf-api
279
+ homepage: https://github.com/berkshelf/berkshelf-api
250
280
  licenses:
251
281
  - Apache 2.0
252
282
  metadata: {}
@@ -266,7 +296,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
266
296
  version: '0'
267
297
  requirements: []
268
298
  rubyforge_project:
269
- rubygems_version: 2.0.3
299
+ rubygems_version: 2.0.14
270
300
  signing_key:
271
301
  specification_version: 4
272
302
  summary: A server which indexes cookbooks from various sources and hosts it over a
@@ -278,6 +308,7 @@ test_files:
278
308
  - spec/support/chef_server.rb
279
309
  - spec/unit/berkshelf/api/application_spec.rb
280
310
  - spec/unit/berkshelf/api/cache_builder/worker/chef_server_spec.rb
311
+ - spec/unit/berkshelf/api/cache_builder/worker/github_spec.rb
281
312
  - spec/unit/berkshelf/api/cache_builder/worker/opscode_spec.rb
282
313
  - spec/unit/berkshelf/api/cache_builder/worker_spec.rb
283
314
  - spec/unit/berkshelf/api/cache_builder_spec.rb