coursemology-evaluator 0.1.1 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. checksums.yaml +4 -4
  2. data/.env +3 -3
  3. data/.gitignore +23 -22
  4. data/.hound.yml +8 -8
  5. data/.idea/Coursemology Evaluator.iml +8 -8
  6. data/.rspec +2 -2
  7. data/.rubocop.unhound.yml +109 -109
  8. data/.rubocop.yml +46 -46
  9. data/.travis.yml +17 -17
  10. data/Gemfile +4 -4
  11. data/Procfile +1 -1
  12. data/README.md +29 -29
  13. data/Rakefile +6 -6
  14. data/bin/evaluator +5 -5
  15. data/coursemology-evaluator.gemspec +37 -37
  16. data/lib/coursemology/evaluator.rb +36 -36
  17. data/lib/coursemology/evaluator/cli.rb +52 -52
  18. data/lib/coursemology/evaluator/client.rb +81 -75
  19. data/lib/coursemology/evaluator/docker_container.rb +59 -59
  20. data/lib/coursemology/evaluator/logging.rb +12 -12
  21. data/lib/coursemology/evaluator/logging/client_log_subscriber.rb +25 -25
  22. data/lib/coursemology/evaluator/logging/docker_log_subscriber.rb +18 -18
  23. data/lib/coursemology/evaluator/models.rb +7 -7
  24. data/lib/coursemology/evaluator/models/base.rb +50 -50
  25. data/lib/coursemology/evaluator/models/programming_evaluation.rb +55 -67
  26. data/lib/coursemology/evaluator/models/programming_evaluation/package.rb +12 -12
  27. data/lib/coursemology/evaluator/services.rb +6 -6
  28. data/lib/coursemology/evaluator/services/evaluate_programming_package_service.rb +151 -151
  29. data/lib/coursemology/evaluator/string_io.rb +14 -14
  30. data/lib/coursemology/evaluator/utils.rb +42 -42
  31. data/lib/coursemology/evaluator/version.rb +5 -5
  32. data/lib/coursemology/polyglot/extensions.rb +3 -3
  33. data/lib/coursemology/polyglot/extensions/language.rb +24 -24
  34. metadata +15 -4
  35. data/Gemfile.lock +0 -114
data/Gemfile CHANGED
@@ -1,4 +1,4 @@
1
- source 'https://rubygems.org'
2
-
3
- # Specify your gem's dependencies in coursemology-evaluator.gemspec
4
- gemspec
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in coursemology-evaluator.gemspec
4
+ gemspec
data/Procfile CHANGED
@@ -1 +1 @@
1
- evaluator: evaluator --host=$HOST --api-token=$API_TOKEN --api-user-email=$API_USER_EMAIL
1
+ evaluator: evaluator --host=$HOST --api-token=$API_TOKEN --api-user-email=$API_USER_EMAIL
data/README.md CHANGED
@@ -1,29 +1,29 @@
1
- # Coursemology Code Evaluator [![Build Status](https://travis-ci.org/Coursemology/evaluator-slave.svg?branch=master)](https://travis-ci.org/Coursemology/evaluator-slave)
2
- [![Code Climate](https://codeclimate.com/github/Coursemology/evaluator-slave/badges/gpa.svg)](https://codeclimate.com/github/Coursemology/evaluator-slave) [![Coverage Status](https://coveralls.io/repos/Coursemology/evaluator-slave/badge.svg?branch=master&service=github)](https://coveralls.io/github/Coursemology/evaluator-slave?branch=master) [![Security](https://hakiri.io/github/Coursemology/evaluator-slave/master.svg)](https://hakiri.io/github/Coursemology/evaluator-slave/master) [![Inline docs](http://inch-ci.org/github/coursemology/evaluator-slave.svg?branch=master)](http://inch-ci.org/github/coursemology/evaluator-slave)
3
-
4
- This is the evaluator program which will query Coursemology for pending evaluation jobs.
5
-
6
- ## Setting up the Evaluator Slave
7
-
8
- ### System Requirements
9
-
10
- 1. Ruby (>= 2.1.0)
11
- 2. Linux (tested on Ubuntu 14.04)
12
- 3. Docker (the user the evaluator runs as must be able to talk to the Docker Remote API endpoint)
13
-
14
- ### Getting Started
15
-
16
- 1. Install the gem
17
-
18
- ```sh
19
- $ gem install coursemology-evaluator
20
- ```
21
-
22
- 2. Modify `.env` to suit your environment. Point to the host to your Coursemology instance, and
23
- specify the API email and API key.
24
-
25
- 1. You might need to configure a new user on your Coursemology instance, enable token
26
- authentication, and grant the `auto_grader` system/instance permission.
27
-
28
- 3. Start the evaluator using the Procfile. You can use [foreman](https://github.com/ddollar/foreman)
29
- or any similar tool to generate system scripts for boot.
1
+ # Coursemology Code Evaluator [![Build Status](https://travis-ci.org/Coursemology/evaluator-slave.svg?branch=master)](https://travis-ci.org/Coursemology/evaluator-slave)
2
+ [![Code Climate](https://codeclimate.com/github/Coursemology/evaluator-slave/badges/gpa.svg)](https://codeclimate.com/github/Coursemology/evaluator-slave) [![Coverage Status](https://coveralls.io/repos/Coursemology/evaluator-slave/badge.svg?branch=master&service=github)](https://coveralls.io/github/Coursemology/evaluator-slave?branch=master) [![Security](https://hakiri.io/github/Coursemology/evaluator-slave/master.svg)](https://hakiri.io/github/Coursemology/evaluator-slave/master) [![Inline docs](http://inch-ci.org/github/coursemology/evaluator-slave.svg?branch=master)](http://inch-ci.org/github/coursemology/evaluator-slave) [![Gem Version](https://badge.fury.io/rb/coursemology-evaluator.svg)](https://badge.fury.io/rb/coursemology-evaluator)
3
+
4
+ This is the evaluator program which will query Coursemology for pending evaluation jobs.
5
+
6
+ ## Setting up the Evaluator Slave
7
+
8
+ ### System Requirements
9
+
10
+ 1. Ruby (>= 2.1.0)
11
+ 2. Linux (tested on Ubuntu 14.04)
12
+ 3. Docker (the user the evaluator runs as must be able to talk to the Docker Remote API endpoint)
13
+
14
+ ### Getting Started
15
+
16
+ 1. Install the gem
17
+
18
+ ```sh
19
+ $ gem install coursemology-evaluator
20
+ ```
21
+
22
+ 2. Modify `.env` to suit your environment. Point to the host to your Coursemology instance, and
23
+ specify the API email and API key.
24
+
25
+ 1. You might need to configure a new user on your Coursemology instance, enable token
26
+ authentication, and grant the `auto_grader` system/instance permission.
27
+
28
+ 3. Start the evaluator using the Procfile. You can use [foreman](https://github.com/ddollar/foreman)
29
+ or any similar tool to generate system scripts for boot.
data/Rakefile CHANGED
@@ -1,6 +1,6 @@
1
- require 'bundler/gem_tasks'
2
- require 'rspec/core/rake_task'
3
-
4
- RSpec::Core::RakeTask.new(:spec)
5
-
6
- task default: :spec
1
+ require 'bundler/gem_tasks'
2
+ require 'rspec/core/rake_task'
3
+
4
+ RSpec::Core::RakeTask.new(:spec)
5
+
6
+ task default: :spec
@@ -1,5 +1,5 @@
1
- #!/usr/bin/env ruby
2
- require 'coursemology/evaluator'
3
-
4
- Coursemology::Evaluator.eager_load!
5
- Coursemology::Evaluator::CLI.start(ARGV)
1
+ #!/usr/bin/env ruby
2
+ require 'coursemology/evaluator'
3
+
4
+ Coursemology::Evaluator.eager_load!
5
+ Coursemology::Evaluator::CLI.start(ARGV)
@@ -1,37 +1,37 @@
1
- # coding: utf-8
2
- lib = File.expand_path('../lib', __FILE__)
3
- $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
- require 'coursemology/evaluator/version'
5
-
6
- Gem::Specification.new do |spec|
7
- spec.name = 'coursemology-evaluator'
8
- spec.version = Coursemology::Evaluator::VERSION
9
- spec.authors = ['Joel Low']
10
- spec.email = ['joel@joelsplace.sg']
11
- spec.license = 'MIT'
12
-
13
- spec.summary = 'Coursemology programming package evaluator'
14
- spec.description = 'Sets up a consistent environment for evaluating programming packages.'
15
- spec.homepage = 'http://coursemology.org'
16
- spec.files = `git ls-files -z`.split("\x0").
17
- reject { |f| f.match(/^(test|spec|features)\//) }
18
- spec.executables = spec.files.grep(/^bin\//) { |f| File.basename(f) }
19
- spec.require_paths = ['lib']
20
-
21
- spec.add_development_dependency 'bundler'
22
- spec.add_development_dependency 'rake'
23
- spec.add_development_dependency 'rspec'
24
- spec.add_development_dependency 'factory_girl'
25
- spec.add_development_dependency 'simplecov'
26
- spec.add_development_dependency 'coveralls'
27
- spec.add_development_dependency 'codeclimate-test-reporter'
28
- spec.add_development_dependency 'vcr'
29
-
30
- spec.add_dependency 'activesupport', '~> 4.2.0'
31
- spec.add_dependency 'flexirest', '~> 1.2'
32
- spec.add_dependency 'faraday_middleware'
33
-
34
- spec.add_dependency 'coursemology-polyglot', '>= 0.0.3'
35
- spec.add_dependency 'docker-api', '>= 1.2.5'
36
- spec.add_dependency 'rubyzip'
37
- end
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'coursemology/evaluator/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = 'coursemology-evaluator'
8
+ spec.version = Coursemology::Evaluator::VERSION
9
+ spec.authors = ['Joel Low']
10
+ spec.email = ['joel@joelsplace.sg']
11
+ spec.license = 'MIT'
12
+
13
+ spec.summary = 'Coursemology programming package evaluator'
14
+ spec.description = 'Sets up a consistent environment for evaluating programming packages.'
15
+ spec.homepage = 'https://github.com/Coursemology/evaluator-slave'
16
+ spec.files = `git ls-files -z`.split("\x0").
17
+ reject { |f| f.match(/^(test|spec|features)\//) }
18
+ spec.executables = spec.files.grep(/^bin\//) { |f| File.basename(f) }
19
+ spec.require_paths = ['lib']
20
+
21
+ spec.add_development_dependency 'bundler'
22
+ spec.add_development_dependency 'rake'
23
+ spec.add_development_dependency 'rspec'
24
+ spec.add_development_dependency 'factory_girl'
25
+ spec.add_development_dependency 'simplecov'
26
+ spec.add_development_dependency 'coveralls'
27
+ spec.add_development_dependency 'codeclimate-test-reporter'
28
+ spec.add_development_dependency 'vcr'
29
+
30
+ spec.add_dependency 'activesupport', '~> 4.2.0', '>= 4.2.2'
31
+ spec.add_dependency 'flexirest', '~> 1.2', '>= 1.2.6'
32
+ spec.add_dependency 'faraday_middleware'
33
+
34
+ spec.add_dependency 'coursemology-polyglot', '>= 0.0.3'
35
+ spec.add_dependency 'docker-api', '>= 1.2.5'
36
+ spec.add_dependency 'rubyzip'
37
+ end
@@ -1,36 +1,36 @@
1
- # frozen_string_literal: true
2
- require 'active_support/all'
3
- require 'flexirest'
4
- require 'faraday_middleware'
5
- require 'docker'
6
- require 'zip'
7
- Docker.validate_version!
8
-
9
- require 'coursemology/polyglot'
10
- require 'coursemology/polyglot/extensions'
11
- require 'coursemology/evaluator/version'
12
-
13
- module Coursemology::Evaluator
14
- extend ActiveSupport::Autoload
15
-
16
- autoload :Client
17
- autoload :DockerContainer
18
- autoload :CLI
19
- autoload :Models
20
- autoload :Services
21
- autoload :StringIO
22
- autoload :Utils
23
-
24
- eager_autoload do
25
- autoload :Logging
26
- end
27
-
28
- # The logger to use for the client.
29
- mattr_reader(:logger) { ActiveSupport::Logger.new(STDOUT) }
30
-
31
- def self.eager_load!
32
- super
33
- Coursemology::Polyglot.eager_load!
34
- Logging.eager_load!
35
- end
36
- end
1
+ # frozen_string_literal: true
2
+ require 'active_support/all'
3
+ require 'flexirest'
4
+ require 'faraday_middleware'
5
+ require 'docker'
6
+ require 'zip'
7
+ Docker.validate_version!
8
+
9
+ require 'coursemology/polyglot'
10
+ require 'coursemology/polyglot/extensions'
11
+ require 'coursemology/evaluator/version'
12
+
13
+ module Coursemology::Evaluator
14
+ extend ActiveSupport::Autoload
15
+
16
+ autoload :Client
17
+ autoload :DockerContainer
18
+ autoload :CLI
19
+ autoload :Models
20
+ autoload :Services
21
+ autoload :StringIO
22
+ autoload :Utils
23
+
24
+ eager_autoload do
25
+ autoload :Logging
26
+ end
27
+
28
+ # The logger to use for the client.
29
+ mattr_reader(:logger) { ActiveSupport::Logger.new(STDOUT) }
30
+
31
+ def self.eager_load!
32
+ super
33
+ Coursemology::Polyglot.eager_load!
34
+ Logging.eager_load!
35
+ end
36
+ end
@@ -1,52 +1,52 @@
1
- # frozen_string_literal: true
2
- require 'optparse'
3
-
4
- class Coursemology::Evaluator::CLI
5
- Options = Struct.new(:host, :api_token, :api_user_email, :one_shot)
6
-
7
- def self.start(argv)
8
- new.start(argv)
9
- end
10
-
11
- def start(argv)
12
- run(argv)
13
- end
14
-
15
- def run(argv)
16
- options = optparse!(argv)
17
- Coursemology::Evaluator::Client.initialize(options.host, options.api_user_email,
18
- options.api_token)
19
- Coursemology::Evaluator::Client.new(options.one_shot).run
20
- end
21
-
22
- private
23
-
24
- # Parses the options specified on the command line.
25
- #
26
- # @param [Array<String>] argv The arguments specified on the command line.
27
- # @return [Coursemology::Evaluator::CLI::Options]
28
- def optparse!(argv) # rubocop:disable Metrics/MethodLength
29
- options = Options.new
30
- option_parser = OptionParser.new do |parser|
31
- parser.banner = "Usage: #{parser.program_name} [options]"
32
- parser.on('-hHOST', '--host=HOST', 'Coursemology host to connect to') do |host|
33
- options.host = host
34
- end
35
-
36
- parser.on('-tTOKEN', '--api-token=TOKEN') do |token|
37
- options.api_token = token
38
- end
39
-
40
- parser.on('-uUSER', '--api-user-email=USER') do |user|
41
- options.api_user_email = user
42
- end
43
-
44
- parser.on('-o', '--one-shot') do
45
- options.one_shot = true
46
- end
47
- end
48
-
49
- option_parser.parse!(argv)
50
- options
51
- end
52
- end
1
+ # frozen_string_literal: true
2
+ require 'optparse'
3
+
4
+ class Coursemology::Evaluator::CLI
5
+ Options = Struct.new(:host, :api_token, :api_user_email, :one_shot)
6
+
7
+ def self.start(argv)
8
+ new.start(argv)
9
+ end
10
+
11
+ def start(argv)
12
+ run(argv)
13
+ end
14
+
15
+ def run(argv)
16
+ options = optparse!(argv)
17
+ Coursemology::Evaluator::Client.initialize(options.host, options.api_user_email,
18
+ options.api_token)
19
+ Coursemology::Evaluator::Client.new(options.one_shot).run
20
+ end
21
+
22
+ private
23
+
24
+ # Parses the options specified on the command line.
25
+ #
26
+ # @param [Array<String>] argv The arguments specified on the command line.
27
+ # @return [Coursemology::Evaluator::CLI::Options]
28
+ def optparse!(argv) # rubocop:disable Metrics/MethodLength
29
+ options = Options.new
30
+ option_parser = OptionParser.new do |parser|
31
+ parser.banner = "Usage: #{parser.program_name} [options]"
32
+ parser.on('-hHOST', '--host=HOST', 'Coursemology host to connect to') do |host|
33
+ options.host = host
34
+ end
35
+
36
+ parser.on('-tTOKEN', '--api-token=TOKEN') do |token|
37
+ options.api_token = token
38
+ end
39
+
40
+ parser.on('-uUSER', '--api-user-email=USER') do |user|
41
+ options.api_user_email = user
42
+ end
43
+
44
+ parser.on('-o', '--one-shot') do
45
+ options.one_shot = true
46
+ end
47
+ end
48
+
49
+ option_parser.parse!(argv)
50
+ options
51
+ end
52
+ end
@@ -1,75 +1,81 @@
1
- # frozen_string_literal: true
2
- class Coursemology::Evaluator::Client
3
- def self.initialize(host, api_user_email, api_token)
4
- Coursemology::Evaluator::Models::Base.base_url = host
5
- Coursemology::Evaluator::Models::Base.api_user_email = api_user_email
6
- Coursemology::Evaluator::Models::Base.api_token = api_token
7
-
8
- Coursemology::Evaluator::Models::Base.initialize
9
- end
10
-
11
- # @param [Boolean] one_shot If the client should only fire one request.
12
- def initialize(one_shot = false)
13
- @terminate = one_shot
14
- end
15
-
16
- def run
17
- Signal.trap('SIGTERM', method(:on_sig_term))
18
-
19
- loop do
20
- allocate_evaluations
21
- break if @terminate
22
-
23
- # :nocov:
24
- # This sleep might not be triggered in the specs, because interruptions to the thread is
25
- # nondeterministically run by the OS scheduler.
26
- sleep(1.minute)
27
- # :nocov:
28
- end
29
- end
30
-
31
- private
32
-
33
- # Requests evaluations from the server.
34
- def allocate_evaluations
35
- evaluations =
36
- ActiveSupport::Notifications.instrument('allocate.client.evaluator.coursemology') do
37
- languages = Coursemology::Polyglot::Language.concrete_languages.map(&:display_name)
38
- Coursemology::Evaluator::Models::ProgrammingEvaluation.allocate(language: languages)
39
- end
40
-
41
- on_allocate(evaluations)
42
- rescue Flexirest::HTTPUnauthorisedClientException => e
43
- ActiveSupport::Notifications.publish('allocate_fail.client.evaluator.coursemology', e: e)
44
- end
45
-
46
- # The callback for handling an array of allocated evaluations.
47
- #
48
- # @param [Array<Coursemology::Evaluator::Models::ProgrammingEvaluation>] evaluations The
49
- # evaluations retrieved from the server.
50
- def on_allocate(evaluations)
51
- evaluations.each do |evaluation|
52
- on_evaluation(evaluation)
53
- end
54
- end
55
-
56
- # The callback for handling an evaluation.
57
- #
58
- # @param [Coursemology::Evaluator::Models::ProgrammingEvaluation] evaluation The evaluation
59
- # retrieved from the server.
60
- def on_evaluation(evaluation)
61
- ActiveSupport::Notifications.instrument('evaluate.client.evaluator.coursemology',
62
- evaluation: evaluation) do
63
- evaluation.evaluate
64
- end
65
-
66
- ActiveSupport::Notifications.instrument('save.client.evaluator.coursemology') do
67
- evaluation.save
68
- end
69
- end
70
-
71
- # The callback for handling SIGTERM sent to the process.
72
- def on_sig_term
73
- @terminate = true
74
- end
75
- end
1
+ # frozen_string_literal: true
2
+ class Coursemology::Evaluator::Client
3
+ def self.initialize(host, api_user_email, api_token)
4
+ Coursemology::Evaluator::Models::Base.base_url = host
5
+ Coursemology::Evaluator::Models::Base.api_user_email = api_user_email
6
+ Coursemology::Evaluator::Models::Base.api_token = api_token
7
+
8
+ Coursemology::Evaluator::Models::Base.initialize
9
+ end
10
+
11
+ # @param [Boolean] one_shot If the client should only fire one request.
12
+ def initialize(one_shot = false)
13
+ @terminate = one_shot
14
+ end
15
+
16
+ def run
17
+ Signal.trap('SIGTERM', method(:on_sig_term))
18
+
19
+ loop do
20
+ evaluations = allocate_evaluations
21
+
22
+ if evaluations && !evaluations.empty?
23
+ on_allocate(evaluations)
24
+ else
25
+ # :nocov:
26
+ # This sleep might not be triggered in the specs, because interruptions to the thread is
27
+ # nondeterministically run by the OS scheduler.
28
+ sleep(1.minute)
29
+ # :nocov:
30
+ end
31
+
32
+ break if @terminate
33
+ end
34
+ end
35
+
36
+ private
37
+
38
+ # Requests evaluations from the server.
39
+ #
40
+ # @return [Array<Coursemology::Evaluator::Models::ProgrammingEvaluation>] The evaluations
41
+ # retrieved from the server.
42
+ def allocate_evaluations
43
+ ActiveSupport::Notifications.instrument('allocate.client.evaluator.coursemology') do
44
+ languages = Coursemology::Polyglot::Language.concrete_languages.map(&:display_name)
45
+ Coursemology::Evaluator::Models::ProgrammingEvaluation.allocate(language: languages)
46
+ end
47
+ rescue Flexirest::HTTPUnauthorisedClientException => e
48
+ ActiveSupport::Notifications.publish('allocate_fail.client.evaluator.coursemology', e: e)
49
+ nil
50
+ end
51
+
52
+ # The callback for handling an array of allocated evaluations.
53
+ #
54
+ # @param [Array<Coursemology::Evaluator::Models::ProgrammingEvaluation>] evaluations The
55
+ # evaluations retrieved from the server.
56
+ def on_allocate(evaluations)
57
+ evaluations.each do |evaluation|
58
+ on_evaluation(evaluation)
59
+ end
60
+ end
61
+
62
+ # The callback for handling an evaluation.
63
+ #
64
+ # @param [Coursemology::Evaluator::Models::ProgrammingEvaluation] evaluation The evaluation
65
+ # retrieved from the server.
66
+ def on_evaluation(evaluation)
67
+ ActiveSupport::Notifications.instrument('evaluate.client.evaluator.coursemology',
68
+ evaluation: evaluation) do
69
+ evaluation.evaluate
70
+ end
71
+
72
+ ActiveSupport::Notifications.instrument('save.client.evaluator.coursemology') do
73
+ evaluation.save
74
+ end
75
+ end
76
+
77
+ # The callback for handling SIGTERM sent to the process.
78
+ def on_sig_term
79
+ @terminate = true
80
+ end
81
+ end