legionio 0.4.3 → 1.2.0

data/NOTICE.txt

@@ -0,0 +1,9 @@
+LegionIO
+Copyright 2021 Optum
+
+Project Description:
+====================
+LegionIO is an automation framework for create IFTTT style relationships, scheduling and managing sync and async tasks/jobs
+
+Author(s):
+Esity

data/README.md

@@ -3,6 +3,7 @@
 LegionIO is a framework for automating and connecting things. You can see all the docs inside confluence  
 https://legionio.atlassian.net/wiki/spaces/LEGION/overview
 https://legionio.atlassian.net/wiki/spaces/LEX/pages/7864551/Extensions
+*Soon to be migrated to GitHub Wiki*
 
 ### What does it do?
 LegionIO is an async job engine designed for scheduling tasks and creating relationships between things that wouldn't 
@@ -26,21 +27,21 @@ After installing gem you can use the commands `legionio` to start legion, `legio
 and `lex_gen` to generate a new legion extension
 
 ### Example Legion Extensions(LEX)
-* [lex-http](https://bitbucket.org/legion-io/lex-http/src/master/) - Gives legion the ability to make http requests, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/12910593/Lex+Http)
-* [lex-influxdb](https://bitbucket.org/legion-io/lex-influxdb/src/master/) - Write, read, and manage influxdb nodes, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614891774/Lex+Influxdb)
-* [lex-log](https://bitbucket.org/legion-io/lex-log/src/master/) - Send log items to either stdout or a file with lex-log, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614858995/Lex+Log)
-* [lex-memcache](https://bitbucket.org/legion-io/lex-memcached/src/master/) - run memcached commands like set, add, append, delete, flush, reset_stats against memcached servers, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614858753/Lex+Memcached)
-* [lex-pihole](https://bitbucket.org/legion-io/lex-pihole/src/master/) - Allows Legion to interact with [Pi-Hole](https://pi-hole.net/). Can do things like get status, add/remove domains from the list, etc 
-* [lex-ping](https://bitbucket.org/legion-io/lex-ping/src/master/) - You can ping things?, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/631373895/Lex+Ping)
-* [lex-pushover](https://bitbucket.org/legion-io/lex-pushover/src/master/) - Connects Legion to [Pushover](https://pushover.net/), [docs]()
-* [lex-redis](https://bitbucket.org/legion-io/lex-redis/src/master/) - similiar to lex-memcached but for redis
-* [lex-sleepiq](https://bitbucket.org/legion-io/lex-sleepiq/src/master/) - Control your SleepIQ bed with Legion!
-* [lex-ssh](https://bitbucket.org/legion-io/lex-ssh/src/master/) - Send commands to a server via SSH in an async fashion, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614891551/Lex+SSH)
+* [lex-http](https://github.com/LegionIO/lex-http/src/master/) - Gives legion the ability to make http requests, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/12910593/Lex+Http)
+* [lex-influxdb](https://github.com/LegionIO/lex-influxdb/src/master/) - Write, read, and manage influxdb nodes, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614891774/Lex+Influxdb)
+* [lex-log](https://github.com/LegionIO/lex-log/src/master/) - Send log items to either stdout or a file with lex-log, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614858995/Lex+Log)
+* [lex-memcache](https://github.com/LegionIO/lex-memcached/src/master/) - run memcached commands like set, add, append, delete, flush, reset_stats against memcached servers, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614858753/Lex+Memcached)
+* [lex-pihole](https://github.com/LegionIO/lex-pihole/src/master/) - Allows Legion to interact with [Pi-Hole](https://pi-hole.net/). Can do things like get status, add/remove domains from the list, etc 
+* [lex-ping](https://github.com/LegionIO/lex-ping/src/master/) - You can ping things?, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/631373895/Lex+Ping)
+* [lex-pushover](https://github.com/LegionIO/lex-pushover/src/master/) - Connects Legion to [Pushover](https://pushover.net/), [docs]()
+* [lex-redis](https://github.com/LegionIO/lex-redis/src/master/) - similiar to lex-memcached but for redis
+* [lex-sleepiq](https://github.com/LegionIO/lex-sleepiq/src/master/) - Control your SleepIQ bed with Legion!
+* [lex-ssh](https://github.com/LegionIO/lex-ssh/src/master/) - Send commands to a server via SSH in an async fashion, [docs](https://legionio.atlassian.net/wiki/spaces/LEX/pages/614891551/Lex+SSH)
 
 Bitbucket repos for extensions that are active or being worked on
-[lex list](https://bitbucket.org/legion-io/workspace/projects/LEX)
+[lex list](https://github.com/LegionIO/workspace/projects/LEX)
 A nice list in the wiki to view all the extensions, their docs and status
-[Legion Extensions](https://legionio.atlassian.net/wiki/spaces/LEX/pages/7864551/Extensions)
+[Legion Extensions](https://github.com/topics/legionio?l=ruby)
 
 ### Scheduling Tasks
 1) Ensure you have the Legion::Data gem installed and configured  

data/SECURITY.md

@@ -0,0 +1,9 @@
+# Security Policy
+
+## Supported Versions
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.x.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+To be added

data/attribution.txt

@@ -0,0 +1 @@
+Add attributions here.

data/legionio.gemspec

@@ -10,22 +10,29 @@ Gem::Specification.new do |spec|
   spec.authors       = ['Esity']
   spec.email         = ['matthewdiverson@gmail.com']
 
-  spec.summary       = 'Legion Core Software to string it all together'
-  spec.description   = 'Legion Core runs Legion Framwork'
-  spec.homepage      = 'https://bitbucket.org/legion-io/legion/'
-  spec.metadata['source_code_uri'] = 'https://bitbucket.org/legion-io/legion/src/master/'
-  spec.metadata['documentation_uri'] = 'https://legionio.atlassian.net/wiki/spaces/LEGION/overview'
-  spec.metadata['bug_tracker_uri'] = 'https://legionio.atlassian.net/browse/LEG'
-  spec.license = 'MIT'
+  spec.summary       = 'The primary gem to run the LegionIO Framework'
+  spec.description   = 'LegionIO is an extensible framework for running, scheduling and building relationships of tasks in a concurrent matter'
+  spec.homepage      = 'https://github.com/Optum/LegionIO'
+  spec.license       = 'Apache-2.0'
+  spec.require_paths = ['lib']
   spec.required_ruby_version = '>= 2.5.0'
 
+  spec.metadata = {
+    'bug_tracker_uri'   => 'https://github.com/Optum/LegionIO/issues',
+    'changelog_uri'     => 'https://github.com/Optum/LegionIO/src/main/CHANGELOG.md',
+    'documentation_uri' => 'https://github.com/Optum/LegionIO',
+    'homepage_uri'      => 'https://github.com/Optum/LegionIO',
+    'source_code_uri'   => 'https://github.com/Optum/LegionIO',
+    'wiki_uri'          => 'https://github.com/Optum/LegionIO'
+  }
+
   spec.files = `git ls-files -z`.split("\x0").reject do |f|
     f.match(%r{^(test|spec|features)/})
   end
+  spec.test_files = spec.files.select { |p| p =~ %r{^test/.*_test.rb} }
 
   spec.bindir        = 'exe'
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = ['lib']
 
   spec.add_dependency 'concurrent-ruby', '>= 1.1.7'
   spec.add_dependency 'concurrent-ruby-ext', '>= 1.1.7'
@@ -33,32 +40,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'oj', '>= 3.10'
   spec.add_dependency 'thor', '>= 1'
 
-  spec.add_dependency 'legion-cache'
+  spec.add_dependency 'legion-cache', '>= 0.2.0'
   spec.add_dependency 'legion-crypt', '>= 0.2.0'
-  spec.add_dependency 'legion-exceptions'
-  spec.add_dependency 'legion-json'
-  spec.add_dependency 'legion-logging'
-  spec.add_dependency 'legion-settings'
+  spec.add_dependency 'legion-json', '>= 0.2.0'
+  spec.add_dependency 'legion-logging', '>= 0.2.0'
+  spec.add_dependency 'legion-settings', '>= 0.2.0'
   spec.add_dependency 'legion-transport', '>= 1.1.9'
 
   spec.add_dependency 'lex-node'
 
-  spec.add_development_dependency 'codecov'
   spec.add_development_dependency 'legion-data'
-  spec.add_development_dependency 'rake'
-  spec.add_development_dependency 'rspec'
-  spec.add_development_dependency 'rspec_junit_formatter'
-  spec.add_development_dependency 'rubocop'
-  spec.add_development_dependency 'rubocop-rake'
-  spec.add_development_dependency 'rubocop-rspec'
-  spec.add_development_dependency 'simplecov'
-
-  spec.add_development_dependency 'lex-conditioner'
-  spec.add_development_dependency 'lex-health'
-  spec.add_development_dependency 'lex-http'
-  spec.add_development_dependency 'lex-lex'
-  spec.add_development_dependency 'lex-log'
-  spec.add_development_dependency 'lex-scheduler'
-  spec.add_development_dependency 'lex-tasker'
-  spec.add_development_dependency 'lex-transformer'
 end

data/lib/legion.rb

@@ -1,7 +1,6 @@
 Process.setproctitle('Legion')
 require 'concurrent'
 require 'securerandom'
-require 'legion/exceptions'
 require 'legion/version'
 require 'legion/process'
 require 'legion/service'

data/lib/legion/cli/trigger.rb

@@ -65,7 +65,7 @@ module Legion
 
         status = options[:delay].zero? ? 'task.queued' : 'task.delayed'
         task = generate_task_id(function_id: trigger_function.values[:id], status: status, runner_id: trigger_runner.values[:id], args: payload,
-delay: options[:delay])
+                                delay: options[:delay])
 
         unless options[:delay].zero?
           say "Task: #{task[:task_id]} is queued and will be run in #{options[:delay]}s"
@@ -74,7 +74,7 @@ delay: options[:delay])
 
         routing_key = "#{trigger_extension.values[:exchange]}.#{trigger_runner.values[:queue]}.#{trigger_function.values[:name]}"
         exchange = Legion::Transport::Messages::Dynamic.new(function: trigger_function.values[:name], function_id: trigger_function.values[:id],
-routing_key: routing_key, args: payload)
+                                                            routing_key: routing_key, args: payload)
         exchange.options[:task_id] = task[:task_id]
         exchange.publish if options[:delay].zero?
 

data/lib/legion/extensions.rb

@@ -59,10 +59,15 @@ module Legion
         Legion::Logging.info "#{@extensions.count} extensions loaded with subscription:#{@subscription_tasks.count},every:#{@timer_tasks.count},poll:#{@poll_tasks.count},once:#{@once_tasks.count},loop:#{@loop_tasks.count}"
       end
 
-      def load_extension(extension, values)
+      def load_extension(extension, values) # rubocop:disable Metrics/PerceivedComplexity, Metrics/CyclomaticComplexity, Metrics/AbcSize
         return unless gem_load(values[:gem_name], extension)
 
         extension = Kernel.const_get(values[:extension_class])
+        extension.extend Legion::Extensions::Core unless extension.singleton_class.included_modules.include? Legion::Extensions::Core
+
+        min_version = Legion::Settings[:extensions][values[:extension_name]][:min_version] || nil
+        Legion::Logging.fatal values if min_version.is_a?(String) && Gem::Version.new(values[:version]) >= Gem::Version.new(min_version)
+
         if extension.data_required? && Legion::Settings[:data][:connected] == false
           Legion::Logging.warn "#{values[:extension_name]} requires Legion::Data but isn't enabled, skipping"
           return false
@@ -73,6 +78,11 @@ module Legion
           return false
         end
 
+        if extension.crypt_required? && Legion::Settings[:crypt][:cs].nil?
+          Legion::Logging.warn "#{values[:extension_name]} requires Legion::Crypt but isn't ready, skipping"
+          return false
+        end
+
         if extension.vault_required? && Legion::Settings[:crypt][:vault][:connected] == false
           Legion::Logging.warn "#{values[:extension_name]} requires Legion::Crypt::Vault but isn't enabled, skipping"
           return false

data/lib/legion/extensions/actors/once.rb

@@ -7,7 +7,7 @@ module Legion
         include Legion::Extensions::Actors::Base
 
         def initialize
-          return if disabled?
+          return unless enabled?
 
           if respond_to? :functions
             functions.each do
@@ -30,7 +30,7 @@ module Legion
         end
 
         def cancel
-          return if disabled?
+          return unless enabled?
 
           @task.cancel unless @task.cancelled?
         end

data/lib/legion/extensions/builders/runners.rb

@@ -31,6 +31,8 @@ module Legion
               class_methods:   {}
             }
 
+            @runners[runner_name.to_sym][:scheduled_tasks] = loaded_runner.scheduled_tasks if loaded_runner.method_defined? :scheduled_tasks
+
             if settings.key?(:runners) && settings[:runners].key?(runner_name.to_sym)
               @runners[runner_name.to_sym][:desc] = settings[:runners][runner_name.to_sym][:desc]
             end
@@ -42,6 +44,8 @@ module Legion
             end
 
             loaded_runner.methods(false).each do |runner_method|
+              next if %i[scheduled_tasks runner_description].include? runner_method
+
               @runners[runner_name.to_sym][:class_methods][runner_method] = {
                 args: loaded_runner.method(runner_method).parameters
               }

data/lib/legion/extensions/core.rb

@@ -84,7 +84,7 @@ module Legion
         end
       end
 
-      def build_settings # rubocop:disable Metrics/AbcSize
+      def build_settings
         if Legion::Settings[:extensions].key?(lex_name.to_sym)
           Legion::Settings[:default_extension_settings].each do |key, value|
             Legion::Settings[:extensions][lex_name.to_sym][key.to_sym] = if Legion::Settings[:extensions][lex_name.to_sym].key?(key.to_sym)

data/lib/legion/extensions/data.rb

@@ -8,6 +8,7 @@ module Legion
       include Legion::Extensions::Helpers::Logger
 
       def build
+        Legion::Logging.fatal 'testing inside run'
         @models = []
         @migrations = []
         if Dir[File.expand_path("#{data_path}/migrations/*.rb")].count.positive?
@@ -43,10 +44,12 @@ module Legion
       end
 
       def migrate_class
-        @migrate_class ||= Legion::Extensions::Data::Migrator.new(migrations_path, lex_class.to_s)
+        @migrate_class ||= Legion::Extensions::Data::Migrator.new(migrations_path, lex_class.to_s, lex_name)
       end
 
       def run
+        Legion::Logging.fatal 'testing inside run'
+
         return true if migrate_class.is_current?
 
         log.debug('Running LEX schema migrator')

data/lib/legion/extensions/data/migrator.rb

@@ -4,7 +4,8 @@ module Legion
   module Extensions
     module Data
       class Migrator < Sequel::IntegerMigrator
-        def initialize(path, extension, **)
+        def initialize(path, extension, _lex_name, **)
+          Legion::Logging.fatal @extension
           @path = path
           @extension = extension
           super(Legion::Data::Connection.sequel, path)
@@ -19,6 +20,10 @@ module Legion
         end
 
         def schema_dataset
+          dataset = Legion::Data::Connection.sequel.from(default_schema_table).where(namespace: @extension)
+          return dataset unless dataset.count.positive?
+
+          Legion::Logging.unknown Legion::Data::Model::Extension.insert(active: 1, namespace: @extension, extension: lex_name)
           Legion::Data::Connection.sequel.from(default_schema_table).where(namespace: @extension)
         end
         alias ds schema_dataset

data/lib/legion/runner.rb

@@ -1,4 +1,3 @@
-require 'legion/exceptions/handled_task'
 require_relative 'runner/log'
 require_relative 'runner/status'
 require 'legion/transport'
@@ -6,14 +5,14 @@ require 'legion/transport/messages/check_subtask'
 
 module Legion
   module Runner
-    def self.run(runner_class:, function:, task_id: nil, args: nil, check_subtask: true, generate_task: true, parent_id: nil, master_id: nil, **opts) # rubocop:disable Metrics/ParameterLists
+    def self.run(runner_class:, function:, task_id: nil, args: nil, check_subtask: true, generate_task: true, parent_id: nil, master_id: nil, catch_exceptions: false, **opts) # rubocop:disable Layout/LineLength, Metrics/CyclomaticComplexity, Metrics/ParameterLists
       runner_class = Kernel.const_get(runner_class) if runner_class.is_a? String
 
       if task_id.nil? && generate_task
         task_gen = Legion::Runner::Status.generate_task_id(
           function: function,
-           runner_class: runner_class,
-         parent_id: parent_id, master_id: master_id, task_id: task_id, **opts
+          runner_class: runner_class,
+          parent_id: parent_id, master_id: master_id, task_id: task_id, **opts
         )
         task_id = task_gen[:task_id] unless task_gen.nil?
       end
@@ -23,6 +22,9 @@ module Legion
       args[:master_id] = master_id unless master_id.nil?
       args[:parent_id] = parent_id unless parent_id.nil?
 
+      # result = Fiber.new { Fiber.yield runner_class.send(function, **args) }
+      raise 'No Function defined' if function.nil?
+
       result = runner_class.send(function, **args)
     rescue Legion::Exception::HandledTask
       status = 'task.exception'
@@ -38,9 +40,9 @@ module Legion
                                     check_subtask: check_subtask)
       status = 'task.exception'
       result = { success: false, status: status, error: { message: e.message, backtrace: e.backtrace } }
-    else
-      status = 'task.completed'
+      raise e unless catch_exceptions
     ensure
+      status = 'task.completed' if status.nil?
       Legion::Runner::Status.update(task_id: task_id, status: status) unless task_id.nil?
       if check_subtask && status == 'task.completed'
         Legion::Transport::Messages::CheckSubtask.new(runner_class:  runner_class,