tcell_agent 1.1.10 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b84894f0c99387e227049af1f0cf23514630037a8a717b43283bcb19c66dfa37
-  data.tar.gz: 7464cb57bee06eea3c94717334ef755b0acbef4768f836f39b8de7cdb85cb505
+  metadata.gz: fef169c3426eb04bcf907ba86a06b56508b3510b5a460dd20bdd0e859c773832
+  data.tar.gz: c60696ff6d4058537d8162cc28c5f956340a5b5a533bf1df7b805f1f8b208f90
 SHA512:
-  metadata.gz: 60dabf811b67b4d762a8c39c96e47f7f3fff2e0287bb8586fec019976680b7c88efcbeb8e8ef541a8c719de0430a1c010e72ebaf7dfaffd03ff38b8c501fabca
-  data.tar.gz: 4974dccf426c1113ffbd238ef8006dc8600fc01bcbd29fbf940c2c78df5f979c80f48888b7565fa8a3c998b71edc60973e2e97ddd08c8648224755b270af9ae9
+  metadata.gz: ffaeef09c1dbf616146f6407abdcd5c53c292098c9e6e3bb9ab32602c8f5cb94ac69e2d958e00ef6fd58b6c1c6e133104935617fd06410752f704132eaa78d81
+  data.tar.gz: 77a9ab760568d411be5535f0c9842376289667892b39b88bed4bfc6646a496d8cf26a1d6704e78c644d7564aa4fc9e1955d6cbbf36ec40a91ff01d3ad37807c1

data/README.md

@@ -1,120 +1,48 @@
-# TCellAgent [![Build Status](https://magnum.travis-ci.com/tcellio/rubyagent-tcell.svg?token=j7YU3iPt38CqCoDeM83P)](https://magnum.travis-ci.com/tcellio/rubyagent-tcell)
-
-TCell Agent - Instruments Rails
-
-## Release Process Checklist
-
-Open a jira with the following checklist (sample: https://jira.tcell.io/browse/RA-98)
-
-
-[ X ] unit tests pass  
-[ X ] integration tests pass  
-[ X ] test-kit tests pass (all variants)  
-[ X ] check test-kit logs for any unexpected errors  
-[ X ] performance tests pass (https://github.com/tcellio/load-testing/tree/master/agents)  
-[ X ] update changelog and version (sample https://github.com/tcellio/rubyagent-tcell/pull/275)  
-
-[ X ] manual install verified working (`make build`, install gem globally: `gem install tcell_agent-x.x.x.gem` works properly)  
-[ X ] run `tcell_agent test` command (might not be needed, since this is run in test-kit for every test-app)  
-
-[ X ] Update Master Branch
-
-[ X ] ensure unknown_options accounts for any new keys (check `lib/tcell_agent/config/unknown_options.rb` file for logic)  
-[ X ] update support page in wiki with any new support (https://tcellio.atlassian.net/wiki/spaces/EN/pages/54460460/Server+Agent+Support)  
- 
-Repeat if any bugs are found and patched
+# RubyAgent
+A Ruby security agent to instrument Ruby web applications. Additional docs can be found [here](https://docs.tcell.io/docs/ruby-agent-install)
+## Installation
 
-[ X ] merge master branch to release branch (this model was followed to make it easy to hotfix released versions)
+Add this line to your Gemfile to install the gem directly from [RubyGems](https://rubygems.org)
 
-[ X ] create release in github repo with release notes and appropriate version tag (https://github.com/tcellio/rubyagent-tcell/releases)  
-[ X ] run `make build` on release branch (`gem unpack tcell_agent-x.x.x.gem` and manually inspect .gem file to make sure no files are missing and no extra files are included)  
-[ X ] upload to rubygems (`gem push tcell_agent-x.x.x.gem`)  
-[ X ] confirm upload worked by running a test app that installs latest version from rubygems and smoke test  
+    $ gem 'tcell_agent'
 
-Common Manual Tests Before Release
+or to use a local version directly,
 
-Run a Test Kit test app with multiple worker processes and run the suite of tests against it (https://github.com/tcellio/tcell/compare/test-kit/run-multiple-worker-processes?expand=1)
+    $ gem 'tcell_agent', :path => '<path-to-gem-repo'
 
-Manually running a test app with multiple worker processes is temporary until this is implemented: https://jira.tcell.io/browse/TCEL-5298
+and to install the gem,
 
-Smoke test a test app. Make sure agent can report appfw, you can post forms, you can login/log out. Test server configurations that you can't test with test-kit. Like multiple worker processes.
+    $ bundle install
 
-Special Things
+## Usage
 
-Config File - For rails applications config files are commonly stored in `config/` directory. Rubyagent is configured to look in `config/tcell_agent.config` by default for its configuration file.
+Download the config file from the tCell dashboard  and move it to the `config` directory of the application.
 
-Agent Home Owner - A customer experienced an issue where their web app master process ran as root but each worker process ran as a different user. This caused problems with the `tcell/` directory. If the master process created the directory then it was owned by root, so worker processes would raise permission exceptions when they tried to do any logging or write to the cache under the `tcell/`.  `config.agent_home_owner` was introduced to allow the customer to specify the name of the user that should own `tcell/` so that worker process would not get permission exceptions when writing to it. This is no longer necessary with the newest agent version. The newest agent version doesn't do anything in the master process, everything is executed in the workers, so `tcell/` should have the correct permissions because it'll always be created by a worker process.
+    $ cp ~/Downloads/tcell_agent.config config/
 
-Rails Initializers: It's common for gems to be configured thru Rails initializers: https://guides.rubyonrails.org/v2.3/configuring.html#using-initializers. TCell rubyagent supports this as well. This means that certain rubyagent startup code that relies on configuration needs to run after Rails initializers have run. The rubyagent can be configured entirely thru initializers (as opposed to using tcell_agent.config file and/or ENV vars). Sample `config/initializers/tcell.rb` file:
+When the server is started, tCell will be running. The log files will be created in the `tcell` directory of the project.
 
+Alternatively, Rails initializers can be used instead of adding the `tcell_agent.config` file. The values for the fields can be found in `tcell_agent.config`. Here's a sample `config/initializers/tcell.rb`:
 ```
 if defined?(TCellAgent)
   TCellAgent.configure do |config|
     config.app_id = "---APP_ID---"
     config.api_key = "---API_KEY---"
-    config.agent_log_dir = "/tmp/tcelllogs"
-    config.agent_home_dir = "/tmp/tcellhome"
-    config.tcell_api_url = "https://api.tcell-preview.io/api/v1"
-    config.tcell_input_url = "https://input.tcell-preview.io/api/v1"
+    config.agent_log_dir = "---path-to-logs-dir---"
+    config.agent_home_dir = "--path-to-tcell-dir---"
+    config.tcell_api_url = ""
+    config.tcell_input_url = ""
     config.allow_payloads = true
-    # Removed for ruby 2.0
-    # config.agent_home_owner = "boris"
-    config.enabled = true
-    config.logging_options = {"enabled" => true, "level" => "DEBUG"}
-    config.host_identifier = "foomyserver"
-    config.log_tag = "TCELL_LOGS"
-    # Removed for ruby 2.0
-    # config.logger can no longer be supported after rubyagent v1.1.4 since logging is now handled by libtcellagent
-    # config.logger = Rails.logger
-    # or
-    # config.logger = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
   end
 end
 ```
 
-Heroku Deployments - Rails apps are commonly deployed to Heroku. The biggest impact that has on the rubyagent is the agent log file. Heroku doesn't provide access to the filesystem, so the rubyagent needs the ability to log to STDOUT instead of a file. `log_tag` and `logger` settings in configuration were introduced to support this.
-
-Log Tag (see above for example `log_tag` set in config) - Since rubyagent log can be printed to STDOUT along with all the other logging, `log_tag` is a custom string added to every log line to be able to filter just tcell log lines.
-
-Logger object - Up until rubyagent v1.1.4 you could set a Rails.logger for the rubyagent to use for its logging. With the migration to have libtcellagent handle logging, that use case can no longer be supported. This feature was mainly used to get the rubyagent to log to STDOUT in heroku deployments. So as long as the agent supports some config to allow to log to STDOUT, this won't be missed.
-
-## Installation
-
-Download the GEM file and unpack it:
+## Troubleshooting
 
-    $ curl -O https://s3-us-west-2.amazonaws.com/tcell-agent-download/duvm4dj/tcell_agent-0.2.0.gem
-    
-In your rails directory
-    
-    $ mv tcell_agent-0.2.0.gem vendor/cache/
-    $ bundle install
-
-Add this line to your application's Gemfile:
-
-
-```ruby 
-gem 'tcell_agent', '0.2.0'
-```
-or if you're using the repository directly.
-```ruby
-gem "tcell_agent", :path => "<path to your tcell_agent repo>"
+Log level can be customized in `tcell_agent.config` by adding a block such as:
 ```
-
-And then execute:
-
-    $ bundle
-
-## Usage
-
-You can download the config file from the Agents section of the application then move it to the config directory
-
-    $ cp ~/Downloads/tcell_agent.config config/
-
-Or run the helper command
-
-    $ bundle exec tcell_agent setup
-
-Or if running from the repo:
-
-    $ bundle exec <path to repo>/rubyagent-tcell/bin/tcell_agent 
-
+"logging_options": {
+    "enabled": true,
+    "level": "INFO"
+}
+```

data/bin/tcell_agent

@@ -257,33 +257,45 @@ elsif command == 'test'
 
   printf '%-50s', 'Requiring configuration library... '
   require 'tcell_agent/configuration'
-  require 'tcell_agent/api'
   puts 'passed'
 
-  printf '%-50s', 'Make test API call for policies... '
-  api = TCellAgent::TCellApi.new
-  if api.poll_api
-    puts 'passed'
-  else
+  printf '%-50s', 'Loading native library... '
+  require 'tcell_agent/rust/native_library'
+  unless TCellAgent::Rust::NativeLibrary.common_lib_available?
     puts 'failed'
     Kernel.exit(1)
   end
+  puts 'passed'
 
-  printf '%-50s', 'Sending a Test event... '
-  send_succeeded = api.send_event_set([])
-  unless send_succeeded
+  printf '%-50s', 'Make test API call for policies... '
+  require 'tcell_agent/rust/native_agent'
+  errors = TCellAgent::Rust::NativeAgent.test_policies
+  if !errors.empty?
     puts 'failed'
+    puts errors
     Kernel.exit(1)
+  else
+    puts 'passed'
   end
-  puts 'passed'
 
-  printf '%-50s', 'Loading native library... '
-  require 'tcell_agent/rust/whisperer'
-  unless TCellAgent::Rust::Wrapper.common_lib_available?
+  printf '%-50s', 'Sending a Test event... '
+  require 'tcell_agent/logger'
+  require 'tcell_agent/sensor_events/server_agent'
+  errors = TCellAgent::Rust::NativeAgent.test_event_sender(
+    [
+      TCellAgent::SensorEvents::ServerAgentDetailsLanguageEvent.new(
+        'Ruby',
+        RUBY_VERSION
+      )
+    ]
+  )
+  if !errors.empty?
     puts 'failed'
+    puts errors
     Kernel.exit(1)
+  else
+    puts 'passed'
   end
-  puts 'passed'
 
   puts
   puts 'all tests passed, looks good.'

data/lib/tcell_agent.rb

@@ -1,20 +1,26 @@
 # See the file "LICENSE" for the full license governing this code.
 
-require 'tcell_agent/utils/passwords'
-require 'tcell_agent/utils/strings'
-require 'tcell_agent/utils/io'
 require 'tcell_agent/logger'
+require 'tcell_agent/utils/strings'
 require 'tcell_agent/configuration'
 
 require 'tcell_agent/agent'
 
-require 'tcell_agent/policies/http_tx_policy'
-require 'tcell_agent/policies/http_redirect_policy'
-require 'tcell_agent/policies/login_fraud_policy'
-require 'tcell_agent/policies/dataloss_policy'
-
-require 'tcell_agent/sensor_events/dlp'
 require 'tcell_agent/sensor_events/util/sanitizer_utilities'
 
 require 'tcell_agent/instrumentation'
-require 'tcell_agent/start_background_thread'
+
+require 'tcell_agent/instrument_servers'
+
+if !TCellAgent.configuration.disable_all && TCellAgent.configuration.should_instrument?
+  require 'tcell_agent/instrumentation/cmdi'
+  require 'tcell_agent/instrumentation/lfi'
+  require 'tcell_agent/instrumentation/monkey_patches/io'
+  require 'tcell_agent/instrumentation/monkey_patches/file'
+  require 'tcell_agent/instrumentation/monkey_patches/kernel'
+
+  require 'tcell_agent/hooks/login_fraud'
+  require 'tcell_agent/rails/on_start' if defined?(Rails)
+  # sinatra used to be supported, but dropped support due to no customers using it
+  # require 'tcell_agent/sinatra' if defined?(Sinatra)
+end

data/lib/tcell_agent/agent.rb

@@ -2,120 +2,44 @@
 
 require 'tcell_agent/logger'
 require 'tcell_agent/version'
-require 'tcell_agent/api'
 require 'tcell_agent/configuration'
 
 require 'tcell_agent/sensor_events/server_agent'
-require 'tcell_agent/utils/queue_with_timeout'
 
-require 'tcell_agent/agent/event_processor'
-require 'tcell_agent/agent/policy_manager'
+require 'tcell_agent/policies/policy_types'
+require 'tcell_agent/policies/policies_manager'
+require 'tcell_agent/policies/policy_polling'
 require 'tcell_agent/agent/static_agent'
-require 'tcell_agent/agent/policy_types'
 require 'tcell_agent/agent/route_manager'
-require 'tcell_agent/agent/fork_pipe_manager'
 
 require 'tcell_agent/routes/table'
 
-require 'net/http'
-require 'thread'
-require 'logger'
+require 'tcell_agent/settings_reporter'
+require 'tcell_agent/rust/native_agent'
+
 require 'json'
-require 'monitor'
 
 module TCellAgent
   class Agent
-    attr_accessor :start_pid
-    attr_accessor :event_queue
-
-    attr_accessor :fork_event_queue
-    attr_accessor :fork_event_thread
-    attr_accessor :fork_event_thread_mutex
-
-    attr_accessor :metrics_event_queue
-    attr_accessor :metrics_event_thread
-    attr_accessor :metrics_event_thread_mutex
-
-    attr_accessor :policies
-    attr_accessor :eventProcessorThread
-    attr_accessor :response_time_table
-    attr_accessor :route_table
-
-    attr_accessor :event_processor_thread
-    attr_accessor :event_processor
-    attr_accessor :worker_mutex
-
-    attr_accessor :policy_polling_thread
-    attr_accessor :policy_polling_worker_mutex
-
-    attr_accessor :event_queue_monitor
-    attr_accessor :event_dispatch_monitor
-
-    attr_accessor :stop_agent
-    attr_accessor :complete_policy_cache
-
-    def initialize(start_pid = Process.pid)
-      @start_pid = start_pid
-      @dispatch_events_timeout = TCellAgent.configuration.event_time_limit_seconds || 55
-      @dispatch_events_limit = TCellAgent.configuration.event_batch_size_limit || 20
-      @worker_mutex = Mutex.new
-      @policy_polling_worker_mutex = Mutex.new
-      @@policy_tapi = TCellApi.new
-
-      # Agent request thread
-      @policies = {
-        TCellAgent::PolicyTypes::RUST => TCellAgent::Policies::RustPolicies.new
-      }
-      @lock = Monitor.new
-
-      initialize_processor_variables
-
-      if TCellAgent.configuration.preload_policy_filename
-        TCellAgent.logger.info('Preloading a policy file')
-        begin
-          policy_file = File.open(TCellAgent.configuration.preload_policy_filename).read
-          policy_jsons = JSON.parse(policy_file)
-          policy_jsons = policy_jsons['result'] if policy_jsons.key?('result')
-          process_policy_json(policy_jsons, false)
-        rescue StandardError => e
-          TCellAgent.logger.error(e.message)
-        end
-      end
-      cached_policies = policies_from_cachefile
-      process_policy_json(cached_policies, false) if cached_policies
-    end
+    include TCellAgent::ModuleLoggerAccess
 
-    def initialize_processor_variables
-      @complete_policy_cache = {}
+    attr_accessor :route_table,
+                  :stop_agent,
+                  :safe_to_check_cmdi
 
+    def initialize
       @stop_agent = false
+      @native_agent = nil
       @route_table = TCellAgent::Routes::RouteTable.new
-
-      @event_queue_monitor = Monitor.new
-      @event_dispatch_monitor = Monitor.new
-      @mutex = Monitor.new
-
-      @response_time_table = {}
-      @sessions_metrics = TCellAgent::SensorEvents::SessionsMetric.new
-      @sessions_metrics_mutex = Monitor.new
-
-      @dispatch_events = []
-      @event_queue = BoundedQueue.new(200)
-
-      @fork_event_queue = Queue.new
-      @fork_event_thread_mutex = Monitor.new
-
-      @metrics_event_queue = Queue.new
-      @metrics_event_thread_mutex = Monitor.new
+      @safe_to_check_cmdi = false
+      @policies_manager = PoliciesManager.new(nil)
     end
 
-    def parent_process?
-      @start_pid == Process.pid
-    end
-
-    def start
+    def validate_config
       if TCellAgent::Utils::Strings.blank?(TCellAgent.configuration.api_key) ||
-         TCellAgent::Utils::Strings.blank?(TCellAgent.configuration.app_id)
+         TCellAgent::Utils::Strings.blank?(TCellAgent.configuration.app_id) ||
+         TCellAgent::Utils::Strings.blank?(TCellAgent.configuration.tcell_input_url) ||
+         TCellAgent::Utils::Strings.blank?(TCellAgent.configuration.tcell_api_url)
         puts ' ********* ********* ********* *********'
         puts '* tCell.io                               *'
         puts '* Configuration info is missing, you may  *'
@@ -123,13 +47,70 @@ module TCellAgent
         puts '* it in the config/ directory            *'
         puts ' ********* ********* ********* *********'
         TCellAgent.configuration.enabled = false
+      end
+    end
+
+    def start(server_name)
+      TCellAgent.thread_agent.validate_config
+      return unless TCellAgent.configuration.should_instrument?
+
+      @native_agent = TCellAgent::Rust::NativeAgent.create_agent(
+        TCellAgent.configuration
+      )
+      if @native_agent.nil?
+        TCellAgent.configuration.enabled = false
         return
       end
 
-      TCellAgent.logger.debug('Starting thread agent')
+      TCellAgent.native_agent = @native_agent
+      @policies_manager = PoliciesManager.new(@native_agent)
+      # if preload_policy_filename is used and policy polling is
+      # disabled, need to call poll policies to make sure
+      # ruby policies are in sync with native agent enablements
+      result = @native_agent.poll_new_policies
+      policies_and_enablements = result['new_policies_and_enablements'] || {}
+      @policies_manager.process_policy_json(
+        policies_and_enablements['enablements'],
+        policies_and_enablements['policies']
+      )
+
+      @policy_polling = PolicyPolling.new(@policies_manager, @native_agent)
+
+      module_logger.info("Starting thread agent: #{server_name}")
+
+      @safe_to_check_cmdi = true
+
+      TCellAgent.report_settings
+      TCellAgent::Instrumentation::Rails.send_framework_info
+      TCellAgent::Instrumentation::Rails.send_settings
+    rescue StandardError => standard_error
+      TCellAgent.configuration.enabled = false
+      module_logger.error("Error starting agent: (#{standard_error.class}) #{standard_error.message}")
+      module_logger.exception(standard_error)
+    end
+
+    def policies
+      @policies_manager.policies
+    end
+
+    def report_metrics(request_time, tcell_context)
+      @native_agent.report_metrics(
+        request_time, tcell_context
+      )
+    rescue StandardError => standard_error
+      module_logger.error("Error reporting metric: (#{standard_error.class}) #{standard_error.message}")
+      module_logger.exception(standard_error)
+    end
+
+    def queue_sensor_event(event)
+      return unless @native_agent
 
-      ensure_policy_polling_running
-      ensure_event_processor_running
+      @native_agent.send_sanitized_events(
+        [event]
+      )
+    rescue StandardError => standard_error
+      module_logger.error("Error sending event: (#{standard_error.class}) #{standard_error.message}")
+      module_logger.exception(standard_error)
     end
   end
 end