appmap 0.41.0 → 0.43.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d747c0e65dc1efb45769ae9118a7b97015b0378d79e20eec855e74c20b622af2
-  data.tar.gz: dcc3b91f32246aaefd02fbcf47672bfdf13acf6cadb715fbc5ff0040dbcfbf72
+  metadata.gz: 4007f0eb67a3a25088e8b3677de8d0388fd1bfec4b326c43a51c503f640325b5
+  data.tar.gz: 1b593251a7a072b7a1483e7af172b9157d6913208658fb324630bc52b0b4fe73
 SHA512:
-  metadata.gz: 5f82a6a0cf8075537b0835e783779c827c5afed81e83e242bc97cbfd5b6ef0320fe352235436f57009a63db321223b9f7f82e8bcdced558d9b935a2ce4e4cbf3
-  data.tar.gz: aaa4676d24b6bb1bcbd34fb7f0e3a0244d005c36dd89782b774b342b7e30e1c9637795d0d492522b2a5e525de413e3adb9da54bd5ace2699373ffdfadf3a8157
+  metadata.gz: 6af64a17bf69655ae8bdc9b4b67b7e2c9bd1b87d07a23e266dcabd2a927311224a4b27a77355674559828a08ae1df2e7231e2a68a2734c0f620de8efc48ec6d4
+  data.tar.gz: 7725cb0c79d8be13fddf7a3b688113f4fd31941c4f08f64ee44d4fbb3f5255dd3a308b08ef578413938052ee3b9eeecedc3e58bc092b12e23bfa872eac119eb7

data/CHANGELOG.md

@@ -1,3 +1,31 @@
+# v0.43.0
+
+* Record `name` and `class` of each entry in Hash-like parameters, messages, and return values.
+* Record client-sent headers in HTTP server request and response.
+* Record HTTP server request `mime_type`.
+* Record HTTP server request `authorization`.
+
+# v0.42.1
+
+* Add missing require `set`.
+* Check `cls.respond_to?(:singleton_class)`, since it oddly, may not.
+
+# v0.42.0
+
+* Remove `feature_group` and `feature` metadata from minitest and RSpec AppMaps.
+* Add `metadata.source_location`.
+
+# v0.41.2
+
+* Don't rely on `gemspec.source_paths` to list all the source locations in a gem. Hook any code that's loaded
+  from inside the `gem_dir`.
+
+# v0.41.1
+
+* Make best effort to ensure that class name is not `null` in the appmap.json.
+* Don't try and instrument gems which are a dependency of the this gem.
+* Fix a nil exception when applying the exclude list to builtins.
+
 # v0.41.0
 
 * Adjust some label names to match `provider.*`, `format.*`.

data/README.md

@@ -9,6 +9,7 @@
   - [Minitest](#minitest)
   - [Cucumber](#cucumber)
   - [Remote recording](#remote-recording)
+  - [Server process recording](#server-process-recording)
 - [AppMap for VSCode](#appmap-for-vscode)
 - [Uploading AppMaps](#uploading-appmaps)
 - [Development](#development)
@@ -81,6 +82,22 @@ If you are using Ruby on Rails, require the railtie after Rails is loaded.
 require 'appmap/railtie' if defined?(AppMap).
 ```
 
+**application.rb**
+
+Add this line to *application.rb*, to enable server recording with `APPMAP_RECORD=true`:
+
+```ruby
+module MyApp
+  class Application < Rails::Application
+    ...
+  
+    config.appmap.enabled = true if ENV['APPMAP_RECORD']
+    
+    ...
+  end
+end
+```
+
 # Configuration
 
 When you run your program, the `appmap` gem reads configuration settings from `appmap.yml`. Here's a sample configuration
@@ -306,6 +323,12 @@ $ bundle exec rails server
 
 6. Open the AppLand browser extension and push `Stop`. The recording will be transferred to the AppLand website and opened in your browser.
 
+## Server process recording
+
+Run your Rails server with `APPMAP_RECORD=true`. When the server exits, an *appmap.json* file will be written to the project directory. This is a great way to start the server, interact with your app as a user (or through it's API), and then view an AppMap of everything that happened.
+
+Be sure and set `WEB_CONCURRENCY=1`, if you are using a webserver that can run multiple processes. You only want there to be one process while you are recording, otherwise they will both try and write *appmap.json* and one of them will clobber the other.
+
 # AppMap for VSCode
 
 The [AppMap extension for VSCode](https://marketplace.visualstudio.com/items?itemName=appland.appmap) is a great way to onboard developers to new code, and troubleshoot hard-to-understand bugs with visuals.
@@ -317,7 +340,7 @@ The [AppMap extension for VSCode](https://marketplace.visualstudio.com/items?ite
 For instructions on uploading, see the documentation of the [AppLand CLI](https://github.com/applandinc/appland-cli).
 
 # Development
-[![Build Status](https://travis-ci.org/applandinc/appmap-ruby.svg?branch=master)](https://travis-ci.org/applandinc/appmap-ruby)
+[![Build Status](https://travis-ci.com/applandinc/appmap-ruby.svg?branch=master)](https://travis-ci.com/applandinc/appmap-ruby)
 
 ## Running tests
 
@@ -346,7 +369,7 @@ The fixture apps in `test/fixtures` are plain Ruby projects that exercise the ba
 
 ### `spec/fixtures`
 
-The fixture apps in `spec/fixtures` are simple Rack, Rails4, and Rails5 apps.
+The fixture apps in `spec/fixtures` are simple Rack, Rails5, and Rails6 apps.
 You can use them to interactively develop and test the recording features of the `appmap` gem.
 These fixture apps are more sophisticated than `test/fixtures`, because they include additional 
 resources such as a PostgreSQL database.
@@ -372,11 +395,15 @@ $ docker-compose run --rm app ./create_app
 Now you can start a development container.
 
 ```sh-session
-$ docker-compose run --rm -v $PWD/../../..:/src/appmap-ruby app bash
+$ docker-compose run --rm -v $PWD:/app -v $PWD/../../..:/src/appmap-ruby app bash
 Starting rails_users_app_pg_1 ... done
-root@6fab5f89125f:/app# cd /src/app
+root@6fab5f89125f:/app# cd /src/appmap-ruby
+root@6fab5f89125f:/src/appmap-ruby# rm ext/appmap/*.so ext/appmap/*.o
+root@6fab5f89125f:/src/appmap-ruby# bundle
+root@6fab5f89125f:/src/appmap-ruby# bundle exec rake compile
+root@6fab5f89125f:/src/appmap-ruby# cd /src/app
 root@6fab5f89125f:/src/app# bundle config local.appmap /src/appmap-ruby
-root@6fab5f89125f:/src/app# bundle update appmap
+root@6fab5f89125f:/src/app# bundle
 ```
 
 At this point, the bundle is built with the `appmap` gem located  in `/src/appmap`, which is volume-mounted from the host.
@@ -391,4 +418,3 @@ Configuring AppMap from path appmap.yml
 Finished in 0.07357 seconds (files took 2.1 seconds to load)
 4 examples, 0 failures
 ```
-

data/lib/appmap.rb

@@ -50,6 +50,11 @@ module AppMap
       end
     end
 
+    # Whether to include source and comments in all class maps.
+    def include_source?
+      ENV['APPMAP_SOURCE'] == 'true'
+    end
+
     # Used to start tracing, stop tracing, and record events.
     def tracing
       @tracing ||= Trace::Tracing.new

data/lib/appmap/command/record.rb

@@ -27,7 +27,7 @@ module AppMap
           event_thread.join
           yield AppMap::APPMAP_FORMAT_VERSION,
                 AppMap.detect_metadata,
-                AppMap.class_map(tracer.event_methods),
+                AppMap.class_map(tracer.event_methods, include_source: AppMap.include_source?),
                 events
         end
 

data/lib/appmap/config.rb

@@ -16,20 +16,20 @@ module AppMap
         end
 
         def build_from_gem(gem, shallow: true, package_name: nil, exclude: [], labels: [])
-          gem_paths(gem).map do |gem_path|
-            Package.new(gem_path, gem, package_name, exclude, labels, shallow)
+          if %w[method_source activesupport].member?(gem)
+            warn "WARNING: #{gem} cannot be AppMapped because it is a dependency of the appmap gem"
+            return
           end
+          Package.new(gem_path(gem), gem, package_name, exclude, labels, shallow)
         end
 
         private_class_method :new
 
         protected
 
-        def gem_paths(gem)
+        def gem_path(gem)
           gemspec = Gem.loaded_specs[gem] or raise "Gem #{gem.inspect} not found"
-          gemspec.source_paths.map do |path|
-            File.join(gemspec.gem_dir, path)
-          end
+          gemspec.gem_dir
         end
       end
 
@@ -57,8 +57,12 @@ module AppMap
     # Methods that should always be hooked, with their containing
     # package and labels that should be applied to them.
     HOOKED_METHODS = {
-      'ActiveSupport::SecurityUtils' => Hook.new(:secure_compare, Package.build_from_path('active_support', package_name: 'active_support', labels: %w[provider.secure_compare])),
-      'ActionView::Renderer' => Hook.new(:render, Package.build_from_path('action_view', package_name: 'action_view', labels: %w[mvc.view]))
+      'ActiveSupport::SecurityUtils' => Hook.new(:secure_compare, Package.build_from_path('active_support', labels: %w[provider.secure_compare])),
+      'ActionView::Renderer' => Hook.new(:render, Package.build_from_path('action_view', labels: %w[mvc.view])),
+      'ActionDispatch::Cookies::CookieJar' => Hook.new(%i[[]= clear update delete recycle], Package.build_from_path('action_pack', labels: %w[provider.http.cookie])),
+      'ActionDispatch::Cookies::EncryptedCookieJar' => Hook.new(%i[[]=], Package.build_from_path('action_pack', labels: %w[provider.http.cookie crypto])),
+      'CanCan::ControllerAdditions' => Hook.new(%i[authorize! can? cannot?], Package.build_from_path('cancancan', labels: %w[provider.authorization])),
+      'CanCan::Ability' => Hook.new(%i[authorize!], Package.build_from_path('cancancan', labels: %w[provider.authorization])),
     }.freeze
 
     BUILTIN_METHODS = {
@@ -71,7 +75,7 @@ module AppMap
       'Net::SMTP' => Hook.new(:send, Package.build_from_path('net/smtp', package_name: 'net/smtp', labels: %w[protocol.smtp protocol.email io])),
       'Net::POP3' => Hook.new(:mails, Package.build_from_path('net/pop3', package_name: 'net/pop', labels: %w[protocol.pop protocol.email io])),
       'Net::IMAP' => Hook.new(:send_command, Package.build_from_path('net/imap', package_name: 'net/imap', labels: %w[protocol.imap protocol.email io])),
-      'Marshal' => Hook.new(%i[dump load], Package.build_from_path('marshal', labels: %w[format.marshal provider.serialization marshal])),
+      'Marshal' => Hook.new(%i[dump load], Package.build_from_path('marshal', labels: %w[format.marshal provider.serialization])),
       'Psych' => Hook.new(%i[dump dump_stream load load_stream parse parse_stream], Package.build_from_path('yaml', package_name: 'psych', labels: %w[format.yaml provider.serialization])),
       'JSON::Ext::Parser' => Hook.new(:parse, Package.build_from_path('json', package_name: 'json', labels: %w[format.json provider.serialization])),
       'JSON::Ext::Generator::State' => Hook.new(:generate, Package.build_from_path('json', package_name: 'json', labels: %w[format.json provider.serialization])),
@@ -105,9 +109,9 @@ module AppMap
             shallow = true if shallow.nil?
             Package.build_from_gem(gem, exclude: package['exclude'] || [], shallow: shallow)
           else
-            [ Package.build_from_path(path, exclude: package['exclude'] || [], shallow: package['shallow']) ]
+            Package.build_from_path(path, exclude: package['exclude'] || [], shallow: package['shallow'])
           end
-        end.flatten
+        end.compact
         Config.new config_data['name'], packages, config_data['exclude'] || []
       end
     end
@@ -137,7 +141,7 @@ module AppMap
       return unless location_file
 
       location_file = location_file[Dir.pwd.length + 1..-1] if location_file.index(Dir.pwd) == 0
-      packages.find do |pkg|
+      packages.select { |pkg| pkg.path }.find do |pkg|
         (location_file.index(pkg.path) == 0) &&
           !pkg.exclude.find { |p| location_file.index(p) }
       end

data/lib/appmap/event.rb

@@ -36,8 +36,29 @@ module AppMap
           (value_string||'')[0...LIMIT].encode('utf-8', invalid: :replace, undef: :replace, replace: '_')
         end
 
+        def object_properties(hash_like)
+          hash = hash_like.to_h
+          hash.keys.map do |key|
+            {
+              name: key,
+              class: hash[key].class.name,
+            }
+          end
+        rescue
+          nil
+        end
+
         protected
 
+        # Heuristic for dynamically defined class whose name can be nil
+        def best_class_name(value)
+          value_cls = value.class
+          while value_cls.name.nil?
+            value_cls = value_cls.superclass
+          end
+          value_cls.name
+        end
+
         def custom_display_string(value)
           case value
           when File
@@ -70,6 +91,12 @@ module AppMap
           end
         end
       end
+
+      protected
+
+      def object_properties(hash_like)
+        self.class.object_properties(hash_like)
+      end
     end
 
     class MethodCall < MethodEvent
@@ -77,6 +104,7 @@ module AppMap
 
       class << self
         def build_from_invocation(mc = MethodCall.new, defined_class, method, receiver, arguments)
+          defined_class ||= 'Class'
           mc.tap do
             static = receiver.is_a?(Module)
             mc.defined_class = defined_class
@@ -110,14 +138,14 @@ module AppMap
                 end
               {
                 name: param_name,
-                class: value.class.name,
+                class: best_class_name(value),
                 object_id: value.__id__,
                 value: display_string(value),
                 kind: param_type
               }
             end
             mc.receiver = {
-              class: receiver.class.name,
+              class: best_class_name(receiver),
               object_id: receiver.__id__,
               value: display_string(receiver)
             }
@@ -172,7 +200,7 @@ module AppMap
           mr.tap do |_|
             if return_value
               mr.return_value = {
-                class: return_value.class.name,
+                class: best_class_name(return_value),
                 value: display_string(return_value),
                 object_id: return_value.__id__
               }
@@ -183,7 +211,7 @@ module AppMap
               while next_exception
                 exception_backtrace = next_exception.backtrace_locations.try(:[], 0)
                 exceptions << {
-                  class: next_exception.class.name,
+                  class: best_class_name(next_exception),
                   message: next_exception.message,
                   object_id: next_exception.__id__,
                   path: exception_backtrace&.path,

data/lib/appmap/hook.rb

@@ -46,7 +46,16 @@ module AppMap
         cls = trace_point.self
 
         instance_methods = cls.public_instance_methods(false) - OBJECT_INSTANCE_METHODS
-        class_methods = cls.singleton_class.public_instance_methods(false) - instance_methods - OBJECT_STATIC_METHODS
+        # NoMethodError: private method `singleton_class' called for Rack::MiniProfiler:Class
+        class_methods = begin
+          if cls.respond_to?(:singleton_class)
+            cls.singleton_class.public_instance_methods(false) - instance_methods - OBJECT_STATIC_METHODS
+          else
+            []
+          end
+        rescue NameError
+          []
+        end
 
         hook = lambda do |hook_cls|
           lambda do |method_id|
@@ -80,7 +89,13 @@ module AppMap
         end
 
         instance_methods.each(&hook.(cls))
-        class_methods.each(&hook.(cls.singleton_class))
+        # NoMethodError: private method `singleton_class' called for Rack::MiniProfiler:Class
+        begin
+          class_methods.each(&hook.(cls.singleton_class)) if cls.respond_to?(:singleton_class)
+        rescue NameError
+          # NameError:
+          #   uninitialized constant Faraday::Connection
+        end
       end
 
       tp.enable(&block)

data/lib/appmap/hook/method.rb

@@ -35,7 +35,7 @@ module AppMap
                 else
                   "#{hook_method.name} (class resolution deferred)"
                 end
-          warn "AppMap: Hooking " + msg
+          warn "AppMap: Hooking #{msg} at line #{(hook_method.source_location || []).join(':')}"
         end
 
         defined_class = @defined_class

data/lib/appmap/middleware/remote_recording.rb

@@ -67,7 +67,7 @@ module AppMap
 
         response = JSON.generate \
           version: AppMap::APPMAP_FORMAT_VERSION,
-          classMap: AppMap.class_map(tracer.event_methods),
+          classMap: AppMap.class_map(tracer.event_methods, include_source: AppMap.include_source?),
           metadata: metadata,
           events: @events
 

data/lib/appmap/minitest.rb

@@ -7,20 +7,25 @@ module AppMap
   # be activated around each test.
   module Minitest
     APPMAP_OUTPUT_DIR = 'tmp/appmap/minitest'
-    LOG = false
+    LOG = ( ENV['APPMAP_DEBUG'] == 'true' || ENV['DEBUG'] == 'true' )
 
     def self.metadata
       AppMap.detect_metadata
     end
 
-    Recording = Struct.new(:test) do
-      def initialize(test)
+    Recording = Struct.new(:test, :test_name) do
+      def initialize(test, test_name)
         super
 
-        warn "Starting recording of test #{test.class}.#{test.name}" if AppMap::Minitest::LOG
+        warn "Starting recording of test #{test.class}.#{test.name}@#{source_location}" if AppMap::Minitest::LOG
         @trace = AppMap.tracing.trace
       end
 
+      def source_location
+        test.method(test_name).source_location.join(':')
+      end
+
+
       def finish
         warn "Finishing recording of test #{test.class}.#{test.name}" if AppMap::Minitest::LOG
 
@@ -31,7 +36,7 @@ module AppMap
 
         AppMap::Minitest.add_event_methods @trace.event_methods
 
-        class_map = AppMap.class_map(@trace.event_methods)
+        class_map = AppMap.class_map(@trace.event_methods, include_source: AppMap.include_source?)
 
         feature_group = test.class.name.underscore.split('_')[0...-1].join('_').capitalize
         feature_name = test.name.split('_')[1..-1].join(' ')
@@ -39,9 +44,8 @@ module AppMap
 
         AppMap::Minitest.save scenario_name,
                               class_map,
-                              events: events,
-                              feature_name: feature_name,
-                              feature_group_name: feature_group
+                              source_location,
+                              events: events
       end
     end
 
@@ -55,8 +59,8 @@ module AppMap
         FileUtils.mkdir_p APPMAP_OUTPUT_DIR
       end
 
-      def begin_test(test)
-        @recordings_by_test[test.object_id] = Recording.new(test)
+      def begin_test(test, name)
+        @recordings_by_test[test.object_id] = Recording.new(test, name)
       end
 
       def end_test(test)
@@ -74,12 +78,11 @@ module AppMap
         @event_methods += event_methods
       end
 
-      def save(example_name, class_map, events: nil, feature_name: nil, feature_group_name: nil, labels: nil)
+      def save(example_name, class_map, source_location, events: nil, labels: nil)
         metadata = AppMap::Minitest.metadata.tap do |m|
           m[:name] = example_name
+          m[:source_location] = source_location
           m[:app] = AppMap.configuration.name
-          m[:feature] = feature_name if feature_name
-          m[:feature_group] = feature_group_name if feature_group_name
           m[:frameworks] ||= []
           m[:frameworks] << {
             name: 'minitest',
@@ -128,7 +131,7 @@ if AppMap::Minitest.enabled?
     alias run_without_hook run
 
     def run
-      AppMap::Minitest.begin_test self
+      AppMap::Minitest.begin_test self, name
       begin
         run_without_hook
       ensure

data/lib/appmap/rails/request_handler.rb

@@ -6,15 +6,38 @@ require 'appmap/hook'
 module AppMap
   module Rails
     module RequestHandler
+      # Host and User-Agent will just introduce needless variation.
+      # Content-Type and Authorization get their own fields in the request.
+      IGNORE_HEADERS = %w[host user_agent content_type authorization].map(&:upcase).map {|h| "HTTP_#{h}"}.freeze
+
+      class << self
+        def selected_headers(env)
+          # Rack prepends HTTP_ to all client-sent headers.
+          matching_headers = env
+            .select { |k,v| k.start_with? 'HTTP_'}
+            .reject { |k,v| IGNORE_HEADERS.member?(k) }
+            .reject { |k,v| v.blank? }
+            .each_with_object({}) do |kv, memo|
+              key = kv[0].sub(/^HTTP_/, '').split('_').map(&:capitalize).join('-')
+              value = kv[1]
+              memo[key] = value
+            end
+          matching_headers.blank? ? nil : matching_headers
+        end
+      end
+
       class HTTPServerRequest < AppMap::Event::MethodEvent
-        attr_accessor :normalized_path_info, :request_method, :path_info, :params
+        attr_accessor :normalized_path_info, :request_method, :path_info, :params, :mime_type, :headers, :authorization
 
         def initialize(request)
           super AppMap::Event.next_id_counter, :call, Thread.current.object_id
 
-          @request_method = request.request_method
-          @normalized_path_info = normalized_path request
-          @path_info = request.path_info.split('?')[0]
+          self.request_method = request.request_method
+          self.normalized_path_info = normalized_path(request)
+          self.mime_type = request.headers['Content-Type']
+          self.headers = RequestHandler.selected_headers(request.env)
+          self.authorization = request.headers['Authorization']
+          self.path_info = request.path_info.split('?')[0]
           # ActionDispatch::Http::ParameterFilter is deprecated
           parameter_filter_cls = \
             if defined?(ActiveSupport::ParameterFilter)
@@ -22,7 +45,7 @@ module AppMap
             else
               ActionDispatch::Http::ParameterFilter
             end
-          @params = parameter_filter_cls.new(::Rails.application.config.filter_parameters).filter(request.params)
+          self.params = parameter_filter_cls.new(::Rails.application.config.filter_parameters).filter(request.params)
         end
 
         def to_h
@@ -30,7 +53,10 @@ module AppMap
             h[:http_server_request] = {
               request_method: request_method,
               path_info: path_info,
-              normalized_path_info: normalized_path_info
+              mime_type: mime_type,
+              normalized_path_info: normalized_path_info,
+              authorization: authorization,
+              headers: headers,
             }.compact
 
             h[:message] = params.keys.map do |key|
@@ -39,8 +65,11 @@ module AppMap
                 name: key,
                 class: val.class.name,
                 value: self.class.display_string(val),
-                object_id: val.__id__
-              }
+                object_id: val.__id__,
+              }.tap do |message|
+                properties = object_properties(val)
+                message[:properties] = properties if properties
+              end
             end
           end
         end
@@ -59,7 +88,7 @@ module AppMap
       end
 
       class HTTPServerResponse < AppMap::Event::MethodReturnIgnoreValue
-        attr_accessor :status, :mime_type
+        attr_accessor :status, :mime_type, :headers
 
         def initialize(response, parent_id, elapsed)
           super AppMap::Event.next_id_counter, :return, Thread.current.object_id
@@ -68,13 +97,15 @@ module AppMap
           self.mime_type = response.headers['Content-Type']
           self.parent_id = parent_id
           self.elapsed = elapsed
+          self.headers = RequestHandler.selected_headers(response.headers)
         end
 
         def to_h
           super.tap do |h|
             h[:http_server_response] = {
               status: status,
-              mime_type: mime_type
+              mime_type: mime_type,
+              headers: headers
             }.compact
           end
         end