piggly-nsd 2.3.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b53364d89c087bf4c2ce49ec451ece2cb1c17694bab9e765e034a05493ef3999
+  data.tar.gz: a46eea287041eeaa4130b730dd409f1f35663e6fca5258e96c639b1f567eeb11
+SHA512:
+  metadata.gz: 1cfd022ed8e03c56d372911943bd7b662a6ff967f63bde7b09a01e010b5117bc41ad0f5c245a30e72139776a5eece380ab9d2d60175462555a874cfb008e73c8
+  data.tar.gz: b8653bb06b92d31eee24e5c9a317fe912dc61e01d46fae649dadeb4610b831f4ad50199683996cddc756727124be68514fc0d919b372d80f70347851539d7751

data/README.md

@@ -0,0 +1,170 @@
+[![CI](https://github.com/sergeiboikov/piggly/actions/workflows/ci.yml/badge.svg)](https://github.com/sergeiboikov/piggly/actions/workflows/ci.yml)
+# Piggly-NSD
+
+Code coverage reports for PostgreSQL PL/pgSQL stored procedures
+
+**Note:** This is a maintained fork of the original [piggly](https://github.com/kputnam/piggly) project by Kyle Putnam.
+
+![Screenshot](https://kputnam.github.io/piggly/images/example.png)
+
+## Purpose
+
+PL/pgSQL doesn't have much in the way of developer tools, and writing automated tests for
+stored procedures can be much easier when you know what you haven't tested. Code coverage
+allows you to see which parts of your code haven't been executed.
+
+Piggly is a tool (written in Ruby, but you can write your tests in any language) to track
+code coverage of PostgreSQL PL/pgSQL stored procedures. It reports on code coverage to help
+you identify untested parts of your code.
+
+## How Does It Work?
+
+Piggly tracks the execution of PostgreSQL's PL/pgSQL stored procedures by recompiling
+the stored procedure with instrumentation code. Basically, RAISE WARNING statements notify the
+client of an execution event (e.g., a branch condition evaluating to true or false). It records
+these events and generates prettified source code that is annotated with coverage details.
+
+## Features
+
+* Readable and easily-navigable reports (see [example](http://kputnam.github.io/piggly/reports/index.html))
+* Language agnostic - write your tests in Ruby, Python, Java, SQL scripts etc
+* Branch, block, and loop coverage analysis
+* Instrumenting source-to-source compiler
+* Low test execution overhead
+* Reduced compilation times by use of disk caching
+* Possible to aggregate coverage across multiple runs
+
+## Limitations
+
+* Not all PL/pgSQL grammar is currently supported, but the grammar is easy to modify
+* Cannot parse nested dollar-quoted strings, eg $A$ ... $B$ ... $B$ ... $A$
+* SQL statements are not instrumented, so their branches (COALESCE, WHERE-clauses, etc) aren't tracked
+
+## Requirements
+
+* [Treetop](http://github.com/nathansobo/treetop): `gem install treetop`
+* The [ruby-pg driver](http://bitbucket.org/ged/ruby-pg/): `gem install pg`
+* The examples require ActiveRecord: `gem install activerecord`
+
+## Versioning
+
+This fork continues the version numbering from the original piggly project:
+- **2.3.1** - Last version of original piggly by Kyle Putnam
+- **2.3.2** - NSD fork with UTF-8 encoding support and updated dependencies
+
+## How to Install
+
+To install the latest from github:
+
+    $ git clone https://github.com/sergeiboikov/piggly.git
+    $ cd piggly
+    $ bundle install
+    $ bundle exec rake spec
+
+    $ gem uninstall piggly-nsd  # If piggly-nsd was installed earlier
+    $ gem build piggly.gemspec
+    $ gem install piggly-nsd-*.gem
+
+To install the latest release:
+
+    $ gem install piggly-nsd
+
+## Usage
+
+Your stored procedures must already be loaded in the database. Configure your database connection in
+a file named `config/database.yml` relative to where you want to run piggly. You can also specify the
+`-d PATH` to an existing configuration file. The contents of the file follow ActiveRecord conventions:
+
+    piggly:
+      adapter: postgresql
+      database: cookbook
+      username: kputnam
+      password: secret
+      host: localhost
+
+Here we'll add some stored procedures to the database we described above:
+
+    $ cat example/proc/*.sql | psql -U kputnam -h localhost cookbook
+
+Note the connection is expected to be named `piggly` but you may specify the `-k DATABASE` option to
+use a different connection name (eg `-k development` in Rails).  See also `example/config/database.yml`.
+
+Now you are ready to recompile and install your stored procedures. This reads the configuration from 
+`./config/database.yml` relative to the current working directory.
+
+    $ piggly trace
+    compiling 5 procedures
+    Compiling scramble
+    Compiling scramble
+    Compiling numberedargs
+    Compiling snippets
+    Compiling iterate
+    tracing 5 procedures
+
+For running the development version use the following command:
+    `ruby bin/piggly trace`
+
+This caches the original version (without instrumentation) in `piggly/cache` so you can restore them
+later. Piggly will only recompile procedures that have changed in the database since it last
+made a copy in `piggly/cache`.
+
+*WARNING*: piggly fetches your code from the database and replaces it (in the database) with the
+instrumented code. If you run `piggly trace` twice consecutively, the second time will cause an error
+because you are trying to re-instrument code that has already been instrumented. You need to run
+`piggly untrace` or restore your original stored procedures manually before you can trace them again.
+
+Now you're ready to execute your tests. Make sure your connection is configured to log `RAISE WARNING`
+messages to a file -- or you can log them to `STDERR` and redirect that to a file. For instance you
+might run:
+
+    $ ant test 2> messages.txt
+    $ make test 2> messages.txt
+    etc.
+
+To build the coverage report, have piggly read that file in by executing `piggly report < messages.txt`,
+ or `piggly report -f messages.txt`. You don't actually need the intermediate file, you can pipe your
+test suite directly in like `ant test 2>&1 | piggly report`.
+
+Once the report is built you can open it in `piggly/reports/index.html`.
+
+## Running the Examples
+
+    $ cd piggly
+    $ bundle install
+    $ cat example/README
+    ...
+
+    $ ./example/run-tests
+    compiling 5 procedures
+    Compiling scramble
+    Compiling scramble
+    Compiling numberedargs
+    Compiling snippets
+    Compiling iterate
+    tracing 5 procedures
+    Loaded suite /home/kputnam/wd/piggly/example/test/iterate_test
+    Started
+    ......
+    Finished in 0.199236 seconds.
+
+    6 tests, 6 assertions, 0 failures, 0 errors, 0 skips
+
+    Test run options: --seed 25290
+    clearing previous coverage
+    storing coverage profile
+    creating index
+    creating reports
+    reporting coverage for scramble
+    reporting coverage for scramble
+    reporting coverage for numberedargs
+    reporting coverage for snippets
+    reporting coverage for iterate: +0.0% block, +0.0% branch, +0.0% loop
+    restoring 5 procedures
+    OK, view /home/kputnam/wd/piggly/example/piggly/reports/index.html
+
+    $ ls -alh example/reports/index.html
+    -rw-r--r-- 1 kputnam kputnam 1.4K 2010-04-28 11:21 example/reports/index.html
+
+## Bugs & Issues
+
+Please report any issues or feature requests on the [github tracker](https://github.com/sergeiboikov/piggly/issues).

data/Rakefile

@@ -0,0 +1,33 @@
+begin
+  require "rubygems"
+  require "bundler/setup"
+rescue LoadError
+  warn "couldn't load bundler:"
+  warn "  #{$!}"
+end
+
+begin # rspec-2
+  require "rspec/core/rake_task"
+  RSpec::Core::RakeTask.new do |t|
+    t.verbose    = false
+    t.pattern    = "spec/**/*_spec.rb"
+    t.rspec_opts = "--color --format=p"
+  end
+rescue LoadError => first
+  begin # rspec-1
+    require "spec/rake/spectask"
+    Spec::Rake::SpecTask.new do |t|
+      t.pattern = "spec/**/*_spec.rb"
+      t.spec_opts << "--color"
+      t.spec_opts << "--format=p"
+    end
+  rescue LoadError => second
+    task :spec do
+      warn "couldn't load rspec version 1 or 2:"
+      warn "  #{first}"
+      warn "  #{second}"
+    end
+  end
+end
+
+task :default => :spec

data/bin/piggly

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+$:.unshift File.expand_path(File.join(File.dirname(__FILE__), *%w(.. lib)))
+require "piggly"
+
+# Number of children to fork for parallel tasks
+Piggly::Util::ProcessQueue.concurrent = 2
+
+Piggly::Command::Base.main(ARGV)

data/lib/piggly/command/base.rb

@@ -0,0 +1,148 @@
+module Piggly
+  module Command
+
+    class Base
+    end
+
+    class << Base
+
+      def main(argv)
+        cmd, argv = command(argv)
+
+        if cmd.nil?
+          abort "usage: #{$0} {report|trace|untrace} --help"
+        else
+          cmd.main(argv)
+        end
+      end
+
+      # @return [(Class, Array<String>)]
+      def command(argv)
+        return if argv.empty?
+        head, *tail = argv
+
+        case head.downcase
+        when "report";  [Report,  tail]
+        when "trace";   [Trace,   tail]
+        when "untrace"; [Untrace, tail]
+        end
+      end
+
+      # @return [PG::Connection]
+      def connect(config)
+        require "pg"
+        require "erb"
+
+        files = Array(config.database_yml ||
+          %w(piggly/database.yml
+             config/database.yml
+             piggly/database.json
+             config/database.json))
+
+        path = files.find{|x| File.exist?(x) } or
+          raise "No database config files found: #{files.join(", ")}"
+
+        specs =
+          if File.extname(path) == ".json"
+            require "json"
+            JSON.load(ERB.new(IO.read(path)).result)
+          else
+            require "yaml"
+            YAML.unsafe_load(ERB.new(IO.read(path)).result)
+          end
+
+        spec = (specs.is_a?(Hash) and specs[config.connection_name]) or
+          raise "Database '#{config.connection_name}' is not configured in #{path}"
+
+        conn = PG::Connection.connect(spec["host"], spec["port"], nil, nil,
+          spec["database"], spec["username"], spec["password"])
+        
+        # Set client encoding to UTF-8 to properly handle Cyrillic and other Unicode characters
+        conn.set_client_encoding('UTF8')
+        
+        conn
+      end
+
+      # @return [Enumerable<SkeletonProcedure>]
+      def filter(config, index)
+        if config.filters.empty?
+          index.procedures
+        else
+          head, _ = config.filters
+
+          start =
+            case head.first
+            when :+; []
+            when :-; index.procedures
+            end
+
+          config.filters.inject(start) do |s, pair|
+            case pair.first
+            when :+; s | index.procedures.select(&pair.last)
+            when :-; s.reject(&pair.last)
+            end
+          end
+        end
+      end
+
+      def o_accumulate(config)
+        lambda{|x| config.accumulate = x }
+      end
+
+      def o_cache_root(config)
+        lambda{|x| config.cache_root = x }
+      end
+
+      def o_report_root(config)
+        lambda{|x| config.report_root = x }
+      end
+
+      def o_include_paths(config)
+        lambda{|x| config.include_paths.concat(x.split(":")) }
+      end
+
+      def o_database_yml(config)
+        lambda{|x| config.database_yml = x }
+      end
+
+      def o_connection_name(config)
+        lambda{|x| config.connection_name = x }
+      end
+
+      def o_version(config)
+        lambda {|x| puts "piggly #{VERSION} #{VERSION::RELEASE_DATE}"; exit! }
+      end
+
+      def o_dry_run(config)
+        lambda {|x| config.dry_run = true }
+      end
+
+      def o_select(config)
+        lambda do |x|
+          filter =
+            if m = x.match(%r{^/([^/]+)/$})
+              lambda{|p| p.name.to_s.match(m.captures.first) }
+            else
+              lambda{|p| p.name.to_s === x }
+            end
+
+          config.filters << [:+, filter]
+        end
+      end
+
+      def o_reject(config)
+        lambda do |x|
+          filter =
+            if m = x.match(%r{^/([^/]+)/$})
+              lambda{|p| p.name.to_s.match(m.captures.first) }
+            else
+              lambda{|p| p.name.to_s === x }
+            end
+
+          config.filters << [:-, filter]
+        end
+      end
+    end
+
+  end
+end

data/lib/piggly/command/report.rb

@@ -0,0 +1,162 @@
+module Piggly
+  module Command
+
+    #
+    # This command reads a given file (or STDIN) which is expected to contain messages like the
+    # pattern Profile::PATTERN, which is probbaly "WARNING:  PIGGLY 0123456789abcdef".
+    #
+    # Lines in the input that match this pattern are profiled and used to generate a report
+    #
+    class Report < Base
+    end
+
+    class << Report
+      def main(argv)
+        require "pp"
+        io, config = configure(argv)
+
+        profile = Profile.new
+        index   = Dumper::Index.new(config)
+
+        procedures = filter(config, index)
+
+        if procedures.empty?
+          if filters.empty?
+            abort "no stored procedures in the cache"
+          else
+            abort "no stored procedures in the cache matched your criteria"
+          end
+        elsif config.dry_run?
+          puts procedures.map{|p| p.signature }
+          exit 0
+        end
+
+        profile_procedures(config, procedures, profile)
+        clear_coverage(config, profile)
+
+        read_profile(config, io, profile)
+        store_coverage(profile)
+
+        create_index(config, index, procedures, profile)
+        create_reports(config, procedures, profile)
+      end
+
+      # Adds the given procedures to Profile
+      #
+      def profile_procedures(config, procedures, profile)
+        # register each procedure in the Profile
+        compiler = Compiler::TraceCompiler.new(config)
+        procedures.each do |p|
+          result = compiler.compile(p)
+          profile.add(p, result[:tags], result)
+        end
+      end
+
+      # Clear coverage after procedures have been loaded
+      #
+      def clear_coverage(config, profile)
+        unless config.accumulate?
+          puts "clearing previous coverage"
+          profile.clear
+        end
+      end
+
+      # Reads +io+ for lines matching Profile::PATTERN and records coverage
+      #
+      def read_profile(config, io, profile)
+        np = profile.notice_processor(config)
+        io.each{|line| np.call(line) }
+      end
+
+      # Store the coverage Profile on disk
+      #
+      def store_coverage(profile)
+        puts "storing coverage profile"
+        profile.store
+      end
+
+      # Create the report's index.html
+      #
+      def create_index(config, index, procedures, profile)
+        puts "creating index"
+        reporter = Reporter::Index.new(config, profile)
+        reporter.install("resources/piggly.css", "resources/sortable.js", "resources/highlight.js")
+        reporter.report(procedures, index)
+      end
+
+      # Create each procedures' HTML report page
+      #
+      def create_reports(config, procedures, profile)
+        puts "creating reports"
+        queue = Util::ProcessQueue.new
+
+        compiler = Compiler::TraceCompiler.new(config)
+        reporter = Reporter::Procedure.new(config, profile)
+
+        Parser.parser
+
+        procedures.each do |p|
+          queue.add do
+            unless compiler.stale?(p)
+              data = compiler.compile(p)
+              path = reporter.report_path(p.source_path(config), ".html")
+
+              unless profile.empty?(data[:tags])
+                changes = ": #{profile.difference(p, data[:tags])}"
+              end
+
+              puts "reporting coverage for #{p.name}#{changes}"
+            # pp data[:tags]
+            # pp profile[p]
+            # puts
+
+              reporter.report(p)
+            end
+          end
+        end
+
+        queue.execute
+      end
+
+      def configure(argv, config = Config.new)
+        io = $stdin
+        p  = OptionParser.new do |o|
+          o.on("-t", "--dry-run",           "only print the names of matching procedures", &o_dry_run(config))
+          o.on("-s", "--select PATTERN",    "select procedures matching PATTERN", &o_select(config))
+          o.on("-r", "--reject PATTERN",    "ignore procedures matching PATTERN", &o_reject(config))
+          o.on("-c", "--cache-root PATH",   "local cache directory", &o_cache_root(config))
+          o.on("-o", "--report-root PATH",  "report output directory", &o_report_root(config))
+          o.on("-a", "--accumulate",        "accumulate data from the previous run", &o_accumulate(config))
+          o.on("-V", "--version",           "show version", &o_version(config))
+          o.on("-h", "--help",              "show this message") { abort o.to_s }
+          o.on("-f", "--input PATH",        "read trace messages from PATH") do |path|
+            io = if path == "-"
+                   $stdin
+                 else
+                   File.open(path, "rb")
+                 end
+          end
+        end
+
+        begin
+          p.parse! argv
+          
+          if io.eql?($stdin) and $stdin.tty?
+            raise OptionParser::MissingArgument,
+              "stdin must be a pipe, or use --input PATH"
+          end
+
+          return io, config
+        rescue OptionParser::InvalidOption,
+               OptionParser::InvalidArgument,
+               OptionParser::MissingArgument
+          puts p
+          puts
+          puts $!
+          exit! 1
+        end
+      end
+    end
+
+  end
+end

data/lib/piggly/command/trace.rb

@@ -0,0 +1,90 @@
+module Piggly
+  module Command
+
+    #
+    # This command connects to the database, dumps all stored procedures, compiles them
+    # with instrumentation code, and finally installs the instrumented code.
+    #
+    class Trace < Base
+    end
+
+    class << Trace
+      def main(argv)
+        config     = configure(argv)
+        connection = connect(config)
+        index      = Dumper::Index.new(config)
+
+        dump(connection, index)
+
+        procedures = filter(config, index)
+
+        if procedures.empty?
+          if config.filters.empty?
+            abort "no stored procedures in the cache"
+          else
+            abort "no stored procedures in the cache matched your criteria"
+          end
+        elsif config.dry_run?
+          puts procedures.map{|p| p.signature }
+          exit 0
+        end
+
+        trace(config, procedures)
+        install(Installer.new(config, connection), procedures, Profile.new)
+      end
+
+      # Writes all stored procedures in the database to disk
+      #   @return [void]
+      def dump(connection, index)
+        index.update(Dumper::ReifiedProcedure.all(connection))
+      end
+
+      # Compiles all the stored procedures on disk and installs them
+      #   @return [void]
+      def trace(config, procedures)
+        puts "compiling #{procedures.size} procedures"
+
+        compiler = Compiler::TraceCompiler.new(config)
+        queue    = Util::ProcessQueue.new
+        procedures.each{|p| queue.add { compiler.compile(p) }}
+
+        # Force parser to load before we start forking
+        Parser.parser
+        queue.execute
+      end
+
+      def install(installer, procedures, profile)
+        puts "tracing #{procedures.size} procedures"
+        installer.install(procedures, profile)
+      end
+
+      # Parses command-line options
+      #   @return [Config]
+      def configure(argv, config = Config.new)
+        p = OptionParser.new do |o|
+          o.on("-t", "--dry-run",           "only print the names of matching procedures", &o_dry_run(config))
+          o.on("-s", "--select PATTERN",    "select procedures matching PATTERN", &o_select(config))
+          o.on("-r", "--reject PATTERN",    "ignore procedures matching PATTERN", &o_reject(config))
+          o.on("-c", "--cache-root PATH",   "local cache directory", &o_cache_root(config))
+          o.on("-o", "--report-root PATH",  "report output directory", &o_report_root(config))
+          o.on("-d", "--database PATH",     "read database adapter settings from YAML/JSON file", &o_database_yml(config))
+          o.on("-k", "--connection NAME",   "use connection adapter NAME", &o_connection_name(config))
+          o.on("-V", "--version",           "show version", &o_version(config))
+          o.on("-h", "--help",              "show this message") { abort o.to_s }
+        end
+
+        begin
+          p.parse! argv
+          config
+        rescue OptionParser::InvalidOption,
+               OptionParser::InvalidArgument,
+               OptionParser::MissingArgument
+          puts p
+          puts
+          puts $!
+          exit! 1
+        end
+      end
+    end
+  end
+end

data/lib/piggly/command/untrace.rb

@@ -0,0 +1,78 @@
+module Piggly
+  module Command
+
+    class Untrace < Base
+    end
+
+    class << Untrace
+      def main(argv)
+        config      = configure(argv)
+        index       = Dumper::Index.new(config)
+        connection  = connect(config)
+        procedures  = filter(config, index)
+
+        if procedures.empty?
+          if config.filters.empty?
+            abort "no stored procedures in the cache"
+          else
+            abort "no stored procedures in the cache matched your criteria"
+          end
+        elsif config.dry_run?
+          puts procedures.map{|x| x.signature }
+          exit 0
+        end
+
+        untrace(Installer.new(config, connection), procedures)
+      end
+
+      #
+      # Restores database procedures from file cache
+      #
+      def untrace(installer, procedures)
+        puts "restoring #{procedures.size} procedures"
+        installer.uninstall(procedures)
+      end
+
+      #
+      # Returns a list of Procedure values that satisfy at least one of the given filters
+      #
+      def find_procedures(filters, index)
+        if filters.empty?
+          index.procedures
+        else
+          filters.inject(Set.new){|set, filter| set | index.procedures.select(&filter) }
+        end
+      end
+
+      #
+      # Parses command-line options
+      #
+      def configure(argv, config = Config.new)
+        p = OptionParser.new do |o|
+          o.on("-t", "--dry-run",         "only print the names of matching procedures", &o_dry_run(config))
+          o.on("-s", "--select PATTERN",  "select procedures matching PATTERN", &o_select(config))
+          o.on("-r", "--reject PATTERN",  "ignore procedures matching PATTERN", &o_reject(config))
+          o.on("-c", "--cache-root PATH", "local cache directory", &o_cache_root(config))
+          o.on("-d", "--database PATH",   "read 'piggly' database adapter settings from YAML file", &o_database_yml(config))
+          o.on("-k", "--connection NAME", "use connection adapter NAME", &o_connection_name(config))
+          o.on("-V", "--version",         "show version", &o_version(config))
+          o.on("-h", "--help",            "show this message") { abort o.to_s }
+        end
+
+        begin
+          p.parse! argv
+          config
+        rescue OptionParser::InvalidOption,
+               OptionParser::InvalidArgument,
+               OptionParser::MissingArgument
+          puts p
+          puts
+          puts $!
+          exit! 1
+        end
+      end
+
+    end
+
+  end
+end