bryton-lite 1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3774eb36e39cd0c1e23784ff1c9b6972aee2cd3228f162183f1b45c7c51ba348
+  data.tar.gz: d12041455299d8b4f19cf83975991168bc6792ff07c227c8c6b5d49e95d34789
+SHA512:
+  metadata.gz: 34cca96d0ce490c263a1e22be5978bb708832940dd0ad7c187f57bfe3195eaa930376957a46c57a61e403284895f8f129786267957c5bc067b45bf0ecc297fc8
+  data.tar.gz: 445ccdbca8dbdba5faf901e2bbcba4eed21214a6e85f8a3fe6376d858b6896752f8dfe2a38cf866678f460243e0075a581770b234fb67623500fa240d7cd50d2

data/README.md

@@ -0,0 +1,296 @@
+# bryton-lite
+Bare bones Ruby implementation of the Bryton testing protocol
+
+## Install
+
+The usual:
+
+```sh
+gem install bryton-lite
+```
+
+## Overview
+
+Bryton is a file-based testing protocol. The point of Bryton is to allow you to
+write your tests without constraints on how they need to be organized. All your
+test scripts have to do is output a JSON object as the last line of STDOUT. At
+a minimum, the JSON object should have a "success" element set to true or false:
+
+```json
+{"success":true}
+{"success":false}
+```
+
+Bryton allows you to start simple, with just a few tests scripts that you can
+run manually to see the results. As you progress to automated testing you can
+use a Bryton runner to run all your tests.
+   
+A Bryton runner (i.e. the routine running the tests) calls each executable file
+in a directory, collecting the results. Those executables can be written in any
+language, Ruby or otherwise. It also recurses into subdirectories. The runner
+then reports the results back to you in either a human readable or machine
+readable format.
+
+Bryton (which is currently under development) will be a full featured testing
+system with a large array of options. Bryton::Lite only implements a few of the
+features. In fact, Bryton::Lite is to be used for testing Bryton.
+
+
+## Use
+
+This gem has two modules, one for building tests and one for running tests. The
+two modules don't depend on each other. You can use one or the other or both.
+
+### Bryton::Lite::Tests
+
+#### Basic example
+
+Bryton is designed to be simple. You don't need any special tools to implement
+the basic protocol. For example, consider the following script:
+
+```ruby
+#!/usr/bin/ruby -w
+require 'json'
+
+# some test
+def some_test()
+   return true
+end
+
+# results hash
+results = {}
+
+# run a test
+results['success'] = some_test()
+
+# output results
+puts JSON.generate(results)
+```
+
+Notice that you don't even need this gem to run that script. Bryton is designed
+to be simple and easy to implement. That test outputs a JSON object as either
+`{"success":true}` or `{"success":false}`. Now let's take a look at a script
+in which a test fails.
+
+```ruby
+#!/usr/bin/ruby -w
+require 'json'
+
+# some test
+def some_test()
+   return false
+end
+
+# results hash
+results = {}
+
+# run a test
+if some_test()
+   results['success'] = true
+else
+   results['errors'] ||= []
+   results['errors'].push({'id'=>'some_test_failure'})
+end
+
+# output results
+puts JSON.generate(results)
+```
+
+In this example one of the tests fails. The script could have simply set
+`success` to false, but instead it gives a little more information by creating
+the `errors` array and adding a message to it. Notice that `success` is never
+explicitly set. That's because the presence of an error implies failure.
+
+#### Bryton::Lite::Tests.assert
+
+Bryton::Lite::Tests provides several tools for building and outputting test
+results. Consider this script:
+
+```ruby
+#!/usr/bin/ruby -w
+require 'bryton/lite'
+
+# some test
+def some_test
+   return false
+end
+
+# test a function
+Bryton::Lite::Tests.assert some_test()
+
+# done
+Bryton::Lite::Tests.try_succeed
+Bryton::Lite::Tests.done
+```
+
+In this test, we create a function that, in this case, always returns false.
+Then we use Bryton::Lite::Tests.assert to check the result of that function.
+If the test fails, then an error is added to the output hash.
+
+Next we call Bryton::Lite::Tests.try_succeed. That function marks the test
+script as successful, but only if there are not errors. Remember that a test run
+is only considered successful if the output explicitly sets `success` as true.
+
+
+Finally, we call Bryton::Lite::Tests.done, which outputs the the results and
+exits. Bryton::Lite::Tests.done should be called as the last line of your
+script.
+
+Here's the output for that run.
+
+```json
+{"errors":[{"line":12,"file":"./basic.rb"}]}
+```
+
+By default, `assert` notes the file name and line number of the error. You can
+also add an id for the error:
+
+```ruby
+Bryton::Lite::Tests.assert some_test(), 'running some_test()'
+```
+
+which outputs
+
+```json
+{"errors":[{"line":11,"file":"./id.rb","id":"running some_test()"}]}
+```
+
+If you want to manually add information to the error, use a do block. `assert`
+yields the error hash:
+
+```ruby
+Bryton::Lite::Tests.assert(some_test()) do |error|
+   error['notes'] = 'failure of some_test'
+end
+```
+
+which outputs a JSON object with whatever you added:
+
+```json
+{"errors":[{"line":12,"file":"./block.rb","notes":"failure of some_test"}]}
+```
+
+#### Bryton::Lite::Tests.fail
+
+`fail` works much like `assert`, but it unconditionally adds an error. `fail`
+has syntax similar to `assert`.
+
+```ruby
+Bryton::Lite::Tests.fail() do |error|
+   error['notes'] = 'failed db test'
+end
+```
+
+which outputs JSON like this:
+
+```json
+{"errors":[{"line":6,"file":"./no-id.rb","notes":"failed db test"}]}
+```
+
+### Bryton::Lite::Runner
+
+Bryton::Lite::Runner runs all the tests in a directory tree. The full Bryton
+protocol will allow you to pick and choose which tests to run and what to do
+on success or failure. Bryton::Lite::Runner just implements the basic concept of
+running all the tests.
+
+To run tests, your script should go to the root of the directory where you have
+your test files. Then just run `Bryton::Lite::Runner.run`.
+
+```ruby
+#!/usr/bin/ruby -w
+require 'bryton/lite'
+
+Dir.chdir '../tests'
+Bryton::Lite::Runner.run
+puts Bryton::Lite::Runner.success?
+```
+
+If you just want to know the success or failure of the tests, output
+`Bryton::Lite::Runner.success?`. If you want more information, you can output
+all the results from the entire test run:
+
+```ruby
+puts JSON.pretty_generate(Bryton::Lite::Tests.hsh)
+```
+
+Notice that we use `Bryton::Lite::Tests.hsh` to get the results. The runner
+stores results with Bryton::Lite::Tests because the run itself is a test. There
+are main three elements to a results hash.
+
+| key     | data type           | explanation                                                                                                                                                                               |
+|---------|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| success | true, false, or nil | Indicates success of failure of a test. Nil is considered false.                                                                                                                          |
+| file    | hash                | Hash of information about the fiule being tested. Should at least contain the path to the file relative to the root directory of the test. If `dir` is true then the file is a directory. |
+| errors  | array               | Each element of the array give details about an error. If any elements are present in the errors array then `success` should not be true.                                                 |
+| nested  | array               | Array of nested test results. Every nested result must be successful or the entire test run is considered to have failed.                                                                 |
+
+As with any hash, you can add your own custom elements.
+
+Here is a sample output from a script run. (JSON doesn't allow comments, but
+I've added some here for clarity.
+
+```javascript
+// root directory for tests
+{
+   // indicates that the test run failed
+   "success": false,
+   
+   // file information about this directory
+   "file": {
+      "path": ".",
+      "dir": true
+   },
+   
+   // nested list of subtests
+   // directories always have an array of nested tests
+   "nested": [
+      {
+         "file": {
+            "path": "./load-tests",
+            "dir": true
+         },
+         
+         "nested": [
+            {
+               // indicates that this file test failed
+               "success": false,
+               
+               // file information about ./load-tests/crash.rb
+               "file": {
+                  "path": "./load-tests/crash.rb"
+               },
+               
+               // array of error messages
+               "errors": [
+                  {
+                     // indicates that execution of the script failed
+                     "id": "execution-failed",
+                     
+                     // STDERR from the execution
+                     "stderr": "./crash.rb:5:in `/': divided by 0 (ZeroDivisionError)\n\tfrom ./crash.rb:5:in `<main>'\n"
+                  }
+               ]
+            },
+            
+            {
+               // indicates that the script succeeded
+               "success": true,
+               
+               // file information
+               "file": {
+                  "path": "./load-tests/success.rb"
+               }
+            }
+         ]
+      },
+      
+      {
+         "success": true,
+         "file": {
+            "path": "./test-a.rb"
+         }
+      }
+   ]
+}
+```

data/lib/bryton/lite.rb

@@ -0,0 +1,427 @@
+require 'open3'
+require 'json'
+
+
+#===============================================================================
+# Bryton, Bryton::Lite
+#
+
+# Not much here. Just initializing the Bryton namespace.
+
+module Bryton
+end
+
+# Not much here either. Just initializing the Bryton::Lite namespace.
+
+module Bryton::Lite
+end
+#
+# Bryton, Bryton::Lite
+#===============================================================================
+
+
+#===============================================================================
+# Bryton::Lite::Runner
+#
+
+# Runs tests in the current directory tree.
+
+module Bryton::Lite::Runner
+	#---------------------------------------------------------------------------
+	# run
+	#
+	
+	# Run the tests. If you pass in a block, each test result is yielded with
+	# the path to the file and the results hash.
+	
+	def self.run(&block)
+		Bryton::Lite::Tests.reset
+		dir('.', &block)
+		Bryton::Lite::Tests.try_succeed
+		Bryton::Lite::Tests.reorder_results
+	end
+	#
+	# run
+	#---------------------------------------------------------------------------
+	
+	
+	#---------------------------------------------------------------------------
+	# dir
+	#
+	
+	# Runs the tests in a directory. Don't call this method directly.
+	
+	def self.dir(path, &block)
+		entries = Dir.entries('./')
+		entries = entries.reject {|entry| entry.match(/\A\.+\z/mu)}
+		
+		# note file and path
+		Bryton::Lite::Tests['file'] = {}
+		Bryton::Lite::Tests['file']['path'] = path
+		Bryton::Lite::Tests['file']['dir'] = true
+		
+		# loop through entries
+		entries.each do |entry|
+			entry_path = "#{path}/#{entry}"
+			
+			# if directory, recurse into it
+			if File.directory?(entry)
+				Dir.chdir(entry) do
+					Bryton::Lite::Tests.nest do
+						self.dir(entry_path, &block)
+					end
+				end
+			
+			# elsif the file is executable then execute it
+			elsif File.executable?(entry)
+				results = execute(entry, entry_path)
+				Bryton::Lite::Tests.add_to_nest results
+				
+				if block_given?
+					yield entry_path, results
+				end
+			end
+		end
+	end
+	#
+	# dir
+	#---------------------------------------------------------------------------
+	
+	
+	#---------------------------------------------------------------------------
+	# execute
+	#
+	
+	# Executes a file, parses the results, and returns the results. Don't call
+	# this method directly.
+	
+	def self.execute(name, path)
+		# init
+		rv = {}
+		rv['file'] = {}
+		rv['file']['path'] = path
+		
+		# execute script
+		cap = Bryton::Lite::Runner::Capture.new("./#{name}")
+		
+		# if script crashed
+		if not cap.success?
+			rv['success'] = false
+			rv['errors'] ||= []
+			
+			# build error
+			error = {'id'=>'execution-failed'}
+			error['id'] = 'execution-failed'
+			error['stderr'] = cap.stderr
+			
+			# add error and return
+			rv['errors'].push(error)
+			return rv
+		
+		# if we get a non-blank line, attempt to parse it as JSON
+		elsif non_blank = self.last_non_blank(cap.stdout)
+			begin
+				# retrieve and parse results
+				results = JSON.parse( non_blank )
+				rv = rv.merge(results)
+				
+				# ensure that if there are errors, success is false
+				if rv['errors'] and rv['errors'].any?
+					rv['success'] = false
+				end
+				
+				# return
+				return rv
+			rescue
+				rv['success'] = false
+				rv['errors'] = []
+				rv['errors'].push({'id'=>'invalid-json'})
+				return rv
+			end
+		
+		# did not get non-blank line
+		else
+			rv['success'] = false
+			rv['errors'] = []
+			rv['errors'].push({'id'=>'no-results'})
+			return rv
+		end
+	end
+	#
+	# execute
+	#---------------------------------------------------------------------------
+	
+	
+	#---------------------------------------------------------------------------
+	# last_non_blank
+	#
+	
+	# Finds the last non-blank line in STDOUT from the file execution. Don't
+	# call this method directly.
+	
+	def self.last_non_blank(str)
+		str.lines.reverse.each do |line|
+			if line.match(/\S/mu)
+				return line
+			end
+		end
+		
+		# didn't find non-blank
+		return nil
+	end
+	#
+	# last_non_blank
+	#---------------------------------------------------------------------------
+	
+	
+	#---------------------------------------------------------------------------
+	# success?
+	#
+	
+	# Returns the success or failure of the test run.
+	
+	def self.success?
+		return Bryton::Lite::Tests.success?
+	end
+	#
+	# success?
+	#---------------------------------------------------------------------------
+end
+#
+# Bryton::Lite::Runner
+#===============================================================================
+
+
+#===============================================================================
+# Bryton::Lite::Runner::Capture
+# class for capturing the output of a script
+#
+
+# Executes the given command and captures the results.
+
+class Bryton::Lite::Runner::Capture
+	# Executes the command with Open3.capture3 and holds on to the results.
+	def initialize(*cmd)
+		@results = Open3.capture3(*cmd)
+	end
+	
+	# Returns the content of STDOUT from the execution.
+	# @return [String]
+	def stdout
+		return @results[0]
+	end
+	
+	# Returns the content of STDERR from the execution.
+	# @return [String]
+	def stderr
+		return @results[1]
+	end
+	
+	# Returns the status of execution.
+	# @return [Process::Status]
+	def status
+		return @results[2]
+	end
+	
+	# Returns the success or failure of the execution.
+	# @return [Boolean]
+	def success?
+		return status.success?
+	end
+end
+#
+# Bryton::Lite::Runner::Capture
+#===============================================================================
+
+
+#===============================================================================
+# Bryton::Lite::Tests
+#
+
+# Utilities for use in the test script.
+
+module Bryton::Lite::Tests
+	# accessors
+	class << self
+		# @return [Hash]
+		attr_reader :hsh
+	end
+	
+	# reset
+	def self.reset()
+		@hsh = {}
+	end
+	
+	# call reset
+	reset()
+	
+	# Get the element with the given key.
+	def self.[](k)
+		return @hsh[k]
+	end
+	
+	# Set the element with the given key and value.
+	def self.[]=(k, v)
+		return @hsh[k] = v
+	end
+	
+	# Set to success. Raises an exception if any errors exist. You probably want
+	# to use try_succeed() instead.
+	def self.succeed
+		if not allow_success?
+			raise 'cannot-set-to-success'
+		end
+		
+		return @hsh['success'] = true
+	end
+	
+	# Returns true if ['success'] is true, false otherwise. Always returns true
+	# or false.
+	# @return [Boolean]
+	def self.success?
+		return @hsh['success'] ? true : false
+	end
+	
+	# Try to set to success, but will not do so if there are errors or if the
+	# test is already set as fail. Does not raise an exception if the success
+	# cannot be set to true.
+	# @return [Boolean]
+	def self.try_succeed
+		return @hsh['success'] = allow_success?()
+	end
+	
+	# Test if try_succeed can set success. You probably don't need to call this
+	# method directly.
+	# @return [Boolean]
+	def self.allow_success?
+		return allow_success_recurse(@hsh)
+	end
+	
+	# Tests each nested result. If any of the nested results are set to failure,
+	# return false. Otherwise returns true.
+	def self.allow_success_recurse(test_hsh)
+		# if any errors
+		if test_hsh['errors'] and test_hsh['errors'].any?
+			return false
+		end
+		
+		# recurse into nested elements
+		if test_hsh['nested']
+			test_hsh['nested'].each do |child|
+				if not allow_success_recurse(child)
+					return false
+				end
+			end
+		end
+		
+		# at this point it should be successful
+		return true
+	end
+	
+	# Returns the errors array, creating it if necessary.
+	def self.errors
+		@hsh['errors'] ||= []
+		return @hsh['errors']
+	end
+	
+	# Creates (if necessary) an array with the key "nested". Creates a results
+	# hash that is nested in that array. In the do block,
+	# Bryton::Lite::Tests.hsh is set to that child results hash.
+	def self.nest()
+		@hsh['nested'] ||= []
+		child = {}
+		@hsh['nested'].push child
+		hold_hsh = @hsh
+		@hsh = child
+		
+		begin
+			yield
+		ensure
+			@hsh = hold_hsh
+		end
+	end
+	
+	# Created the "nested" array and adds the given results to that array
+	def self.add_to_nest(child)
+		@hsh['nested'] ||= []
+		@hsh['nested'].push child
+	end
+	
+	# done
+	def self.done
+		try_succeed()
+		reorder_results()
+		STDOUT.puts JSON.generate(@hsh)
+		exit
+	end
+	
+	# to_json
+	def self.to_json
+		return JSON.generate(@hsh)
+	end
+	
+	# assert
+	def self.assert(bool, id=nil, level=0, &block)
+		if not bool
+			fail id, 1, &block
+		end
+	end
+	
+	# Mark the test as failed.
+	def self.fail(id=nil, level=0, &block)
+		loc = caller_locations[level]
+		
+		# create error object
+		error = {}
+		error['line'] = loc.lineno
+		id and error['id'] = id
+		
+		# add to errors
+		@hsh['errors'] ||= []
+		@hsh['errors'].push error
+		
+		# if block
+		if block_given?
+			yield error
+		end
+	end
+	
+	# reorder_results
+	# This method is for improving the readability of a result by putting the
+	# "success" element first and "nested" last.
+	def self.reorder_results
+		@hsh = reorder_results_recurse(@hsh)
+	end
+	
+	# reorder_results_recurse
+	# This method rcurses through test results, putting "success" first and
+	# "nested" last.
+	def self.reorder_results_recurse(old_hsh)
+		new_hsh = {}
+		nested = old_hsh.delete('nested')
+		
+		# add success if it exists
+		if old_hsh.has_key?('success')
+			new_hsh['success'] = old_hsh.delete('success')
+		end
+		
+		# add every other element
+		new_hsh = new_hsh.merge(old_hsh)
+		
+		# add nested last
+		if nested
+			new_hsh['nested'] = nested
+			
+			# recurse through nested results
+			new_hsh['nested'].map! do |child|
+				reorder_results_recurse child
+			end
+		end
+		
+		# return reformatted hash
+		return new_hsh
+	end
+end
+#
+# Bryton::Lite::Tests
+#===============================================================================

metadata

@@ -0,0 +1,44 @@
+--- !ruby/object:Gem::Specification
+name: bryton-lite
+version: !ruby/object:Gem::Version
+  version: '1.0'
+platform: ruby
+authors:
+- Mike O'Sullivan
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2023-05-18 00:00:00.000000000 Z
+dependencies: []
+description: Bare bones implementation of the Bryton testing protocol
+email: mike@idocs.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/bryton/lite.rb
+homepage: https://github.com/mikosullivan/bryton-lite
+licenses:
+- MIT
+metadata: {}
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.1.2
+signing_key:
+specification_version: 4
+summary: Bryton::Lite
+test_files: []