bones-compiler 1.1.0

data/CHANGELOG

@@ -0,0 +1,117 @@
+###################
+### v1.1        ###
+###################
+
+General:
+- Added support for a compiler optimisation pass: array substitution with a local register copy in the case of chunk to element species.
+- Added support for a compiler optimisation pass: thread-merging, potentially improving re-use through locality at the cost of parallelism.
+- Updated and added examples.
+- Tuned the skeletons.
+
+Skeletons performance/readability/usability tuning:
+- Added a special GPU-CUDA 'chunk(1,N)' skeleton with pre-shuffling (e.g. atax, bicg, mvt, syrk). Added support for a corresponding transformation in Bones.
+- Added a special GPU-CUDA 2x'chunk(1,N)' skeleton with pre-shuffling if there are two chunk inputs (e.g. gesummv,syr2k).
+- Updated the various OpenCL skeletons to create only once the 'context', 'command queue' and 'program'. This saves significant time for programs executing a single kernel multiple times or multiple kernels subsequently.
+- Tune GPU-CUDA skeletons to prefer L1 cache above scratchpad memory.
+
+Examples:
+- Updated several benchmarks with additional species.
+- Added the 'saxpy' benchmark.
+- Added 'example12' to demonstrate the use of species in a separate function.
+
+Bug fixes:
+- Removed the generation of global code in case no species are detected.
+
+Miscellaneous:
+- Renamed the original main function and created a new 'main' in 'common/globals.c' with initialisation and clean-up. The use of '#pragma species initialize' is now deprecated, but a main function is required in the original code.
+- Added the identification of a species' function, making variable definition detection local.
+- Improved the compiler run-time for algorithms for which a skeleton is unavailable.
+- Added a command-line option to be able to generate code for a single species only (-only_alg_number).
+- Added a command-line option to set a fixed thread-merge factor (-merge_factor).
+- Added a gemspec file.
+- Updated the documentation and readme.
+
+###################
+### v1.0        ###
+###################
+
+General:
+- Added a benchmark set to the examples directory based on the PolyBench/C benchmark set.
+- Performed major code refactoring, improving maintainability and performance of Bones.
+- Updated all examples to work with the upcoming 'automatic species extraction tool' (ASET).
+
+Skeletons performance/readability/usability tuning:
+- Updated the 'GPU-CUDA' reduction skeleton to support initial values, loops not starting at zero or not a power of 2, and more (see the new 'shared/example5.c').
+- Improved the way skeletons are matched with species. Introduced a 'default' skeleton and modified others to accomodate these changes.
+
+Improved code/species support:
+- Added atomic support for OpenCL based targets
+- Added support for loop iterator variables inside the 'algorithmic species' with selective memory copy for the 'GPU-CUDA' target, see 'element/example11.c'.
+
+Bug fixes:
+- Fixed a compatibility problem with CAST version 0.2.0.
+
+Miscellaneous:
+- Added proper error messaging to catch various exceptions 
+- Reversed the order of loop-flattening to obtain coalesced memory accesses (in particular for the 'GPU-CUDA' target).
+- Renamed 'tile' into 'chunk', since 'tile' might imply using a 2D chunk of data.
+- Changed '#pragma bones' into '#pragma species' to match the algorithmic species naming.
+- Added a warning message for negative or zero range dimensions.
+- Organized the search-and-replace parameters found in skeletons, renamed most of them and added a few new.
+- Added headers to the code examples and cleaned them up.
+- Added a warning message if an 'endkernel' pragma is missing.
+- Improved the way error messages are thrown.
+
+###################
+### v0.9 (Beta) ###
+###################
+
+General:
+- Implemented the new 'algorithmic species' using ranges, thus creating support for for-loops with affine functions as loop-bounds.
+- Updated the examples
+
+Supported targets:
+- Changed the names of the targets 'GPU-OPENCL' and 'CPU-OPENCL' into 'GPU-OPENCL-AMD' and 'CPU-OPENCL-INTEL' respectively.
+- Added a new target 'CPU-C' which implements a C-to-C pass-through.
+- Added a new target 'CPU-OPENMP', which uses OpenMP to create 4 CPU threads.
+- Added a new target 'CPU-OPENCL-AMD'. This target is similar to 'CPU-OPENCL-INTEL', but targets the AMD APP.
+
+Skeletons performance/readability/usability tuning:
+- Added a basic prefetching technique in the local memory for the neighbourhood skeleton for the 'GPU-CUDA' target.
+- Removed the first entry in the transformation settings, which was used in previous versions to set the dimensions (now automatically detected).
+- Changed the 'GPU-CUDA' skeletons such that host files can be compiled with a C99 compiler.
+- Tuned 'CPU-OPENCL-INTEL' performance for Intel's OpenCL SDK.
+- Created aligned memory allocation functions to enable zero-copy possibilities for the 'CPU-OPENCL-INTEL' target.
+- Completed the addition of 'bones_' for every variable in the skeletons.
+
+Verification code:
+- Create a verification function specific to each output.
+- Moved the verification code (including the original code) to a separate file, which is only generated if '-c' is provided as a flag to Bones.
+
+Bug fixes:
+- Fixed a bug which would create function names starting with a digit.
+- Adjusted the use of directory structures in the code for Windows-compatibility.
+- Fixed a bug where variables for verification would have duplicate names.
+- Fixed a bug where verification code would not compile for 'unsigned int' types.
+- Fixed a memory leak which would occur when verification is enabled.
+- Fixed a bug where statements of the form 'a[i]++' would not be recognized as input nor as output. They are now rewritten as 'a[i]=a[i]+1'.
+
+Miscellaneous:
+- Added support for selective copying-out (based on array access ranges)
+- Added support for defines found in header functions (pre-processor now also pre-processes the header files)
+- Added the possibility to specify the order of inputs/outputs in the classification by giving their names (if not given, the default ordering is assumed).
+- Writing to a specific location in an array followed by a read no longer considers the array as input and output, it is now output only.
+- Added a check to see if for-loops start and end as expected (as provided by the ranges given through the 'algorithmic species').
+- Create a 'simplify' function, which simplifies math expressions to a certain extend. A test is included to give a few examples of what it can do.
+- Clean-up of the Rakefile, addition of stub tasks to compile and execute example code, and the addition of an 'add new target' task.
+- Changed the code such that the core components of Bones (the 'lib' folder) do not have to be adjusted to add a new target.
+- Added performance measurement for original code in case verification is enabled.
+- Renamed 'Tribe' into 'Species' and 'Primitive' into 'Algorithm'.
+
+###################
+### v0.8 (Beta) ###
+###################
+
+Initial release.
+
+###################

data/LICENSE

@@ -0,0 +1,9 @@
+Copyright (c) 2012 Cedric Nugteren, Eindhoven University of Technology, The Netherlands
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+

data/README.rdoc

@@ -0,0 +1,126 @@
+= Bones
+Recent advances in multi-core and many-core processors requires programmers to exploit an increasing amount of parallelism from their applications. Data parallel languages such as CUDA and OpenCL make it possible to take advantage of such processors, but still require a large amount of effort from programmers. To address the challenge of parallel programming, we introduce Bones.
+
+Bones is a source-to-source compiler based on algorithmic skeletons and a new algorithm classification (named 'algorithmic species'). The compiler takes C-code annotated with class information as input and generates parallelized target code. Targets include NVIDIA GPUs (through CUDA), AMD GPUs (through OpenCL) and CPUs (through OpenCL and OpenMP). Bones is open-source, written in the Ruby programming language, and is available through our website. The compiler is based on the C-parser CAST, which is used to parse the input code into an abstract syntax tree (AST) and to generate the target code from a transformed AST.
+
+== Usage
+The usage is as follows:
+      bones --application <input> --target <target> [OPTIONS]
+With the following flags:
+    --application, -a <s>:   Input application file
+         --target, -t <s>:   Target processor (choose from: GPU-CUDA, GPU-OPENCL-AMD,
+                             CPU-OPENCL-INTEL, CPU-OPENCL-AMD, CPU-OPENMP, CPU-C)
+       --measurements, -m:   Enable/disable timers
+             --verify, -c:   Verify correctness of the generated code
+            --version, -v:   Print version and exit
+               --help, -h:   Show this message
+
+Bones can be invoked from the command-line. Two arguments (-a and -t) are mandatory, others are optional. This is an example of the usage of Bones assuming the file '+example.c+' to be present:
+  bones -a example.c -t GPU-CUDA -c
+
+== Examples
+The best place to start experimenting with Bones is the '+examples+' directory. A large number of examples are available in this folder, grouped by algorithmic species (either element, neighbourhood, shared or chunk). The examples illustrate different kinds of coding styles and give a large number of different classes to work with. The folder '+benchmarks+' gives more examples, taken from the PolyBench/C benchmark set. Additionally, a folder '+applications+' is included, containing example complete applications. Currently, the following examples are available:
+ |-- element                      |-- applications
+ |   |-- example1.c               |   \-- ffos.c
+ |   |-- example2.c               \-- benchmarks
+ |   |-- example3.c                   |-- 2mm.c
+ |   |-- example4.c                   |-- 3mm.c
+ |   |-- example5.c                   |-- adi.c
+ |   |-- example6.c                   |-- atax.c
+ |   |-- example7.c                   |-- bicg.c
+ |   |-- example8.c                   |-- cholesky.c
+ |   |-- example9.c                   |-- correlation.c
+ |   |-- example10.c                  |-- covariance.c
+ |   |-- example11.c                  |-- doitgen.c
+ |   \-- example12.c                  |-- durbin.c
+ |-- neighbourhood                    |-- dynprog.c
+ |   |-- example1.c                   |-- fdtd-2d-apml.c
+ |   |-- example2.c                   |-- fdtd-2d.c
+ |   |-- example3.c                   |-- floyd-warshall.c
+ |   \-- example4.c                   |-- gemm.c
+ |-- shared                           |-- gemver.c
+ |   |-- example1.c                   |-- gesummv.c
+ |   |-- example2.c                   |-- jacobi-1d-imper.c
+ |   |-- example3.c                   |-- jacobi-2d-imper.c
+ |   |-- example4.c                   |-- lu.c
+ |   \-- example5.c                   |-- ludcmp.c
+ |-- chunk                            |-- mvt.c
+ |   |-- example1.c                   |-- reg_detect.c
+ |   |-- example2.c                   |-- saxpy.c
+ |   |-- example3.c                   |-- seidel-2d.c
+ |   |-- example4.c                   |-- syr2k.c
+ |   \-- example5.c                   |-- syrk.c
+                                      |-- trisolv.c
+                                      \-- trmm.c
+
+All examples can be ran through Bones for a specific target using an automated Rake task. Executing '<tt>rake examples:generate</tt>' or simply '+rake+' will execute Bones for all examples for a given target. The target can be changed in the '+Rakefile+' found in the root directory of Bones.
+
+== Limitations
+Bones takes C99 source code as input. However, several coding styles are unsupported as of now or might yield worse performance compared to others. The numerous examples provided should give the user an idea of the possibilities and limitations of the tool. A complete list of coding guidelines and limitations will follow in the future. Currently, an initial list of major limitations and guidelines is given below. In this list, we use '+algorithm+' to denote an algorithm captured by an algorithmic species.
+* If the algorithm works on a N-dimensional data structure, use N-dimensional arrays (don't flatten it yourself, e.g. use '<tt>example[i][j]</tt>' instead of '<tt>example[i+j*A]</tt>') and specify an N-dimensional algorithmic species.
+* Write your while-loops as for-loops if possible. For-loops should have a unit increment, other loops (e.g. decrementing loops) must be re-written.
+* Loops can have affine bounds containing constants, defines and variables. Variables should not include loop variables of loops that are part of the '+algorithm+'.
+* Function calls are not allowed within the '+algorithm+'. Some mathematical functions are allowed.
+* Variables are allowed in the definition of an algorithmic species. If they are used, the should also be used somewhere in the body of the '+algorithm+'.
+* Bones is designed to work on a single input file with at least a function called 'main'. If your (to-be-accelerated) code spawns over multiple C-files, Bones could either be applied multiple times, or the code could be merged into a single file.
+
+
+
+= Installation procedure
+Installation of Bones is a simple matter of extracting the Bones package to a directory of your choice or installing the gem ('<tt>gem install bones-compiler</tt>'). However, there are a number of prerequisites.
+
+== Prerequisites
+Bones requires the installation of Ruby, the Rubygems gem package manager and two gems:
+1. Any version of *Ruby* *1.8* or *1.9*. Information on Ruby is found at http://www.ruby-lang.org
+   * [OS X]: Ruby is pre-installed on any OS X system since Tiger (10.4).
+   * [Linux]: Ruby is pre-installed on some Linux based systems. Most Linux package managers (yum, apt-get) will be able to provide a Ruby installation. Make sure that the ruby development package ('+ruby-devel+') is also installed, as it is required by one of the gems.
+   * [Windows]: Ruby for Windows can be obtained from http://rubyinstaller.org/
+2. The *Rubygems* gem package manager. Information on Rubygems can be found at http://rubygems.org
+   * [OS X]: Rubygems is pre-installed on any OS X system since Tiger (10.4).
+   * [Linux]: Most Linux package managers will be able to provide a Rubygems installation by installing the package '+rubygems+'.
+   * [Windows]: Rubygems for Windows is obtained automatically when installing from http://rubyinstaller.org/
+3. Bones requires two gems, *trollop* and *cast*. Both gems can be installed by calling Rubygems from the command line, i.e.: '<tt>gem install trollop cast</tt>'.
+
+For example, all prerequisites can be installed as follows on a Fedora, Red-Hat or CentOS system:
+  yum install ruby ruby-devel rubygems
+  gem install trollop cast
+For an Ubuntu, Debian or Mint system, the equivalent commands are:
+  apt-get install ruby ruby-devel rubygems
+  gem install trollop cast
+
+== Installing Bones
+To install the compiler, simply extract the '<tt>bones\_x.x.tar.gz</tt>' package to a directory of your choice. The Bones executable is found in the '+bin+' subdirectory. Including the path to the '+bin+' directory to your environmental variable '+PATH+' will make Bones available from any directory on your machine. Starting at version 1.1, Bones is also available as a gem ('<tt>gem install bones-compiler</tt>').
+
+
+= Documentation
+There are two ways to go to obtain more information regarding Bones. To obtain more information about the compiler itself, the ideas behind it and the algorithm classification, it is a good idea to read scientific publications. To get more information about the code structure, HTML documentation can be generated automatically using RDoc.
+
+== Code documentation
+Code documentation can be generated automatically using RDoc. Navigate to the installation root of Bones and use Rake to generate documentation: '<tt>rake rdoc</tt>'. More information on using Rake is provided later in this document. Next, open '<tt>rdoc/index.html</tt>' to navigate through the documentation. The same documentation is also available on the web at http://parse.ele.tue.nl/tools/bones/rdoc/.
+
+== Scientific publications
+Scientific publications related to Bones can be obtained from http://parse.ele.tue.nl/publications. Two publications are relevant:
+1. <b>A Modular and Parameterisable Classification of Algorithms</b>, which provides details on the used algorithm classification. When refering to the algorithm classification in scientific work, you are kindly asked to include the following citations:
+
+    @TECHREPORT{Nugteren2011,
+      author      = {Cedric Nugteren and Henk Corporaal},
+      title       = {{A Modular and Parameterisable Classification of Algorithms}},
+      institution = {Eindhoven University of Technology},
+      year        = {2011},
+      number      = {No. ESR-2011-02},
+    }
+2. <b>Introducing 'Bones': A Parallelizing Source-to-Source Compiler Based on Algorithmic Skeletons</b>, which introduces the Bones source-to-source compiler. When refering to Bones in scientific work, you are kindly asked to include the following citations:
+
+    @INPROCEEDINGS{Nugteren2012,
+      author    = {Cedric Nugteren and Henk Corporaal},
+      title     = {{Introducing `Bones': A Parallelizing Source-to-Source Compiler
+                  Based on Algorithmic Skeletons}},
+      booktitle = {{GPGPU-5: 5th Workshop on General Purpose Processing on
+                  Graphics Processing Units}},
+      year      = {2012},
+    }
+
+
+
+= Questions
+Questions can be directed by email. You can find contact details on the personal page of the author at http://parse.ele.tue.nl/cnugteren or on the project page at github.

data/Rakefile

@@ -0,0 +1,107 @@
+require 'rake/testtask'
+require 'rdoc/task'
+require 'rake/clean'
+
+# Set the location of the examples
+EXAMPLES = File.join('examples','benchmarks','*.c')
+
+# Set the clean/clobber tasks
+CLOBBER.include(Dir[File.join('examples','*','*_*-*')])
+
+# Pick a target from a list of possible targets
+#               0             1                2                  3              4         5   
+TARGETS = ['GPU-CUDA','GPU-OPENCL-AMD','CPU-OPENCL-INTEL','CPU-OPENCL-AMD','CPU-OPENMP','CPU-C']
+TARGET = TARGETS[0]
+
+# Settings for Bones
+MEASUREMENTS = true
+VERIFICATION = true
+
+# Small helper function to display text on screen
+def display(text)
+	print '[Rake] ### '+text+': '
+	p
+end
+
+# Set the default task
+task :default => [:examples]
+
+# Rake tasks related to the examples
+namespace :examples do
+	
+	# Task to process and test everything (generating code, compiling code, executing)
+	desc 'Run all examples through Bones, compile them, and execute them'
+	task :verify, [:file] => [:generate, :compile, :execute] do |t, args|
+	end
+	
+	# Task to pass examples through Bones
+	desc 'Generate target code using Bones'
+	task :generate, :file do |t, args|
+		args.with_defaults(:file => EXAMPLES)
+		Dir[args.file].sort.each do |file|
+			display('Generating')
+			options = (MEASUREMENTS ? '-m ' : '') + (VERIFICATION ? '-c ' : '')
+			sh "bin/bones -a #{file} -t #{TARGET} #{options}"
+		end
+	end
+	
+	# Task to compile the generated code for the examples (NOTE: this task is a stub)
+	desc 'Compile all examples (using gcc/nvcc)'
+	task :compile, :file do |t, args|
+		args.with_defaults(:file => EXAMPLES)
+		Dir[args.file].sort.each do |file|
+			compile(file,TARGET)
+		end
+	end
+	
+	# Task to execute the compiled code for the examples (NOTE: this task is a stub)
+	desc 'Execute all examples'
+	task :execute, :file do |t, args|
+		args.with_defaults(:file => EXAMPLES)
+		Dir[args.file].sort.each do |file|
+			execute(file,TARGET)
+		end
+	end
+
+	# Helper function to compile code
+	#def compile(file,target)
+		# (system-specific, to be filled in by the user)
+	#end
+
+	# Helper function to execute code
+	#def execute(file,target)
+		# (system-specific, to be filled in by the user)
+	#end
+	
+end
+task :examples => ['examples:generate']
+
+# Task which adds a new target to the skeleton library based on an existing target
+desc 'Adds a new target to the skeleton library'
+task :add_target, :name, :base do |t, args|
+	args.with_defaults(:name => 'NEW-TARGET', :base => 'CPU-OPENMP')
+	base = 'skeletons/'+args.base
+	name = 'skeletons/'+args.name
+	if File.exists?(base) && !File.exists?(name)
+		sh "cp -r #{base} #{name}"
+	else
+		puts '[Rake] ### Error adding new target'
+	end
+end
+
+# Test individual parts of the code
+Rake::TestTask.new do |test|
+	test.test_files = FileList[File.join('test','*','test_*.rb')]
+	test.verbose = false
+end
+
+# Generate HTML documentation using RDoc
+RDoc::Task.new do |rdoc|
+	rdoc.title = 'Bones'
+	rdoc.options << '--line-numbers'
+	rdoc.rdoc_files.include(File.join('lib','**','*.rb'))
+	rdoc.rdoc_files.include('README.rdoc')
+	rdoc.rdoc_dir = 'rdoc'
+	rdoc.main = 'README.rdoc'
+end
+

data/VERSION

@@ -0,0 +1 @@
+1.1.0

data/bin/bones

@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+
+# Set the path for the libraries and the installation directory
+BONES_DIR = File.dirname(__FILE__) + '/../'
+lib_dir = File.join(BONES_DIR,'lib')
+$LOAD_PATH.unshift lib_dir unless $LOAD_PATH.include?(lib_dir)
+
+# Enable verbose output
+VERBOSE = true
+
+# Load the libraries
+require 'castaddon.rb'
+require 'bones.rb'
+
+# Run the main function
+bones = Bones::Engine.new
+bones.process
+bones.write_output
+
+

data/examples/applications/ffos.c

@@ -0,0 +1,552 @@
+//
+// This file is part of the Bones source-to-source compiler examples. This C-code
+// demonstrates the use of Bones for an example (real) application: 'Fast Focus
+// on Structures' (FFOS). For more information on the application or on Bones
+// please use the contact information below.
+//
+// == More information on the FFOS application
+// Contact............Yifan He / Zhenyu Ye
+// Web address........http://zhenyu-ye.net/publications/acivs2011/yifan2011acivs.pdf
+// 
+// == More information on Bones
+// Contact............Cedric Nugteren <c.nugteren@tue.nl>
+// Web address........http://parse.ele.tue.nl/bones/
+//
+// == File information
+// Filename...........applications/ffos.c
+// Author.............Cedric Nugteren
+// Last modified on...22-May-2012
+//
+
+//########################################################################
+//### Includes
+//########################################################################
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <math.h>
+#include <time.h>
+
+//########################################################################
+//### Defines
+//########################################################################
+
+#define XVECTORS 10
+#define YVECTORS 10
+
+//########################################################################
+//### Forward declarations
+//########################################################################
+
+void SaveBMPFile(unsigned char ** image, const char * outputdestination, int width, int height);
+unsigned char ** LoadBMPFile(int *width, int *height);
+void CPU_FindCenters(int* vector, int *coordinates, int size);
+void CPU_Visualize(unsigned char** image0, int* Xcoordinates, int* Ycoordinates, unsigned char **image3, int width, int height);
+void CPU_BCV(int *histogram, float *BCVtable, int size);
+
+//########################################################################
+//### Global variables
+//########################################################################
+
+int messages = 2;
+
+//########################################################################
+//### Start of the main function
+//########################################################################
+
+int main(void) {
+
+	// Declare loop variables
+	int i,h,w,a;
+
+	// Set other variables
+	int threshold = 0;
+	int hist[256];
+	for (i=0;i<256;i++) {	hist[i] = 0; }	
+	float * BCVtable = (float *)malloc(256*sizeof(float));
+
+	// Loading image0 from disk
+	if (messages == 2) { printf("### Loading image0 from disk.\n"); }
+	int width = 0;
+	int height = 0;
+	unsigned char ** image0 = LoadBMPFile(&width, &height);
+	
+	// Create space for image1
+	if (messages == 2) { printf("### Allocating space for image1.\n"); }
+	unsigned char ** image1 = (unsigned char **)malloc(width*sizeof(*image1));
+	unsigned char * image1_1D = (unsigned char *)malloc(width*height*sizeof(unsigned char));
+	for(i=0;i<width;i++) {	image1[i] = &image1_1D[i*height];	}
+	
+	// Create space for image2
+	if (messages == 2) { printf("### Allocating space for image2.\n"); }
+	unsigned char ** image2 = (unsigned char **)malloc(width*sizeof(*image2));
+	unsigned char * image2_1D = (unsigned char *)malloc(width*height*sizeof(unsigned char));
+	for(i=0;i<width;i++) { image2[i] = &image2_1D[i*height];	}
+	
+	// Create space for image3
+	if (messages == 2) { printf("### Allocating space for image3.\n"); }
+	unsigned char ** image3 = (unsigned char **)malloc(width*sizeof(*image3));
+	unsigned char * image3_1D = (unsigned char *)malloc(width*height*sizeof(unsigned char));
+	for(i=0;i<width;i++) { image3[i] = &image3_1D[i*height];	}
+	
+	// Create space for projection vectors
+	if (messages == 2) { printf("### Allocating space for projection vectors.\n"); fflush(stdout); }
+	int * Xvector = (int *)malloc(width*sizeof(int));
+	int * Yvector = (int *)malloc(height*sizeof(int));
+	
+	// Create coordinate arrays
+	if (messages == 2) { printf("### Allocating space for coordinate arrays.\n"); fflush(stdout); }
+	int Xcoordinates[XVECTORS]; for(i=0;i<XVECTORS;i++) { Xcoordinates[i] = 0; }
+	int Ycoordinates[YVECTORS]; for(i=0;i<YVECTORS;i++) { Ycoordinates[i] = 0; }
+	
+	//########################################################################
+	//### PART1: Histogramming (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART1: Histogramming.\n"); fflush(stdout); }
+	
+	#pragma species kernel 0:height-1,0:width-1|element -> 0:255|shared
+	for (h=0;h<height;h++) {
+		for (w=0;w<width;w++) {
+			hist[image0[h][w]] = hist[image0[h][w]] + 1;
+		}
+	}
+	#pragma species endkernel histogram
+	
+	//########################################################################
+	//### Between class variance (CPU)
+	//########################################################################
+	if (messages == 2) { printf("### Create a between class variance table.\n"); fflush(stdout); }
+	CPU_BCV(hist, BCVtable, width*height);
+
+	//########################################################################
+	//### PART2: Search for the maximum (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART2: Search for the maximum value.\n"); fflush(stdout); }
+	float maximum[1];
+	maximum[0] = 10;
+	int length = 256;
+	
+	//#pragma species kernel 0:255|element -> 0:0|shared
+	for (i=0;i<length;i++) {
+		maximum[0] = (BCVtable[i] > maximum[0]) ? BCVtable[i] : maximum[0];
+	}
+	//#pragma species endkernel maximum_1
+	
+	if (messages == 2) { printf("### Maximum is %.3lf.\n",maximum[0]); fflush(stdout); }
+	
+	//########################################################################
+	//### PART3: Search for the maximum - larger synthetic example (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART3: Search for the maximum value (synthetic example).\n"); fflush(stdout); }
+	int vector_size = 2097152; // 2048x1024
+	float* synthetic_vector = (float*)malloc(sizeof(float)*vector_size);
+	srand(time(NULL));for (i=0;i<vector_size;i++) {	synthetic_vector[i] = (rand() % 7777777) / 1000.0; }
+	float result[1];
+	result[0] = 0;
+	
+	//#pragma species kernel 0:2097151|element -> 0:0|shared
+	for (i=0;i<vector_size;i++) {
+		result[0] = (synthetic_vector[i] > result[0]) ? synthetic_vector[i] : result[0];
+	}
+	//#pragma species endkernel maximum_2
+	
+	if (messages == 2) { printf("### Maximum is %.3lf.\n",result[0]); fflush(stdout); }
+	
+	//########################################################################
+	//### Search for the index of the maximum (CPU)
+	//########################################################################
+	if (messages == 2) { printf("### Search for the index of the maximum value.\n"); fflush(stdout); }
+	for (i=0;i<256;i++) {
+		if (BCVtable[i] == maximum[0]) {
+			threshold = i;
+			break;
+		}
+	}
+	
+	//########################################################################
+	//### PART4: Binarization (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART4: Binarization with treshold at %d.\n",threshold); fflush(stdout); }
+	
+	#pragma species kernel 0:height-1,0:width-1|element -> 0:height-1,0:width-1|element
+	for (h=0;h<height;h++) {
+		for (w=0;w<width;w++) {
+			if (image0[h][w] > threshold) { image1[h][w] = 1; }
+			else {                          image1[h][w] = 0; }
+		}
+	}
+	#pragma species endkernel threshold
+	
+	//########################################################################
+	//### PART5: Erosion 7x7 (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART5: Perform the erode kernel.\n"); fflush(stdout); }
+	
+	int condition;
+	#pragma species kernel 7:height-8,7:width-8|neighbourhood(-3:3,-3:3) -> 0:height-1,0:width-1|element
+	for (h=0;h<height;h++) {
+		for (w=0;w<width;w++) {
+			if (w >= 7 && h >= 7 && w <= width-7 && h <= height-7) {
+				condition = 1;
+				for(a=-3;a<=3;a++) {
+					condition = condition
+											* image1[(h-3)][(w+a)]
+											* image1[(h-2)][(w+a)]
+											* image1[(h-1)][(w+a)]
+											* image1[(h+0)][(w+a)]
+											* image1[(h+1)][(w+a)]
+											* image1[(h+2)][(w+a)]
+											* image1[(h+3)][(w+a)]
+					;
+				}
+				if (condition == 1) { image2[h][w] = 255; }
+				else {                image2[h][w] = 0; }
+			}
+			else {
+				image2[h][w] = 0;
+			}
+		}
+	}
+	#pragma species endkernel erosion
+	
+	//########################################################################
+	//### PART6: 1D erosion(7) synthetic example (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART6: Perform the erode kernel (1D - synthetic).\n"); fflush(stdout); }
+	int vector_size2 = 2097152; // 2048x1024
+	int* vector2a = (int*)malloc(sizeof(int)*vector_size2);
+	int* vector2b = (int*)malloc(sizeof(int)*vector_size2);
+	srand(time(NULL));
+	for (i=0;i<vector_size2;i++) {
+		if (rand()%15 > 1) { vector2a[i] = 1; }
+		else              { vector2a[i] = 0; }
+	}
+	
+	//#pragma species kernel 0:2097151|neighbourhood(-3:3) -> 0:2097151|element
+	for (i=0;i<vector_size2;i++) {
+		if (i >= 7 && i <= vector_size2-7) {
+			condition = 1;
+			for(a=-3;a<=3;a++) {
+				condition = condition * vector2a[i+a];
+			}
+			if (condition == 1) { vector2b[i] = 255; }
+			else {                vector2b[i] = 0; }
+		}
+		else {
+			vector2b[i] = 0;
+		}
+	}
+	//#pragma species endkernel erosion1d
+	
+	// Compute a gold reference
+	int gold = 0;
+	int gold_condition = 1;
+	for(a=-3;a<=3;a++) { gold_condition = gold_condition * vector2a[10+a]; }
+	if (gold_condition == 1) { gold = 255; }
+	if (messages == 2) { printf("### Result at index 10 is %d and should be %d.\n",vector2b[10],gold); fflush(stdout); }
+	
+	//########################################################################
+	//### PART7: Y-projection (accelerated)
+	//########################################################################	
+	if (messages >= 1) { printf("### PART7: Starting the Y-projection algorithm.\n"); fflush(stdout); }
+	
+	int result_yp;
+	#pragma species kernel 0:height-1,0:width-1|chunk(0:height-1,0:0) -> 0:width-1|element
+	for (w=0;w<width;w++) {
+		result_yp = 0;
+		for (h=0;h<height;h++) {
+			if (image2[h][w] == 255) {
+				result_yp = 255;
+			}
+		}
+		Yvector[w] = result_yp;
+	}
+	#pragma species endkernel y_projection
+	
+	//########################################################################
+	//### PART8: X-projection (accelerated)
+	//########################################################################
+	if (messages >= 1) { printf("### PART8: Starting the X-projection algorithm.\n"); fflush(stdout); }
+	
+	int result_xp;
+	#pragma species kernel 0:height-1,0:width-1|chunk(0:0,0:width-1) -> 0:height-1|element
+	for (h=0;h<height;h++) {
+		result_xp = 0;
+		for (w=0;w<width;w++) {
+			if (image2[h][w] == 255) {
+				result_xp = 255;
+			}
+		}
+		Xvector[h] = result_xp;
+	}
+	#pragma species endkernel x_projection
+	
+	//########################################################################
+	//### Search for the centers of the projection vectors (CPU)
+	//########################################################################	
+	if (messages == 2) { printf("### Search for X- and Y-projection vectors.\n"); fflush(stdout); }
+	CPU_FindCenters(Xvector, Xcoordinates, width);
+	CPU_FindCenters(Yvector, Ycoordinates, height);
+		
+	//########################################################################
+	//### Visualize, save to disk and finalize the program
+	//########################################################################
+	CPU_Visualize(image0, Xcoordinates, Ycoordinates, image3, width, height);
+	SaveBMPFile(image1, "output1.bmp", width, height);
+	SaveBMPFile(image2, "output2.bmp", width, height);
+	SaveBMPFile(image3, "output3.bmp", width, height);
+	free(image0);
+	free(image1);
+	free(image1_1D);
+	free(image2);
+	free(image2_1D);
+	free(image3);
+	free(image3_1D);
+	free(Xvector);
+	free(Yvector);
+	free(BCVtable);
+	if (messages == 2) { printf("### End of program\n"); fflush(stdout); }
+	return 0;
+}
+
+//########################################################################
+//### Structures used in the BMP functions
+//########################################################################
+
+typedef struct {
+	int size;
+	int reserved;
+	int offset;
+} BMPHeader;
+typedef struct {
+	int size;
+	int width;
+	int height;
+	int planesBitsPerPixel;
+	int compression;
+	int imageSize;
+	int xPelsPerMeter;
+	int yPelsPerMeter;
+	int clrUsed;
+	int clrImportant;
+} BMPInfoHeader;
+
+//########################################################################
+//### Function to save BMP data to a file
+//########################################################################
+
+void SaveBMPFile(unsigned char ** image, const char * outputdestination, int width, int height)
+{
+	// Variable declarations
+	int x,y,j;
+	FILE *fd_out;
+  unsigned long ulBitmapSize = (height * width * 3)+54; 
+  char ucaBitmapSize[4];
+  ucaBitmapSize[3]= (ulBitmapSize & 0xFF000000) >> 24;
+  ucaBitmapSize[2]= (ulBitmapSize & 0x00FF0000) >> 16;
+  ucaBitmapSize[1]= (ulBitmapSize & 0x0000FF00) >> 8;
+  ucaBitmapSize[0]= (ulBitmapSize & 0x000000FF);
+  	
+	// Load output file
+	fd_out = fopen(outputdestination, "wb");
+ 		
+	// Write BMP header
+	fprintf(fd_out,"%c%c%c%c%c%c%c%c%c%c", 66, 77, ucaBitmapSize[0], ucaBitmapSize[1], ucaBitmapSize[2], ucaBitmapSize[3], 0, 0, 0, 0); 
+	fprintf(fd_out,"%c%c%c%c%c%c%c%c%c%c", 54, 0, 0, 0, 40, 0 , 0, 0, (width & 0x00FF), (width & 0xFF00)>>8); 
+	fprintf(fd_out,"%c%c%c%c%c%c%c%c%c%c", 0, 0, (height & 0x00FF), (height & 0xFF00) >> 8, 0, 0, 1, 0, 24, 0); 
+	fprintf(fd_out,"%c%c%c%c%c%c%c%c%c%c", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0); 
+	fprintf(fd_out,"%c%c%c%c%c%c%c%c%c%c", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
+	fprintf(fd_out,"%c%c%c%c", 0, 0 ,0, 0);
+
+	// Save RGB data to output file	
+	for(y=0;y<height;y++) {
+		for(x=0;x<width;x++)	{
+			fputc(image[x][y],fd_out);
+			fputc(image[x][y],fd_out);
+			fputc(image[x][y],fd_out);
+		}
+		int over = width%4;
+		if (over != 0) {
+			for(j=0;j<over;j++) {
+				fputc(0,fd_out);
+			}
+		}
+	}
+
+	// Clean up
+	fclose(fd_out);
+}
+
+//########################################################################
+//### Function to load BMP data from disk
+//########################################################################
+
+unsigned char ** LoadBMPFile(int *width, int *height)
+{
+	// Variable declarations
+	short type;
+	int temp;
+	BMPHeader hdr;
+	BMPInfoHeader infoHdr;
+	FILE *fd;
+	int i, y, x;
+	
+	// Open the file stream
+	fd = fopen("../../../input.bmp","rb");
+
+	// Open the file and scan the contents
+	if(!(fd)) { printf("***BMP load error: file access denied***\n"); exit(0);	}
+	temp = fread(&type, sizeof(short), 1, fd);
+	temp = fread(&hdr, sizeof(hdr), 1, fd);
+	if(type != 0x4D42) { printf("***BMP load error: bad file format***\n"); exit(0); }
+	temp = fread(&infoHdr, sizeof(infoHdr), 1, fd);
+	if((infoHdr.planesBitsPerPixel>>16) != 24) { printf("***BMP load error: invalid color depth (%d)*** \n",(infoHdr.planesBitsPerPixel>>16)); exit(0); }
+	if(infoHdr.compression) { printf("***BMP load error: compressed image***\n"); exit(0); }
+	(*width)  = infoHdr.width;
+	(*height) = infoHdr.height;
+
+	// Allocate memory to store the BMP's contents
+	unsigned char ** image = (unsigned char **)malloc((*width) * sizeof(*image));
+	unsigned char * image_1D = (unsigned char *)malloc((*width) * (*height) * sizeof(unsigned char));
+	for(i=0; i<(*width); i++) {
+		image[i] = &image_1D[i*(*height)];
+	}
+
+	// Read the BMP file and store the contents
+	fseek(fd, hdr.offset - sizeof(hdr) - sizeof(infoHdr), SEEK_CUR);
+	for(y = 0; y < (*height); y++) {
+		for(x = 0; x < (*width); x++)	{
+			image[x][y] = ((int)fgetc(fd));
+			fgetc(fd);
+			fgetc(fd);
+		}
+		int over = (4 - ((*width)*3) % 4) % 4;
+		if (over != 0) {
+			for(x = 0; x < over; x++)	{
+				fgetc(fd);
+			}
+		}
+	}
+
+	// Exit the function and clean-up
+	if(ferror(fd)) {
+		printf("***Unknown BMP load error.***\n");
+		free(image[0]);
+		free(image);
+		exit(0);
+	}
+	fclose(fd);
+	return image;
+}
+
+//########################################################################
+//### Find the center of a projection vector (using a state machine)
+//########################################################################		
+void CPU_FindCenters(int* vector, int *coordinates, int size) {
+	int s;
+	int state = 0;
+	int count = 0;
+	int coordinate = 0;
+	for (s=0;s<size;s++) {
+		if (state == 0) {                           // Last thing I found was a zero
+			if (vector[s] == 255) {                  // I found a 255 now
+				state = 1;
+				count = 0;
+			}
+		}		
+		if (state == 1) {                           // Last thing I found was 255
+			if (vector[s] == 0) {                    	// I found a zero now
+				state = 0;
+				if (count > 4) {                        // To filter out noise
+					coordinates[coordinate] = s-(count/2);
+					coordinate++;
+				}
+			}	
+			else {                                    // I found a 255 again
+				count++;
+			}
+		}
+	}
+}
+
+//########################################################################
+//### CPU kernel to visualize the results
+//########################################################################
+void CPU_Visualize(unsigned char** image0, int* Xcoordinates, int* Ycoordinates, unsigned char **image3, int width, int height) {
+
+	// Loop variables
+	int h, w, x, y;
+
+	// Copy the whole image
+	for (h=0;h<height;h++) {
+		for (w=0;w<width;w++) {
+			unsigned char value = image0[h][w];
+			image3[h][w] = value;
+		}
+	}
+	
+	// Replace the centers with white pixels
+	for (x=0;x<XVECTORS;x++) {
+		for (y=0;y<YVECTORS;y++) {
+			image3[Xcoordinates[x]][Ycoordinates[y]] = 255;
+		}
+	}
+}
+
+//########################################################################
+//### CPU kernel function for between class variance (BCV), part of Otsu thresholding
+//########################################################################		
+void CPU_BCV(int *histogram, float *BCVtable, int size) {
+	int i;
+	
+	// Initialize the BCV table to zero
+	for (i=0;i<256;i++) {
+		BCVtable[i] = 0;
+	}
+	
+	// Pre-calculated the total of the weigthed sums
+	int wsumtotal = 0;
+	for (i=0;i<256;i++) {
+		wsumtotal = wsumtotal + i*histogram[i];
+	}
+	
+	// Set the initial values
+	int sumb = 0;
+	int sumf = size;	
+	int wsumb = 0;
+	int wsumf = wsumtotal;
+	
+	float wb;
+	float wf;
+	float meanb;
+	float meanf;
+
+	// Iterate over all possible threshold values
+	for (i=0;i<256;i++)	{
+
+		// Update the weighted sums
+		wsumb = wsumb + i*histogram[i];
+		wsumf = wsumtotal - wsumb;
+		
+		// Calculate the necessary components
+		wb = sumb / (float)size;
+		wf = sumf / (float)size;
+		meanb = wsumb / (float)sumb;
+		meanf = wsumf / (float)sumf;
+	
+		// Stop if the sum of foreground is equal to zero
+		if (sumf == 0) { break; }
+		
+		// Output the BCV value
+		BCVtable[i] = wb*wf*(meanb-meanf)*(meanb-meanf);		
+		
+		// If the sum of the background was equal to zero, BCV table will be NaN and must be set to zero
+		if (sumb == 0) { BCVtable[i] = 0; }		
+		
+		// Update the sum of the background (all darker pixels compared to the current pixel) and foreground pixels (the rest)
+		sumb = sumb + histogram[i];
+		sumf = size - sumb;
+	}
+}
+
+//########################################################################