openstudio-workflow 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: cc2c789cb6f241ce8a9693f4fd52c2cea857a6ab
+  data.tar.gz: 8ead2192e95b45850c00a0b0e048d326eef05ead
+SHA512:
+  metadata.gz: 31c526b09291097df08aef677a51e1dd46bfb64710c0a3a0fef1b6e2a8f9fc2ab0d23973afcdd651dedfe7f17f77266fcb6ef7d5df67a80d39addb590ef9b358
+  data.tar.gz: acaee1bf5ae39b1f4be3f67b4d292dfe790f0415e1f1aa04552bdae09ad1003ad0b44c45939a4b5f0f4ed76ff76ec1522871a61dd2fd5c508c8848b88685cbec

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+OpenStudio::Workflow Change Log
+==================================
+
+Unreleased
+--------------
+
+Version 0.0.1
+--------------
+
+* Initial release with basic workflow implemented to support running OpenStudio measure-based workflows

data/README.md

@@ -0,0 +1,97 @@
+# OpenStudio::Workflow
+
+Run an EnergyPlus simulation using a file-based workflow that is read from a Local or MongoDB adapter.
+
+## Installation
+
+This applications has the following dependencies
+
+* Ruby 2.0
+* OpenStudio with Ruby 2.0 bindings
+* EnergyPlus 8.1 (assuming OpenStudio ~> 1.3.1)
+* MongoDB if using MongoDB Adapter (or when running rspec)
+
+[OpenStudio](http://developer.nrel.gov/downloads/buildings/openstudio/builds/) needs to be installed
+and in your path.  On Mac/Linux it is easiest to add the following to your .bash_profile or /etc/profile.d in order
+to make sure that OpenStudio can be loaded.
+
+    export OPENSTUDIO_ROOT=/usr/local
+    export RUBYLIB=$OPENSTUDIO_ROOT/lib/ruby/site_ruby/2.0.0
+
+Add this line to your application's Gemfile:
+
+    gem 'OpenStudio-workflow'
+
+Use this line if you want the bleeding edge:
+
+    gem 'OpenStudio-workflow', :git => 'git@github.com:NREL/OpenStudio-workflow-gem.git'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install OpenStudio-workflow
+
+## Usage
+
+There are currently two adapters to run OpenStudio workflow. The first is a simple Local adapter
+allowing the user to pass in the directory to simulation. The directory must have an
+[analysis/problem JSON file](spec/files/local_ex1/analysis_1.json) and a [datapoint JSON file](spec/files/local_ex1/datapoint_1.json).
+The workflow manager will use these data (and the measures, seed model, and weather data) to assemble and
+execute the standard workflow of (preflight->openstudio measures->energyplus->postprocess).
+
+    r = OpenStudio::Workflow.load 'Local', '/home/user/a_directory', options
+    r.run
+
+The workflow manager can also use MongoDB to receive instructions on the workflow to run and the data point values.
+
+## Caveats and Todos
+
+### Caveats
+
+* There are currently several hard coded workflow options
+* Must use OpenStudio with Ruby 2.0 support
+* Using MongoDB as the Adapter requires a command line zip (gnuzip) utility
+
+### Todos
+
+* Read the analysis.json file to determine the states that are going to run instead of (or inaddition to) passing them into the constructor
+* Implement better error handling with custom exception classes
+* Implement a different measure directory, seed model directory, and weather file directory option
+* Dynamically add other "states" to the workflow
+* Create and change into a unique directory when running measures
+* ~~Implement Error State~~
+* ~~Implement MongoDB Adapter~~
+* ~~Implement remaining Adapter states (i.e. communicate success, communicate failure etc~~
+* Add a results adapter to return a string as the last call based on the source of the call. (e.g. R, command line, C++, etc).
+* Implement a logger in the Adapters, right now they are unable to log
+* Hook up the measure groups based workflows
+* More tests!
+* ~~Add xml workflow item~~
+
+## Testing and Development
+
+Depending on what adapter is being tested it may be preferable to skip installing various gems.  This can be done by calling
+
+    bundle install --without mongo
+
+On Windows it is recommended to bundle without mongo nor ci as they may require native extensions.
+
+    bundle install --without mongo ci
+
+### Testing
+
+Run `rspec` or `rake` to execute the tests.
+
+## Contributing
+
+1. Fork it ( https://github.com/NREL/OpenStudio-workflow/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+## Development
+

data/Rakefile

@@ -0,0 +1,20 @@
+require 'bundler'
+require 'bundler/gem_tasks'
+begin
+  Bundler.setup
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts 'Run `bundle install` to install missing gems'
+  exit e.status_code
+end
+
+require 'rake'
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+#require 'rubocop/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+task default: [:spec]

data/lib/openstudio/workflow/adapter.rb

@@ -0,0 +1,68 @@
+######################################################################
+#  Copyright (c) 2008-2014, Alliance for Sustainable Energy.
+#  All rights reserved.
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 2.1 of the License, or (at your option) any later version.
+#
+#  This library is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#  Lesser General Public License for more details.
+#
+#  You should have received a copy of the GNU Lesser General Public
+#  License along with this library; if not, write to the Free Software
+#  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+######################################################################
+
+# Adapter class to decide where to obtain instructions to run the simulation workflow
+module OpenStudio
+  module Workflow
+    class Adapter
+
+      attr_accessor :options
+
+      def initialize(options={})
+        @options = options
+        @log = nil
+      end
+
+      #class << self
+      #attr_reader :problem
+
+      def load(filename, options={})
+        instance.load(filename, options)
+      end
+
+      def communicate_started(id, options = {})
+        instance.communicate_started id
+      end
+
+      def get_datapoint(id, options={})
+        instance.get_datapoint id, options
+      end
+
+      def get_problem(id, options = {})
+        instance.get_problem id, options
+      end
+
+      def communicate_results(id, results)
+        instance.communicate_results id, results
+      end
+
+      def communicate_complete(id)
+        instance.communicate_complete id
+      end
+
+      def communicate_failure(id)
+        instance.communicate_failure id
+      end
+
+      def get_logger(file, options={})
+        instance.get_logger file, options
+      end
+    end
+  end
+end

data/lib/openstudio/workflow/adapters/local.rb

@@ -0,0 +1,110 @@
+######################################################################
+#  Copyright (c) 2008-2014, Alliance for Sustainable Energy.
+#  All rights reserved.
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 2.1 of the License, or (at your option) any later version.
+#
+#  This library is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#  Lesser General Public License for more details.
+#
+#  You should have received a copy of the GNU Lesser General Public
+#  License along with this library; if not, write to the Free Software
+#  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+######################################################################
+
+require_relative '../adapter'
+
+# Local file based workflow
+module OpenStudio
+  module Workflow
+    module Adapters
+      class Local < Adapter
+        def initialize(options={})
+
+          super
+        end
+
+        # Tell the system that the process has started
+        def communicate_started(directory, options = {})
+          # Watch out for namespace conflicts (::Time is okay but Time is OpenStudio::Time)
+          File.open("#{directory}/started.job", 'w') { |f| f << "Started Workflow #{::Time.now}" }
+        end
+
+        # Get the data point from the path
+        def get_datapoint(directory, options={})
+          defaults = {datapoint_filename: 'datapoint.json', format: 'json'}
+          options = defaults.merge(options)
+
+          # how do we log within this file?
+          if File.exist? "#{directory}/#{options[:datapoint_filename]}"
+            ::MultiJson.load(File.read("#{directory}/#{options[:datapoint_filename]}"), symbolize_names: true)
+          else
+            fail "Data point file does not exist for #{directory}/#{options[:datapoint_filename]}"
+          end
+        end
+
+        # Get the Problem/Analysis definition from the local file
+        # TODO: rename this to get_analysis_definintion (or something like that)
+        def get_problem(directory, options = {})
+          defaults = {problem_filename: 'problem.json', format: 'json'}
+          options = defaults.merge(options)
+
+          if File.exist? "#{directory}/#{options[:problem_filename]}"
+            ::MultiJson.load(File.read("#{directory}/#{options[:problem_filename]}"), symbolize_names: true)
+          else
+            fail "Problem file does not exist for #{directory}/#{options[:problem_filename]}"
+          end
+        end
+
+        def communicate_intermediate_result(directory)
+          # noop
+        end
+
+        def communicate_complete(directory)
+          File.open("#{directory}/finished.job", 'w') { |f| f << "Finished Workflow #{::Time.now}" }
+        end
+
+        # Final state of the simulation. The os_directory is the run directory and may be needed to
+        # zip up the results of the simuation.
+        def communicate_failure(directory)
+          File.open("#{directory}/failed.job", 'w') { |f| f << "Failed Workflow #{::Time.now}" }
+          #@communicate_module.communicate_failure(@communicate_object, os_directory)
+        end
+
+        def communicate_results(directory, results)
+          if results.is_a? Hash
+            File.open("#{directory}/datapoint_out.json", 'w') { |f| f << JSON.pretty_generate(results) }
+          else
+            pp "Unknown datapoint result type. Please handle #{results.class}"
+            #data_point_json_path = OpenStudio::Path.new(run_dir) / OpenStudio::Path.new('data_point_out.json')
+            #os_data_point.saveJSON(data_point_json_path, true)
+          end
+          #end
+        end
+
+        # TODO: can this be deprecated in favor a checking the class?
+        def communicate_results_json(eplus_json, analysis_dir)
+          # noop
+        end
+
+        def reload
+          # noop
+        end
+
+        # For the local adapter send back a handle to a file to append the data. For this adapter
+        # the log messages are likely to be the same as the run.log messages.
+        # TODO: do we really want two local logs from the Local adapter? One is in the run dir and the other is in the root
+        def get_logger(directory, options={})
+          @log ||= File.open("#{directory}/local_adapter.log", "w")
+          @log
+        end
+
+      end
+    end
+  end
+end

data/lib/openstudio/workflow/adapters/mongo.rb

@@ -0,0 +1,259 @@
+######################################################################
+#  Copyright (c) 2008-2014, Alliance for Sustainable Energy.
+#  All rights reserved.
+#
+#  This library is free software; you can redistribute it and/or
+#  modify it under the terms of the GNU Lesser General Public
+#  License as published by the Free Software Foundation; either
+#  version 2.1 of the License, or (at your option) any later version.
+#
+#  This library is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#  Lesser General Public License for more details.
+#
+#  You should have received a copy of the GNU Lesser General Public
+#  License along with this library; if not, write to the Free Software
+#  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+######################################################################
+
+require_relative '../adapter'
+
+module OpenStudio
+  module Workflow
+    module Adapters
+      class MongoLog
+        def initialize(datapoint_model)
+          @dp = datapoint_model
+          @dp.sdp_log_file ||= []
+        end
+
+        def write(msg)
+          @dp.sdp_log_file << msg.gsub('\n', '')
+          @dp.save!
+        end
+      end
+
+      class Mongo < Adapter
+        attr_reader :datapoint
+
+        def initialize(options={})
+          super
+
+          require 'mongoid'
+          require 'mongoid_paperclip'
+          require 'delayed_job_mongoid'
+          base_path = @options[:mongoid_path] ? @options[:mongoid_path] : "#{File.dirname(__FILE__)}/mongo"
+
+          Dir["#{base_path}/models/*.rb"].each { |f| require f }
+          Mongoid.load!("#{base_path}/mongoid.yml", :development)
+
+          @datapoint = nil
+        end
+
+        # Tell the system that the process has started
+        def communicate_started(directory, options={})
+          # Watch out for namespace conflicts (::Time is okay but Time is OpenStudio::Time)
+          File.open("#{directory}/started.job", 'w') { |f| f << "Started Workflow #{::Time.now}" }
+
+          @datapoint ||= get_datapoint_model(options[:datapoint_id])
+          @datapoint.status = 'started'
+          @datapoint.status_message = ''
+          @datapoint.run_start_time = ::Time.now
+
+
+          # TODO: use a different method to determine if this is an amazon account
+          # TODO: use the ComputeNode model to pull out the information so that we can reuse the methods
+          # Determine what the IP address is of the worker node and save in the data point
+          require 'socket'
+          if Socket.gethostname =~ /os-.*/
+            # Maybe use this in the future: /sbin/ifconfig eth1|grep inet|head -1|sed 's/\:/ /'|awk '{print $3}'
+            # Must be on vagrant and just use the hostname to do a lookup
+            map = {
+                'os-server' => '192.168.33.10',
+                'os-worker-1' => '192.168.33.11',
+                'os-worker-2' => '192.168.33.12'
+            }
+            @datapoint.ip_address = map[Socket.gethostname]
+            @datapoint.internal_ip_address = @datapoint.ip_address
+
+            # TODO: add back in the instance id
+          elsif Socket.gethostname =~ /instance-data.ec2.internal.*/i
+            # On amazon, you have to hit an API to determine the IP address because
+            # of the internal/external ip addresses
+
+            # NL: add the suppress
+            public_ip_address = `curl -sL http://169.254.169.254/latest/meta-data/public-ipv4`
+            internal_ip_address = `curl -sL http://169.254.169.254/latest/meta-data/local-ipv4`
+            # instance_information = `curl -sL http://169.254.169.254/latest/meta-data/instance-id`
+            # instance_information = `curl -sL http://169.254.169.254/latest/meta-data/ami-id`
+            @datapoint.ip_address = public_ip_address
+            @datapoint.internal_ip_address = internal_ip_address
+            # @datapoint.server_information = instance_information
+          else
+            if Gem.loaded_specs["facter"]
+              # TODO: add hostname via the facter gem (and anything else?)
+              @datapoint.ip_address = Facter.fact(:ipaddress).value
+              @datapoint.internal_ip_address = Facter.fact(:hostname).value
+            end
+          end
+
+          @datapoint.save!
+        end
+
+        # Get the data point from the path
+        def get_datapoint(directory, options={})
+          # TODO : make this a conditional on when to create one vs when to error out.
+          # keep @datapoint as the model instance
+          @datapoint = DataPoint.find_or_create_by(uuid: options[:datapoint_id])
+
+          # convert to JSON for the workflow - and rearrange the version (fix THIS)
+          datapoint_hash = {}
+          unless @datapoint.nil?
+            datapoint_hash[:data_point] = @datapoint.as_document.to_hash
+            # TODO: Can i remove this openstudio_version stuff?
+            #datapoint_hash[:openstudio_version] = datapoint_hash[:openstudio_version]
+
+            # TODO: need to figure out how to get symbols from mongo.
+            datapoint_hash = MultiJson.load(MultiJson.dump(datapoint_hash, pretty: true), symbolize_keys: true)
+          else
+            fail "Could not find datapoint"
+          end
+
+          datapoint_hash
+        end
+
+        # TODO: cleanup these options.  Make them part of the class. They are just unwieldly here.
+        def get_problem(directory, options = {})
+          defaults = {format: 'json'}
+          options = defaults.merge(options)
+
+          get_datapoint(directory, options) unless @datapoint
+
+          if @datapoint
+            analysis = @datapoint.analysis.as_document.to_hash
+          else
+            fail "Cannot retrieve problem because datapoint was nil"
+          end
+
+          analysis_hash = {}
+          if analysis
+            analysis_hash[:analysis] = analysis
+            analysis_hash[:openstudio_version] = analysis[:openstudio_version]
+
+            # TODO: need to figure out how to get symbols from mongo.
+            analysis_hash = MultiJson.load(MultiJson.dump(analysis_hash, pretty: true), symbolize_keys: true)
+          end
+          analysis_hash
+        end
+
+        def communicate_intermediate_result(directory)
+          # noop
+        end
+
+        def communicate_complete(directory)
+          @datapoint.run_end_time = ::Time.now
+          @datapoint.status = 'completed'
+          @datapoint.status_message = 'completed normal'
+          @datapoint.save!
+        end
+
+        # Final state of the simulation. The os_directory is the run directory and may be needed to
+        # zip up the results of the simuation.
+        def communicate_failure(directory)
+          # zip up the folder even on datapoint failures
+          if directory && File.exist?(directory)
+            zip_results(directory)
+          end
+
+          @datapoint.run_end_time = ::Time.now
+          @datapoint.status = 'completed'
+          @datapoint.status_message = 'datapoint failure'
+          @datapoint.save!
+        end
+
+        def communicate_results(directory, results)
+          zip_results(directory, 'workflow')
+
+          #@logger.info 'Saving EnergyPlus JSON file'
+          if results
+            @datapoint.results ? @datapoint.results.merge!(eplus_json) : @datapoint.results = results
+          end
+          result = @datapoint.save! # redundant because next method calls save too.
+          if result
+            #@logger.info 'Successfully saved result to database'
+          else
+            #@logger.error 'ERROR saving result to database'
+          end
+        end
+
+        # TODO: can this be deprecated in favor a checking the class?
+        def communicate_results_json(eplus_json, analysis_dir)
+          # noop
+        end
+
+        # TODO: not needed anymore i think...
+        def reload
+          # noop
+        end
+
+        # TODO: Implement the writing to the mongo_db for logging
+        def get_logger(directory, options={})
+          # get the datapoint object
+          get_datapoint(directory, options) unless @datapoint
+          @log = OpenStudio::Workflow::Adapters::MongoLog.new(@datapoint)
+          @log
+        end
+
+        private
+
+        def get_datapoint_model(uuid)
+          # TODO : make this a conditional on when to create one vs when to error out.
+          # keep @datapoint as the model instance
+          DataPoint.find_or_create_by(uuid: uuid)
+        end
+
+        # TODO: this uses a system call to zip results at the moment
+        def zip_results(analysis_dir, analysis_type = 'workflow')
+          eplus_search_path = nil
+          current_dir = Dir.pwd
+          FileUtils.mkdir_p "#{analysis_dir}/reports"
+          case analysis_type
+            when 'workflow'
+              eplus_search_path = "#{analysis_dir}/*run*/eplustbl.htm"
+            when 'runmanager'
+              eplus_search_path = "#{analysis_dir}/*EnergyPlus*/eplustbl.htm"
+          end
+
+          # copy some files into a report folder
+          eplus_html = Dir.glob(eplus_search_path).last || nil
+          if eplus_html
+            #@logger.info "Checking for HTML Report: #{eplus_html}"
+            if File.exist? eplus_html
+              # do some encoding on the html if possible
+              html = File.read(eplus_html)
+              html = html.force_encoding('ISO-8859-1').encode('utf-8', replace: nil)
+              File.open("#{analysis_dir}/reports/eplustbl.html", 'w') { |f| f << html }
+            end
+          end
+
+          # create zip file using a system call
+          #@logger.info "Zipping up Analysis Directory #{analysis_dir}"
+          if File.directory? analysis_dir
+            Dir.chdir(analysis_dir)
+            `zip -9 -r --exclude=*.rb* data_point_#{@datapoint.uuid}.zip .`
+          end
+
+          # zip up only the reports folder
+          report_dir = "#{analysis_dir}"
+          #@logger.info "Zipping up Analysis Reports Directory #{report_dir}/reports"
+          if File.directory? report_dir
+            Dir.chdir(report_dir)
+            `zip -r data_point_#{@datapoint.uuid}_reports.zip reports`
+          end
+          Dir.chdir(current_dir)
+        end
+      end
+    end
+  end
+end