lorj 0.1.0

data/README.md

@@ -0,0 +1,652 @@
+# Lorj
+
+**Lorj** library is a process/controller framework used to build inter-operable process functions against any kind of controllers.
+
+It has been designed to provide cloud agnostics feature to forj command line tool and FORJ maestro gardener.
+
+This framework helps to design any kind of solution which needs to maintain a generic process logic, fully controller independant.
+
+## Implementations examples
+
+### How forj cli implemented Lorj lib
+
+This case was implemented by forj cli to implement services on any kind of cloud solution.
+
+forj cli needs to build a forge (collection of servers) on a cloud.
+It requires to ensure that everything is in place (network properly configured if not create it, router exists, flavors available, etc.) to create the first server.
+
+For **Lorj**, how to create the forge and ensure that everything is in place is the **process**.<br>
+And then how to manipulate the cloud (get the network, create the network, create the server, etc...) is the **controller**. <br>
+Usually, the controller is a wrapper to an API, which do actions, like 'create server'.
+
+Then, at runtime, while changing the controller to aws, hpcloud, openstack, or even docker or decker, forj cli process will be still the same.
+It simplifies forj cli extension, by just adding new controller to support more clouds providers.
+
+Currently, forj cli has implemented *hpcloud* and *openstack* providers, using *fog*. Docker controller is under development.
+
+## Getting started
+
+As playing by example is better than a long discussion, let's start playing with a simple example.
+
+### Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'lorj'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install lorj
+
+
+### Write your first Lorj code
+
+#### Student database management example context:
+
+Imagine you have an API code which today manages students database in yaml format.<br>
+We have written a small class to illustrate this API. (See examples/yaml_students/yaml_students.rb)
+
+This api has everything to create, query, edit and delete a student in a yaml file.
+
+Using Lorj, you want to propose multiple storage system (not only using the yaml format) to your application, by supporting several storage systems, like mysql DB.
+
+next, we will write 3 versions, which will introduce how to deal with process and controllers:
+
+* version 1:
+     Writing your first 'do nothing' process, with no controller.
+
+* version 2:
+     Complete the process to have create/query/get and delete capability, with mock controller.
+     The mock controller is basically a controller keeping data in memory.
+
+* version 3:
+     Update the version 2 to fully implement the example of an yaml_student API.
+
+     The main is `examples/yaml_students/students.rb`<BR>
+     The API is `examples/yaml_students/yaml_students.rb`
+
+We have created a full example version 4 which implements some interesting way to improve your code available in `examples/students_4/`
+
+---
+#### Writing student version 1
+
+**NOTE**: following example is available in `examples/students_1/`
+
+First of all, write your first main. Create a file `example.rb` with following content:
+
+##### File `students.rb`
+
+``` ruby
+    #!/usr/bin/env ruby
+
+    $APP_PATH = File.dirname(__FILE__)
+    require 'lorj'
+
+    # If you want to see what is happening in the framework, uncomment debug settings.
+    # PrcLib.level = Logger::DEBUG # Printed out to your console.
+    # PrcLib.core_level = 3 # framework debug levels.
+
+    # Initialize the framework
+    hProcesses = [ File.join($APP_PATH, 'process', 'students.rb')]
+    oStudentCore = Lorj::Core.new( nil, hProcesses)
+
+    # Ask the framework to create the object student 'Robert Redford'
+    oStudentCore.Create(:student, {:student_name => "Robert Redford"})
+```
+
+##### File `process/Students.rb`
+
+Now, let's write our first process. We are going to create a file Students.rb under a sub-directory 'process'.
+
+``` ruby
+  # Students process
+  class StudentsProcess
+    def create_student(sObjectType, hParams)
+        puts "Running creation process for object '%s' = '%s'" % [sObjectType, hParams[:student_name] ]
+
+        # If you prefer to print out to the log system instead:
+        # PrcLib::debug("Running creation process for object '%s' = '%s'" % [sObjectType, hParams[:student_name] ])
+    end
+  end
+
+  # Declaring your data model and handlers.
+  class Lorj::BaseDefinition
+
+     # We need to define the student object and the handler to use while we need to create it.
+     define_obj(:student,
+     {
+        :create_e => :create_student # The function to call in the class Students
+     })
+
+     obj_needs    :data,         :student_name,        { :for => [:create_e] }
+
+  end
+```
+
+What did we wrote?
+
+* We defined a *StudentsProcess* class
+
+    This is the core of your process. It describes how to handle the object requested. The data to use is available in `hParams`.
+
+    **IMPORTANT !!** There is no reference to any files or Database connection.
+
+    **NOTE**: The framework requires you to define a class name composed by the name of the process and 'Process'.
+    Here, it is 'Students' + **'Process'**
+
+
+* We declared the data model and the process handlers with declaration in *Lorj::BaseDefinition* class.
+
+     In short, we declared `:student` object, which needs `:student_name` at creation step.
+
+     **NOTE** Implicitly, the framework consider that the object contains at least 2 data fields, :id (should be unique) and :name (string)
+
+
+Currently, this model do nothing except printing out.
+
+#####typical output
+
+    $ example/students_1/students.rb
+    WARNING: PrcLib.app_defaults is not set. Application defaults won't be loaded. 
+    Running creation process for object 'student' = 'Robert Redford'
+    WARNING: 'create_student' has returned no data for object Lorj::Data 'student'!
+
+There is 2 warnings. 
+
+* **PrcLib.app_defaults** represents the application name. It is used to keep your application configuration data in ~/.{PrcLib.app_defaults}/ directory. Defining this data in your main, will eliminate this warning.
+* **create_student** function in your process has returned nothing. while lorj is called to create an object, it assumes to get the object data created. Lorj keep those data in a cache. In this example, **create_student** returned nil, and lorj raise a warning about that. 
+
+---
+#### Writing student version 2
+
+**NOTE**: following example is available in `examples/students_2/`
+
+lorj comes with a `:mock` controller. This one is really basic. It keep data in an Hash in MockController class.<br>
+In this example, we add create/get/query/delete capability in the process and use the `:mock` controller to store the data.
+
+##### File `process/Students.rb`
+
+Add 3 handlers query_e, get_e and delete_e like this. Add the :mapping case as well.
+
+``` ruby
+  class Lorj::BaseDefinition
+     define_obj(:student,
+     {
+        :create_e => :create_student,   # The function to call in the class Students
+        :query_e  => :controller_query, # We use predefined call to the controller query
+        :get_e    => :controller_get,   # We use predefined call to the controller get
+        :delete_e => :controller_delete # We use predefined call to the controller delete
+     })
+
+  # Note about mapping. This is usually done by the controller. We will see this later.
+  obj_needs    :data,         :student_name,        { :for => [:create_e], :mapping => :name }
+
+  end
+```
+
+Update the handler `create_student`:
+
+``` ruby
+  def create_student(sObjectType, hParams)
+      PrcLib::state ("Running creation process for object '%s' = '%s'" % [sObjectType, hParams[:student_name] ])
+
+      oObject = controler.create(sObjectType)
+      raise "Student '%s' not created." % hParams[:student_name] if oObject.nil?
+      PrcLib::info ("'%s': '%s' created with id %s" % [sObjectType, hParams[:student_name],  oObject[:id]])
+      oObject
+  end
+```
+
+
+
+##### File `students.rb`
+The update your main and add those lines. In the following, we create 3 students, while 2 are duplicated.
+
+``` ruby
+
+    # Want to create a duplicated student 'Robert Redford'?
+    oStudentCore.Create(:student)
+    # no problem. The key is the key in the Mock controller array.
+
+    oStudentCore.Create(:student, :student_name => "Anthony Hopkins")
+
+    # Let's create a third different student.
+    oStudents = oStudentCore.Query(:student, { :name => "Robert Redford" } )
+
+    puts "%s students found" % oStudents.length
+
+    oStudents.each { | oStudent |
+       puts "%s: %s" % [oStudent[:id], oStudent[:name]]
+    }
+
+    # let's check the get function, who is the ID 2?
+    oStudent = oStudentCore.Get(:student, 2)
+
+    puts "The student ID 2 is %s" % oStudent[:name]
+
+```
+
+#####typical output:
+
+    $ example/students_2/students.rb
+    WARNING: PrcLib.app_defaults is not set. Application defaults won't be loaded. 
+    2 students found
+    0: Robert Redford
+    1: Robert Redford
+    The student ID 2 is Anthony Hopkins
+
+
+Cool! But everything is in memory! We would like to write this in a file.
+Writing our controller, means that the process should not be updated anymore!
+
+Let's move to the most interesting version. Translate an API example to use lorj!
+
+#### Writing student version 3
+
+**NOTE**: following example is available in `examples/students_3/`
+
+A controller is like a wrapper to an API which offer what we would like to use to save our yaml file.<br>
+We already have a running API that we can use to manage students in yaml format.
+
+This API is written in `examples/yaml_students/yaml_students.rb`. <br>
+`examples/yaml_students/students.rb` is the main program calling the api and create a basic yaml file example in `/tmp/students.yaml`
+
+You can review the main 'students.rb' to see the simple code.
+
+In short, this code do:
+
+* create 3 students, only if they do not exists
+* remove a wrong one
+* Identify list of students on a specific training
+* Identify students removed.
+
+We will do the same thing with lorj.
+
+Using lorj means you are going to split your code in different pieces. You will need to integrate a layer between your main and the API. This layer is composed by :
+
+* 1 generic data model
+* 1 generic process handlers
+* 1 controller definition - expanding the data model for our Student yaml API
+* 1 controller code - wrapper to the student yaml API.
+
+So, we are going to update following files:
+
+* students.rb - Adapt the main to do the same as original students script using the student yaml API.
+* process/students.rb - Student data model - Update the process data model.
+* process/students.rb - Student process handler - Update student process handlers.
+* controller/yaml_students.rb - controller definition - Extend the student data model with controller definition.
+* controller/yaml_students_controller.rb - controller code - Write the student yaml API wrapper.
+
+First of all, let me re-visit the :student data model, then the student process handlers.
+
+We need to define more parameters for handlers, and define the data model.
+
+##### File `process/students.rb`
+Update the class Lorj::BaseDefinition, and the data model and handlers parameters.
+
+*The generic data model:*
+
+``` ruby
+    class Lorj::BaseDefinition
+       define_obj(:student,
+       {
+          :create_e => :create_student,   # The function to call in the class Students
+          :query_e  => :controller_query, # We use predefined call to the controller query
+          :delete_e => :controller_delete # We use predefined call to the controller delete
+       })
+
+       # obj_needs is used to declare parameters to pass to handlers.
+       # :for indicates those parameters to be passed to create_e handler only.
+       # Those data (or objects) will be collected and passed to the process handler as hParams.
+
+       obj_needs   :data,   :student_name,          { :for => [:create_e] }
+
+       # By default, all data are required.
+       # You can set it as optional. Your process will need to deal with this optional data.
+       obj_needs_optional
+       obj_needs   :data,   :first_name,            { :for => [:create_e] }
+       obj_needs   :data,   :last_name,             { :for => [:create_e] }
+       obj_needs   :data,   :course  # Note that in this model, the training is renamed as course.
+                                     # the controller will need to map it to 'training'.
+
+       # def_attribute defines the data model.
+       # The process will be able to access those data
+       # if the controller has mapped them.
+       # For the exercice, I have changed the name of the training field to become :course instead.
+       def_attribute :course
+       # Same thing for the student name. instead of 'name', we defined :student_name
+       def_attribute :student_name
+       def_attribute :first_name
+       def_attribute :last_name
+       def_attribute :status
+
+       undefine_attribute :name
+   end
+```
+
+Then, update the process to manage student duplicates. Update function `create_student`
+
+*The generic process handlers:*
+
+``` ruby
+    class StudentsProcess
+       def create_student(sObjectType, hParams)
+          PrcLib::state ("Running creation process for object '%s' = '%s'" % [sObjectType, hParams[:student_name] ])
+
+          # config object is a reference to runtime/config data.
+          oList = Query(sObjectType, {:name => hParams[:student_name]})
+          case oList.length
+             when 0
+                oObject = controller_create(sObjectType)
+                raise "Student '%s' not created." % hParams[:student_name] if oObject.nil?
+                PrcLib::info ("'%s': '%s' created with id %s" % [sObjectType, hParams[:student_name],  oObject[:id]])
+             when 1
+                oObject = oList[0]
+                PrcLib::info ("'%s': '%s' loaded with id %s" % [sObjectType, hParams[:student_name],  oObject[:id]])
+             else
+                oObject = oList[0]
+                PrcLib::warning("More than one student named '%s' is found: %s records. Selecting the first one and removing duplicates." % [hParams[:student_name], oList.length])
+                iCount = 0
+                oList[1..-1].each { | elem |
+                   register(elem)
+                   iCount += controller_delete(sObjectType)
+                }
+                PrcLib::info ("'%s': %s duplicated '%s' removed. First loaded with id %s" % [sObjectType, iCount, hParams[:student_name],  oObject[:id]])
+          end
+          oObject
+       end
+    end
+```
+
+Here you see that we query, check list, create if missing, or delete if duplicates found.
+You can run it now, as we still uses mock controller. It should work.
+
+##### File `students.rb`
+
+Now, let's update the main to be close to what we have on `examples/yaml_students/students.rb`
+
+This is a simple basic translation of `examples/yaml_students/students.rb`
+
+``` ruby
+    #!/usr/bin/env ruby
+
+    $APP_PATH = File.dirname(__FILE__)
+    require 'lorj'
+    require 'ansi'
+
+    # If you want to see what is happening in the framework, uncomment debug settings.
+    # PrcLib.level = Logger::DEBUG # Printed out to your console.
+    # PrcLib.core_level = 3 # framework debug levels. Values between 0 to 5.
+
+    # Initialize the framework
+    hProcesses = [ File.join($APP_PATH, 'process', 'students.rb')]
+
+    # You can try with mock instead. uncomment the next line and comment 2 next lines.
+    #~ oStudentCore = Lorj::Core.new( nil, hProcesses, :mock)
+    oStudentCore = Lorj::Core.new( nil, hProcesses, File.join($APP_PATH, 'controller', 'yaml_students.rb'))
+    oStudentCore.Create(:connection, :connection_string => "/tmp/students.yaml")
+
+    puts ANSI.bold("Create 1st student:")
+
+    oStudentCore.Create(:student, {
+       student_name:  'Robert Redford',
+       first_name:    'Robert',
+       last_name:     'Redford',
+       course:        'Art Comedy'
+    })
+
+    puts ANSI.bold("Create 2nd student:")
+    oStudentCore.Create(:student, {
+       student_name:  'Anthony Hopkins',
+       first_name:    'Anthony',
+       last_name:     'Hopkins',
+       course:        'Art Drama'
+    })
+
+    puts ANSI.bold("Create 3rd student:")
+    oStudentCore.Create(:student, {
+       student_name: "Marilyn Monroe",
+       first_name:   'Marilyn',
+       last_name:    'Monroe',
+       course:       'Art Drama'
+    })
+
+    puts ANSI.bold("Create mistake")
+    oStudentCore.Create(:student, {
+       :student_name  => "Anthony Mistake",
+       :first_name    => 'Anthony',
+       :last_name     => 'Mistake',
+       :course        => 'what ever you want!!!'
+    })
+
+    # Because the last student was the mistake one, we can directly delete it.
+    # Usually, we use get instead.
+    puts ANSI.bold("Remove mistake")
+    oStudentCore.Delete(:student)
+
+    puts ANSI.bold("List of students for 'Art Drama':")
+    puts oStudentCore.Query(:student, { :course => "Art Drama"}).to_a
+
+    puts ANSI.bold("Deleted students:")
+    puts oStudentCore.Query(:student,{ :status => :removed}).to_a
+```
+
+We have reproduced the code. Note that there is no if while creating. It is embedded in the create_student process.
+
+##### File `controller/yaml_students_controller.rb`
+
+We need to write the controller part, now. As I said, it is like a wrapper. Let's have a look:
+
+*The controller code:*
+
+``` ruby
+    # declare yaml student API to the controller
+    cur_file = File.expand_path(File.join(File.dirname(File.dirname(__FILE__)), "..", "yaml_students", 'yaml_students.rb'))
+    require cur_file
+
+    # The controller is a combination of 2 elements:
+    # - Controller class
+    #   Code which will interfere with the external API.
+    #
+    #   The file name must respect the name of the class. 1st letter already capitalized and letter after _ is capitalized.
+    #   file: my_code.rb => needs to create MyCodeController class
+    #
+    # - Definition class
+    #   This class declare any kind of mapping or additional fields to consider.
+    #   Additionnal fields are unknow by the process. So, those fields will needs to be setup before.
+    #
+    #   file name convention is identical than controller class.
+    #   file: my_code.rb => needs to create MyCode class
+
+    class YamlStudentsController
+       def initialize()
+          @@valid_attributes = [:name, :first_name, :last_name, :id, :status, :training]
+       end
+
+       def create(sObjectType, hParams)
+          case sObjectType
+             when :connection
+                required?(hParams, :hdata, :file_name)
+                YamlSchool.new(hParams[:hdata, :file_name])
+             when :student
+                required?(hParams, :connection)
+                required?(hParams, :student_name)
+
+                # We use the hdata built by the lorj. See set_hdata in the next file.
+                hParams[:connection].create_student(hParams[:student_name], hParams[:hdata])
+             else
+                Error "'%s' is not a valid object for 'create'" % sObjectType
+          end
+       end
+
+       # This function return a collection which have to provide:
+       # functions: [], length, each
+       def query(sObjectType, sQuery, hParams)
+          case sObjectType
+             when :student
+                required?(hParams, :connection)
+
+                hParams[:connection].query_student(sQuery)
+             else
+                Error "'%s' is not a valid object for 'create'" % sObjectType
+          end
+
+       end
+
+       def delete(sObjectType, hParams)
+          case sObjectType
+             when :student
+                required?(hParams, :connection)
+
+                hParams[:connection].delete_student(hParams[sObjectType][:id])
+             else
+                Error "'%s' is not a valid object for 'create'" % sObjectType
+          end
+       end
+
+       def get_attr(oControlerObject, key)
+          # This controller function read the data and
+          # extract the information requested by the framework.
+          # Those data will be mapped to the process data model.
+          # The key is an array, to get data from a level tree.
+          # [data_l1, data_l2, data_l3] => should retrieve data from structure like data[ data_l2[ data_l3 ] ]
+          begin
+             attributes = oControlerObject
+             raise "get_attr: attribute '%s' is unknown in '%s'. Valid one are : '%s'" % [key[0], oControlerObject.class, @@valid_attributes ] unless @@valid_attributes.include?(key[0])
+             Lorj::rhGet(attributes, key)
+          rescue => e
+             Error "get_attr: Unable to map '%s'. %s" % [key, e.message]
+          end
+       end
+
+       def set_attr(oControlerObject, key, value)
+          begin
+             attributes = oControlerObject
+             raise "set_attr: attribute '%s' is unknown in '%s'. Valid one are : '%s'" % [key[0], oControlerObject.class, @@valid_attributes ] unless @@valid_attributes.include?(key[0])
+             Lorj::rhSet(attributes, value, key)
+          rescue => e
+             Error "set_attr: Unable to map '%s' on '%s'" % [key, sObjectType]
+          end
+       end
+
+       def update(sObjectType, oObject, hParams)
+          case sObjectType
+             when :student
+                required?(hParams, :connection)
+
+                hParams[:connection].update_student(oObject)
+             else
+                Error "'%s' is not a valid object for 'create'" % sObjectType
+          end
+       end
+
+    end
+
+```
+In short, we wrap:
+
+- create with YamlSchool.create_student
+- delete with YamlSchool.delete_student
+- query with YamlSchool.query_student
+- update with YamlSchool.update_student
+
+And we have defined 2 additional functions
+
+- get_attr: to extract data from a YamlSchool Object
+- set_attr: to set data to a YamlSchool Object
+
+##### File `controller/yaml_students.rb`
+
+And we need to write some mapping stuff to the controller. We have to add this
+
+*The controller definition:*
+
+``` ruby
+    class YamlStudents
+       # This is a new object which is known by the controller only.
+       # Used to open the yaml file. Generically, I named it :connection.
+       # But this can be any name you want. Only the controller will deal with it.
+       define_obj(:connection,{
+        :create_e => :controller_create # Nothing complex to do. So, simply call the controller create.
+       })
+    
+       obj_needs   :data,   :connection_string,  :mapping => :file_name
+       undefine_attribute :id    # Do not return any predefined ID
+       undefine_attribute :name  # Do not return any predefined NAME
+    
+       # The student model have to be expanded.
+       define_obj(:student)
+       # It requires to create a connection to the data, ie opening the yaml file.
+       # So, before working with the :student object, the controller requires a connection
+       # This connection will be loaded in the memory and provided to the controller
+       # when needed.
+       obj_needs   :CloudObject,              :connection
+    
+       # To simplify controller wrapper, we use hdata built by lorj, and passed to the API
+       # This hdata is a hash containing mapped data, thanks to set_hdata.
+       set_hdata :first_name
+       set_hdata :last_name
+       # Instead of 'course', the yaml API uses 'training'
+       set_hdata :course, :mapping => :training
+        
+       get_attr_mapping :course, :training
+       # instead of 'student_name', the yaml API uses 'name'
+       get_attr_mapping :student_name, :name
+    
+       # This controller will know how to manage a student file with those data.
+       # But note that the file can have a lot of more data than what the process
+       # usually manage. It is up to you to increase your process to manage more data.
+       # Then each controller may need to define mapping fields.
+    end
+```
+
+That's it!
+
+#####typical output:
+    $ example/students_3/students.rb 
+    WARNING: PrcLib.app_defaults is not set. Application defaults won't be loaded. 
+    Create 1st student:
+    Create 2nd student:
+    Create 3rd student:
+    Create mistake
+    Student created '{:id=>3, :course=>"what ever you want!!!", :student_name=>"Anthony Mistake", :first_name=>"Anthony", :last_name=>"Mistake", :status=>:active}'
+    Remove mistake
+    Wrong student to remove: 3 = Anthony Mistake
+    List of students for 'Art Drama':
+    {:id=>1, :course=>"Art Drama", :student_name=>"Anthony Hopkins", :first_name=>"Anthony", :last_name=>"Hopkins", :status=>:active}
+    {:id=>2, :course=>"Art Drama", :student_name=>"Marilyn Monroe", :first_name=>"Marilyn", :last_name=>"Monroe", :status=>:active}
+    Deleted students:
+    {:id=>3, :course=>"what ever you want!!!", :student_name=>"Anthony Mistake", :first_name=>"Anthony", :last_name=>"Mistake", :status=>:removed}
+    
+## What next?
+
+Check the code in the `examples/students_3/`. It provides a little more than is written here.
+
+In short, it introduces the usage of Lorj::Config, before initializing Lorj::Core.
+
+## next features
+
+lorj provides 2 differents configuration class.
+
+* Lorj::Config  - Basic config systems. Data are retrieved in **application defaults**, then **local config**, then **runtime**.
+* Lorj::Account - Based on Lorj::Config, it introduces an **account** file, between **local config** and **runtime**.
+* Logging system - Based on logger. A log file contains all data until debug. It prints to the console only data specified by PrcLib::.level
+* lorj.setup - This function works with Lorj::Account. Thanks an application data definition (defaults.yaml), lorj ask the user information to build an account. Those data will be loaded and used by your process/controller flow.
+
+For details, see API documentation.
+
+# Contributing to Lorj
+
+We welcome all types of contributions.  Checkout our website (http://docs.forj.io/en/latest/dev/contribute.html)
+to start hacking on Lorj or Forj.  Also join us in our community (https://www.forj.io/community/) to help grow and foster Lorj and Forj for your development today!
+
+
+#License:
+
+Lorj lib is licensed under the Apache License, Version 2.0.  See LICENSE for full license text.

data/Rakefile

@@ -0,0 +1,24 @@
+# encoding: UTF-8
+#
+# (c) Copyright 2014 Hewlett-Packard Development Company, L.P.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+desc 'Run the specs.'
+RSpec::Core::RakeTask.new do |t| 
+  t.pattern = 'spec/*_spec.rb'
+end
+