flok 0.0.12 → 0.0.14

data/docs/rakefile.md

@@ -0,0 +1,13 @@
+#Rakefile
+
+##Rubygem
+`rake push` - Increment the version of the gem by `0.0.1`, push to *Github*, tag the commit with the version, and then push the update to *RubyGems.org*
+
+##Testing
+Tasks for testing are listed in [Testing](./testing.md)
+
+##Interactive
+Tasks for interactive (pipes, etc) are listed in [Interactive](./interactive.md)
+
+##Building
+Tasks for ubilding are listed in [Compilation](./)

data/docs/scheduling.md

@@ -0,0 +1,45 @@
+#Scheduling
+**This document has incorrect information. It is a document still under development**
+Flok is capable of loading 10k images simultaneously while maintaining 1ms response time on typical JS environments for UI. This is all accomplished through its scheduling system.
+
+##History
+Traditional operating systems have a *scheduler* which puts *tasks* into a *runnable* state based on the amount of *CPU* resources that task has used.  Task *fairness*, which is the amount of CPU time a task is granted, is a [heated debate](http://yarchive.net/comp/linux/fairness.html).  Our scheduler is based on the concepts of the `ULE` scheduler from [FreeBSD](freebsd.org)
+
+##Not CPU time slices.  Resources.
+The first departure from a traditional operating system scheduler is that *Flok* schedules **resources**, a super-set of **cpu time slices**.  Additionally, Flok schedules with an evented system that does not rely on a timer (as many modern OS schedulers are moving to).
+
+##The standard Flok resources are defined with the labels:
+  0. `main` - User-interface displaying, etc.
+  1. `net` - Downloading, Uploading, Get requests, etc.
+  2. `disk` - Transferring things to/from disk
+  3. `cpu` - Tasks that tax the cpu
+  4. `gpu` - Tasks that tax the gpu
+
+##Flok also has two priority classes (Does not effect `main` queue):
+  * `high` - Things that need to happen soon.
+  * `low` - Things that can probably wait.
+
+Despite the similarities among these classes, there are three distinct groups of classes.  `rt` is a queue that is **never** deferred. it has the lowest latency response time and is used for UI operations. `high`, `norm`, and `low` are all **asynchronous** queues.  They are **always** deferred. `idle` is also, **always** deferred, but `idle` tasks *only* run when **nothing** else is running.
+
+##Flok tasks
+Each `task` in flok is a transaction to the native device that contains a call-back and or fixed-run-time. Each `task` belongs to *N* labels and `1` class.
+
+##Sharing the pie
+Scheduling is tricky because you can't write a scheduler that just ignores everything that isn't a *high* task if there are *high* tasks queud. That kind of scheduler will never execute other tasks, even if they are of lower importance.  Like `ULE`, `Flok` cuts the scheduling groups up into a pie so that a higher percentage of `high` tasks are given simultaneously and a low percentage of `med`, and lowest for `low` are given.
+
+##Messages from the server
+Messages sent via `if_dispatch` to the server have a special format that looks like this:
+```javascript
+  msg = [
+    [0, 0, "ping", 1, "ping2", "hello"], => <main queue>: [0, "ping"] and [1, "ping2", "hello"]
+    [1, 1, "download_image", "http://testimage.com/test.png"],
+    [4, 1, "blur_button", 23]
+  ]
+```
+
+The message is broken up into *3* distinct queues.  The first queue, queue 0, is the **main** queue. Each queue should be interpreted in order. That
+means the *main* queue will always be synchronously executed before the rest of the queues are asynchronously dispatched. The `download_image` is
+apart of the `net` queue, and the *gpu* is part of queue 4.  Look above at *Resource Labels* to see what each queue is.
+
+## message queue count
+Each time a message is sent, the queue the message to belongs to is decremented by 1.

data/docs/services.md

@@ -0,0 +1,74 @@
+#Services
+Services are entirely built into the Flok kernel. If you need custom actions for your services, they must be created as a driver and then
+you must adhere to the services API.
+
+###Request
+You initiate a service request via `Request`. This may be called only in the controller at this time. This macro takes several parameters
+parameters, all of which must either be strict variable names or double quoted strings:
+
+  * `Request(name, info, event_name)`
+    * `name` - The name of the service you are making a request from
+    * `info` - The information to give the service
+    * `event_name` - The name of the event when a callback occurs
+
+Callbacks are handled by redirecting them through the [event](./mod/event.md) interface `int_event`.  You should register with `reg_evt` to receive
+events back from a service. The request itself is not made via the event system, only the response is.  This may seem a little odd because normally
+events come in from the outside while this is an internally generated event.
+
+###Request API
+Registering a service involves adding files to the `./app/kern/services` folder. All *ruby* files in this folder are used in the compilation of 
+services. You declare services in the following format. Your service should not depend on anything except particular module functions.
+
+```
+service("rest") do
+  on_init %{
+    #Code inserted here is put into the global space, add initialization
+    #procedures, functions that need to be called, etc.
+    #
+    #You may use the function respond(info) within here to 
+
+    #Store the in-progress requests as a list of hashes
+    #that contain an array in the order of [event_pointer, evenct_name]
+    var service_rest_tp_to_einfo = {}
+
+    function service_rest_callback(tp, success, info) {
+      //Lookup event info
+      var einfo = service_rest_tp_to_einfo[tp];
+
+      //Send info back to service
+      int_event(einfo[0], einfo[1], {:success => success, :info => info});
+
+      //Remove entries in telepointer table and rest service info
+      tel_del(tp);
+      delete service_rest_tp_to_einfo[tp];
+    }
+  }
+
+  on_request %{
+    #Code that handles a payload goes here.
+    #You have access to `info`, `ep`, and `ename` which was given in the Request macro
+
+    //Create a GET request that will respond to the telepointer
+    var tp = tel_reg(service_rest_callback);
+
+    //Now register the event information to respond to when a callback is received
+    service_rest_tp_to_einfo[tp] = [ep, ename]
+
+    //Start the request
+    SEND("net", "if_net_req", "GET", info.url, info.params, tp)
+  }
+end
+```
+
+This is then compiled down to
+```js
+//***************************************
+//on_init code is appended to the outside
+//....
+//***************************************
+
+//Your request code is put inside a function
+function service_rest_req(info, ep, ename) {
+  //on_request code
+}
+```

data/docs/services/rest.md

@@ -0,0 +1,21 @@
+#Rest Service
+This is the RESTful network request service. Designed for small payloads of RESTful network requests.
+
+###Info to start a request
+```js
+var info = {
+  url: "http://my_rest_endpoint",
+  params: {
+    my_param_0: "test",
+  }
+}
+```
+
+When you receive a request back, you will receive 
+```js
+{
+  success: true,
+  info: 
+}
+```
+Where if success is true, then info is a hash. If success is false, then info is a failure message.

data/docs/services/timer.md

@@ -0,0 +1,21 @@
+#Timer service
+This service manages the `int_timer` function (which is a little different than how most modules work). When an `int_timer` request
+comes in, the timer service dispatches it to an apprpriate controller event handler.
+
+Timer service mantains an `timer_evt` which contains an array of arrays for interval timers that will be called (recurring) periodically.
+The elements of `timer_evt` are an array with the following information: `[tps, base_pointer, event_name]`.  When `int_timer` is called,
+it will send events to all entries in `timer_evt` which have a `ticks` that is a modulo of `ttick`; the global tick counter.
+
+###Info to start a request
+```js
+var info = {
+  ticks: 4,
+}
+```
+`ticks` the number of ticks to wait between fires.
+
+When you receive a request back, you will receive 
+```js
+{
+}
+```

data/docs/testing.md

@@ -1 +1,80 @@
-# Testing
+#Testing
+All tests that relate to the kernel are automatically built with `FLOK_ENV=DEBUG`
+
+##./spec layout
+The `spec` folder contains all tests with the exception of **Driver** tests (See below for information).
+
+  * `spec/`
+    - `etc/`  - All tests that do not fit any categories.
+      -`lib/` - Tests that pertain to the ./lib/flok ruby library
+    - `iface/`
+      - `kern/` - All tests that *only* need the kernel (server) pipe.
+      - `driver/` - All tests that *only* need the driver (client) pipe.
+      -  `all/` - All tests that need both the kernel and driver pipe.
+    - `kern/` - All tests for the kernel that do not need the pipe.
+    - `lib/` - A folder of useful subroutines.
+    - `env/` - Files that are automatically loaded at the beginning of each test that provide the environment.
+      - `kern.rb` (for `./spec/kern/*_spec.rb`)
+      - `iface.rb` (for `./spec/iface/**/*_spec.rb`)
+      - `etc.rb` (for `./spec/etc/*_spec.rb`)
+
+For each `_spec.rb` file in the `./spec` folder, make sure to change the directory root to the project root. For example, if your spec file was
+`./spec/iface/kern/pipe_spec.rb` then you would put `Dir.chdir File.join File.dirname(__FILE__), '../../../'` at the top. RSpec needs to be able to
+execute anywhere.
+
+##Drivers
+See the section on *Spec* in [Drivers](./drivers.md). Driver tests can be executed from the global rakefile by the convenience task `rake spec:driver PLATFORM=$PLATFORM`
+
+##Kernel
+Kernel tests only operate on the kernel and should **never** include tests that can be accomplished through using the kernel's pipe interface. Kernel tests are given a pre-build `V8` context that has the `application.js` loaded through `therubyracer`.
+
+ * Running - `rake spec:kern PLATFORM=$PLATFORM`
+ * Requires - `./spec/etc/kern.rb`
+ * Location - `./spec/kern/*_spec.rb`
+ * Environment
+   * `include_context "kern"` - Access `@ctx`, a V8 runtime that has `application.js` preloaded. See [therubyracer](https://github.com/cowboyd/therubyracer)
+   * `mock(function_name, &block)` - Override a function of `@ctx` with a ruby method
+   * `$PLATFORM` - Current platform the `application.js` was built for.
+   * `$PWD` - `./`
+
+##Interface
+Interface tests are used for things that can be tested through either only the kernel pipe, driver pipe, or both.
+  * Running - `rake spec:iface PLATFORM=$PLATFORM`
+  * Requires - `./spec/etc/iface.rb`
+  * Location
+    * `./spec/iface/kern` - Tests only need the server pipe
+    * `./spec/iface/driver` - Tests only need the client pipe
+    * `./spec/iface/all` - Tests need both pipes
+  * Variables
+    * `@mods` - The list of modules this driver suppotrs in `DEBUG` mode
+  * Environment
+   * `include_context "iface:kern"` - Access `@pipe` matching the description for kernel pipe [Interactive](./interactive.md)
+   * `include_context "iface:driver"` - Access `@pipe` matching the description for driver pipe [Interactive](./interactive.md)
+   * `$PLATFORM` - The name of the platform of the driver and the parameters for the kernel.
+   * `$PWD` - `./`
+
+For each test, including both `kern` and `driver`, please use the `module_dep(module_name)` to prevent tests from running on unspported modules
+
+##Etc
+Tests that do not fit into any of the categories (e.g. build output tests), should be placed in `./spec/etc`
+These tests are runnable via `rake spec:etc PLATFORM=$PLATFORM`
+
+  * Running - `rake spec:etc PLATFORM=$PLATFORM`
+  * Requires - `./spec/etc/etc.rb`
+  * Location - `./spec/etc/`
+  * Environment
+   * `$PLATFORM` - The name of the platform of the driver and the parameters for the kernel.
+   * `$PRODUCTS_PATH` - Location of the `application.js` file and `drivers` folder.
+   * `$PWD` - `./`
+
+##World & All
+
+####World (Per platform)
+The `rake spec:world PLATFORM=$PLATFORM` executes the following tests:
+  1. The etc tests
+  2. The kernel tests
+  4. The interface tests
+  5. The drivers tests
+
+####All (All platforms)
+The `rake spec` executes the same thing as `rake spec:world PLATFORM=$PLATFORM` but uses every platform automatically. This will test the kernel multiple times, but that is ok, as different platforms have different modules enabled for the kernel.

data/flok.gemspec

@@ -26,8 +26,12 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'webrick', '~> 1.3'
   spec.add_development_dependency "closure-compiler", "~> 1.1"
   spec.add_development_dependency "phantomjs", "~> 1.9"
+  spec.add_development_dependency "pry", "~> 0.10"
   spec.add_development_dependency "rspec-wait", "~> 0.0"
   spec.add_development_dependency "os", "0.9.6"
+  spec.add_development_dependency "boojs", "~> 0.0"
+  spec.add_development_dependency "activesupport", "~> 4.2"
+  spec.add_development_dependency "cakery", "~> 0.0"
   spec.add_runtime_dependency "thor", "~> 0.19"
   spec.executables << 'flok'
 end

data/lib/flok.rb

@@ -1,5 +1,11 @@
 require "flok/version"
+require "flok/utilities"
+require "flok/interactive"
 require "flok/build"
+require "flok/user_compiler"
+require "flok/services_compiler"
+require "flok/platform"
+require "flok/project"
 
 module Flok
 end

data/lib/flok/build.rb

@@ -1,6 +1,6 @@
 require 'yaml'
 require 'json'
-
+require 'erb'
 
 ##################################################################
 #This file contains everything relating to compiling source files
@@ -21,9 +21,32 @@ module Flok
     File.write(output_path, out)
   end
 
+  def self.src_glob_r type, dir_path, output_path
+    out = ""
+    Dir.chdir dir_path do
+      FileUtils.mkdir_p(dir_path)
+      FileUtils.mkdir_p(File.dirname(output_path))
+      nodes = []
+      nodes += Dir["./init/**/*.#{type}"].select{|e| File.file?(e)} 
+      nodes += Dir["./config/**/*.#{type}"].select{|e| File.file?(e)} 
+      final_nodes = Dir["./final/**/*.#{type}"].select{|e| File.file?(e)} 
+      nodes += Dir["./*.#{type}"].select{|e| File.file?(e)}
+      nodes += (Dir["./**/*"] - nodes - final_nodes).select{|e| File.file?(e)}
+      nodes += final_nodes
+      nodes.each do |f|
+        out << File.read(f) << "\n"
+      end
+    end
+
+    File.write(output_path, out)
+  end
+
   #Build the whole world for a certain platform
-  def self.build_world build_path, platform
-    #What platform are we working with?
+  def self.build_world build_path, platform, environment
+    #Environment must be either DEBUG or RELEASE
+    raise "$FLOK_ENV must either be DEBUG or RELEASE, got #{environment.inspect}" unless %w{DEBUG RELEASE}.include? environment
+
+    #Clean up previous build
     `rm -rf #{build_path}`
 
     #1. `rake build` is run inside `./app/drivers/$platform`
@@ -36,44 +59,70 @@ module Flok
     #2. All js files in `./app/kern/config/*.js` are globbed togeather and sent to `./products/$platform/glob/1kern_config.js`
     Flok.src_glob("js", './app/kern/config', "#{build_path}/glob/1kern_config.js")
 
-    #3. All js files in `./app/kern/*.js` are globbed togeather and sent to `./products/$platform/glob/2kern.js`
-    Flok.src_glob("js", './app/kern', "#{build_path}/glob/2kern.js")
+    #3. All js files in `./app/kern/*.js` are globbed togeather and sent to `./products/$platform/glob/2kern.pre_macro.js`
+    Flok.src_glob("js", './app/kern', "#{build_path}/glob/2kern.pre_macro.js")
+
+
+    #4. All rb files inside `./app/kern/services/` are globbed into `./products/$PLATFORM/glob/kern_services.rb`
+    Flok.src_glob("rb", './app/kern/services/', "#{build_path}/glob/kern_services.rb")
 
-    #4. All js files are globbed from `./products/$platform/glob` and combined into `./products/$platform/application.js`
-    Flok.src_glob("js", "#{build_path}/glob", "#{build_path}/application.js")
+    #5. `./products/$PLATFORM/glob/kern_serivces.rb` is processed via `Flok::Services` and then exported as `./products/$PLATFORM/glob/kern_services.pre_macro.js`
+    File.write("#{build_path}/glob/kern_services.pre_macro.js", Flok::ServicesCompiler.compile(File.read("#{build_path}/glob/kern_services.rb")))
 
-    #5. Add custom commands
+    #6. All js files in `./products/$PLATFORM/glob/2kern.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to ./products/$PLATFORM/glob/2kern.js 
+    require './app/kern/macro.rb'
+    File.write("#{build_path}/glob/2kern.pre_macro.js", macro_process(File.read("#{build_path}/glob/2kern.pre_macro.js")))
+
+    #7. All js files in `./products/$PLATFORM/glob/kern_services.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to ./products/$PLATFORM/glob/kern_services.pre_macro.js
+    File.write("#{build_path}/glob/kern_services.pre_macro.js", macro_process(File.read("#{build_path}/glob/kern_services.pre_macro.js")))
+
+    #8. All js files are globbed from `./products/$platform/glob` and combined into `./products/$platform/glob/application.js.erb`
+    Flok.src_glob("js", "#{build_path}/glob", "#{build_path}/glob/application.js.erb")
+
+    #9. Add custom commands
     ################################################################################################################
-    #lsiface() - List interfaces for driver config.yml
+    #MODS - List mods listed in config.yml
     #---------------------------------------------------------------------------------------
     #Load the driver config.yml
     driver_config = YAML.load_file("./app/drivers/#{platform}/config.yml")
     raise "No config.yml found in your 'platform: #{platform}' driver" unless  driver_config
 
     #Create array that looks like a javascript array with single quotes
-    iface_arr = "[" + driver_config['ifaces'].map!{|e| "'#{e}'"}.join(", ") + "]"
+    mods = Flok::Platform.mods(platform, environment)
+    mods_js_arr = "[" + mods.map{|e| "'#{e}'"}.join(", ") + "]"
 
     #Append this to our output file
-    `echo "IFACES = #{iface_arr};" >> #{build_path}/application.js`
-    `echo "PLATFORM = \'#{platform}\';" >> #{build_path}/application.js`
+    `echo "MODS = #{mods_js_arr};" >> #{build_path}/glob/application.js.erb`
+    `echo "PLATFORM = \'#{platform}\';" >> #{build_path}/glob/application.js.erb`
     #---------------------------------------------------------------------------------------
     ################################################################################################################
 
-    `rm -rf #{build_path}/glob`
+    #10. Append relavent mods code in kernel with macros
+    mods.each do |mod|
+      s = File.read("./app/kern/mod/#{mod}.js")
+      open("#{build_path}/glob/application.js.erb", "a") do |f|
+        f.puts macro_process(s)
+      end
+    end
+
+    #11. The compiled `glob/application.js.erb` file is run through the ERB compiler and formed into `application.js`
+    erb_src = File.read "#{build_path}/glob/application.js.erb"
+    renderr = ERB.new(erb_src)
+    context = ApplicationJSERBContext.new()
+    new_src = renderr.result(context.get_binding)
+    File.write "#{build_path}/application.js", new_src
   end
 
-  def self.system! cmd
-    res = system(cmd)
-    out = ""
-    out << "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n"
-    out << "SHELL ERROR\n"
-    out << "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n\n"
-      out << "\t(user@localhost) #{cmd}\n"
-      out << "\t(user@localhost) echo $?\n"
-      out << "\t#{res}\n"
-      out << "\t(user@localhost) pwd\n\t"
-      out << `pwd`
-    out << "\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n"
-    raise out unless res
+  class ApplicationJSERBContext
+    def get_binding
+      return binding
+    end
+
+    def initialize
+      #Debug / Release
+      @debug = (ENV['FLOK_ENV'] == "DEBUG")
+      @release = (ENV['FLOK_ENV'] == "RELEASE")
+      @mods = Flok::Platform.mods(ENV['PLATFORM'], ENV['FLOK_ENV'])
+    end
   end
 end

data/lib/flok/interactive.rb

@@ -0,0 +1,58 @@
+require 'therubyracer'
+require 'v8'
+
+#Interactive pipe for testing. 
+#Creates a server that routes $stdin into int_dispatch and $stdout to inf_dispatch.
+
+module Flok
+  class InteractiveServer
+    def initialize app_js_path
+      @app_js_path = app_js_path
+      @ctx = V8::Context.new
+      @ctx.load app_js_path
+
+      inject_ruby_into_js
+      inject_dispatch_shims
+    end
+
+    #Will take over any remaining IO
+    def begin_pipe
+      loop do
+        begin
+          q = JSON.parse(gets)
+          @ctx[:int_dispatch].call(q)
+        rescue Errno::EIO
+          #Can't say anything here, we don't have a pipe
+        rescue V8::Error => e
+          $stderr.puts "\n--------------------------------------------------" 
+          $stderr.puts "V8 Boo-Boo for #{@app_js_path}"
+          $stderr.puts "--------------------------------------------------\n"
+          $stderr.puts e.message
+          $stderr.puts "--------------------------------------------------\n" 
+          exit 1
+        end
+      end
+    end
+
+    #JS Functions#####################################################
+    def inject_ruby_into_js 
+      #Output to stdout
+      @ctx["write"] = lambda do |this, str|
+        str.each_line do |line|
+          $stdout.puts line
+          $stdout.flush
+        end
+      end
+    end
+
+    #Make calls to if_dispatch go to $stdout, make $stdin call int_dispatch
+    def inject_dispatch_shims
+      @ctx.eval %{
+        function if_dispatch(q) {
+          write(JSON.stringify(q));
+        }
+      }
+    end
+    ##################################################################
+  end
+end