flok 0.0.36 → 0.0.38

data/docs/scheduling.md

@@ -14,6 +14,7 @@ The first departure from a traditional operating system scheduler is that *Flok*
   2. `disk` - Transferring things to/from disk
   3. `cpu` - Tasks that tax the cpu
   4. `gpu` - Tasks that tax the gpu
+  5. `async` - Tasks that shouldn't run on the main thread, but are not scheduled limited.
 
 ##Flok also has two priority classes (Does not effect `main` queue):
   * `high` - Things that need to happen soon.

data/docs/services.md

@@ -1,74 +1,159 @@
 #Services
-Services are entirely built into the Flok kernel. If you need custom actions for your services, they must be created as a driver and then
-you must adhere to the services API.
+Services are helper hubs that send and receive information typically from other services or directly from interrupts. They are meant
+to act as a glue between controllers and devices. Services can receive events and can run periodic events on longer intervals. They are
+very similar to controllers except they do not contain actions and are meant to be used as singletons (although they are instantized, the 
+instances are globally shared).
 
-###Request
-You initiate a service request via `Request`. This may be called only in the controller at this time. This macro takes several parameters
-parameters, all of which must either be strict variable names or double quoted strings:
+##Datatypes
+Services maintain the following datatypes per instance:
+  * `instance_name_sessions` - A hash that contains the connection view controllers as keys and 'true' as the values.
+  * `instance_name_n_sessions` - A count of the current number of active sessions
 
-  * `Request(name, info, event_name)`
-    * `name` - The name of the service you are making a request from
-    * `info` - The information to give the service
-    * `event_name` - The name of the event when a callback occurs
+##Code
+All kernel service classes are placed in `./app/kern/services/` and are ruby files. Here is an example:
+```ruby
+#A sample service, all services have capital letters because they are like classes and are instantized
+service :sample do
+  #Global space#######################################################################################################################
+  #In this space, you may define functions that are accessible to anything
+  global %{
+    function <%= @name %>_cache_save(x) {
+    };
+  }
+  ####################################################################################################################################
 
-Callbacks are handled by redirecting them through the [event](./mod/event.md) interface `int_event`.  You should register with `reg_evt` to receive
-events back from a service. The request itself is not made via the event system, only the response is.  This may seem a little odd because normally
-events come in from the outside while this is an internally generated event.
+  #Initialization#####################################################################################################################
+  #When a service is woken_up, this function is called. A service instances is guaranteed to never be woken up
+  on_wakeup %{
+  }
 
-###Request API
-Registering a service involves adding files to the `./app/kern/services` folder. All *ruby* files in this folder are used in the compilation of 
-services. You declare services in the following format. Your service should not depend on anything except particular module functions.
+  #When an agent (currently all) service is no longer needed by a controller, AND the service has flushed all of it's transaction queues,
+  the service will receive a sleep request. At this point, you should remove all initialized data. If your service is
+  too expensive to destroy all initialized data each time it is woken and slept, then it is too expensive to wakeup at all
+  and you should reconsider your design. After this function calls, this service should act like it never existed and clear
+  all of it's initalized variables.
+  on_sleep %{
+  }
+  ####################################################################################################################################
 
-```
-service("rest") do
-  on_init %{
-    #Code inserted here is put into the global space, add initialization
-    #procedures, functions that need to be called, etc.
-    #
-    #You may use the function respond(info) within here to 
-
-    #Store the in-progress requests as a list of hashes
-    #that contain an array in the order of [event_pointer, evenct_name]
-    var service_rest_tp_to_einfo = {}
-
-    function service_rest_callback(tp, success, info) {
-      //Lookup event info
-      var einfo = service_rest_tp_to_einfo[tp];
-
-      //Send info back to service
-      int_event(einfo[0], einfo[1], {:success => success, :info => info});
-
-      //Remove entries in telepointer table and rest service info
-      tel_del(tp);
-      delete service_rest_tp_to_einfo[tp];
-    }
+  #Session management#################################################################################################################
+  #Things 'connect' to a service, which is just a function call that objects, like controllers, make to a service instance
+  #that notify the service that that object is now connected. You may use this to start things like automatically sending
+  #events to controller instances. You have a session list called $NAME_sessions that is an array of currently connected clients.
+  on_connect %{
   }
 
-  on_request %{
-    #Code that handles a payload goes here.
-    #You have access to `info`, `ep`, and `ename` which was given in the Request macro
+  #When an object is destroyed, this notifies the service that that object no longer wishes to receive things from the service.
+  on_disconnect %{
+  }
+  ####################################################################################################################################
 
-    //Create a GET request that will respond to the telepointer
-    var tp = tel_reg(service_rest_callback);
+  #Do things##########################################################################################################################
+  #Services are a lot like controllers, they have a mechanism to handle events
+  on :event, %{
+    #Services maintain their own context variables through using <%= @name %> macros to prefix variables, each instance will have a different name
+    <%= @name %>_hello = "hi";
 
-    //Now register the event information to respond to when a callback is received
-    service_rest_tp_to_einfo[tp] = [ep, ename]
+    #Inside here you receive...
+      bp - The base pointer of the 'thing' that invoked this function.
+      params - The parameters that were 'sent' (i.e. called)
+  }
 
-    //Start the request
-    SEND("net", "if_net_req", "GET", info.url, info.params, tp)
+  #Do something every 5 seconds if this service (a) has clients and (b) has nothing left in a transaction queue
+  every 5.seconds %{
   }
+  ####################################################################################################################################
 end
 ```
 
-This is then compiled down to
-```js
-//***************************************
-//on_init code is appended to the outside
-//....
-//***************************************
+###Variables accesible
+For each service function, these are what you can access
+  * `on_wakeup`
+    * No variables yet
+  * `on_sleep`
+    * No variables left over
+  * `on_connect`
+    * `bp` - The base address of the controller that connected
+    * `sessions` - A hash of currently active sessions where each key is a base pointer
+  * `on_disconnect`
+    * `bp` - The base address of the controller that disconnected
+    * `sessions` - A hash of currently active sessions where each key is a base pointer
+  * `every x.seconds`
+    * `sessions` - A hash of currently active sessions where each key is a base pointer
+  * `on`
+    * `bp` - The base address of the controller that sent event
+    * `params` - The parameters sent with the message
+    * `sessions` - A hash of currently active sessions where each key is a base pointer
+
+###Service function in controllers
+When you are inside a controller, you may make as service request through `Request`. **You must have already declared usage of the service through
+`services`**:
+  * `Request(service_insatnce_name, ename, params)`
+    * `service_instance_name` - The instance name of the service you are making a request from
+    * `ename` - The name of the event to 'send' the service. (`on` handlers for service)
+    * `params` - Any information you'd like to send along with the service.
+
+###Example controller
+```ruby
+controller :controller do 
+  spots "content"
+  services :my_service
+
+  action :index do
+    on_entry %{
+      //Request can be placed anywhere in the controller
+      Request("my_service", "hello", {});
+    }
+  end
+end
+```
+
+###Services when compiled
+Services get compiled through the `services_compiler` which generates the following functions
+```
+$INAME_on_wakeup() {
+}
+
+$INAME_on_sleep() {
+}
+
+$INAME_on_connect(bp) {
+}
+
+$INAME_on_disconnect(bp) {
+}
 
-//Your request code is put inside a function
-function service_rest_req(info, ep, ename) {
-  //on_request code
+//For each 'on' function
+$INAME_on_XXXXX(bp, params) = {
 }
+
+//Event handler
+$INAME_event_handler(ep, event_name, info) {
+}
+```
+
+###Services config for projects
+Inside a project, `./config/services.rb` holds the services configuration. This configuration file tells flok exactly what services it should
+instantize based on which class. The file contains a list of `service_instance` commands with the following format:
+```ruby
+service_instance :instance_name, :service_class
 ```
+
+Additionally, you may pass in a hash at the end of `service_instance` that will be available as `@options` inside the service definition `rb` file.
+
+###Spec service
+By default, there is a spec service class available called 'test' when compiled with debug. This service contains a function named `$iname_function(x)` that
+sets `$iname_function_args` to the input of that function.
+
+###Roughly how the services system works
+The services are all hard-coded function calls that are initialized with names like `my_instance_on_wakeup`.  You have a service *class* defined in
+either the kernel `./app/kern/services` or `./app/services` in a project. These files are then used as a template when you define a service in
+`./config/services.rb` in your project. The flok library `ServicesCompiler` then takes the config and services file and generates the output
+javascript code to support a service.  Services are talked to through the simple function naming scheme above with the exception of timers which open
+a new `evt` record every time in wakes up and stops the `evt` when it goes to sleep with a new base pointer. This means, a timer will not fire against
+a service if the `evt` is no longer active. Regular `on` requests are not events because there would be way to much overhead.
+
+The controller compiler, `UserCompiler`, mentioned in [Project](./project.md) provides the `services` method when defining a controller. This
+`services` method is used in the `UserCompiler` to inject service functions directly into the controller's `ctable` definition. `_embed` and `Goto
+macro` of the controllers then call `__init__` and `__dealloc__` of the `ctable` which is augmented with the necessary `services` like calling
+`connect` and `disconnect`.

data/docs/services/vm.md

@@ -0,0 +1,128 @@
+#Virtual Memory (vm.js)
+Virtual memory provides something akin to operating system virtual memory systems with an emphasis on the paging infrastructure.  Unlike an operating system, flok has the concept of a grand-unified namespaced address space that extends the concepts of caching and semantics across asynchronous and even networked systems.  This vm systems is increadibly powerful because it allows you to create custom paging devices; this allows you to use one set of semantics to perform very complicated activities like pulling a news feed or list; having that news feed cached to disk automatically; etc.
+
+Additionally, flok introduces a notification system that works with the demand paging schemes and caching schemes that allow you to grab data *now* and then be notified whenever a fresh copy is available from the server.
+
+Each pager belongs to a *namespace*; page faults hit a namespace and then the pager takes over. The pager can choose to service a request; or even throw an exception if a certain semantic is not supported in it's namespace; for example, you may want to disable write semantics for a network pager you called `net` because you expect people to make ordinary network requests.
+
+Fun aside; Because of the hashing schemantics; this paging system solves the age old problem of ... how do you show that data has changed *now* when to be assured that you have perferctly synchronized data with the server?;... you need to do a 3-way handshake with the updates.  You could have a network server pager that supports writes but dosen't forward those to the network. That way, you can locally modify the page and then if the modifications were guessed correctly, the server would not even send back a page modification update! (Locally, the page would have been propogated as well).  In the meantime, after modifying the local page, you would send a real network request to the server which would in turn update it's own paging system but at that point, the server would check in with you about your pages, but miraculously, because you gussed the updated page correctly, no modifications will need to be made. You could even purposefully put a 'not_synced' key in and actually show the user when the page was correctly synchronized.
+
+##Pages
+Each page is a dictionary containing a list of entries.
+```ruby
+page_example = {
+  _head: <<uuid STR>>,
+  _next: <<uuid STR>,
+  _id: <<uuid STR>,
+  entries: [
+    {_id: <<uuid STR>>, _timestmap: <<epoch_milliseconds STR>>},
+    ...
+  ],
+  _hash: <<CRC32 >
+}
+```
+
+  * `_head (optional)` - An optional pointer that indicates a *head* page. The head pages are special pages that contain 0 elements in the entries array, no `_head` key, and `_next` points to the *head* of the list. A head page might be used to pull down the latest news where the head will tell you whether or not there is anything left for you to receive.
+  * `_next (optional)` - The next element on this list. If `_next` is non-existant, then this page is the endpoint of the list.
+  * `_id` - The name of this page. Even if every key changed, the `_id` will not change. This is supposed to indicate, semantically, that this page still *means* the same thing.  For example, imagine a page.  If all entries were to be **removed** from this page and new entries were **inserted** on this page, then it would be semantically sound to say that the entries were **changed**.
+  * `entries` - An array of dictionaries. Each element contains a `_id` that is analogous to the page `_id`. (These are not the same, but carry the same semantics).  Entries also have a `_timestamp` based on their creation or edit time from the unix epoch milliseconds.
+  * `_hash` - All entry `_id's`, `_next`, the page `_id`, and `head` are hashed togeather. Any changes to this page will cause this `_hash` to change which makes it a useful way to check if a page is modified and needs to be updated. The hash function is an ordered CRC32 function run in the following order.  See [Calculating Page Hash](#calculating_page_hash).
+
+------
+
+## <a name='calculating_page_hash'></a>Calculating Page Hash
+The `_hash` value of a page is calculated in the following way:
+  1. `z = crc32(0, _head)`
+  2. `z = crc32(z, _next)`
+  3. `z = crc32(z, _id)`
+  4. `z = crc32(z, entriesN._timestamp)` where N goes through all entries in order.
+
+Assuming a crc function of `crc32(seed, string)`
+
+------
+
+##Configuration
+The paging service may be configured in your `./config/services.rb`. You must set an array of pagers where each pager is responsible for a particular
+namespace. See [VM Pagers](./vm/pagers.md) for more info.
+
+```ruby
+service_instance :vm, :vm, {
+  :pagers => [
+    {
+      :name => "spec0",
+      :namespace => "user",
+      :options => {  //Passed to pager init function
+      }
+    }
+  ]
+}
+```
+Each pager can only be used once. In the future, using multiple copies of pagers would be a welcome addition. If you need to duplicate functionality of a pager,
+you will want to copy your pager into a seperate piece of code and rename it so that it contains unique function names and variables, e.g. `my_pager0_read()` -> `my_pager1_read()`
+
+  * Pager options
+    * `name` - The name of the pager, this is used to create functions to each pager like `$NAME_read_sync`
+    * `namespace` - The namespace of the pager, this is used during requests to the pager, each pager is bound to a namespace
+    * `options` - A hash that is given to the pager's init function.
+
+
+##Requests
+
+###`watch`
+This is how you asynchronously **read a page** and request notifications for any updates to a page. When you first watch a page, you will receive a local cached copy if it is available. For the first watch of a page, pagers will typically update that page so you will get another read as soon as it is available.  For pages that are not locally cached, you will have to wait for a response.
+
+**To re-iterate, flok has a different concept of what constitutes a read. Flok does not distinguish between a read and the want to know about changes to a page. Flok considers controllers that have just watched a page to have an invalid copy of that page, and thus need to be notified that the page has changed for first read**
+  * Parameters
+    * `ns` - The namespace of the page, e.g. 'user'
+    * `id` - Watching the page that contains this in the `_id` field
+  * Event Responses
+    * `read_res` - Whenever a change occurs to a page or the first read.
+      * `ns` - Namespace of the fault
+      * `first` - A boolean that indicates whether this page was ever received on `page_update` before. i.e. is it a change after we were already given this page previously in a `page_update` for this receiver?
+      * `page` - A dictionary object that is a reference to the page. This should be treated as immutable as it is a shared resource.
+
+###`read_sync`
+Request a page of memory synchronously. This will only trigger one `read_res`. If a page does not exist, that should be considered an error. You would normally use this with a blank pager that relies on the cache system to recover data that is either resident in RAM or load it from disk. For example, maybe you would like to display the user's name when they first login without waiting.
+  * Parameters
+    * `ns` - The namespace of the page, e.g. 'user'
+    * `id` - Watching the page that contains this in the `_id` field
+  * Event Responses
+    * `read_res` - Whenever a change occurs to a page or the first read.
+      * `ns` - Namespace of the fault
+      * `first` - A boolean that indicates whether this page was ever received on `page_update` before. i.e. is it a change after we were already given this page previously in a `page_update` for this receiver?
+      * `page` - A dictionary object that is a reference to the page. This should be treated as immutable as it is a shared resource.
+  * Debug quirks
+    * Sets `vm_read_sync_called` to true when called
+
+###`create`
+Creates a new page or overrides an existing one. Will automatically add timestamps to entries. If you need to modify an existing page, see [User page modification helpers](#user_page_modification_helpers)
+  * Parameters
+    * `ns` - The namespace of the page, e.g. 'user'
+    * `id` - Watching the page that contains this in the `_id` field
+    * `next` - The next pointer of the page
+    * `head` - The head pointer of the page
+    * `entries` - An array of dictionary entries for the array
+
+##Helper Methods
+###Pager specific
+  * `vm_cache_write(ns, key, page)` - Save a page to cache memory. This will not recalculate the page hash.
+
+###Page modification
+  * `vm_rehash_page(page)` - Calculates the hash for a page and modifies that page with the new `_hash` field.
+
+### <a name='user_page_modification_helpers'></a>User page modification helpers
+You should never directly edit a page in user land; if you do; the pager has no way of knowing that you made modifications. Additionally, if you have multiple controllers watching a page, and it is modified in one controller, those other controllers
+will not receive the notifications of the page modifications.
+
+**These are only for existing pages; that is, pages that have been received through `read_res`. If you need to create a new page, do so through `create`**
+####Per entry
+  * `entry_del(page, eindex)` - Remove a single entry from a page.
+  * `entry_insert(page, eindex, entry)` - Insert an entry, entry should be a dictionary value. It will automatically have the timestamp added.
+  * `entry_dma(page, eindex)` - Returns a mutable entry at a specific index. In addition, it updates the entries `_timestamp` of the entry.
+
+####Page attributes
+  * `set_page_next(page, hash)` - Sets the `_next` hash for the page
+  * `set_page_head(page, hash)` - Sets the `_head` hash for the page
+
+##Spec helpers
+The variable `vm_did_wakeup` is set to true in the wakeup part of the vm service.

data/docs/services/vm/pagers.md

@@ -0,0 +1,46 @@
+#VM Pagers
+Here is a list of default pagers for the vm system.
+
+=======
+
+##How to make your own pager
+A new pager can be created by adding the pager to the `./app/kern/services` folder or `./app/services/pagers` if you are in a project.
+
+**For all operations that are cacheable, you must write to vm_cache[ns][key]**
+
+Each pager must implement the following functions:
+  * `init(options)` - Initialize a pager structure, passes options given in vm options hash for this pager in `./config/services.rb`
+  * `read(bp, key)`
+  * `read_sync(bp, key)`
+  * `write(key, page)`
+
+##Caching
+For pagers that wish to have their pages cached, they must set their `read` and `read_sync` to write to vm_cache.
+```js
+  vm_cache_write(ns, key, spec0_data[key])
+```
+
+##Default pagers
+###`mem` - Default memory pager
+This pager dosen't do anything beyond allow you to set pages, write to them, and delete them.
+  * Supported operations
+    * `read`
+    * `read_sync`
+    * `write`
+
+###`sockio` - Network pager
+  * Supported operations
+    * `read`
+
+###Spec pagers
+###`spec0` 
+This pager assists with specs in ./spec/kern/vm_service_spec.js
+  * Supported operations
+    * `init(options)` - Will set the `spec0_init_options` to be what ever options it got.
+    * `read` - Will set the `spec0_read_sync_called` to be true.
+    * `read_sync` - Will set the `spec0_read_sync_called` to be true.
+###`spec1` 
+This pager is designed to test the read-sync-notify notification system. When this function is first called,
+it will return 'a' for any value. The second call to read will return `b`.
+  * Supported operations
+    * `init(options)`

data/lib/flok/build.rb

@@ -62,24 +62,18 @@ module Flok
     #3. All js files in `./app/kern/*.js` are globbed togeather and sent to `./products/$platform/glob/2kern.pre_macro.js`
     Flok.src_glob("js", './app/kern', "#{build_path}/glob/2kern.pre_macro.js")
 
+    #4. All js files in `./app/kern/pagers/*.js` are appended to `./products/$PLATFORM/glob/3kern.pre_macro.js`
+    Flok.src_glob("js", './app/kern/pagers/', "#{build_path}/glob/3kern.pre_macro.js")
 
-    #4. All rb files inside `./app/kern/services/` are globbed into `./products/$PLATFORM/glob/kern_services.rb`
-    Flok.src_glob("rb", './app/kern/services/', "#{build_path}/glob/kern_services.rb")
-
-    #5. `./products/$PLATFORM/glob/kern_serivces.rb` is processed via `Flok::Services` and then exported as `./products/$PLATFORM/glob/kern_services.pre_macro.js`
-    File.write("#{build_path}/glob/kern_services.pre_macro.js", Flok::ServicesCompiler.compile(File.read("#{build_path}/glob/kern_services.rb")))
-
-    #6. All js files in `./products/$PLATFORM/glob/2kern.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to ./products/$PLATFORM/glob/2kern.js 
+    #5. All js files in `./products/$PLATFORM/glob/{2,3}kern.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to ./products/$PLATFORM/glob/{2,3}kern.js
     require './app/kern/macro.rb'
     File.write("#{build_path}/glob/2kern.pre_macro.js", macro_process(File.read("#{build_path}/glob/2kern.pre_macro.js")))
+    File.write("#{build_path}/glob/3kern.pre_macro.js", macro_process(File.read("#{build_path}/glob/3kern.pre_macro.js")))
 
-    #7. All js files in `./products/$PLATFORM/glob/kern_services.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to ./products/$PLATFORM/glob/kern_services.pre_macro.js
-    File.write("#{build_path}/glob/kern_services.pre_macro.js", macro_process(File.read("#{build_path}/glob/kern_services.pre_macro.js")))
-
-    #8. All js files are globbed from `./products/$platform/glob` and combined into `./products/$platform/glob/application.js.erb`
+    #6. All js files are globbed from `./products/$platform/glob` and combined into `./products/$platform/glob/application.js.erb`
     Flok.src_glob("js", "#{build_path}/glob", "#{build_path}/glob/application.js.erb")
 
-    #9. Add custom commands
+    #7. Add custom commands
     ################################################################################################################
     #MODS - List mods listed in config.yml
     #---------------------------------------------------------------------------------------
@@ -88,7 +82,7 @@ module Flok
     raise "No config.yml found in your 'platform: #{platform}' driver" unless  driver_config
 
     #Create array that looks like a javascript array with single quotes
-    mods = Flok::Platform.mods(platform, environment)
+    mods = Flok::Platform.mods(environment)
     mods_js_arr = "[" + mods.map{|e| "'#{e}'"}.join(", ") + "]"
 
     #Append this to our output file
@@ -97,7 +91,7 @@ module Flok
     #---------------------------------------------------------------------------------------
     ################################################################################################################
 
-    #10. Append relavent mods code in kernel with macros
+    #8. Append relavent mods code in kernel with macros
     mods.each do |mod|
       s = File.read("./app/kern/mod/#{mod}.js")
       open("#{build_path}/glob/application.js.erb", "a") do |f|
@@ -105,7 +99,7 @@ module Flok
       end
     end
 
-    #11. The compiled `glob/application.js.erb` file is run through the ERB compiler and formed into `application.js`
+    #9. The compiled `glob/application.js.erb` file is run through the ERB compiler and formed into `application.js`
     erb_src = File.read "#{build_path}/glob/application.js.erb"
     renderr = ERB.new(erb_src)
     context = ApplicationJSERBContext.new()
@@ -122,7 +116,8 @@ module Flok
       #Debug / Release
       @debug = (ENV['FLOK_ENV'] == "DEBUG")
       @release = (ENV['FLOK_ENV'] == "RELEASE")
-      @mods = Flok::Platform.mods(ENV['PLATFORM'], ENV['FLOK_ENV'])
+      @mods = Flok::Platform.mods(ENV['FLOK_ENV'])
+      @defines = Flok::Platform.defines(ENV['FLOK_ENV'])
     end
   end
 end