flok 0.0.40 → 0.0.41

data/docs/callout.md

@@ -5,4 +5,4 @@ and interval timers and to send a custom event to the given port.
 ##Registration
 You may register for a timer event via `reg_timeout(ep, ename, ticks)`. This will wait `ticks` before firing.
 To continually fire, you may use `reg_interval(ep, ename, ticks)` which will continue to fire every `ticks`. If `ep` is no longer in the `evt`, then
-the entry will no longer exist.
+the entry will no longer exist.  (the timer will automatically be de-registered)

data/docs/client_api.md

@@ -17,7 +17,10 @@ Client API covers controller action event handlers.
     * `params` - What was passed in the event
     * `__base__` - The address of the controller
     * `__info__` - Holds the `context`, current action, etc. See [Datatypes](./datatypes.md)
-### Controller on_entry
+### Controller on_entry (actions)
+    * `context` - The information for the controllers context
+    * `__base__` - The address of the controller
+    * `__info__` - Holds the `context`, current action, etc. See [Datatypes](./datatypes.md)
+### Controller on_entry (global)
     * `context` - The information for the controllers context
     * `__base__` - The address of the controller
-    * `__info__` - Holds the `context`, current action, etc. See [Datatypes](./datatypes.md)

data/docs/config_yml.md

@@ -0,0 +1,41 @@
+#config.yml
+Config.yml stores the configuration, per platform, for each project. The `config.yml` file contains information relating to:
+  1. What *modules* to include.  A *module* directive (`MODS`) tells flok what files to include from it's `$FLOK_GEM/app/kern/mods` directory. The specific platform driver may also read the modules list; but it is common for platform drivers to just have built-in modules that would be working if paired with the correct kernel modules interrupt handlers.
+  2. What `defines` to use. A `define` shows up in the `@defines` array for kernel source code and may enable/disable sections of code. Documentation will indicate whether features need to have a `define` derective to be enabled. An example of this is simulation support of the `speech` module through `speech_sim`.
+  3. `debug_attach` - A special directive used by the specs suite to understand the scheme used for the debugging server.
+
+##Example
+```yml
+DEBUG:
+  debug_attach: socket_io
+  mods:
+    - ui
+    - event
+    - net
+    - segue
+    - controller
+    - debug
+    - sockio
+    - persist
+    - timer
+  defines:
+    - mem_pager
+    - sockio_pager
+RELEASE:
+  mods:
+    - ui
+    - event
+    - net
+    - segue
+    - controller
+    - sockio
+    - persist
+    - timer
+  defines:
+    - mem_pager
+    - sockio_pager
+
+```
+
+##Where does a config.yml come from
+The `config.yml` starts its life inside the flok gem's `$FLOK_GEM/app/drivers/$PLATFORM/config.yml` where it is later copied into new flok projects when created with `flok new` into your `$PROJECT/config/platforms/$PLATFORM/config.yml` via the `$FLOK_GEM/lib/flok/project_template` directory where an `erb` file called `config.yml` reads straight from the `$FLOK_GEM/app/drivers/$PLATFORM/config.yml` file.

data/docs/controllers.md

@@ -26,6 +26,10 @@ controller "tab_controller" do
   spots "content"
   services "my_service" #See docs on services for what this means
 
+  #Global on_entry, will only be run once on the first action
+  on_entry %{
+  }
+
   #You can also define macros for shared action traits
   macro "my_macro" do
     on "shared_clicked" do

data/docs/datatypes.md

@@ -12,7 +12,9 @@ ctable_entry {
   actions,       //A dictionary [String:action_info] that corresponds to a dictionary of action_info object's based on the action's name.
   spots,         //An array fo spot names for this controller
   name,          //The name of the controller, useful for certain lookup operations, this is also the ctable key
-  __init__,      //A function that is called when this controller is created. Signals service connection
+  __init__,      //A function that is called when this controller is created. Signals service connection and the controller on_entry bits.
+  Additionally, all interval timers are configured here based on their unique names. Actions that are not active will not receive these events (they
+  will be ignored).
   __dealloc__    //A function that is called when this controller is destroyed via parent controller switching actions in Goto. Signals services d/c
 }
 ```
@@ -23,7 +25,8 @@ dictate what happends when events come in, etc.
 ```javascript
 action_info {
   on_entry      //A function that is called when this action is initialized.
-  handlers      //A dictionary [String:f(context, info)] of event handlers for events that occur
+  handlers      //A dictionary [String:f(base)] of event handlers for events that occur. Timer events are given a unique name and stored here like
+  `3toht_5_sec`
 }
 ```
 

data/docs/debug_server.md

@@ -5,6 +5,8 @@ information. The key `debug_attach` holds a protocol key like `socket_io`. This
 The `debug` module implements the `debug server`. Although, the server is a seperate
 piece of code that dosen't fit into the same `if` and `int` paradigm. The specs in the `debug server` also use various `debug` module helpers.
 
+======
+
 ## `socket_io` Protocol
 This protocol states that the driver must repeadeately attempt to connect to the socket.io port located at `localhost:9999`.
 

data/docs/dispatch.md

@@ -7,7 +7,7 @@ we will be forced to block anyway, so it makes sense to allow large tranfers (bu
 
 In order to relieve this problem, *flok* restricts the number of pipelined messages **per queue** to 5 with the exception of the `main` queue (the only synchronous queue). That means you
 can have a total of `(N*5)` messages assuming there are `N` queue types (at the time of this writing, there are 5 not including the `main` queue). It is unlikely that all queues will be used
-as most requests on the flok client will not use multiple resources in one pipelined stage. The client is responsible for requesting more data until no more data is available.
+as most requests on the flok client will not use multiple resources in one pipelined stage. The client is responsible for requesting more data until no more data is available. The client knows about more data waiting by receiving the string "i" at the start of the response array (see below).
 
 ##Confusion about synchronous and asynchronous
 There are various stages of message processing so it can be confusing as to what is excatly synchronous and asynchronous. Flok assumes a few things
@@ -66,10 +66,15 @@ And it receives this in `res`:
 ```
 
 Notice how it's the same as the int_dispatch from the server except that queue 1 (`net_q`) is missing 1 message ([1, "download", "..."]). The 'i' at the start
-indicates that the request is 'incomplete' and the client should request with a blank request array following completion of dequing all these events.
-So the flok server still ha the following in it's queues. The `net_q` will be transfered after the next client request which will take place
+indicates that the request is 'incomplete' and the client should request with a blank request array following completion of dequing all these events. The second request should be **asynchronous** w.r.t to the frist request and the third request asynchronous w.r.t to the second request. This is because, if there were enough events, the main thread would be blocked until the full queue was finished de-queing. This raises an issue; what happends if a request comes through before the asynchronous request comes through? Nothing special. Clients should prioritize the request queue to dispatch things that need `int_event` *now* instead of wait until the queue is drained. While the requests will do the same thing either which way; you don't want to pre-empt the higher priority request while it's waiting for a low priority queue drain else you lose the benefits.
+So the flok server still has the following in it's queues. The `net_q` will be transfered after the next client request which will take place
 after the `int_dispatch` call as the client should always call `int_dispatch` as many times until it gets a blank que `int_dispatch` as many times until it gets a blank queue.
 
+Additionally, the `i` flag can be used with no information initially given. This arises when the `event` module en-queues an incomplete request because the `event` module needs to support defering the next event call.
+
+
+**Again, we can not stress how important it is to ensure that incompletion de-queueing is asynchronous. Behaviorally, your program will be the same; but it has a much larger opportunity to cause latency as the de-queing itself (and remmeber, requests that request incompleteness but are not incomplete themselves are **still** synchronous**
+
 Note that:
 While at first you might think we need to test that int_dispatch called intra-respond of our if_event needs to test whether or not we still send
 out blank [] to int_dispatch; this is not the case. In the real world, flok is supposed to also make any necessary if_disptach calls during all

data/docs/known_issues.md

@@ -0,0 +1,6 @@
+#Known issues & Bugs
+
+  0000. A `every` event used in a controller's action will time correctly on the first action, but subsequent actions will be off by at most 3 ticks.
+  This is because we do not have any way (currently) to reset the timing queue in the controller as it reisters for all timing events at init. See
+  "Does not call intervals of other actions; and still works when switching back actions" in `spec/kern/controller_spec.rb` 
+  0001. `Goto` macro does not recursively 'dealloc' everything.

data/docs/mod/event.md

@@ -1,20 +1,25 @@
-#Event (event.js)
-
-### Functions
-`if_event(ep, event_name, event)` - Receive an event at some object located at `ep`.  This is a platform defined opaque pointer.
-
-### Interrupts
-`int_event(ep, event_name, event)` - Send an event back to *Flok* through an event. The `ep` in this case is dependent on the sub-system. Dispatching is provided through the `evt` (event vector table). On the flok kernel, using, `reg_evt` and `dereg_evt` will determine what happens post int_event.  If `ep` is no longer valid, the event in ignored. Returns `false` if the destination does not exist and `true` otherwise.
-For example, the `vc` (view controller) subsystem will receive any events sent when the `ep` is an opaque pointer to a
-./app/driver/$PLATFORM/config.yml`)
-file is used to compile only the modules into the flok kernel that the driver supports.rface controller from `ui`.
-
-### Helper function
-`reg_evt(ep, f)` - Register a function to be called when `ep` is sent a message, function looks like function f(ep, ename, info)
-`dereg_evt(ep)` - Disable notifications to a function
-
-### Kernel spec related
-  * `spec_event_handler(ep, event_name, event)` - This function should send the message `spec_event_handler_res(ep, event_name, event)` if called
-  * `int_spec_event_dereg` - This function should de-register 3848392 from being an event 
-
-Additionally, you should register the event pointer `3848392` to call the spec_event_handler_res.
+#Event (event.js)
+
+### Functions
+`if_event(ep, event_name, event)` - Receive an event at some object located at `ep`.  This is a platform defined opaque pointer.
+
+### Interrupts
+  * `int_event(ep, ename, event)` - Send an event back to *Flok* through an event. The `ep` in this case is dependent on the sub-system. Dispatching is provided through the `evt` (event vector table). On the flok kernel, using, `reg_evt` and `dereg_evt` will determine what happens post int_event.  If `ep` is no longer valid, the event in ignored. Returns `false` if the destination does not exist and `true` otherwise.
+  For example, the `vc` (view controller) subsystem will receive any events sent when the `ep` is an opaque pointer to a
+./app/driver/$PLATFORM/config.yml`)
+file is used to compile only the modules into the flok kernel that the driver supports.rface controller from `ui`.
+
+  * `int_event_defer(ep, ename, event)` - Same as `int_event` except that the event will be sent to the appropriate receiver at some point in the future, and guaranteed not in the current thread of execution.  This is used internally by the flok kernel; so the fact that it's an external interface dosen't make a lot of sense.
+
+#Deferred (asynchronous) events
+When you call `int_event_defer`, as you should be calling it since it's not really meant to be used outside the kernel, you enqueue the event on the `edefer_q` array as a array containing [`ep`, `ename`, `event`]. Shifting this array will yield the oldest `ep`, followed by `ename`, and then the oldest `event` until nothing remains. At this point, nothing happens.  When `int_dispatch` is called, it first checks to see if there is anything on the `edefer_q`. If there is, it takes one thing off the queue and executes it. If there are things remaining on `edefer_q` after completion of `int_dispatch`, the next request going out is marked `incomplete` with a leading `i` (see [dispatch](./dispatch.md)) so that another request will be made to `int_dispatch` in the near future.
+
+### Helper function
+`reg_evt(ep, f)` - Register a function to be called when `ep` is sent a message, function looks like function f(ep, ename, info)
+`dereg_evt(ep)` - Disable notifications to a function
+
+### Kernel spec related
+  * `spec_event_handler(ep, event_name, event)` - This function should send the message `spec_event_handler_res(ep, event_name, event)` if called
+  * `int_spec_event_dereg` - This function should de-register 3848392 from being an event 
+
+Additionally, you should register the event pointer `3848392` to call the spec_event_handler_res.

data/docs/mod/persist.md

@@ -3,7 +3,7 @@ Persistance management. Loosely based on redis.
 
 ###Driver messages
 `if_per_set(ns, key, value)` - Set a key and value
-`if_per_get(s, ns, key)` - Get a key's value, a message `int_get_res` will be sent back, `s` is the session key that will also be sent back
+`if_per_get(s, ns, key)` - Get a key's value, a message `int_get_res` will be sent back, `s` is the session key that will also be sent back. If there is no key, `null` will be sent back.
 `if_per_del(ns, key)` - Delete a particular key
 `if_per_del_ns(ns)` - Delete an entire namespace
 

data/docs/mod/speech.md

@@ -0,0 +1,12 @@
+#Speech (speech.js)
+There is a software-simulation of the `speech` module. To enable it, put `speech_sim` in the `enable` section of your config file. The speech sim will replicate the rate at which speech progresses but will not produce audio (obviously).
+
+###Client interface
+`if_speech_say(text)` - Start speaking this text. Multiple things will never be queued.
+`if_speech_cancel()` - Stop any speaking in progress; if there is no speaking then nothing should happend
+
+###Kernel interrupts
+`int_speech_cancelled()` - The speech was cancelled
+`int_speech_finished()` - The speech completed succesfully
+`int_speech_started()` - The speech has started
+`int_will_speak_range(offset, count)` - The speech is in the process of talking over a range of the text

data/docs/project.md

@@ -6,7 +6,7 @@ You create and build projects via the `flok` command. You must set the `$FLOK_EN
 
  * `flok new <path>` - Create a new flok project
  * `flok build` - Build a flok project. Generates files in `./products`
- * `flok server` - Trigger auto-rebuild when a file is modified in the `./app` folder and hosts the `products/$PLATFORM` folder on `http://localhost:9992/`. e.g. `http://localhost:9992/application_user.js`
+ * `flok server` - Trigger auto-rebuild when a file is requested in the `./app` folder and hosts the `products/$PLATFORM` folder on `http://localhost:9992/`. e.g. `http://localhost:9992/application_user.js`. Outputs `SERVER STARTED` when server is fully launched.
 
 ###Folder structure
   * `app/`
@@ -29,7 +29,7 @@ You create and build projects via the `flok` command. You must set the `$FLOK_EN
 ###User build process
   1. The gem is build via `build:world` using the platform given in build.
   2. Copy everything in the gems ./flok/products/$PLATFORM -> $PROJECT/products/$PLATFORM and ./flok/app/kern/services/*.rb -> $PROJECT/products/$PLATFORM/services/kern_services.rb
-  3. The controllers in `./app/controllers/*.rb` are run through the `user_compiler` in `./lib/flok/user_compiler.rb` and then saved to the projects
+  3. The controllers in `./app/controllers/**/*.rb` are run through the `user_compiler` in `./lib/flok/user_compiler.rb` and then saved to the projects
   `./products/$PLATFORM/glob/user_compiler.js` and the `./app/services/*.rb` are globbed into `./products/$PLATFORM/services/user_services.rb`
   4. The `./products/$PLATFORM/services/*.rb` file are globbed into `./products/$PLATFORM/services/combined_services.rb`
   5. The service configuration in `./config/services.rb` is read and run through `services_compiler` and files from

data/docs/services/vm.md

@@ -8,10 +8,13 @@ Each pager belongs to a *namespace*; page faults hit a namespace and then the pa
 Fun aside; Because of the hashing schemantics; this paging system solves the age old problem of ... how do you show that data has changed *now* when to be assured that you have perferctly synchronized data with the server?;... you need to do a 3-way handshake with the updates.  You could have a network server pager that supports writes but dosen't forward those to the network. That way, you can locally modify the page and then if the modifications were guessed correctly, the server would not even send back a page modification update! (Locally, the page would have been propogated as well).  In the meantime, after modifying the local page, you would send a real network request to the server which would in turn update it's own paging system but at that point, the server would check in with you about your pages, but miraculously, because you gussed the updated page correctly, no modifications will need to be made. You could even purposefully put a 'not_synced' key in and actually show the user when the page was correctly synchronized.
 
 ##Pages
-Each page is a dictionary containing a list of entries.
+Each page is either of a `array` type or `hash` type.
+
+###Array type
 ```ruby
 page_example = {
   _head: <<uuid STR or NULL>>,
+  _type: "array",
   _next: <<uuid STR or NULL>,
   _id: <<uuid STR>,
   entries: [
@@ -20,12 +23,31 @@ page_example = {
   ],
   _hash: <<CRC32 >
 }
+```
+
+###Hash type
+```ruby
+page_example = {
+  _head: <<uuid STR or NULL>>,
+  _type: "hash",
+  _next: <<uuid STR or NULL>,
+  _id: <<uuid STR>
+  entries: {
+    "my_id0" => {_sig: <<random_signature for inserts and modifies STR>>},
+    ...
+  },
+  _hash: <<CRC32 >
+}
 ```
 
   * `_head (string or null)` - An optional pointer that indicates a *head* page. The head pages are special pages that contain 0 elements in the entries array, no `_head` key, and `_next` points to the *head* of the list. A head page might be used to pull down the latest news where the head will tell you whether or not there is anything left for you to receive.
   * `_next (string or null)` - The next element on this list. If `_next` is non-existant, then this page is the endpoint of the list.
   * `_id (string)` - The name of this page. Even if every key changed, the `_id` will not change. This is supposed to indicate, semantically, that this page still *means* the same thing.  For example, imagine a page.  If all entries were to be **removed** from this page and new entries were **inserted** on this page, then it would be semantically sound to say that the entries were **changed**.
-  * `entries (array of hashes)` - An array of dictionaries. Each element contains a `_id` that is analogous to the page `_id`. (These are not the same, but carry the same semantics).  Entries also have a `_sig` based on their creation or edit time from the unix epoch milliseconds.
+  * `entries`
+    * `_type == 'array'`
+      * An array of dictionaries. Each element contains a `_id` that is analogous to the page `_id`. (These are not the same, but carry the same semantics).  Entries also have a `_sig` which should be a generated hash value that changes when the entry changes.
+    * `_type == 'hash'`
+      * A dictionary of dictionaries. Entries have a `_sig` which should be a generated hash value that changes when the entry changes.
   * `_hash (string)` - All entry `_id's`, `_next`, the page `_id`, and `head` are hashed togeather. Any changes to this page will cause this `_hash` to change which makes it a useful way to check if a page is modified and needs to be updated. The hash function is an ordered CRC32 function run in the following order.  See [Calculating Page Hash](#calculating_page_hash).
 
 ------
@@ -36,7 +58,12 @@ The `_hash` value of a page is calculated in the following way:
   1. `z = crc32(z, _head) if _head`
   2. `z = crc32(z, _next) if _next`
   3. `z = crc32(z, _id)`
-  4. `z = crc32(z, entriesN._sig)` where N goes through all entries in order.
+  4. `_type` dependent
+    * For `_type == 'array'` 
+      * `z = crc32(z, entriesN._sig)` where N goes through all entries in order.
+    * For `_type == 'hash'`
+      * `R = crc32(0, entries[key]._sig)` is calcuated for each entry; R is an array.
+      * `z = crc32(z, r0+r1+r2+...)` where `r0, r1, ...` are the elements of the array R we just calculated. This makes order not important.
 
 If a key is null, then the crc step is skipped for that key.  e.g. if `_head` was null, then `z = crc32(0, _head)` would be skipped
 
@@ -103,12 +130,10 @@ if (page is not resident in memory && not_synchronous) {
   * Parameters
     * `ns` - The namespace of the page, e.g. 'user'
     * `id` - Watching the page that contains this in the `_id` field
-    * `sync (optional)` - If set to `true` then the disk read will be performed synchronously.
+    * `sync (optional)` - If set to `true` then the disk read and cache read will be performed synchronously. Additionally, all future cache reads / updates will be performed synchronously.
   * Event Responses
     * `read_res` - Whenever a change occurs to a page or the first read.
     * Returns an immutable page in params
-  * Debug mode
-    * When `@debug`, an exception will be thrown if you attempt to watch the same key from one controller multiple times.
 
 ###`unwatch`
 This is how you **unwatch** a page. For view controllers that are destroyed, it is not necessary to manually `unwatch` as the `vm` service will be notified on it's disconnection and automatically remove any watched pages for it's base pointer. This should be used for thingcs like scroll lists where the view controller is no longer interested in part of a page-list.
@@ -127,7 +152,7 @@ use the modification helpers. These modification helpers implement copy on write
     * If in `@debug` mode, the variable `vm_write_list` contains an array dictionary of the last page passed to the pager (tail is latest).
 
 ##Cache
-See below with `vm_cache_write` for how to write to the cache. Each pager can choose whether or not to cache; some pagers may cache only reads while others will cache writes.  Failure to write to the cache at all will cause `watch` to never trigger. Some pagers may use a trick where writes are allowed, and go directly to the cache but nowhere else. This is to allow things like *pending* transactions where you can locally fake data until a server response is received which will both wipe the fake write and insert the new one. Cache writes will trigger `watch`; if you write to cache with `vm_cache_write` with a page that has the same `_hash` as a page that already exists in cache, no `watch` events will be triggered. Additionally, calling `vm_cache_write` with a non-modified page will result in no performance penalty.
+See below with `vm_cache_write` for how to write to the cache. Each pager can choose whether or not to cache; some pagers may cache only reads while others will cache writes.  Failure to write to the cache at all will cause `watch` to never trigger. Some pagers may use a trick where writes are allowed, and go directly to the cache but nowhere else. This is to allow things like *pending* transactions where you can locally fake data until a server response is received which will both wipe the fake write and insert the new one. Cache writes will trigger `watch`; if you write to cache with `vm_cache_write` with a page that has the same `_hash` as a page that already exists in cache, no `watch` events will be triggered. Additionally, calling `vm_cache_write` with a non-modified page will result in no performance penalty. `vm_cache_write` notifies controllers asynchronously and is not effected by the `watch` flag on controllers.
 
 ###Pageout & Cache Synchronization
 Cache will periodically be synchronized to disk via the `pageout` service. When flok reloads itself, and the `vm` service gets a `watch` or `watch_sync` request, the `vm` service will attempt to read from the `vm_cache` first and then read the page from disk (write that disk read to cache). The only difference between `watch_sync` and `watch` is that `watch_sync` will synchronously pull from disk and panic if there is no cache available for the page). (Both `watch` and `watch_sync` will always call the pager's after the cache read as well.)
@@ -135,8 +160,8 @@ Cache will periodically be synchronized to disk via the `pageout` service. When
 Pageout is embodied in the function named `vm_pageout()`. This will asynchronously write `vm_dirty` to disk and clear `vm_dirty` once the write has been commited. `vm_pageout()` is called every minute by the interval timer in this service.
 
 ###Datatypes & Structures (Opaque, do not directly modify)
-  * `vm_cache` - The main area for storing the cache. Stored in `vm_cache[ns][key]`
-  * `vm_dirty` - Pages recently written to cache go on the dirty list so that they may be written when the pageout handler runs. Dictionary contains map for `vm_dirty[ns][page._id] => page` for all dirty pages. Pages are removed from the dictionary when they are written in the pageout.
+  * `vm_cache` - The main area for storing the cache. Stored in `vm_cache[ns][key]`. Contains all namespaces by default with blank hashes.
+  * `vm_dirty` - Pages recently written to cache go on the dirty list so that they may be written when the pageout handler runs. Dictionary contains map for `vm_dirty[ns][page._id] => page` for all dirty pages. Pages are removed from the dictionary when they are written in the pageout. Contains all namespaces by default with blank hashes.
   * `vm_notify_map` - The dictionary used to lookup what controllers need to be notified about changes. Stored in `vm_notify_map[ns][id]` which yields an array of controller base pointers.
   * `vm_bp_to_nmap` - A dictionary that maps a `bp` key (usually from a controller) to a dictionary. This dictionary contains a mapping of `bp => ns => id` to an array that contains `[node, index]` where `node` is a reference to `vm_notify_map[ns][id]`. This inverted map must (a) provide a way for `unwatch` to quickly remove entries from itself and (b) provide a way for all entries in `vm_notify_map` to be removed when something (usually a controller) disconrnects.
     must support `unwatch` removal which we only receive the `bp`, `ns`, and `key`.
@@ -155,18 +180,22 @@ will not receive the notifications of the page modifications. Once using these m
 
 Aside, modifying a page goes against the semantics of the vm system; you're thinking of it wrong if you think that's ok. The VM system lets the pager decide what the semantics of a `write` actually means. That may mean it does not directly modify the page; maybe it sends the write request to a server which then validates the request, and then the response on the watched page that was modified will then update your controller.
 
-If you're creating a new page, please use these macros as well; just switch out `CopyPage` for `NewPage`. 
+If you're creating a new page, please use these macros as well; just switch out `CopyPage` for `NewPage`.
 
 ####Per entry
-  * `NewPage(id)` - Returns a new blank page; internally creates a page that has a null `_next`, `_head`, and `entries` array with 0 elements.
-      `_id` is generated if it is not passed.
+  * `NewPage(type, id)` - Returns a new blank page; internally creates a page that has a null `_next`, `_head`, and `entries` array with 0 elements. type can either be `array` or `hash`. `_id` is generated if it is not passed.
   * `CopyPage(page)` - Copies a page and returns the new page. Internally this copies the entire page with the exception of the
       `_hash` field.
-  * `EntryDel(page, eindex)` - Remove a single entry from a page. (Internally this deletes the array entry)
-  * `EntryInsert(page, eindex, entry)` - Insert an entry, entry should be a dictionary value. (Internally this inserts the entry with a unique `_sig` and creates a unique `_id`)
-  * `EntryMutable(page, eindex)` - Returns a mutable entry at a specific index which you can then modify.
-  * `SetPageNext(page, id)` - Sets the `_next` id for the page
-  * `SetPageHead(page, id)` - Sets the `_head` id for the page
+      
+  * For both `array` and `hashes`, the following functions work (albeit different semantics). For array types, the `eindex` is an integer in the array, For hash types, the `eindex` is a key inside the dictionary.
+    * `EntryDel(page, eindex)` - Remove a single entry from a page. (Internally this deletes the array entry).
+    * `EntryInsert(page, eindex, entry)` - Insert an entry, entry should be a dictionary value. 
+      * For arrays, this generates the `_sig` and `_id` for you.
+      * For hashes, this generates the `_sig` for you.
+    * `EntryMutable(page, eindex)` - Set a mutable entry at a specific index which you can then modify. The signature is changed for you. You can not
+        use this with dot syntax like `EntryMutable(page, eindex).id = 'foo'`, you may only get a variable.
+    * `SetPageNext(page, id)` - Sets the `_next` id for the page
+    * `SetPageHead(page, id)` - Sets the `_head` id for the page
 
 Here is an example of a page being modified inside a controller after a `read_res`
 ```js

data/docs/services/vm/pagers.md

@@ -28,11 +28,31 @@ Please name your pagers `pg_XXXX` to help make it clear that you are writing a p
 ####Default memory pager | `pg_mem0`
 The *default memory pager* does not do anything on `watch` or `unwatch`. It depends on the cache to reply to `watch` and `watch_sync` requests created by controllers. Controllers may write to this pager via `write` which this pager will then send directly to `vm_cache_write`. This pager is always compiled into the kernel.
 
-####Spec pager | `pg_spec0`
+####Spec pager | `pg_spec0`, `pg_spec1`
 This pager does the following when calls are made to it's functions, it's designed to assist with `vm` kernel specs.
   * `init` - Sets `pg_spec0_init_params` to `{ns: ns, options: options}`
   * `watch` - Appends `{id: id, hash: hash}` to `pg_spec0_watchlist`
   * `unwatch` - appends id to `pg_spec0_unwatchlist`
   * `write` - Writes the given page to `vm_cache_write`
 
-This pager only exists if the environment is in `DEBUG` mode (`@debug` is enabled).
+These pagers only exists if the environment is in `DEBUG` mode (`@debug` is enabled).
+
+####Net Sim pager | `pg_net_sim`
+This pager is designed to simulate a slowly loading data like a network interface. It does not allow writes.
+  * `init` - Sets `pg_net_sim_spec_did_init` to `true`
+  * `watch` - Set a timer that will elapse after 2 seconds at which point the data will be loaded into the cache.
+  * `unwatch` - Does nothing
+  * `write` - Does nothing
+  * Functions
+    * `pg_net_sim_load_pages(arr)` - Given an array of pages, the net sim will load these pages and then when `watch` is called,
+      it will load those pages, after 2 seconds, into the `vm_cache`
+
+These pagers only exists if the environment is in `DEBUG` mode (`@debug` is enabled).
+
+####Mem pager | `pg_mem0`, `pg_mem1`, `pg_mem2`
+This pager provides you with local memory that will be automatically cached to disk. It has 3 copies
+(`pg_mem0`, `pg_mem'`) etc. because you can use each one in a different namespace.
+  * `init` - Sets `pg_mem0_spec_did_init` to `true` if `@debug`
+  * `watch` - Does nothing
+  * `unwatch` - Does nothing
+  * `write` - Writes the given page to `vm_cache_write`

data/lib/flok/build.rb

@@ -76,10 +76,6 @@ module Flok
     ################################################################################################################
     #MODS - List mods listed in config.yml
     #---------------------------------------------------------------------------------------
-    #Load the driver config.yml
-    driver_config = YAML.load_file("./app/drivers/#{platform}/config.yml")
-    raise "No config.yml found in your 'platform: #{platform}' driver" unless  driver_config
-
     #Create array that looks like a javascript array with single quotes
     mods = Flok::Platform.mods(environment)
     mods_js_arr = "[" + mods.map{|e| "'#{e}'"}.join(", ") + "]"

data/lib/flok/user_compiler.rb

@@ -1,5 +1,7 @@
 #Compile a controller ruby file into a javascript string
 
+require 'active_support'
+require 'active_support/core_ext/numeric'
 require 'erb'
 module Flok
   module UserCompiler
@@ -60,36 +62,8 @@ module Flok
     attr_accessor :controller_name, :action_name, :name
   end
 
-  class UserCompilerAction
-    attr_accessor :controller, :name, :on_entry_src, :ons
-
-    def initialize controller, name, ctx, &block
-      @controller = controller
-      @name = name
-      @ctx = ctx
-      @ons = [] #Event handlers
-
-      self.instance_eval(&block)
-    end
-
-    def on_entry js_src
-      #returns a string
-      @on_entry_src = macro(js_src)
-    end
-
-    def on name, js_src
-      @ons << {:name => name, :src => macro(js_src)}
-    end
-
-    def macro js_src
-      lines = js_src.split("\n").map do |line|
-        
-      end
-
-      return lines.join("\n")
-    end
-
-    def macro text
+  module UserCompilerMacro
+    def _macro text
       out = StringIO.new
 
       text.split("\n").each do |l|
@@ -184,7 +158,8 @@ module Flok
             var old_action = __info__.action;
             __info__.action = "#{action_name}";
 
-            //Remove all views
+            //Remove all views, we don't have to recurse because removal of a view
+            //is supposed to remove *all* view controllers of that tree as well.
             var embeds = __info__.embeds;
             for (var i = 0; i < __info__.embeds.length; ++i) {
               for (var j = 0; j < __info__.embeds[i].length; ++j) {
@@ -251,21 +226,33 @@ module Flok
 
           #For CopyPage(original_page), page_var is original_page
           #This only supports variable names at this time
-          exp.match /\((.*)\);?/
+          exp.match /\((.*?),(.*?)\);?/
+          exp.match /\((.*)\)/ if $1 == nil
+
 
           #Get the id value the user wants, but we have to be careful
           #because if nothing is passed, then we need to set it to null
-          id_var = $1.strip
-          if id_var == ""
-            id_var = "null"
-          end
+          type_var = $1
+          id_var = $2
+
+          type_var = type_var.gsub(/"/, "").strip
+          id_var = (id_var || "null").strip
+
+          raise "NewPage was not given a type" if type_var == ""
+          raise "NewPage type is not valid #{type_var.inspect}" unless ["array", "hash"].include? type_var
+
+          type_var_to_entries = {
+            "array" => "[]",
+            "hash" => "{}",
+          }
 
           out << %{
             #{lvar} {
               _head: null,
               _next: null,
-              entries: [],
+              entries: #{type_var_to_entries[type_var]},
               _id: #{id_var},
+              _type: "#{type_var}",
             }
           }
         elsif l =~ /CopyPage/
@@ -279,18 +266,29 @@ module Flok
           page_var = $1
 
           out << %{
+            
             var __page__ = {
               _head: #{page_var}._head,
               _next: #{page_var}._next,
               _id: #{page_var}._id,
-              entries: [],
+              _type: #{page_var}._type,
             }
 
             //This is a shallow clone, but we own this array
             //When a mutable entry needs to be created, an entry will be cloned
             //and swappend out
-            for (var i = 0; i < #{page_var}.entries.length; ++i) {
-              __page__.entries.push(#{page_var}.entries[i]);
+            if (#{page_var}._type === "array") {
+              __page__.entries = [];
+              for (var i = 0; i < #{page_var}.entries.length; ++i) {
+                __page__.entries.push(#{page_var}.entries[i]);
+              }
+            } else if (#{page_var}._type === "hash") {
+              __page__.entries = {};
+              var keys = Object.keys(#{page_var}.entries);
+              for (var i = 0; i < keys.length; ++i) {
+                var key = keys[i];
+                __page__.entries[key] = #{page_var}.entries[key];
+              }
             }
 
             #{lvar} __page__;
@@ -307,7 +305,11 @@ module Flok
           index_var = $2
 
           out << %{
-            #{page_var}.entries.splice(#{index_var}, 1);
+            if (#{page_var}._type === "array") {
+              #{page_var}.entries.splice(#{index_var}, 1);
+            } else if (#{page_var}._type === "hash") {
+              delete #{page_var}.entries[#{index_var}];
+            }
           }
 
         elsif l =~ /EntryInsert/
@@ -322,10 +324,21 @@ module Flok
           index_var = $2
           entry_var = $3
 
+          page_var.strip!
+          index_var.strip!
+          entry_var.strip!
+
           out << %{
-            #{entry_var}._id = gen_id();
-            #{entry_var}._sig = gen_id();
-            #{page_var}.entries.splice(#{index_var}, 0, #{entry_var});
+
+            if (#{page_var}._type === "array") {
+              #{entry_var}._id = gen_id();
+              #{entry_var}._sig = gen_id();
+              #{page_var}.entries.splice(#{index_var}, 0, #{entry_var});
+            } else if (#{page_var}._type === "hash") {
+              #{entry_var}._sig = gen_id();
+              #{page_var}.entries[#{index_var}] = #{entry_var};
+            }
+
           }
 
         elsif l =~ /SetPageNext/
@@ -369,9 +382,28 @@ module Flok
           page_var = $1
           index_var = $2
 
+
           out << %{
-            //Duplicate entry
-            #{page_var}.entries.splice(#{index_var}, 1, JSON.parse(JSON.stringify(#{page_var}.entries[#{index_var}])));
+            if (#{page_var}._type === "array") {
+              //Duplicate entry
+              #{page_var}.entries.splice(#{index_var}, 1, JSON.parse(JSON.stringify(#{page_var}.entries[#{index_var}])));
+
+              //Here's our new entry
+              var ne = #{page_var}.entries[#{index_var}];
+              ne._sig = gen_id();
+
+              #{lvar} #{page_var}.entries[#{index_var}];
+            } else if (#{page_var}._type === "hash") {
+              //Duplicate entry
+              #{page_var}.entries[#{index_var}] = JSON.parse(JSON.stringify(#{page_var}.entries[#{index_var}]));
+
+              //Here's our new entry
+              var ne = #{page_var}.entries[#{index_var}];
+              ne._sig = gen_id();
+
+              #{lvar} #{page_var}.entries[#{index_var}];
+
+            }
           }
         else
           out.puts l
@@ -381,6 +413,44 @@ module Flok
       return out.string
     end
 
+  end
+
+  class UserCompilerAction
+    attr_accessor :controller, :name, :ons, :every_handlers
+    include UserCompilerMacro
+
+    def initialize controller, name, ctx, &block
+      @controller = controller
+      @name = name
+      @ctx = ctx
+      @_on_entry_src = ""
+      @ons = [] #Event handlers
+      @every_handlers = []
+
+      self.instance_eval(&block)
+    end
+
+    def on_entry js_src
+      #returns a string
+      @_on_entry_src = _macro(js_src)
+    end
+
+    def on_entry_src
+      return @_on_entry_src
+    end
+
+    def on name, js_src
+      @ons << {:name => name, :src => _macro(js_src)}
+    end
+
+    def every seconds, str
+      @every_handlers << {
+        :name => "#{seconds}_sec_#{SecureRandom.hex[0..6]}",
+        :ticks => seconds*4,
+        :src => _macro(str)
+      }
+    end
+
     #You can def things in controller and use them as macros inside actions
     #But these defs. live in the UserCompilerController instance and we need
     #to delegate these calls to the controller that are not available in the action
@@ -395,7 +465,9 @@ module Flok
   end
 
   class UserCompilerController
-    attr_accessor :name, :spots, :macros, :_services
+    include UserCompilerMacro
+
+    attr_accessor :name, :spots, :macros, :_services, :_on_entry
     def initialize name, ctx, &block
       @name = name
       @ctx = ctx
@@ -411,6 +483,10 @@ module Flok
       @macros[name] = block
     end
 
+    def on_entry str
+      @_on_entry = _macro(str)
+    end
+
     #Names of spots
     def spots *spots
       @spots += spots