flok 0.0.38 → 0.0.39

data/docs/client_api.md

@@ -8,6 +8,8 @@ Client API covers controller action event handlers.
   * Send(event_name, info) - Send a custom event on the main queue.
   * Raise(event_name, info) - Will send an event to the parent view controller (and it will bubble up, following `event_gw` which is set in `Embed` as the parent controller
   * Lower(spot_name, event_name, info) - Send an event to a particular spot
+  * Helpers
+    * Page Modification - See [User Page Modification Helpers](./vm.md#user_page_modification_helpers) for a list of functions available.
 
 ### Controller Event Handlers
   * Variables
@@ -18,4 +20,4 @@ Client API covers controller action event handlers.
 ### Controller on_entry
     * `context` - The information for the controllers context
     * `__base__` - The address of the controller
-    * `__info__` - Holds the `context`, current action, etc. See [Datatypes](./datatypes.md)
+    * `__info__` - Holds the `context`, current action, etc. See [Datatypes](./datatypes.md)

data/docs/compilation.md

@@ -14,7 +14,7 @@ as necessary.*
  2. All js files in `./app/kern/config/*.js` are globbed togeather and sent to `./products/$PLATFORM/glob/1kern_config.js`
  3. All js files in `./app/kern/*.js` are globbed togeather and sent to `./products/$PLATFORM/glob/2kern.pre_macro.js`
  4. All js files in `./app/kern/pagers/*.js` are globbed togeather and sent to `./products/$PLATFORM/glob/3kern.pre_macro.js`
- 5. All js files in `./products/$PLATFORM/glob/{2,3}kern.pre_macro.js` are run through `./app/kern/macro.rb's macro_process` and then sent to `./products/$PLATFORM/glob/{2,3}kern.js`
+ 5. All js files in `./products/$PLATFORM/glob/{2,3}kern.pre_macro.js` are run through `./lib/flok/macro.rb's macro_process` and then sent to `./products/$PLATFORM/glob/{2,3}kern.js`
  6. All js files are globbed from `./products/$PLATFORM/glob` and combined into `./products/$PLATFORM/glob/application.js.erb`
  7. Auto-generated code is placed at the end (like PLATFORM global)
  8. The module specific code in `./kern/mod/.*js` are added when the name of the file (without the js part) is mentioned in the `./app/drivers/$PLATFORM/config.yml` `mods` section and appended to `glob/application.js.erb`

data/docs/dispatch.md

@@ -0,0 +1,91 @@
+#Dispatching of messages
+Most javascript implementations implement a sandbox where messages between the javascript core and the client is done via an access controlled xpc system. These xpc systems generally serialize 
+the data to be transferred and then join the requesting processes run queue to complete the request so that the process is charged with the xpc transfer. The longer this XPC transfer takes,
+the more likely the process is going to get pre-empted in the middle of the transfer and have to wait to continue the transfer until the process is rescheduled. It is in our best interest
+to avoid this as it adds large amounts of latency to the application; many small transfers are preferrable to large transfers unless it is a synchronous request.  For synchronous requests,
+we will be forced to block anyway, so it makes sense to allow large tranfers (but caution againts them) in synchronous requests.
+
+In order to relieve this problem, *flok* restricts the number of pipelined messages **per queue** to 5 with the exception of the `main` queue (the only synchronous queue). That means you
+can have a total of `(N*5)` messages assuming there are `N` queue types (at the time of this writing, there are 5 not including the `main` queue). It is unlikely that all queues will be used
+as most requests on the flok client will not use multiple resources in one pipelined stage. The client is responsible for requesting more data until no more data is available.
+
+##Confusion about synchronous and asynchronous
+There are various stages of message processing so it can be confusing as to what is excatly synchronous and asynchronous. Flok assumes a few things
+  1. The disptach mechanism, `int_dispatch`, is always called by the client synchronously, and the javascript core will always respond synchronously to `if_disptach`. 
+  2. The client `if_dispatch` handler will then process the main queue on it's same synchronous thread and then dispatch, asynchronously, the remaining queues; the queues may either each dispatch messages asynchronously or synchronously w.r.t to the original queue. (out of order and parallel are supported)
+
+Additionally, it is always ok, but not suggested, to downgrade an asynchronous request to a synchronous request.  But you can **never** downgrade a synhcronous request to an asynchronous request. Synchronous requests must be done in order and on a single thread; additionally, they can be UI requests which are typically handled on the main thread.
+
+For example, if we dispatch on the `main` queue a disk read request, flok would expect that the disk read would block the javascript core and return execution as soon as the disk read completed. Flok would also presume that the disk read was done at the fastest
+and highest priority of IO and CPU.
+
+Flok would expect that same disk requets, dispatched on an asynhcronous queue, like `disk`, that the request would not execute on the same thread of execution and could execute out of order.
+
+##The standard Flok queues (resources) are defined with the labels:
+  0. `main` - User-interface displaying, etc.
+  1. `net` - Downloading, Uploading, Get requests, etc.
+  2. `disk` - Transferring things to/from disk
+  3. `cpu` - Tasks that tax the cpu
+  4. `gpu` - Tasks that tax the gpu
+
+##Messages from the server
+Messages sent via `if_dispatch` to the server have a special format that looks like this:
+```javascript
+  msg = [
+    [0, 0, "ping", 1, "ping2", "hello"],
+    [1, 1, "download_image", "http://testimage.com/test.png"],
+    [4, 1, "blur_button", 23]
+  ]
+```
+
+The message is broken up into *3* distinct queues.  The first queue, queue 0, is the **main** queue. Each queue should be interpreted in order. That
+means the *main* queue will always be synchronously executed before the rest of the queues are asynchronously dispatched. The `download_image` is
+apart of the `net` queue, and the *gpu* is part of queue 4.  Look above at *Resource Labels* to see what each queue is.
+
+##Example of a session where the flok server does not respond with all messages right away to a client
+Imagine that a flok server has the following available in it's queues for transfer in int_dispatch
+```javascript
+    main_q = [[0, "ping", [0, "ping"], [0, "ping"], [0, "ping"], [0, "ping"], [0, "ping"],
+    net_q = [[1, "download", "..."], [1, "download", "..."], [1, "download", "..."], [1, "download", "..."], [1, "download", "..."], [1, "download", ...]  ,
+    gpu_q = [[1, "blur_button", 23]]
+```
+The `main_q` contains over 5 messages. However, because the `main_q` is dispatched synchronously, we will send those all at once. The `net_q` has
+6 messages; so we will only send 5 of those at once. The `gpu_q` only contains 1 message, so we will send that at once.
+
+The client then calls `int_dispatch`:
+```javascript
+  res = int_disptach(...)
+```
+
+And it receives this in `res`:
+```javascript
+  'i',
+  [0, 0, "ping", 0, "ping", 0, "ping", 0, "ping", 0, "ping", 0, "ping"],
+  [1, 1, "download", "..."], 1, "download", "...", 1, "download", "...", 1, "download", "...", 1, "download", "..."]
+  [4, 1, "blur_button", 23]
+```
+
+Notice how it's the same as the int_dispatch from the server except that queue 1 (`net_q`) is missing 1 message ([1, "download", "..."]). The 'i' at the start
+indicates that the request is 'incomplete' and the client should request with a blank request array following completion of dequing all these events.
+So the flok server still ha the following in it's queues. The `net_q` will be transfered after the next client request which will take place
+after the `int_dispatch` call as the client should always call `int_dispatch` as many times until it gets a blank que `int_dispatch` as many times until it gets a blank queue.
+
+Note that:
+While at first you might think we need to test that int_dispatch called intra-respond of our if_event needs to test whether or not we still send
+out blank [] to int_dispatch; this is not the case. In the real world, flok is supposed to also make any necessary if_disptach calls during all
+int_dispatch calls. We would always receive back if_dispatch; and thus it would follow the same rules as layed out here
+
+```javascript
+    main_q = [0]
+    net_q = [1, 1, "download", ...]
+    gpu_q = [4]
+```
+
+##Spec helpers
+
+###Kernel
+The kernel has the function in `@debug`
+  * `spec_dispatch_q(queue, count)` - Which will internally queue the message [0, "spec"] to the queue given in `queue` `count` times
+
+###Driver 
+`dispatch_spec` to assist with testing of the 'i' re-request behavior.

data/docs/kernel_api.md

@@ -12,6 +12,9 @@ instead.
 ##CRC32
   * `crc32(seed, str)` - Will calculate a CRC32 based on a seed and a string
 
+##Random string
+  * `gen_id()` - Will return a random unique id (8 character string).
+
 ##Events
   * `reg_evt(ep, f)` - Register a function to be called when an event is processed by `int_event`. The function will receive `(ep, event_name, info)`.
 
@@ -36,5 +39,3 @@ variables in here.  If you need to pass a hash literal, array literal, etc, plea
 var payload = {from: null, to: action};
 SEND("main", "if_event", base, "action", payload);
 ```
-
-

data/docs/messaging.md

@@ -36,12 +36,13 @@ live in `./app/kern/mod/` and have the convention of being called `int_*`.
 
 On the client, the driver decides on how messages are handled. At a minimum, the client must support the `if_dispatch` function
 call. The driver is given a queue suggestion based on the first number for each message queue in the `if_dispatch` call. See
-[Scheduling](./scheduling.md) for more information.
+[Dispatching](./disptach.md) for more information.
 
 ### Ping
 Both the client and server are responsible for being able to reply to a few test messages.
 
 #####For the client
+  - Given `[[0, 0, "ping_nothing"]]`, do nothing. Used for `dispatch_spec`
   - Given `[[0, 0, "ping"]]` respond with `[0, pong]`
   - Given `[[0, 1, "ping1", arg]]` respond with `[1, pong1, arg]`
   - Given `[[0, 2, "ping2", arg1, arg2]]` respond with `[1, "pong2", arg1]` and `[2, "pong2", arg1, arg2]`
@@ -73,6 +74,10 @@ Both the client and server are responsible for being able to reply to a few test
 	  - Given `[0, "ping4_int"]` respond with `[[queue_index, 0, "pong4"]]`
   - *If the queue_index is 0 (main), it should queue all 6*
 
+### Dispatch Spec
+
+    - Given `['i', *]` for a queue will force the client to request another queue after it is done processing.
+
 ### Protocols
 Protocols are informal conventions used in Flok when sending certain messages.
 

data/docs/mod/persist.md

@@ -3,7 +3,7 @@ Persistance management. Loosely based on redis.
 
 ###Driver messages
 `if_per_set(ns, key, value)` - Set a key and value
-`if_per_get(s, ns, key)` - Get a key's value, a message `int_get_res` will be sent back
+`if_per_get(s, ns, key)` - Get a key's value, a message `int_get_res` will be sent back, `s` is the session key that will also be sent back
 `if_per_del(ns, key)` - Delete a particular key
 `if_per_del_ns(ns)` - Delete an entire namespace
 
@@ -15,5 +15,6 @@ It is expected that the kernel should manage the write-back cache and that the d
 it is convenient to do so.
 
 ###Kernel interrupts
-`int_per_get_res(s, res)` - A response retrieved from `if_per_get` that contains the session key and result dictionary. If the key
-does not exist, null is returned.
+`int_per_get_res(s, ns, res)` - A response retrieved from `if_per_get` that contains the session key and result dictionary. Currently,
+the service `vm` owns this function; so session does not have an effect on the outcome; but the string `"vm"` should be used for now for any
+session keys involving persist.

data/docs/project_layout.md

@@ -5,5 +5,5 @@
      * `app/drivers/$PLATFORM/` - Platform specific way to implement the interface. See [platform drivers](./platform_drivers.md) for information.
    * `app/kern` - The remaining part, your app, the kernel, etc. all live under here.
      * `app/kern/mod` - Interrupt handlers for drivers and associated code.
-     * `app/kern/macro.rb` - Contains code that is called by `./lib/flok/build.rb` to run all kernel *js* code through (except `./lib/flok/services`)
-       * This macro file provides various macros used in the kernel *and* is what controls the `DEBUG` or `RELEASE` mode ERB variables
+   * `/lib/kern/macro.rb` - Contains code that is called by `./lib/flok/build.rb` to run all kernel *js* code through as well as the `services_compiler`
+     * This macro file provides various macros used in the kernel and services like `SEND`.

data/docs/services/vm.md

@@ -11,31 +11,34 @@ Fun aside; Because of the hashing schemantics; this paging system solves the age
 Each page is a dictionary containing a list of entries.
 ```ruby
 page_example = {
-  _head: <<uuid STR>>,
-  _next: <<uuid STR>,
+  _head: <<uuid STR or NULL>>,
+  _next: <<uuid STR or NULL>,
   _id: <<uuid STR>,
   entries: [
-    {_id: <<uuid STR>>, _timestmap: <<epoch_milliseconds STR>>},
+    {_id: <<uuid STR>>, _sig: <<random_signature for inserts and modifies STR>>},
     ...
   ],
   _hash: <<CRC32 >
 }
 ```
 
-  * `_head (optional)` - An optional pointer that indicates a *head* page. The head pages are special pages that contain 0 elements in the entries array, no `_head` key, and `_next` points to the *head* of the list. A head page might be used to pull down the latest news where the head will tell you whether or not there is anything left for you to receive.
-  * `_next (optional)` - The next element on this list. If `_next` is non-existant, then this page is the endpoint of the list.
-  * `_id` - The name of this page. Even if every key changed, the `_id` will not change. This is supposed to indicate, semantically, that this page still *means* the same thing.  For example, imagine a page.  If all entries were to be **removed** from this page and new entries were **inserted** on this page, then it would be semantically sound to say that the entries were **changed**.
-  * `entries` - An array of dictionaries. Each element contains a `_id` that is analogous to the page `_id`. (These are not the same, but carry the same semantics).  Entries also have a `_timestamp` based on their creation or edit time from the unix epoch milliseconds.
-  * `_hash` - All entry `_id's`, `_next`, the page `_id`, and `head` are hashed togeather. Any changes to this page will cause this `_hash` to change which makes it a useful way to check if a page is modified and needs to be updated. The hash function is an ordered CRC32 function run in the following order.  See [Calculating Page Hash](#calculating_page_hash).
+  * `_head (string or null)` - An optional pointer that indicates a *head* page. The head pages are special pages that contain 0 elements in the entries array, no `_head` key, and `_next` points to the *head* of the list. A head page might be used to pull down the latest news where the head will tell you whether or not there is anything left for you to receive.
+  * `_next (string or null)` - The next element on this list. If `_next` is non-existant, then this page is the endpoint of the list.
+  * `_id (string)` - The name of this page. Even if every key changed, the `_id` will not change. This is supposed to indicate, semantically, that this page still *means* the same thing.  For example, imagine a page.  If all entries were to be **removed** from this page and new entries were **inserted** on this page, then it would be semantically sound to say that the entries were **changed**.
+  * `entries (array of hashes)` - An array of dictionaries. Each element contains a `_id` that is analogous to the page `_id`. (These are not the same, but carry the same semantics).  Entries also have a `_sig` based on their creation or edit time from the unix epoch milliseconds.
+  * `_hash (string)` - All entry `_id's`, `_next`, the page `_id`, and `head` are hashed togeather. Any changes to this page will cause this `_hash` to change which makes it a useful way to check if a page is modified and needs to be updated. The hash function is an ordered CRC32 function run in the following order.  See [Calculating Page Hash](#calculating_page_hash).
 
 ------
 
 ## <a name='calculating_page_hash'></a>Calculating Page Hash
 The `_hash` value of a page is calculated in the following way:
-  1. `z = crc32(0, _head)`
-  2. `z = crc32(z, _next)`
+  0. `z = 0`
+  1. `z = crc32(z, _head) if _head`
+  2. `z = crc32(z, _next) if _next`
   3. `z = crc32(z, _id)`
-  4. `z = crc32(z, entriesN._timestamp)` where N goes through all entries in order.
+  4. `z = crc32(z, entriesN._sig)` where N goes through all entries in order.
+
+If a key is null, then the crc step is skipped for that key.  e.g. if `_head` was null, then `z = crc32(0, _head)` would be skipped
 
 Assuming a crc function of `crc32(seed, string)`
 
@@ -69,60 +72,132 @@ you will want to copy your pager into a seperate piece of code and rename it so
 ##Requests
 
 ###`watch`
-This is how you asynchronously **read a page** and request notifications for any updates to a page. When you first watch a page, you will receive a local cached copy if it is available. For the first watch of a page, pagers will typically update that page so you will get another read as soon as it is available.  For pages that are not locally cached, you will have to wait for a response.
+This is how you **read a page** and request notifications for any updates to a page. The following happens when you watch a page:
+```js
+if (page is resident in memory from previous cache write)
+  send the caller a read_res event *now*
+
+increment_page_ref()
+
+//Synchronously request disk load from cache; this will block
+//Even if we have a request in progress; the synchronous
+//may pre-empt that event because the disk queue might be loaded;
+//so we need to send this anyway
+if (page is not redisent in memory and synchronous) {
+  try_sync_load_from_disk_and_update_cache()
+}
+
+//Only notify if this is the first reference, other controllers who attempt a watch will not signal the pager because the pager already knows
+//about this page
+if first_reference {
+  pager_watch()
+}
 
-**To re-iterate, flok has a different concept of what constitutes a read. Flok does not distinguish between a read and the want to know about changes to a page. Flok considers controllers that have just watched a page to have an invalid copy of that page, and thus need to be notified that the page has changed for first read**
+//Again, only attempt this if the page is not requested by anyone else and is not synchronous (because we would have already tried). The pager will be notified in the meantime, if the disk
+//comes after the pager notification; then the disk will not do anything.
+if (page is not resident in memory && not_synchronous) {
+  //This is an asynchronous request
+  try_load_from_disk_and_update_cache()
+}
+```
   * Parameters
     * `ns` - The namespace of the page, e.g. 'user'
     * `id` - Watching the page that contains this in the `_id` field
+    * `sync (optional)` - If set to `true` then the disk read will be performed synchronously.
   * Event Responses
     * `read_res` - Whenever a change occurs to a page or the first read.
       * `ns` - Namespace of the fault
       * `first` - A boolean that indicates whether this page was ever received on `page_update` before. i.e. is it a change after we were already given this page previously in a `page_update` for this receiver?
       * `page` - A dictionary object that is a reference to the page. This should be treated as immutable as it is a shared resource.
+  * Debug mode
+    * When `@debug`, an exception will be thrown if you attempt to watch the same key from one controller multiple times.
+
+###`unwatch`
+This is how you **unwatch** a page. For view controllers that are destroyed, it is not necessary to manually `unwatch` as the `vm` service will be notified on it's disconnection and automatically remove any watched pages for it's base pointer. This should be used for thingcs like scroll lists where the view controller is no longer interested in part of a page-list.
 
-###`read_sync`
-Request a page of memory synchronously. This will only trigger one `read_res`. If a page does not exist, that should be considered an error. You would normally use this with a blank pager that relies on the cache system to recover data that is either resident in RAM or load it from disk. For example, maybe you would like to display the user's name when they first login without waiting.
   * Parameters
     * `ns` - The namespace of the page, e.g. 'user'
-    * `id` - Watching the page that contains this in the `_id` field
-  * Event Responses
-    * `read_res` - Whenever a change occurs to a page or the first read.
-      * `ns` - Namespace of the fault
-      * `first` - A boolean that indicates whether this page was ever received on `page_update` before. i.e. is it a change after we were already given this page previously in a `page_update` for this receiver?
-      * `page` - A dictionary object that is a reference to the page. This should be treated as immutable as it is a shared resource.
-  * Debug quirks
-    * Sets `vm_read_sync_called` to true when called
+    * `id` - Unwatch the page that contains this in the `_id` field
 
-###`create`
-Creates a new page or overrides an existing one. Will automatically add timestamps to entries. If you need to modify an existing page, see [User page modification helpers](#user_page_modification_helpers)
-  * Parameters
+###`write`
+Creates a new page or overrides an existing one. If you are modifying an existing page, it is imperative that you do not modify the page yourself and
+use the modification helpers. These modification helpers implement copy on write (COW) as well as adjust sigs on specific entries and create ids for new entries.  The proper way to do it is (a) edit the page with the modification helpers mentioned in [User page modification helpers](#user_page_modification_helpers) and (b) perform a write request. This request updates the `_hash` field. Additionally, if you are creating a page, it is suggested that you still use the modification helpers; just use the `NewPage` macro insead of `CopyPage`. Additionally, modifiying a page after making a write request is prohibited as the `vm` service may alter your page.
+  * Parameters 
     * `ns` - The namespace of the page, e.g. 'user'
-    * `id` - Watching the page that contains this in the `_id` field
-    * `next` - The next pointer of the page
-    * `head` - The head pointer of the page
-    * `entries` - An array of dictionary entries for the array
+    * `page` - The page to write (create or update)
+  * Spec helpers
+    * If in `@debug` mode, the variable `vm_write_list` contains an array dictionary of the last page passed to the pager (tail is latest).
+
+##Cache
+See below with `vm_cache_write` for how to write to the cache. Each pager can choose whether or not to cache; some pagers may cache only reads while others will cache writes.  Failure to write to the cache at all will cause `watch` to never trigger. Some pagers may use a trick where writes are allowed, and go directly to the cache but nowhere else. This is to allow things like *pending* transactions where you can locally fake data until a server response is received which will both wipe the fake write and insert the new one. Cache writes will trigger `watch`; if you write to cache with `vm_cache_write` with a page that has the same `_hash` as a page that already exists in cache, no `watch` events will be triggered. Additionally, calling `vm_cache_write` with a non-modified page will result in no performance penalty.
+
+###Pageout & Cache Synchronization
+Cache will periodically be synchronized to disk via the `pageout` service. When flok reloads itself, and the `vm` service gets a `watch` or `watch_sync` request, the `vm` service will attempt to read from the `vm_cache` first and then read the page from disk (write that disk read to cache). The only difference between `watch_sync` and `watch` is that `watch_sync` will synchronously pull from disk and panic if there is no cache available for the page). (Both `watch` and `watch_sync` will always call the pager's after the cache read as well.)
+
+Pageout is embodied in the function named `vm_pageout()`. This will asynchronously write `vm_dirty` to disk and clear `vm_dirty` once the write has been commited. `vm_pageout()` is called every minute by the interval timer in this service.
+
+###Datatypes & Structures (Opaque, do not directly modify)
+  * `vm_cache` - The main area for storing the cache. Stored in `vm_cache[ns][key]`
+  * `vm_dirty` - Pages recently written to cache go on the dirty list so that they may be written when the pageout handler runs. Dictionary contains map for `vm_dirty[ns][page._id] => page` for all dirty pages. Pages are removed from the dictionary when they are written in the pageout.
+  * `vm_notify_map` - The dictionary used to lookup what controllers need to be notified about changes. Stored in `vm_notify_map[ns][id]` which yields an array of controller base pointers.
+  * `vm_bp_to_nmap` - A dictionary that maps a `bp` key (usually from a controller) to a dictionary. This dictionary contains a mapping of `bp => ns => id` to an array that contains `[node, index]` where `node` is a reference to `vm_notify_map[ns][id]`. This inverted map must (a) provide a way for `unwatch` to quickly remove entries from itself and (b) provide a way for all entries in `vm_notify_map` to be removed when something (usually a controller) disconrnects.
+    must support `unwatch` removal which we only receive the `bp`, `ns`, and `key`.
 
 ##Helper Methods
 ###Pager specific
-  * `vm_cache_write(ns, key, page)` - Save a page to cache memory. This will not recalculate the page hash.
+  * `vm_cache_write(ns,  page)` - Save a page to cache memory. This will not recalculate the page hash. The page will be stored in `vm_cache[ns][id]` by.
 
 ###Page modification
-  * `vm_rehash_page(page)` - Calculates the hash for a page and modifies that page with the new `_hash` field.
+  * `vm_rehash_page(page)` - Calculates the hash for a page and modifies that page with the new `_hash` field. If the `_hash` field does not exist, it
+      will create it
 
-### <a name='user_page_modification_helpers'></a>User page modification helpers
+### <a name='user_page_modification_helpers'></a>User page modification helpers (Controller Macros)
 You should never directly edit a page in user land; if you do; the pager has no way of knowing that you made modifications. Additionally, if you have multiple controllers watching a page, and it is modified in one controller, those other controllers
-will not receive the notifications of the page modifications.
+will not receive the notifications of the page modifications. Once using these modifications, you must make a request for `write`. You should not use the information you updated to update your controller right away; you should wait for a `read_res` back because you `watched` the page you just updated. This will normally be performed right away if it's something like the memory pager.
+
+Aside, modifying a page goes against the semantics of the vm system; you're thinking of it wrong if you think that's ok. The VM system lets the pager decide what the semantics of a `write` actually means. That may mean it does not directly modify the page; maybe it sends the write request to a server which then validates the request, and then the response on the watched page that was modified will then update your controller.
+
+If you're creating a new page, please use these macros as well; just switch out `CopyPage` for `NewPage`. 
 
-**These are only for existing pages; that is, pages that have been received through `read_res`. If you need to create a new page, do so through `create`**
 ####Per entry
-  * `entry_del(page, eindex)` - Remove a single entry from a page.
-  * `entry_insert(page, eindex, entry)` - Insert an entry, entry should be a dictionary value. It will automatically have the timestamp added.
-  * `entry_dma(page, eindex)` - Returns a mutable entry at a specific index. In addition, it updates the entries `_timestamp` of the entry.
+  * `NewPage(id)` - Returns a new blank page; internally creates a page that has a null `_next`, `_head`, and `entries` array with 0 elements.
+      `_id` is generated if it is not passed.
+  * `CopyPage(page)` - Copies a page and returns the new page. Internally this copies the entire page with the exception of the
+      `_hash` field.
+  * `EntryDel(page, eindex)` - Remove a single entry from a page. (Internally this deletes the array entry)
+  * `EntryInsert(page, eindex, entry)` - Insert an entry, entry should be a dictionary value. (Internally this inserts the entry with a unique `_sig` and creates a unique `_id`)
+  * `EntryMutable(page, eindex)` - Returns a mutable entry at a specific index which you can then modify.
+  * `SetPageNext(page, id)` - Sets the `_next` id for the page
+  * `SetPageHead(page, id)` - Sets the `_head` id for the page
+
+Here is an example of a page being modified inside a controller after a `read_res`
+```js
+on "read_res", %{
+  //Copy page and modify it
+  var page = Copy(params.page);
+  
+  //Remove first entry
+  EntryDel(page, 0);
+  
+  //Insert an entry
+  var my_entry = {
+    z = 4;
+  }
+  EntryInsert(page, 0, my_entry);
+  
+  //Change an entry
+  var e = EntryMutate(page, 1);
+  e.k = 4;
+  e.z = 5;
+  
+  //Write back page
+  var info = {page: page, ns: "user"};
+  Request("vm", "write", info);
+}
+```
 
-####Page attributes
-  * `set_page_next(page, hash)` - Sets the `_next` hash for the page
-  * `set_page_head(page, hash)` - Sets the `_head` hash for the page
+##Pagers
+See [Pagers](./vm/pagers.md) for information for pager responsibilities and how to implement them.
 
 ##Spec helpers
 The variable `vm_did_wakeup` is set to true in the wakeup part of the vm service.

data/docs/services/vm/pagers.md

@@ -1,46 +1,38 @@
-#VM Pagers
-Here is a list of default pagers for the vm system.
-
-=======
-
-##How to make your own pager
-A new pager can be created by adding the pager to the `./app/kern/services` folder or `./app/services/pagers` if you are in a project.
-
-**For all operations that are cacheable, you must write to vm_cache[ns][key]**
-
-Each pager must implement the following functions:
-  * `init(options)` - Initialize a pager structure, passes options given in vm options hash for this pager in `./config/services.rb`
-  * `read(bp, key)`
-  * `read_sync(bp, key)`
-  * `write(key, page)`
-
-##Caching
-For pagers that wish to have their pages cached, they must set their `read` and `read_sync` to write to vm_cache.
-```js
-  vm_cache_write(ns, key, spec0_data[key])
-```
-
-##Default pagers
-###`mem` - Default memory pager
-This pager dosen't do anything beyond allow you to set pages, write to them, and delete them.
-  * Supported operations
-    * `read`
-    * `read_sync`
-    * `write`
-
-###`sockio` - Network pager
-  * Supported operations
-    * `read`
-
-###Spec pagers
-###`spec0` 
-This pager assists with specs in ./spec/kern/vm_service_spec.js
-  * Supported operations
-    * `init(options)` - Will set the `spec0_init_options` to be what ever options it got.
-    * `read` - Will set the `spec0_read_sync_called` to be true.
-    * `read_sync` - Will set the `spec0_read_sync_called` to be true.
-###`spec1` 
-This pager is designed to test the read-sync-notify notification system. When this function is first called,
-it will return 'a' for any value. The second call to read will return `b`.
-  * Supported operations
-    * `init(options)`
+#Virtual Memory Pagers
+If you haven't already, read [VM Service](../vm.md) for context on pagers.
+
+------
+##Functions required for a pager
+  * `$NAME_init(ns, options)` - Initialize your pager with a namespace (`ns`) and a set of options passed in the `service :vm` options for this pager (See [VM Service](../vm.md)) for example of options hash.
+  * `$NAME_watch(id, page)` - A watch request has been placed for a page id. Multiple watch requests in the *vm service* **will not show up here**.
+      You will only get one watch rquest until you receive an unwatch request. You should attempt to update the page for that key as soon as possible
+      and then wait for future updates. Page is the either a cached page or `undefined`. You should never modify this directly, most pagers should use
+      `_hash` to check with a server if the page needs updating at this point. Some pagers may pre-fetch more pages if there is a `_next`.
+  * `$NAME_unwatch(id)` - There are no controllers that are watching the page with a page that contains this in the `_id` field
+  * `$NAME_write(page)` - You should write this page, e.g. to network, and/or write to `vm_cache_write`.  Alternatively, you can write the page over the network and then let the response from that call `vm_cache_write` in what ever listening code you have.
+    * `page` - A fully constructed page with correctly calculated `_hash` and _sigs on entries.
+
+
+ 
+##When are pagers invoked?
+Pagers handle all requests from controllers except for the following conditions:
+  1. There is a `watch` request placed but a previous `watch` request already exists for the requested page. The pager is already aware of the page watch request and is already waiting for a response. Cached pages would have been returned to the controller that made the `watch` request.
+
+##Where to put pagers
+A new pager class can be created by adding the pager to the `./app/kern/services/pagers/*.js`. Please remember that we do not currently support multiple pager instances for each class; while there is a namespace distinction that could be used to instantize the pager; we do not support statically generating multiple copies of the global variables needed per instance.
+
+Please name your pagers `pg_XXXX` to help make it clear that you are writing a pager.
+
+##Built-in Pagers
+
+####Default memory pager | `pg_mem0`
+The *default memory pager* does not do anything on `watch` or `unwatch`. It depends on the cache to reply to `watch` and `watch_sync` requests created by controllers. Controllers may write to this pager via `write` which this pager will then send directly to `vm_cache_write`. This pager is always compiled into the kernel.
+
+####Spec pager | `pg_spec0`
+This pager does the following when calls are made to it's functions, it's designed to assist with `vm` kernel specs.
+  * `init` - Sets `pg_spec0_init_params` to `{ns: ns, options: options}`
+  * `watch` - Appends `{id: id, hash: hash}` to `pg_spec0_watchlist`
+  * `unwatch` - appends id to `pg_spec0_unwatchlist`
+  * `write` - Writes the given page to `vm_cache_write`
+
+This pager only exists if the environment is in `DEBUG` mode (`@debug` is enabled).