ernie 1.3.0 → 2.0.0

data/History.txt

@@ -1,3 +1,11 @@
+= 2.0.0 / 2010-02-16
+  * Major Changes
+    * Use configuration file for defining handlers
+    * Add Native Erlang modules
+    * Abstract handler logic to support handlers in any language
+    * Add High/Low connection queues
+    * Remove Ruby DSL (must use Ernie.expose now)
+
 = 1.3.0 / 2009-11-30
   * API Additions
     * Add loglevel for setting log level

data/README.md

@@ -3,7 +3,25 @@ Ernie
 
 By Tom Preston-Werner (tom@mojombo.com)
 
-Ernie is a BERT-RPC server implementation that uses an Erlang server to accept incoming connections, and then delegates the request to Ruby handlers.
+Ernie is a BERT-RPC server implementation that uses an Erlang server to accept
+incoming connections, and then delegates the request to custom modules that
+you can write in any language (currently only Ruby and Erlang support is
+included).
+
+Modules that are written in Ruby or any non-Erlang language are known as
+"external" modules and you must specify how many workers of each module should
+be spawned. Requests against these modules are balanced between the workers.
+Modules that are written in Erlang are known as "native" modules and run
+within the Erlang server's runtime. Since these are spawned as lightweight
+processes, there is no balancing necessary and much less communication
+overhead when compared to external modules.
+
+Ernie supports multiple heterogenous modules. For instance, you can have an
+external Ruby module running 10 workers *and* a native Erlang module running
+simultaneously. Ernie keeps track of sending requests to the proper module.
+Using a technique called "shadowing," you can selectively optimize certain
+external module functions with native code and Ernie will handle selecting the
+correct function.
 
 See the full BERT-RPC specification at [bert-rpc.org](http://bert-rpc.org).
 
@@ -12,70 +30,173 @@ Ernie currently supports the following BERT-RPC features:
 * `call` requests
 * `cast` requests
 
-Ernie was developed for GitHub and is currently in production use serving millions of RPC requests every day. The stability and performance have been exemplary.
+Ernie was developed for GitHub and is currently in production use serving
+millions of RPC requests every day. The stability and performance have been
+exemplary.
 
+Ernie follows [Semantic Versioning](http://semver.org/) for release
+versioning.
 
 Installation
 ------------
 
-You must have Erlang installed before installing Ernie.
+Step 1: Install Erlang.
 
-    $ gem install ernie -s http://gemcutter.org
+    http://www.erlang.org/download.html
+
+Step 2: Install Ernie:
+
+    $ gem install ernie
 
 
 Running
 -------
 
     Usage: ernie [command] [options]
-        -h, --handler HANDLER            Handler file
-        -p, --port PORT                  Port
-        -n, --number NUMBER              Number of handler instances
-        -d, --detached                   Run as a daemon
+        -c, --config CONFIG              Config file.
+        -p, --port PORT                  Port.
+        -l, --log-level                  Log level (0-4).
+        -d, --detached                   Run as a daemon.
         -P, --pidfile PIDFILE            Location to write pid file.
 
     Commands:
       <none>                Start an Ernie server.
-      reload-handlers       Gracefully reload all of the the ruby handlers
+      reload-handlers       Gracefully reload all of the external handlers
                             and use the new code for all subsequent requests.
       stats                 Print a list of connection and handler statistics.
 
     Examples:
-      ernie -d -p 9999 -n 10 -h calc.rb
-        Start the ernie server in the background on port 9999 with ten
-        handlers, using the calc.rb handler file.
+      ernie -d -p 9999 -c example.cfg
+        Start the ernie server in the background on port 9999 using the
+        example.cfg configuration file.
 
       ernie reload-handlers -p 9999
         Reload the handlers for the ernie server currently running on
         port 9999.
 
-Example Handler
----------------
+
+Configuration File
+------------------
+
+Ernie configuration files are written as a series of dotted Erlang terms. Each
+term is a list of 2-tuples that specify options for a module.
+
+### Native Modules
+
+The form for native modules is:
+
+    [{module, Module},
+     {type, native},
+     {codepaths, CodePaths}].
+
+Where Module is an atom corresponding to the module name and CodePaths is a
+list of strings representing the file paths that should be added to the
+runtime's code path. These paths will be prepended to the code path and must
+include the native module's directory and the directories of any dependencies.
+
+### External Modules
+
+The form for external modules is:
+
+    [{module, Module},
+     {type, external},
+     {command, Command},
+     {count, Count}].
+
+Where Module is an atom corresponding to the module name, Command is a string
+specifying the command to be executed in order to start a worker, and Count is
+the number of workers to spawn.
+
+### Shadowing
+
+If you specify a native module and an external module of the same name (and in
+that order), Ernie will inspect the native module to see if it has the
+requested function exported and use that if it does. If it does not, then it
+will fall back on the external module. This can be used to selectively
+optimize certain functions in a module without any modifications to your
+client code.
+
+### Predicate Shadowing
+
+In some circumstances it can be nice to conditionally shadow a function in an
+external module based on the nature of the arguments. For example, you might
+want requests for `math:fib(X)` to be routed to the external module when X is
+less than 10, but to be handled by the native module when X is 10 or greater.
+This can be accomplished by implementing a function `math:fib_pred(X)` in the
+native module. Notice the `_pred` appended to the normal function name (pred
+is short for predicate). If a function like this is present, Ernie will call
+it with the requested arguments and if the return value is `true` the native
+module will be used. If the return value is `false` the external module will
+be used.
+
+
+Example Configuration File
+--------------------------
+
+The following example config file informs Ernie of two modules. The first term
+identifies a native module 'nat' that resides in the nat.beam file under the
+'/path/to/app/ebin' directory. The second term specifies an external module
+'ext' that will have 2 workers started with the command 'ruby
+/path/to/app/ernie/ext.rb'.
+
+    [{module, nat},
+     {type, native},
+     {codepaths, ["/path/to/app/ebin"]}].
+
+    [{module, ext},
+     {type, external},
+     {command, "ruby /path/to/app/ernie/ext.rb"},
+     {count, 2}].
+
+
+Native (Erlang) Handler
+-----------------------
+
+Native handlers are written as normal Erlang modules. The exported functions
+will become available to BERT-RPC clients.
+
+### Example
+
+    -module(nat).
+    -export([add/2]).
+
+    add(A, B) ->
+      A + B.
+
+### BERT-RPC Sequence Example
+
+    -> {call, nat, add, [1, 2]}
+    <- {reply, 3}
+
+
+External (Ruby) Handler
+-----------------------
+
+Included in this gem is a library called `ernie` that makes it easy to write
+Ernie handlers in Ruby. All you have to do is write a standard Ruby module and
+expose it to Ernie and the functions of that module will become available to
+BERT-RPC clients.
+
+### Example
 
 Using a Ruby module and Ernie.expose:
 
     require 'ernie'
     
-    module Calc
+    module Ext
       def add(a, b)
         a + b
       end
     end
     
-    Ernie.expose(:calc, Calc)
-    
-Using the DSL (this will be deprecated in a future release):
+    Ernie.expose(:ext, Ext)
 
-    require 'ernie'
+### BERT-RPC Sequence Example
 
-    mod(:calc) do
-      fun(:add) do |a, b|
-        a + b
-      end
-    end
+    -> {call, nat, add, [1, 2]}
+    <- {reply, 3}
 
-
-Logging
--------
+### Logging
 
 You can have logging sent to a file by adding these lines to your handler:
 
@@ -86,21 +207,31 @@ This will log startup info, requests, and error messages to the log. Choosing
 Logger::DEBUG will include the response (be careful, doing this can generate
 very large log files).
 
+### Autostart
 
-Autostart
----------
-
-Normally Ernie handlers will become active after the file has been loaded in.
-you can disable this behavior by setting:
+Normally Ruby Ernie handlers will become active after the file has been loaded
+in. you can disable this behavior by setting:
 
     Ernie.auto_start = false
 
 
-Example BERT-RPC call for above example
----------------------------------------
+Selecting Queue Priority
+------------------------
+
+Ernie maintains High and Low priority queues for incoming connections. If
+there are any connections in the High priority queue, they will always be
+processed first. If the High priority queue is empty, connections will be
+processed from the Low priority queue. By default, connections go into the
+High priority queue. To select a queue, an info BERP of the following form
+must be sent preceding the call.
+
+    -- {info, priority, Priority}
 
-    -> {call, calc, add, [1, 2]}
+Where `Priority` is either the `high` or `low` atom. An example sequence where
+the low priority queue is being selected would look like the following.
 
+    -> {info, priority, low}
+    -> {call, nat, add, [1, 2]}
     <- {reply, 3}
 
 
@@ -112,7 +243,7 @@ You can make BERT-RPC calls from Ruby with the [BERTRPC gem](http://github.com/m
     require 'bertrpc'
 
     svc = BERTRPC::Service.new('localhost', 8000)
-    svc.call.calc.add(1, 2)
+    svc.call.ext.add(1, 2)
     # => 3
 
 

data/VERSION.yml

@@ -1,5 +1,5 @@
 --- 
-:minor: 3
-:build: 
+:major: 2
+:minor: 0
 :patch: 0
-:major: 1
+:build: 

data/bin/ernie

@@ -42,18 +42,14 @@ OptionParser.new do |opts|
   opts.banner = help
   opts.version = version
 
-  opts.on("-h HANDLER", "--handler HANDLER", "Handler ruby file") do |x|
-    options[:handler] = x
+  opts.on("-c CONFIG", "--config CONFIG", "Config file") do |x|
+    options[:config] = x
   end
 
   opts.on("-p PORT", "--port PORT", "Port") do |x|
     options[:port] = x
   end
 
-  opts.on("-n NUMBER", "--number NUMBER", "Number of handler instances") do |x|
-    options[:number] = x
-  end
-
   opts.on("-l LOGLEVEL", "--log-level LOGLEVEL", "Log level (0-4)") do |x|
     options[:log_level] = x
   end
@@ -81,14 +77,13 @@ if command = ARGV[0]
   svc = BERTRPC::Service.new('localhost', port)
   puts svc.call.__admin__.send(command.gsub(/-/, '_'))
 else
-  if !options[:handler]
-    puts "A handler must be specified: ernie -h /path/to/handler.rb"
+  if !options[:config]
+    puts "A config file must be specified: ernie -c /path/to/config.yml"
     exit(1)
   end
 
-  handler = options[:handler]
+  config = options[:config]
   port = options[:port] || DEFAULT_PORT
-  number = options[:number] || 1
   log_level = options[:log_level] || 2
   pidfile = options[:pidfile] ? "-ernie_server_app pidfile \"'#{options[:pidfile]}'\"" : ''
   detached = options[:detached] ? '-detached' : ''
@@ -101,8 +96,7 @@ else
                #{code_paths}
                #{pidfile} \
                -ernie_server_app port #{port} \
-               -ernie_server_app handler '"#{handler}"' \
-               -ernie_server_app number #{number} \
+               -ernie_server_app config '"#{config}"' \
                -ernie_server_app log_level #{log_level} \
                -run ernie_server_app boot}.squeeze(' ')
   puts cmd

data/contrib/ebench.erl

@@ -0,0 +1,76 @@
+% erlc *.erl && erl ebench.beam -run ebench start 10000 20 ext add
+
+-module(ebench).
+-export([start/1]).
+
+start([Ni, Ci, Modi, Funi]) ->
+  Nt = list_to_integer(Ni),
+  C = list_to_integer(Ci),
+  Mod = list_to_atom(Modi),
+  Fun = list_to_atom(Funi),
+  N = round(Nt / C),
+  T0 = erlang:now(),
+  Waiter = spawn(fun() -> wait(T0, N * C) end),
+  spawner(Waiter, N, C, Mod, Fun).
+
+spawner(_Waiter, _N, 0, _Mod, _Fun) ->
+  ok;
+spawner(Waiter, N, C, Mod, Fun) ->
+  spawn(fun() -> loop(Waiter, N, Mod, Fun) end),
+  spawner(Waiter, N, C - 1, Mod, Fun).
+
+% X is the total number of responses to wait for
+wait(T0, XTotal, 0) ->
+  T1 = erlang:now(),
+  Diff = timer:now_diff(T1, T0),
+  Mean = Diff / XTotal,
+  io:format("~p requests completed in ~.2fs~n", [XTotal, Diff / 1000000]),
+  io:format("Mean request time: ~.2fms (~.2f r/s)~n", [Mean / 1000, XTotal / (Diff / 1000000)]),
+  init:stop();
+wait(T0, XTotal, X) ->
+  receive
+    done -> wait(T0, XTotal, X - 1)
+  end.
+
+wait(T0, X) ->
+  wait(T0, X, X).
+
+loop(_Waiter, 0, _Mod, _Fun) ->
+  ok;
+loop(Waiter, N, Mod, Fun) ->
+  hit(Waiter, Mod, Fun),
+  loop(Waiter, N - 1, Mod, Fun).
+
+hit(Waiter, Mod, Fun) ->
+  % io:format("outgoing!~n", []),
+  Host = "localhost",
+  case gen_tcp:connect(Host, 8000, [binary, {packet, 4}]) of
+    {ok, Sock} -> process(Waiter, Mod, Fun, Sock);
+    Any ->
+      io:format("Unable to establish connection: ~p~n", [Any]),
+      Waiter ! done
+  end.
+
+process(Waiter, Mod, Fun, Sock) ->
+  % Info = term_to_binary({info, priority, [low]}),
+  % ok = gen_tcp:send(Sock, Info),
+  Request = term_to_binary({call, Mod, Fun, args(Fun)}),
+  ok = gen_tcp:send(Sock, Request),
+  receive
+    {tcp, _Port, Reply} ->
+      % io:format("~p~n", [Reply]),
+      Res = res(Fun),
+      {reply, Res} = binary_to_term(Reply);
+    {tcp_closed, Port} ->
+      io:format("Connection closed after sending data: ~p~n", [Port]);
+    Any ->
+      io:format("Unexpected message: ~p~n", [Any])
+  end,
+  Waiter ! done,
+  ok = gen_tcp:close(Sock).
+
+args(add) -> [1, 2];
+args(fib) -> [20].
+
+res(add) -> 3;
+res(fib) -> 10946.

data/elib/asset_pool.erl

@@ -2,7 +2,7 @@
 -behaviour(gen_server).
 
 %% api
--export([start_link/1, start/1, lease/0, return/1, reload_assets/0, idle_worker_count/0]).
+-export([start_link/2, lease/1, return/2, reload_assets/1, idle_worker_count/1]).
 
 %% gen_server callbacks
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2,
@@ -16,23 +16,20 @@
 %% API
 %%====================================================================
 
-start_link(Args) ->
-  gen_server:start_link({global, ?MODULE}, ?MODULE, Args, []).
+start_link(Handler, Count) ->
+  gen_server:start_link(?MODULE, [Handler, Count], []).
 
-start(Args) ->
-  gen_server:start({global, ?MODULE}, ?MODULE, Args, []).
+lease(Pid) ->
+  gen_server:call(Pid, lease).
 
-lease() ->
-  gen_server:call({global, ?MODULE}, {lease}).
+return(Pid, Asset) ->
+  gen_server:call(Pid, {return, Asset}).
 
-return(Asset) ->
-  gen_server:call({global, ?MODULE}, {return, Asset}).
+reload_assets(Pid) ->
+  gen_server:call(Pid, reload_assets).
 
-reload_assets() ->
-  gen_server:call({global, ?MODULE}, {reload_assets}).
-
-idle_worker_count() ->
-  gen_server:call({global, ?MODULE}, {idle_worker_count}).
+idle_worker_count(Pid) ->
+  gen_server:call(Pid, idle_worker_count).
 
 %%====================================================================
 %% gen_server callbacks
@@ -45,11 +42,12 @@ idle_worker_count() ->
 %%                         {stop, Reason}
 %% Description: Initiates the server
 %%--------------------------------------------------------------------
-init([Count, Handler]) ->
+init([Handler, Count]) ->
   process_flag(trap_exit, true),
   error_logger:info_msg("~p starting~n", [?MODULE]),
   Token = make_ref(),
   Assets = start_handlers(Count, Handler, Token),
+  logger:debug("Assets = ~p~n", [Assets]),
   {ok, #state{assets = Assets, handler = Handler, token = Token}}.
 
 %%--------------------------------------------------------------------
@@ -61,7 +59,8 @@ init([Count, Handler]) ->
 %%                                      {stop, Reason, State}
 %% Description: Handling call messages
 %%--------------------------------------------------------------------
-handle_call({lease}, _From, State) ->
+handle_call(lease, _From, State) ->
+  logger:debug("Leasing...~n", []),
   Token = State#state.token,
   case queue:out(State#state.assets) of
     {{value, Asset}, Assets2} ->
@@ -91,10 +90,10 @@ handle_call({return, Asset}, _From, State) ->
   end,
   Assets2 = queue:in(NewAsset, State#state.assets),
   {reply, ok, State#state{assets = Assets2}};
-handle_call({reload_assets}, _From, State) ->
+handle_call(reload_assets, _From, State) ->
   Token = make_ref(),
   {reply, ok, State#state{token = Token}};
-handle_call({idle_worker_count}, _From, State) ->
+handle_call(idle_worker_count, _From, State) ->
   WorkerCount = queue:len(State#state.assets),
   {reply, WorkerCount, State};
 handle_call(_Request, _From, State) ->
@@ -140,4 +139,4 @@ start_handlers(Assets, Count, Handler, Token) ->
   start_handlers(Assets2, Count - 1, Handler, Token).
 
 create_asset(Handler, Token) ->
-  {asset, port_wrapper:wrap_link("ruby " ++ Handler), Token}.
+  {asset, port_wrapper:wrap_link(Handler), Token}.

data/elib/asset_pool_sup.erl

@@ -1,15 +1,13 @@
 -module(asset_pool_sup).
 -behaviour(supervisor).
--export([start_link/0, init/1]).
+-export([start_link/2, init/1]).
 
-start_link() ->
-  supervisor:start_link({local, ?MODULE}, ?MODULE, []).
+start_link(Handler, Number) ->
+  supervisor:start_link(?MODULE, [Handler, Number]).
 
-init([]) ->
-  {ok, Handler} = application:get_env(ernie_server_app, handler),
+init([Handler, Number]) ->
   io:format("Using handler ~p~n", [Handler]),
-  {ok, Number} = application:get_env(ernie_server_app, number),
   io:format("Using ~p handler instances~n", [Number]),
   {ok, {{one_for_one, 1, 60},
-    [{asset_pool, {asset_pool, start_link, [[Number, Handler]]},
+    [{asset_pool, {asset_pool, start_link, [Handler, Number]},
     permanent, brutal_kill, worker, [asset_pool]}]}}.

data/elib/bert.erl

@@ -0,0 +1,69 @@
+%%% See http://github.com/mojombo/bert.erl for documentation.
+%%% MIT License - Copyright (c) 2009 Tom Preston-Werner <tom@mojombo.com>
+
+-module(bert).
+-version('1.1.0').
+-author("Tom Preston-Werner").
+
+-export([encode/1, decode/1]).
+
+-ifdef(TEST).
+-include("test/bert_test.erl").
+-endif.
+
+%%---------------------------------------------------------------------------
+%% Public API
+
+-spec encode(term()) -> binary().
+
+encode(Term) ->
+  term_to_binary(encode_term(Term)).
+
+-spec decode(binary()) -> term().
+
+decode(Bin) ->
+  decode_term(binary_to_term(Bin)).
+
+%%---------------------------------------------------------------------------
+%% Encode
+
+-spec encode_term(term()) -> term().
+
+encode_term(Term) ->
+  case Term of
+    [] -> {bert, nil};
+    true -> {bert, true};
+    false -> {bert, false};
+    Dict when is_record(Term, dict, 8) ->
+      {bert, dict, dict:to_list(Dict)};
+    List when is_list(Term) ->
+      lists:map((fun encode_term/1), List);
+    Tuple when is_tuple(Term) ->
+      TList = tuple_to_list(Tuple),
+      TList2 = lists:map((fun encode_term/1), TList),
+      list_to_tuple(TList2);
+    _Else -> Term
+  end.
+
+%%---------------------------------------------------------------------------
+%% Decode
+
+-spec decode_term(term()) -> term().
+
+decode_term(Term) ->
+  case Term of
+    {bert, nil} -> [];
+    {bert, true} -> true;
+    {bert, false} -> false;
+    {bert, dict, Dict} ->
+      dict:from_list(Dict);
+    {bert, Other} ->
+      {bert, Other};
+    List when is_list(Term) ->
+      lists:map((fun decode_term/1), List);
+    Tuple when is_tuple(Term) ->
+      TList = tuple_to_list(Tuple),
+      TList2 = lists:map((fun decode_term/1), TList),
+      list_to_tuple(TList2);
+    _Else -> Term
+  end.

data/elib/ernie.hrl

@@ -0,0 +1,10 @@
+-record(state, {lsock = undefined,      % the listen socket
+                hq = queue:new(),       % high priority queue
+                lq = queue:new(),       % low priority queue
+                count = 0,              % total request count
+                map = undefined}).      % module map. tuples of {Mod, Id}
+
+-record(request, {sock = undefined,     % connection socket
+                  infos = [],           % list of info binaries
+                  action = undefined,   % action binary
+                  priority = high}).    % priority [ high | low ]

data/elib/ernie_admin.erl

@@ -0,0 +1,60 @@
+-module(ernie_admin).
+-export([process/4]).
+-include_lib("ernie.hrl").
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% Process entry point
+
+process(Sock, reload_handlers, _Args, State) ->
+  spawn(fun() -> process_reload_assets(Sock, State) end),
+  State;
+process(Sock, stats, _Args, State) ->
+  spawn(fun() -> process_stats(Sock, State) end),
+  State;
+process(Sock, _Fun, _Args, State) ->
+  gen_tcp:send(Sock, term_to_binary({reply, <<"Admin function not supported.">>})),
+  ok = gen_tcp:close(Sock),
+  State.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% Reload handlers
+
+process_reload_assets(Sock, State) ->
+  lists:map((fun reload/1), State#state.map),
+  gen_tcp:send(Sock, term_to_binary({reply, <<"Handlers reloaded.">>})),
+  ok = gen_tcp:close(Sock).
+
+reload({_Mod, native}) ->
+  ok;
+reload({_Mod, Pid}) ->
+  asset_pool:reload_assets(Pid).
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% Stats
+
+process_stats(Sock, State) ->
+  CountString = stat(count, State),
+  IdleWorkersString = stat(idle, State),
+  QueueLengthString = stat(queue, State),
+  StatString = list_to_binary([CountString, IdleWorkersString, QueueLengthString]),
+  Data = term_to_binary({reply, StatString}),
+  gen_tcp:send(Sock, Data),
+  ok = gen_tcp:close(Sock).
+
+stat(count, State) ->
+  Count = State#state.count,
+  list_to_binary([<<"connections.total=">>, integer_to_list(Count), <<"\n">>]);
+stat(idle, State) ->
+  IdleMap = lists:map((fun idle/1), State#state.map),
+  list_to_binary(IdleMap);
+stat(queue, State) ->
+  HighQueueLength = queue:len(State#state.hq),
+  LowQueueLength = queue:len(State#state.lq),
+  list_to_binary([<<"queue.high=">>, integer_to_list(HighQueueLength), <<"\n">>,
+                  <<"queue.low=">>, integer_to_list(LowQueueLength), <<"\n">>]).
+
+idle({Mod, native}) ->
+  list_to_binary([<<"workers.idle.">>, atom_to_list(Mod), <<"=native\n">>]);
+idle({Mod, Pid}) ->
+  IdleCount = integer_to_list(asset_pool:idle_worker_count(Pid)),
+  list_to_binary([<<"workers.idle.">>, atom_to_list(Mod), <<"=">>, IdleCount, <<"\n">>]).