passenger 2.1.3 → 2.2.0

data/doc/users_guide_snippets/analysis_and_system_maintenance_tools.txt

@@ -0,0 +1,144 @@
+Phusion Passenger provides a set of tools, which are useful for system analysis,
+maintenance and troubleshooting.
+
+
+=== Inspecting memory usage ===
+
+Process inspection tools such as `ps` and `top` are useful, but they
+link:http://groups.google.com/group/phusion-passenger/msg/1fd1c233456d3180[rarely show the correct memory usage].
+The real memory usage is usually lower than what `ps` and `top` report.
+
+There are many technical reasons why this is so, but an explanation is beyond
+the scope of this Users Guide. We kindly refer the interested reader to
+operating systems literature about 'virtual memory' and 'copy-on-write'.
+
+The tool `passenger-memory-stats` allows one to easily analyze Phusion Passenger's
+and Apache's real memory usage. For example:
+
+-------------------------------------------------------
+[bash@localhost root]# passenger-memory-stats
+------------- Apache processes --------------.
+PID    PPID  Threads  VMSize   Private  Name
+---------------------------------------------.
+5947   1     9        90.6 MB  0.5 MB   /usr/sbin/apache2 -k start
+5948   5947  1        18.9 MB  0.7 MB   /usr/sbin/fcgi-pm -k start
+6029   5947  1        42.7 MB  0.5 MB   /usr/sbin/apache2 -k start
+6030   5947  1        42.7 MB  0.5 MB   /usr/sbin/apache2 -k start
+6031   5947  1        42.5 MB  0.3 MB   /usr/sbin/apache2 -k start
+6033   5947  1        42.5 MB  0.4 MB   /usr/sbin/apache2 -k start
+6034   5947  1        50.5 MB  0.4 MB   /usr/sbin/apache2 -k start
+23482  5947  1        82.6 MB  0.4 MB   /usr/sbin/apache2 -k start
+### Processes: 8
+### Total private dirty RSS: 3.50 MB
+
+--------- Passenger processes ---------.
+PID    Threads  VMSize   Private  Name
+---------------------------------------.
+6026   1        10.9 MB  4.7 MB   Passenger spawn server
+23481  1        26.7 MB  3.0 MB   Passenger FrameworkSpawner: 2.0.2
+23791  1        26.8 MB  2.9 MB   Passenger ApplicationSpawner: /var/www/projects/app1-foobar
+23793  1        26.9 MB  17.1 MB  Rails: /var/www/projects/app1-foobar
+### Processes: 4
+### Total private dirty RSS: 27.76 M
+-------------------------------------------------------
+
+The 'Private' or 'private dirty RSS' field shows the *real* memory usage of processes. Here,
+we see that all the Apache worker processes only take less than 1 MB memory each.
+This is a lot less than the 50 MB-ish memory usage as shown in the 'VMSize' column
+(which is what a lot of people think is the real memory usage, but is actually not).
+
+NOTE: Private dirty RSS reporting only works on Linux. Unfortunately other operating systems
+don't provide facilities for determining processes' private dirty RSS. On non-Linux systems,
+the Resident Set Size is reported instead.
+
+
+=== Inspecting Phusion Passenger's internal status ===
+
+One can inspect Phusion Passenger's internal status with the tool `passenger-status`.
+This tool must typically be run as root. For example:
+
+--------------------------------------------------
+[bash@localhost root]# passenger-status
+----------- General information -----------
+max      = 6
+count    = 1
+active   = 0
+inactive = 1
+
+----------- Domains -----------
+/var/www/projects/app1-foobar: 
+  PID: 9617      Sessions: 0    Processed: 7       Uptime: 2m 23s
+--------------------------------------------------
+
+The 'general information' section shows the following information:
+
+max:: The maximum number of application instances that Phusion Passenger will
+spawn. This equals the value given for <<PassengerMaxPoolSize,PassengerMaxPoolSize>> (Apache)
+or <<PassengerMaxPoolSize,passenger_max_pool_size>> (Nginx).
+count:: The number of application instances that are currently alive. This value
+is always less than or equal to 'max'.
+active:: The number of application instances that are currently processing
+requests. This value is always less than or equal to 'count'.
+inactive:: The number of application instances that are currently *not* processing
+requests, i.e. are idle. Idle application instances will be shutdown after a while,
+as can be specified with <<PassengerPoolIdleTime,PassengerPoolIdleTime (Apache)>>/<<PassengerPoolIdleTime,passenger_pool_idle_time (Nginx)>> (unless this
+value is set to 0, in which case application instances are never shut down via idle
+time). The value of 'inactive' equals `count - active`.
+
+The 'domains' section shows, for each application directory, information about running
+application instances:
+
+Sessions:: Shows how many HTTP client are currently in the queue of that application
+Instance, waiting to be processed.
+Processed:: Indicates how many requests the instance has served until now. *Tip:* it's
+possible to limit this number with the <<PassengerMaxRequests,PassengerMaxRequests>>
+configuration directive.
+Uptime:: Shows for how long the application instance has been running.
+
+Since Phusion Passenger uses fair load balancing by default, the number of sessions for the
+application instances should be fairly close to each other. For example, this is fairly
+normal:
+--------------------------------
+  PID: 4281      Sessions: 2      Processed: 7      Uptime: 5m 11s
+  PID: 4268      Sessions: 0      Processed: 5      Uptime: 4m 52s
+  PID: 4265      Sessions: 1      Processed: 6      Uptime: 5m 38s
+  PID: 4275      Sessions: 1      Processed: 7      Uptime: 3m 14s
+--------------------------------
+
+But if you see a "spike", i.e. an application instance has an unusually high number of
+sessions compared to the others, then there might be a problem:
+--------------------------------
+  PID: 4281      Sessions: 2      Processed: 7      Uptime: 5m 11s
+  PID: 17468     Sessions: 8 <-+  Processed: 2      Uptime: 4m 47s
+  PID: 4265      Sessions: 1   |  Processed: 6      Uptime: 5m 38s
+  PID: 4275      Sessions: 1   |  Processed: 7      Uptime: 3m 14s
+                               |
+                               +---- "spike"
+--------------------------------
+
+Possible reasons why spikes can occur:
+
+. Your application is busy processing a request that takes a very long time.
+  If this is the case, then you might want to turn
+  <<PassengerUseGlobalQueue,global queuing>> on.
+. Your application is frozen, i.e. has stopped responding. See
+  <<debugging_frozen,Debugging frozen applications>> for tips.
+
+
+[[debugging_frozen]]
+=== Debugging frozen applications ===
+
+If one of your application instances is frozen (stopped responding), then you
+can figure out where it is frozen by killing it with 'SIGABRT'. This will cause the
+application to raise an exception, with a backtrace.
+
+The exception (with full backtrace information) is normally logged into the Apache
+error log. But if your application or if its web framework has its own exception logging
+routines, then exceptions might be logged into the application's log files instead.
+This is the case with Ruby on Rails. So if you kill a Ruby on Rails application with
+'SIGABRT', please check the application's 'production.log' first (assuming that you're
+running it in a 'production' environment). If you don't see a backtrace there, check
+the Apache error log.
+
+NOTE: It is safe to kill application instances, even in live environments. Phusion Passenger
+will restart killed application instances, as if nothing bad happened.

data/doc/users_guide_snippets/appendix_a_about.txt

@@ -0,0 +1,13 @@
+== Appendix A: About this document ==
+
+The text of this document is licensed under the
+link:http://creativecommons.org/licenses/by-sa/3.0/[Creative Commons
+Attribution-Share Alike 3.0 Unported License].
+
+image:images/by_sa.png[link="http://creativecommons.org/licenses/by-sa/3.0/"]
+
+Phusion Passenger is brought to you by link:http://www.phusion.nl/[Phusion].
+
+image:images/phusion_banner.png[link="http://www.phusion.nl/"]
+
+Phusion Passenger is a trademark of Hongli Lai & Ninh Bui.

data/doc/users_guide_snippets/appendix_b_terminology.txt

@@ -0,0 +1,63 @@
+== Appendix B: Terminology ==
+
+[[application_root]]
+=== Application root ===
+The root directory of an application that's served by Phusion Passenger.
+
+In case of Ruby on Rails applications, this is the directory that contains
+'Rakefile', 'app/', 'config/', 'public/', etc. In other words, the directory
+pointed to by `RAILS_ROOT`. For example, take the following directory structure:
+
+-----------------------------------------
+/apps/foo/       <------ This is the Rails application's application root!
+   |
+   +- app/
+   |   |
+   |   +- controllers/
+   |   |
+   |   +- models/
+   |   |
+   |   +- views/
+   |
+   +- config/
+   |   |
+   |   +- environment.rb
+   |   |
+   |   +- ...
+   |
+   +- public/
+   |   |
+   |   +- ...
+   |
+   +- ...
+-----------------------------------------
+
+In case of Rack applications, this is the directory that contains 'config.ru'.
+For example, take the following directory structure:
+
+-----------------------------------------
+/apps/bar/      <----- This is the Rack application's application root!
+   |
+   +- public/
+   |    |
+   |    +- ...
+   |
+   +- config.ru
+   |
+   +- ...
+-----------------------------------------
+
+In case of Python (WSGI) applications, this is the directory that contains
+'passenger_wsgi.py'. For example, take the following directory structure:
+
+-----------------------------------------
+/apps/baz/      <----- This is the WSGI application's application root!
+   |
+   +- public/
+   |    |
+   |    +- ...
+   |
+   +- passenger_wsgi.py
+   |
+   +- ...
+-----------------------------------------

data/doc/users_guide_snippets/appendix_c_spawning_methods.txt

@@ -0,0 +1,304 @@
+[[spawning_methods_explained]]
+== Appendix C: Spawning methods explained ==
+
+At its core, Phusion Passenger is an HTTP proxy and process manager. It spawns
+Ruby on Rails/Rack/WSGI worker processes (which may also be referred to as
+'backend processes'), and forwards incoming HTTP request to one of the worker
+processes.
+
+While this may sound simple, there's not just one way to spawn worker processes.
+Let's go over the different spawning methods. For simplicity's sake, let's
+assume that we're only talking about Ruby on Rails applications.
+
+=== The most straightforward and traditional way: conservative spawning ===
+
+Phusion Passenger could create a new Ruby process, which will then load the
+Rails application along with the entire Rails framework. This process will then
+enter an request handling main loop.
+
+This is the most straightforward way to spawn worker processes. If you're
+familiar with the Mongrel application server, then this approach is exactly
+what mongrel_cluster performs: it creates N worker processes, each which loads
+a full copy of the Rails application and the Rails framework in memory. The Thin
+application server employs pretty much the same approach.
+
+Note that Phusion Passenger's version of conservative spawning differs slightly
+from mongrel_cluster. Mongrel_cluster creates entirely new Ruby processes. In
+programmers jargon, mongrel_cluster creates new Ruby processes by forking the
+current process and exec()-ing a new Ruby interpreter. Phusion Passenger on the
+other hand creates processes that reuse the already loaded Ruby interpreter. In
+programmers jargon, Phusion Passenger calls fork(), but not exec().
+
+=== The smart spawning method ===
+
+NOTE: Smart spawning is only available for Ruby on Rails applications, not for
+      Rack applications or WSGI applications.
+
+While conservative spawning works well, it's not as efficient as it could be
+because each worker process has its own private copy of the Rails application
+as well as the Rails framework. This wastes memory as well as startup time.
+
+image:images/conservative_spawning.png[Worker processes and conservative spawning] +
+'Figure: Worker processes and conservative spawning. Each worker process has its
+own private copy of the application code and Rails framework code.'
+
+It is possible to make the different worker processes share the memory occupied
+by application and Rails framework code, by utilizing so-called
+copy-on-write semantics of the virtual memory system on modern operating
+systems. As a side effect, the startup time is also reduced. This is technique
+is exploited by Phusion Passenger's 'smart' and 'smart-lv2' spawn methods.
+
+==== How it works ====
+
+When the 'smart-lv2' spawn method is being used, Phusion Passenger will first
+create a so-called 'ApplicationSpawner server' process. This process loads the
+entire Rails application along with the Rails framework, by loading
+'environment.rb'. Then, whenever Phusion Passenger needs a new worker process,
+it will instruct the ApplicationSpawner server to do so. The ApplicationSpawner
+server will create a worker new process
+that reuses the already loaded Rails application/framework. Creating a worker
+process through an already running ApplicationSpawner server is very fast, about
+10 times faster than loading the Rails application/framework from scratch. If
+the Ruby interpreter is copy-on-write friendly (that is, if you're running
+<<reducing_memory_usage,Ruby Enterprise Edition>>) then all created worker
+processes will share as much common
+memory as possible. That is, they will all share the same application and Rails
+framework code.
+
+image:images/smart-lv2.png[] +
+'Figure: Worker processes and the smart-lv2 spawn method. All worker processes,
+as well as the ApplicationSpawner, share the same application code and Rails
+framework code.'
+
+The 'smart' spawn method goes even further, by caching the Rails framework in
+another process called the 'FrameworkSpawner server'. This process only loads
+the Rails framework, not the application. When a FrameworkSpawner server is
+instructed to create a new worker process, it will create a new
+ApplicationSpawner to which the instruction will be delegated. All those
+ApplicationSpawner servers, as well as all worker processes created by those
+ApplicationSpawner servers, will share the same Rails framework code.
+
+The 'smart-lv2' method allows different worker processes that belong to the same
+application to share memory. The 'smart' method allows different worker
+processes - that happen to use the same Rails version - to share memory, even if
+they don't belong to the same application.
+
+Notes:
+
+- Vendored Rails frameworks cannot be shared by different applications, even if
+  both vendored Rails frameworks are the same version. So for efficiency reasons
+  we don't recommend vendoring Rails.
+- ApplicationSpawner and FrameworkSpawner servers have an idle timeout just
+  like worker processes. If an ApplicationSpawner/FrameworkSpawner server hasn't
+  been instructed to do anything for a while, it will be shutdown in order to
+  conserve memory. This idle timeout is configurable.
+
+==== Summary of benefits ====
+
+Suppose that Phusion Passenger needs a new worker process for an application
+that uses Rails 2.2.1.
+
+- If the 'smart-lv2' spawning method is used, and an ApplicationSpawner server
+  for this application is already running, then worker process creation time is
+  about 10 times faster than conservative spawning. This worker process will also
+  share application and Rails framework code memory with the ApplicationSpawner
+  server and the worker processes that had been spawned by this ApplicationSpawner
+  server.
+- If the 'smart' spawning method is used, and a FrameworkSpawner server for
+  Rails 2.2.1 is already running, but no ApplicationSpawner server for this
+  application is running, then worker process creation time is about 2 times
+  faster than conservative spawning. If there is an ApplicationSpawner server
+  for this application running, then worker process creation time is about 10
+  times faster. This worker process will also share application and Rails
+  framework code memory with the ApplicationSpawner and FrameworkSpawner
+  servers.
+
+You could compare ApplicationSpawner and FrameworkSpawner servers with stem
+cells, that have the ability to quickly change into more specific cells (worker
+process).
+
+In practice, the smart spawning methods could mean a memory saving of about 33%,
+assuming that your Ruby interpreter is <<reducing_memory_usage,copy-on-write friendly>>.
+
+Of course, smart spawning is not without gotchas. But if you understand the
+gotchas you can easily reap the benefits of smart spawning.
+
+=== Smart spawning gotcha #1: unintential file descriptor sharing ===
+
+Because worker processes are created by forking from an ApplicationSpawner
+server, it will share all file descriptors that are opened by the
+ApplicationSpawner server. (This is part of the semantics of the Unix
+'fork()' system call. You might want to Google it if you're not familiar with
+it.) A file descriptor is a handle which can be an opened file, an opened socket
+connection, a pipe, etc. If different worker processes write to such a file
+descriptor at the same time, then their write calls will be interleaved, which
+may potentially cause problems.
+
+The problem commonly involves socket connections that are unintentially being
+shared. You can fix it by closing and reestablishing the connection when Phusion
+Passenger is creating a new worker process. Phusion Passenger provides the API
+call `PhusionPassenger.on_event(:starting_worker_process)` to do so. So you
+could insert the following code in your 'environment.rb':
+
+[source, ruby]
+-----------------------------------------
+if defined?(PhusionPassenger)
+    PhusionPassenger.on_event(:starting_worker_process) do |forked|
+        if forked
+            # We're in smart spawning mode.
+            ... code to reestablish socket connections here ...
+        else
+            # We're in conservative spawning mode. We don't need to do anything.
+        end
+    end
+end
+-----------------------------------------
+
+Note that Phusion Passenger automatically reestablishes the connection to the
+database upon creating a new worker process, which is why you normally do not
+encounter any database issues when using smart spawning mode.
+
+==== Example 1: Memcached connection sharing (harmful) ====
+
+Suppose we have a Rails application that connects to a Memcached server in
+'environment.rb'. This causes the ApplicationSpawner to have a socket connection
+(file descriptor) to the Memcached server, as shown in the following figure:
+
+ +--------------------+
+ | ApplicationSpawner |-----------[Memcached server]
+ +--------------------+
+
+Phusion Passenger then proceeds with creating a new Rails worker process, which
+is to process incoming HTTP requests. The result will look like this:
+
+ +--------------------+
+ | ApplicationSpawner |------+----[Memcached server]
+ +--------------------+      |
+                             |
+ +--------------------+      |
+ | Worker process 1   |-----/
+ +--------------------+
+
+Since a 'fork()' makes a (virtual) complete copy of a process, all its file
+descriptors will be copied as well. What we see here is that ApplicationSpawner
+and Worker process 1 both share the same connection to Memcached.
+
+Now supposed that your site gets Slashdotted and Phusion Passenger needs to
+spawn another worker process. It does so by forking ApplicationSpawner. The
+result is now as follows:
+
+ +--------------------+
+ | ApplicationSpawner |------+----[Memcached server]
+ +--------------------+      |
+                             |
+ +--------------------+      |
+ | Worker process 1   |-----/|
+ +--------------------+      |
+                             |
+ +--------------------+      |
+ | Worker process 2   |-----/
+ +--------------------+
+
+As you can see, Worker process 1 and Worker process 2 have the same Memcache
+connection.
+
+Suppose that users Joe and Jane visit your website at the same time. Joe's
+request is handled by Worker process 1, and Jane's request is handled by Worker
+process 2. Both worker processes want to fetch something from Memcached. Suppose
+that in order to do that, both handlers need to send a "FETCH" command to Memcached.
+
+But suppose that, after worker process 1 having only sent "FE", a context switch
+occurs, and worker process 2 starts sending a "FETCH" command to Memcached as
+well. If worker process 2 succeeds in sending only one bye, 'F', then Memcached
+will receive a command which begins with "FEF", a command that it does not
+recognize. In other words: the data from both handlers get interleaved. And thus
+Memcached is forced to handle this as an error.
+
+This problem can be solved by reestablishing the connection to Memcached after forking:
+
+ +--------------------+
+ | ApplicationSpawner |------+----[Memcached server]
+ +--------------------+      |                   |
+                             |                   |
+ +--------------------+      |                   |
+ | Worker process 1   |-----/|                   |
+ +--------------------+      |                   |  <--- created this
+                             X                   |       new
+                                                 |       connection
+                             X <-- closed this   |
+ +--------------------+      |     old           |
+ | Worker process 2   |-----/      connection    |
+ +--------------------+                          |
+           |                                     |
+           +-------------------------------------+
+
+Worker process 2 now has its own, separate communication channel with Memcached.
+The code in 'environment.rb' looks like this:
+
+[source, ruby]
+-----------------------------------------
+if defined?(PhusionPassenger)
+    PhusionPassenger.on_event(:starting_worker_process) do |forked|
+        if forked
+            # We're in smart spawning mode.
+            reestablish_connection_to_memcached
+        else
+            # We're in conservative spawning mode. We don't need to do anything.
+        end
+    end
+end
+-----------------------------------------
+
+==== Example 2: Log file sharing (not harmful) ====
+
+There are also cases in which unintential file descriptor sharing is not harmful.
+One such case is log file file descriptor sharing. Even if two processes write
+to the log file at the same time, the worst thing that can happen is that the
+data in the log file is interleaved.
+
+To guarantee that the data written to the log file is never interleaved, you
+must synchronize write access via an inter-process synchronization mechanism,
+such as file locks. Reopening the log file, like you would have done in the
+Memcached example, doesn't help.
+
+=== Smart spawning gotcha #2: the need to revive threads ===
+
+Another part of the 'fork()' system call's semantics is the fact that threads
+disappear after a fork call. So if you've created any threads in environment.rb,
+then those threads will no longer be running in newly created worker process.
+You need to revive them when a new worker process is created. Use the
+`:starting_worker_process` event that Phusion Passenger provides, like this:
+
+[source, ruby]
+-----------------------------------------
+if defined?(PhusionPassenger)
+    PhusionPassenger.on_event(:starting_worker_process) do |forked|
+        if forked
+            # We're in smart spawning mode.
+            ... code to revive threads here ...
+        else
+            # We're in conservative spawning mode. We don't need to do anything.
+        end
+    end
+end
+-----------------------------------------
+
+=== Smart spawning gotcha #3: code load order ===
+
+This gotcha is only applicable to the 'smart' spawn method, not the 'smart-lv2'
+spawn method.
+
+If your application expects the Rails framework to be not loaded during the
+beginning of 'environment.rb', then it can cause problems when an
+ApplicationSpawner is created from a FrameworkSpawner, which already has the
+Rails framework loaded. The most common case is when applications try to patch
+Rails by dropping a modified file that has the same name as Rails's own file,
+in a path that comes earlier in the Ruby search path.
+
+For example, suppose that we have an application which has a patched version
+of 'active_record/base.rb' located in 'RAILS_ROOT/lib/patches', and
+'RAILS_ROOT/lib/patches' comes first in the Ruby load path. When conservative
+spawning is used, the patched version of 'base.rb' is properly loaded. When
+'smart' (not 'smart-lv2') spawning is used, the original 'base.rb' is used
+because it was already loaded, so a subsequent `require "active_record/base"`
+has no effect.