passenger 4.0.60 → 5.0.0.beta1

data/dev/vagrant/provision.sh

@@ -62,8 +62,10 @@ fi
 
 pushd /vagrant
 if [[ ! -e ~/.test_deps_installed ]]; then
-	rake test:install_deps SUDO=1
+	rake test:install_deps SUDO=1 DEPS_TARGET=~/bundle
 	touch ~/.test_deps_installed
+else
+	bundle install --path ~/bundle
 fi
 popd
 

data/doc/DebuggingAndStressTesting.md

@@ -7,7 +7,7 @@ This guide tells you:
 
 ## Process output
 
-All Phusion Passenger agent processes (PassengerWatchdog, PassengerHelperAgent, PassengerLoggingAgent) as well as all spawned application processes have their stdout and stderr redirected to the _global web server error log_ (that is, _not_ the per-virtual host error log). This is usually '/var/log/apache2/error.log' or '/var/log/nginx/error.log'.
+All PassengerAgent processes as well as all spawned application processes have their stdout and stderr redirected to the _global web server error log_ (that is, _not_ the per-virtual host error log). This is usually '/var/log/apache2/error.log' or '/var/log/nginx/error.log'.
 
 Note that in case of Nginx, Phusion Passenger prints to the error log specified in the server context, not the "http" context. If the server context does not contain an `error_log` directive then the default error log location will be used. The default location depends on how Nginx is configured during compilation, but it is usually either '$PREFIX/logs/error.log' or '/var/log/nginx/error.log'. For example, if your Nginx configuration looks like this:
 
@@ -55,6 +55,6 @@ To enable, set the environment variable `PASSENGER_SIMULATE_SYSCALL_FAILURES`. T
 
 `program_nameN` specifies the name of the Phusion Passenger process for which system call failure simulation should be enabled. This is followed by a list of system call `errno` names and the respective probabilities (between 0 and 1). For example:
 
-    export PASSENGER_SIMULATE_SYSCALL_FAILURES='PassengerWatchdog=ENOSPC:0.01;PassengerHelperAgent=EMFILE:0.001,ECONNREFUSED:0.02'
+    export PASSENGER_SIMULATE_SYSCALL_FAILURES='watchdog=ENOSPC:0.01;server=EMFILE:0.001,ECONNREFUSED:0.02'
 
-This will enable system call failure simulation only for PassengerWatchdog and PassengerHelperAgent, but not for PassengerLoggingAgent. All system calls in PassengerWatchdog will have a 1% probability of throwing ENOSPC. All system calls in PassengerHelperAgent will have a 0.1% probability of throwing EMFILE, and a 2% probability of throwing ECONNREFUSED.
+This will enable system call failure simulation only for the watchdog and the HTTP server, but not for the logging agent. All system calls in the watchdog will have a 1% probability of throwing ENOSPC. All system calls in the HTTP server will have a 0.1% probability of throwing EMFILE, and a 2% probability of throwing ECONNREFUSED.

data/doc/Design and Architecture.txt

@@ -152,7 +152,7 @@ Phusion Passenger initializes as follows.
  * `ext/apache2/Hooks.cpp`, in the constructor for the `Hooks` class.
  * `ext/common/AgentsStarter.h` and `AgentsStarter.cpp`. Most of the logic pertaining starting the Watchdog is in this file.
 
-3. The Watchdog first initializes a <<instance_state_and_communication,"server instance directory">>, which is a temporary directory containing files that will be used during the life time of this Phusion Passenger instance. For example, the directory contains Unix domain socket files, so that the different Phusion Passenger processes can communicate with each other. The Watchdog is implemented in `ext/common/agents/Watchdog/Main.cpp`.
+3. The Watchdog first initializes a <<instance_state_and_communication,"instance directory">>, which is a temporary directory containing files that will be used during the life time of this Phusion Passenger instance. For example, the directory contains Unix domain socket files, so that the different Phusion Passenger processes can communicate with each other. The Watchdog is implemented in `ext/common/agents/Watchdog/Main.cpp`.
 
 4. The Watchdog starts the HelperAgent and the LoggingAgent simultaneously. Each performs its own initialization.
 
@@ -174,7 +174,7 @@ The HelperAgent consists of two subsystems. One is the *request handling subsyst
 
 Recall that requests are first received from the web server. The web server serializes the request into a slightly modified version of link:http://en.wikipedia.org/wiki/SCGI[the SCGI format], and sends it to the HelperAgent's RequestHandler. The RequestHandler performs some work, and eventually sends back a regular HTTP response. The web server parses the RequestHandler response, and sends a response to the original HTTP client.
 
-The RequestHandler listens on a Unix domain socket file. This Unix domain socket file is called `request`, and is located in <<instance_state_and_communication,the server instance directory>>.
+The RequestHandler listens on a Unix domain socket file. This Unix domain socket file is called `request`, and is located in <<instance_state_and_communication,the instance directory>>.
 
 ==== One client per request
 
@@ -313,8 +313,7 @@ All (pre)loaders are located in the `helper-scripts` directory in the source tre
 [options="header"]
 |================================================================================
 | Language/Framework         | Loader                  | Preloader
-| Ruby Rack and Rails >= 3.x | rack-loader.rb          | rack-preloader.rb
-| Ruby on Rails 1.x and 2.x  | classic-rails-loader.rb | classic-rails-preloader.rb
+| Ruby Rack and Rails        | rack-loader.rb          | rack-preloader.rb
 | Python                     | wsgi-loader.py          | -
 | Node.js and bundled Meteor | node-loader.js          | -
 | Unbundled Meteor           | meteor-loader.rb        | -
@@ -484,9 +483,9 @@ Every time you start Phusion Passenger, you've created a new *instance*. Every i
 
 Clearly, the processes can't listen on a specific TCP port for communication. Nor can they listen on a fixed Unix domain socket filename.
 
-That is where the 'server instance directory' comes in. Every Phusion Passenger instance has its own, unique temporary directory. That directory is removed when the instance halts. The directory contains Unix domain socket files that the processes listen on. Every Phusion Passenger related process knows where its own server instance directory is, and thus, knows how to communicate with other processes belonging to the same instance. The server instance directory is implemented in `ext/common/ServerInstanceDir.h`.
+That is where the 'instance directory' comes in. Every Phusion Passenger instance has its own, unique temporary directory. That directory is removed when the instance halts. The directory contains Unix domain socket files that the processes listen on. Every Phusion Passenger related process knows where its own instance directory is, and thus, knows how to communicate with other processes belonging to the same instance. The instance directory is implemented in `ext/common/InstanceDirectory.h`.
 
-Administration tools such as `passenger-status` query information using server instance directories. First, they check which server instance directories exist on the system. If they find only one, then they query the sockets inside that sole server instance directory. Otherwise, they abort with an error and ask the user to specifically select the instance to query.
+Administration tools such as `passenger-status` query information using instance directories. First, they check which instance directories exist on the system. If they find only one, then they query the sockets inside that sole instance directory. Otherwise, they abort with an error and ask the user to specifically select the instance to query.
 
 
 [appendix]

data/doc/Packaging.txt.md

@@ -24,8 +24,8 @@ to be compiled by the user. Phusion Passenger looks for binaries in, and (if
 the user initiates the compilation process) stores binaries in, the following
 directories:
 
- * Normally, binaries are to be located in the `agents` and `libout`
-   subdirectories under the source root.
+ * Normally, binaries are to be located in the `buildout` subdirectory under
+   the source root.
  * Phusion Passenger Standalone does things a little differently. Binaries are
    to be located in one of the following directories, whichever it finds first:
 
@@ -86,7 +86,7 @@ The location configuration file is an ini file that looks as follows:
     [locations]
     natively_packaged=true
     bin_dir=/usr/bin
-    agents_dir=/usr/lib/phusion-passenger/agents
+    support_binaries_dir=/usr/lib/phusion-passenger/support-binaries
     lib_dir=/usr/lib/phusion-passenger
     helper_scripts_dir=/usr/share/phusion-passenger/helper-scripts
     resources_dir=/usr/share/phusion-passenger
@@ -165,14 +165,14 @@ a list of all possible assets and asset directories.
 
    Value when originally packaged: `<SOURCE_ROOT>/bin`
 
- * `agents_dir`
+ * `support_binaries_dir`
 
    A directory that contains (platform-dependent) binaries that Phusion Passenger
    uses, but that should not be directly invoked from the command line. Things like
-   PassengerHelperAgent are located here.
+   PassengerAgent are located here.
 
    Value when originally packaged:
-   - Normally: `<SOURCE_ROOT>/buildout/agents`
+   - Normally: `<SOURCE_ROOT>/buildout/support-binaries`
    - Passenger Standalone: `~/.passenger/standalone/<VERSION>/support-<ARCH>`
 
  * `helper_scripts_dir`
@@ -246,6 +246,35 @@ a list of all possible assets and asset directories.
 
    Value when originally packaged: `<SOURCE_ROOT>/ext/nginx`.
 
+Optional fields:
+
+ * `build_system_dir`
+
+   The directory that contains the Phusion Passenger main Rakefile, used for
+   compiling the Apache module and the agent executable. This field is only
+   present if Phusion Passenger is compilable. Native packages usually do not
+   have this field because they ship a precompiled Apache module and a
+   precompiled agent executable.
+
+   Value when originally packaged: `<SOURCE_ROOT>`
+
+ * `download_cache_dir`
+
+   The directory that contains cached downloaded agent executables. Its main
+   use case is to speed up agent executable downloading when Phusion Passenger
+   is installed from a Ruby gem. When the user installs the gem, a script is
+   invoked which downloads agent executables from the Phusion Passenger
+   websites. The downloaded files are stored in this directory. Then later,
+   when the user runs `passenger start` or any other command which requires
+   the agent executable, the executable will be copied from this cache
+   directory instead of downloaded.
+
+   Native packages ship precompiled executables, so they can omit this field.
+
+   Value when originally packaged: `<SOURCE_ROOT>/download_cache`
+
+
+
 
 # Vendoring of libraries
 

data/doc/ServerOptimizationGuide.txt.md

@@ -0,0 +1,339 @@
+# Server optimization guide
+
+There are two aspects with regard to optimizing Phusion Passenger performance.
+
+The first aspect is *settings tuning*. Phusion Passenger's default settings are not aimed at optimizing, but at *safety*. The defaults are designed to conserve resources, to prevent server overload and to keep web apps up and running. To optimize for performance, you need to tweak some settings whose values depend on your hardware and your environment.
+
+Besides Phusion Passenger settings, you may also want to tune kernel-level settings.
+
+The second aspect is *using performance-enhancing features*. This requires small application-level changes.
+
+If you are optimizing Phusion Passenger for the purpose of **benchmarking** then you should also follow the benchmarking recommendations.
+
+<a name="minimizing_process_spawning"></a>
+
+## Minimizing process spawning
+
+By default, Phusion Passenger spawns and shuts down application processes according to traffic. This allows it to use more resources during busy times, while conserving resources during idle times. This is especially useful if you host more than 1 app on a single server: if not all apps are used at the same time, then you don't have to keep all apps running at the same time.
+
+However, spawning a process takes a lot of time (in the order of 10-20 seconds for a Rails app), and CPU usage will be near 100% during spawning. Therefore, while spawning, your server will be slower at performing other activities, such as handling requests.
+
+For consistent performance, it is thus recommended that you configure a static process pool: telling Phusion Passenger to use a fixed number of processes, instead of spawning and shutting them down dynamically.
+
+### Standalone
+
+Run `passenger start` with `--max-pool-size=N --min-instances=N`, where `N` is the number of processes you want.
+
+### Nginx
+
+Let `N` be the number of processes you want. Set the following configuration in your `http` block:
+
+    passenger_max_pool_size N;
+
+Set the following configuration in your `server` block:
+
+    passenger_min_instances N;
+
+You should also configure `passenger_pre_start` in the `http` block so that your app is started during web server launch:
+
+    # Refer to the Users Guide for more information about passenger_pre_start.
+    passenger_pre_start http://your-website-url.com;
+
+### Apache
+
+Let `N` be the number of processes you want. Set the following configuration in the global context:
+
+    PassengerMaxPoolSize N
+
+Set the following configuration in your virtual host block:
+
+    PassengerMinInstances N
+
+You should also configure `PassengerPreStart` in the global context so that your app is started during web server launch:
+
+    # Refer to the Users Guide for more information about PassengerPreStart.
+    PassengerPreStart http://your-website-url.com
+
+## Maximizing throughput
+
+This section provides guidance on maximizing Phusion Passenger's throughput. The amount of throughput that Phusion Passenger handles is proportional to the number of processes or threads that you've configured. More processes/threads generally means more throughput, but there is an upper limit. Past a certain value, further increasing the number of processes/threads won't help. If you increase the number of processes/threads even further, then performance may even go down.
+
+The optimal value depends on the hardware and the environment. This section will provide you with formulas to calculate that optimal value. The following factors are involved in calculation:
+
+ * **Memory**. More processes implies a higher memory usage. If too much memory is used then the machine will hit swap, which slows everything down. You should only have as many processes as memory limits comfortably allow. Threads use less memory, so prefer threads when possible. You can create tens of threads in place of one process.
+ * **Number of CPUs**. True (hardware) concurrency cannot be higher than the number of CPUs. In theory, if all processes/threads on your system use the CPUs constantly, then:
+
+    * You can increase throughput up to `NUMBER_OF_CPUS` processes/threads.
+    * Increasing the number of processes/threads after that point will increase virtual (software) concurrency, but will not increase true (hardware) concurrency and will not increase maximum throughput.
+
+   Having more processes than CPUs may decrease total throughput a little thanks to context switching overhead, but the difference is not big because OSes are good at context switching these days.
+
+   On the other hand, if your CPUs are not used constantly, e.g. because they’re often blocked on I/O, then the above does not apply and increasing the number of processes/threads does increase concurrency and throughput, at least until the CPUs are saturated.
+ * **Blocking I/O**. This covers all blocking I/O, including hard disk access latencies, database call latencies, web API calls, etc. Handling input from the client and output to the client does not count as blocking I/O, because Phusion Passenger has buffering layers that relief the application from worrying about this.
+
+   The more blocking I/O calls your application process/thread makes, the more time it spends on waiting for external components. While it’s waiting it does not use the CPU, so that’s when another process/thread should get the chance to use the CPU. If no other process/thread needs CPU right now (e.g. all processes/threads are waiting for I/O) then CPU time is essentially wasted. Increasing the number processes or threads decreases the chance of CPU time being wasted. It also increases concurrency, so that clients do not have to wait for a previous I/O call to be completed before being served.
+
+The formulas in this section assume that your machine is dedicated to Phusion Passenger. If your machine also hosts other software (e.g. a database) then you'll need to tweak the formulas a little bit.
+
+### Tuning the application process and thread count
+
+#### Step 1: determining the application's memory usage
+
+The amount of memory that your application uses on a per-process basis, is key to our calculation. You should first figure out how much memory your application typically needs. Every application has different memory usage patterns, so the typical memory usage is best determined by observation.
+
+Run your app for a while, then run `passenger-status` at different points in time to examine memory usage. Then calculate the average of your data points. In the rest of this section, we'll refer to the amount of memory (in MB) that an application process needs, as `RAM_PER_PROCESS`.
+
+In our experience, a typical medium-sized single-threaded Rails application process can use 150 MB of RAM on a 64-bit machine, even when the spawning method is set to "smart".
+
+#### Step 2: determine the system's limits
+
+First, let's define the maximum number of (single-threaded) processes, or the number of threads, that you can comfortably have given the amount of RAM you have. This is a reasonable upper limit that you can reach without degrading system performance. This number is not the final optimal number, but is merely used for further caculations in later steps.
+
+There are two formulas that we can use, depending on what kind of concurrency model your application is using in production.
+
+**Purely single-threaded multi-process formula**
+
+If you didn't explicitly configure multithreading, then you are using using this concurrency model. Or, if you are *not* using Ruby (e.g. if you using **Python, Node.js or Meteor**), then you are also using this concurrency model, because Phusion Passenger only supports multithreading for Ruby apps.
+
+The formula is then as follows:
+
+    max_app_processes = (TOTAL_RAM * 0.75) / RAM_PER_PROCESS
+
+It is derived as follows:
+
+ * `(TOTAL_RAM * 0.75)`: We can assume that there must be at least 25% of free RAM that the operating system can use for other things. The result of this calculation is the RAM that is freely available for applications. If your system runs a lot of services and thus has less memory available for Phusion Passenger and its apps, then you should lower `0.75` to some constant that you think is appropriate.
+ * `/ RAM_PER_PROCESS`: Each process consumes a roughly constant amount of RAM, so the maximum number of processes is a single devision between the aforementioned calculation and this constant.
+
+**Multithreaded formula**
+
+The formula for multithreaded concurrency is as follows:
+
+    max_app_threads_per_process =
+      ((TOTAL_RAM * 0.75) - (CHOSEN_NUMBER_OF_PROCESSES * RAM_PER_PROCESS * 0.9)) /
+      (RAM_PER_PROCESS / 10)
+
+Here, `CHOSEN_NUMBER_OF_PROCESSES` is the number of application processes you want to use. In case of Ruby, Python, Node.js and Meteor, this should be equal to `NUMBER_OF_CPUS`. This is because all these languages can only utilize a single CPU core per process. If you're using a language runtime that does not have a Global Interpreter Lock, e.g. JRuby or Rubinius, then `CHOSEN_NUMBER_OF_PROCESSES` can be 1.
+
+The formula is derived as follows:
+
+ * `(TOTAL_RAM * 0.75)`: The same as explained earlier.
+ * `(CHOSEN_NUMBER_OF_PROCESSES * RAM_PER_PROCESS)`: In multithreaded scenarios, the application processes consume a constant amount of memory, so we deduct this from the RAM that is available to applications. The result is the amount of RAM available to application threads.
+ * `/ (RAM_PER_PROCESS / 10)`: A thread consumes about 10% of the amount of memory a process would, so we divide the amount of RAM available to threads with this number. What we get is the number of threads that the system can handle.
+
+On 32-bit systems, `max_app_threads_per_process` should not be higher than about 200. Assuming an 8 MB stack size per thread, you will run out of virtual address space if you go much further. On 64-bit systems you don’t have to worry about this problem.
+
+#### Step 3: derive the applications' needs
+
+The earlier two formulas were not for calculating the number of processes or threads that application needs, but for calculating how much the system can handle without getting into trouble. Your application may not actually need that many processes or threads! If your application is CPU-bound, then you only need a small multiple of the number of CPUs you have. Only if your application performs a lot of blocking I/O (e.g. database calls that take tens of milliseconds to complete, or you call to Twitter) do you need a large number of processes or threads.
+
+Armed with this knowledge, we derive the formulas for calculating how many processes or threads we actually need.
+
+ * If your application performs a lot of blocking I/O then you should give it as many processes and threads as possible:
+
+       # Use this formula for purely single-threaded multi-process scenarios.
+       desired_app_processes = max_app_processes
+
+       # Use this formula for multithreaded scenarios.
+       desired_app_threads_per_process = max_app_threads_per_process
+
+ * If your application doesn’t perform a lot of blocking I/O, then you should limit the number of processes or threads to a multiple of the number of CPUs to minimize context switching:
+
+       # Use this formula for purely single-threaded multi-process scenarios.
+       desired_app_processes = min(max_app_processes, NUMBER_OF_CPUS)
+
+       # Use this formula for multithreaded scenarios.
+       desired_app_threads_per_process = min(max_app_threads_per_process, 2 * NUMBER_OF_CPUS)
+
+#### Step 3: configure Phusion Passenger
+
+**Purely single-threaded multi-process scenarios**
+
+ * When using Phusion Passenger Standalone, run `passenger start` with `--max-pool-size=<desired_app_processes> --min-instances=<desired_app_processes>`.
+ * When using Phusion Passenger for Nginx, configure:
+   * `passenger_max_pool_size <desired_app_processes>;`
+   * `passenger_min_instances <desired_app_processes>;`
+   * `passenger_pre_start` to have your app started automatically at web server boot.
+ * When using Phusion Passenger for Apache, configure:
+   * `PassengerMaxPoolSize <desired_app_processes>`
+   * `PassengerMinInstances <desired_app_processes>`
+   * `PassengerPreStart` to have your app started automatically at web server boot.
+
+**Multithreaded scenarios**
+
+In order to use multithreading you must use Phusion Passenger Enterprise. The open source version of Phusion Passenger does not support multithreading.
+
+ * When using Phusion Passenger Standalone:
+   * Run `passenger start` with `--max-pool-size=<CHOSEN_NUMBER_OF_PROCESSES> --min-instances=<CHOSEN_NUMBER_OF_PROCESSES> --concurrency-model=thread --thread-count=<desired_app_threads_per_process>`
+   * If `desired_app_processes` is 1, then you should also add `--spawn-method=direct`. By using direct spawning instead of smart spawning, Phusion Passenger will not keep a Preloader process around, saving you some memory. This is because a Preloader process is useless when there's only 1 application process.
+ * When using Phusion Passenger for Nginx, configure:
+   * `passenger_max_pool_size <CHOSEN_NUMBER_OF_PROCESSES>;`
+   * `passenger_min_instances <CHOSEN_NUMBER_OF_PROCESSES>;`
+   * `passenger_concurrency_model thread;`
+   * `passenger_thread_count <desired_app_threads_per_process>;`
+   * `passenger_pre_start` to have your app started automatically at web server boot.
+   * If `desired_app_processes` is 1, then you should set `passenger_spawn_method direct`. By using direct spawning instead of smart spawning, Phusion Passenger will not keep a Preloader process around, saving you some memory. This is because a Preloader process is useless when there's only 1 application process.
+ * When using Phusion Passenger for Apache, configure:
+   * `PassengerMaxPoolSize <desired_app_processes>`
+   * `PassengerMinInstances <desired_app_processes>`
+   * `PassengerConcurrencyModel thread`
+   * `PassengerThreadCount <desired_app_threads_per_process>`
+   * `PassengerPreStart` to have your app started automatically at web server boot.
+   * If `desired_app_processes` is 1, then you should set `PassengerSpawnMethod direct`. By using direct spawning instead of smart spawning, Phusion Passenger will not keep a Preloader process around, saving you some memory. This is because a Preloader process is useless when there's only 1 application process.
+
+#### Possible step 4: configure Rails
+
+Only if you're using the multithreaded concurrency model do you need to configure Rails. You need to enable thread-safety by setting `config.thread_safe!` in `config/environments/production.rb`. In Rails 4.0 this is on by default for the production environment, but in earlier versions you had to enable it manually.
+
+You should also increase the ActiveRecord pool size because it limits concurrency. You can configure it in `config/database.yml`. Set the `pool` value to the number of threads per application process. But if you believe your database cannot handle that much concurrency, keep it at a low value.
+
+#### Example 1: purely single-threaded multi-process scenario with lots of blocking I/O, in a low-memory server
+
+Suppose you have:
+
+ * 1 GB of RAM.
+ * 150 MB of memory usage per application process.
+ * Lots of blocking I/O in the application.
+ * A purely single-threaded multi-process scenario.
+
+Then the calculation is as follows:
+
+    # Use this formula for purely single-threaded multi-process deployments.
+    max_app_processes = (1024 * 0.75) / 150 = 5.12
+    desired_app_processes = max_app_processes = 5.12
+
+Conclusion: you should use 5 or 6 processes. Phusion Passenger should be configured as follows:
+
+    # Standalone
+    passenger start --max-pool-size=5 --min-instances=5
+
+    # Nginx
+    passenger_max_pool_size 5;
+    passenger_min_instances 5;
+
+    # Apache
+    PassengerMaxPoolSize 5
+    PassengerMinInstances 5
+
+However a concurrency of 5 or 6 is way too low if your application performs a lot of blocking I/O. You should use a multithreaded deployment instead, or you need to get more RAM so you can run more processes.
+
+#### Example 2: purely single-threaded multi-process scenario with lots of blocking I/O, in a high-memory server
+
+Suppose you have:
+
+ * 8 CPUs.
+ * 32 GB of RAM.
+ * 150 MB of memory usage per application process.
+ * Lots of blocking I/O in the application.
+ * A purely single-threaded multi-process scenario.
+
+Then the calculation is as follows:
+
+    # Use this formula for purely single-threaded multi-process deployments.
+    max_app_processes = (1024 * 32 * 0.75) / 150 = 163.84
+    desired_app_processes = max_app_processes = 163.84
+
+Conclusion: you should use 163 or 164 processes. This number seems high, but the value is correct. Because your app performs a lot of blocking I/O, you need a lot of I/O concurrency. The more concurrency the better. The amount of concurrency scales linearly with the number of processes, which is why you end up with such a large number.
+
+Phusion Passenger should be configured as follows:
+
+    # Standalone
+    passenger start --max-pool-size=163 --min-instances=163
+
+    # Nginx
+    passenger_max_pool_size 163;
+    passenger_min_instances 163;
+
+    # Apache
+    PassengerMaxPoolSize 163
+    PassengerMinInstances 163
+
+Note that in this example, 163-164 processes is merely the maximum number of processes that you can run, without overloading your RAM. It does not mean that you have enough concurrency for your application! If you need more concurrency, you should use a multithreaded deployment instead.
+
+#### Example 3: multithreaded deployment with lots of blocking I/O
+
+Consider the same machine as in example 2:
+
+ * 8 CPUs.
+ * 32 GB of RAM.
+ * 150 MB of memory usage per application process.
+ * Lots of blocking I/O in the application.
+ * A purely single-threaded multi-process scenario.
+
+But this time you're using multithreading with 8 application processes (because you have 8 CPUs). How many threads do you need per process?
+
+    # Use this formula for multithreaded deployments.
+    max_app_threads_per_process
+    = ((1024 * 32 * 0.75) - (8 * 150)) / (150 / 10)
+    = 1558.4
+
+Conclusion: you should use 1558 threads per process.
+
+    # Standalone
+    passenger start --max-pool-size=8 --min-instances=8 --concurrency-model=thread --thread-count=1558
+
+    # Nginx
+    passenger_max_pool_size 8;
+    passenger_min_instances 8;
+    passenger_concurrency_model thread;
+    passenger_thread_count 1558;
+
+    # Apache
+    PassengerMaxPoolSize 8
+    PassengerMinInstances *
+    PassengerConcurrencyModel thread
+    PassengerThreadCount 1558
+
+Because of the huge number of threads, this only works on a 64-bit platform. If you're on a 32-bit platform, consider lowering the number of threads while raising the number of processes. For example, you can double the number of processes (to 16) and halve the number of threads (to 779).
+
+### Configuring the web server
+
+If you're using Nginx then it does not need additional configuration. Nginx is evented and already supports a high concurrency out of the box.
+
+If you're using Apache, then prefer the worker MPM (which uses a combination of processes and threads) or the event MPM (which is similar to the worker MPM, but better) over the prefork MPM (which only uses processes) whenever possible. PHP requires prefork, but if you don't use PHP then you can probably use one of the other MPMs. Make sure you set a low number of processes and a moderate to high number of threads.
+
+Because Apache performs a lot of blocking I/O (namely HTTP handling), you should give it a lot of threads so that it has a lot of concurrency. Apache's concurrency *must* be somewhat larger than the total number of application processes or total number of application threads. When considering example 3, the Apache concurrency must be larger than `8 * 1558 = 12464`.
+
+If you cannot use the event MPM, consider putting Apache behind an Nginx reverse proxy, with response buffering turned on on the Nginx side. This reliefs a lot of concurrency problems from Apache. If you can use the event MPM then adding Nginx to the mix does not provide many advantages.
+
+### Summary
+
+ * If your application performs a lot of blocking I/O, use lots of processes/threads. You should move away from single-threaded multiprocessing in this case, and start using multithreading.
+ * If your application is CPU-bound, use a small multiple of the number of CPUs.
+ * Do not exceed the number of processes/threads your system can handle without swapping.
+
+## Performance-enhancing features
+
+<a name="turbocaching"></a>
+
+### Turbocaching
+
+Phusion Passenger supports turbocaching since version 4. Turbocaching is an HTTP cache built inside Phusion Passenger. When used correctly, the cache can accelerate your app tremendously.
+
+To utilize turbocaching, you only need to set HTTP caching headers. Please refer to [Google's HTTP caching tutorial](https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching). Phusion Passenger takes advantage of the HTTP headers automatically.
+
+### Out-of-band garbage collection
+
+Phusion Passenger supports out-of-band garbage collection for Ruby apps. With this feature enabled, Phusion Passenger can run the garbage collector in between requests, so that the garbage collector doesn't delay the app as much. Please refer to the Users Guide for more information about this feature.
+
+## Benchmarking recommendations
+
+### Tooling recommendations
+
+ * Use [wrk](https://github.com/wg/wrk) as benchmarking tool.
+   - We do not recommend `ab` because it's slow and buggy.
+   - We do not recommend `siege` and `httperf` because they cannot utilize multiple CPU cores.
+ * Enable HTTP keep-alive in both the server and in your benchmarking tool. Otherwise you will end up benchmarking how quickly the kernel can set up TCP connections, which is a non-trivial part of the request time.
+ * Warm up the server before benchmarking. That is, run a mini-benchmark before the actual benchmark, and discard the result of the mini-benchmark.
+
+### Operating system recommendations
+
+ * Don't benchmark on OS X. OS X's TCP stack and process scheduler are horrible from a performance point of view. We recommend Linux.
+ * When on Linux, be sure to [tune your kernel socket settings](http://www.joedog.org/articles-tuning/) so that they don't stall the benchmark.
+
+### Server and application recommendations
+
+ * If the purpose of your benchmark is to compare against Puma, Unicorn or other app servers, be sure to benchmark against Phusion Passenger Standalone, not Phusion Passenger for Nginx or Phusion Passenger for Apache.
+   - This is because Puma, Unicorn and other app servers are standalone servers, while Phusion Passenger for Nginx and Phusion Passenger for Apache introduce additional layers (namely Nginx and Apache), making the comparison unfair.
+   - Be sure to start Phusion Passenger Standalone with the `builtin` engine. This is the default.
+ * Configure Phusion Passenger to [use a static number of processes](#minimizing_process_spawning). The number of processes should be a multiple of the number of CPU cores.
+ * Ensure that your app outputs HTTP caching headers, so that Phusion Passenger's [turbocaching](#turbocaching) can kick in.