eventmachine 0.7.2 → 0.8.0

data/EPOLL

@@ -0,0 +1,142 @@
+EventMachine now supports epoll, bringing large increases in performance and scalability to Ruby programs.
+
+Epoll(7) is a alternative mechanism for multiplexed I/O that is available in Linux 2.6 kernels.
+It features significantly greater performance than the standard select(2) mechanism, when used in
+applications that require very large numbers of open I/O descriptors.
+
+EventMachine has always used select(2) because its behavior is well standardized and broadly supported.
+But select becomes unreasonably slow when a program has a
+very large number of file descriptors or sockets. Ruby's version of select hardcodes a limit
+of 1024 descriptors per process, but heavily loaded processes will start to show performance
+degradation even after only a few hundred descriptors are in use.
+
+Epoll is an extended version of the poll(2) call, and it solves the problems with select. Programs
+based on epoll can easily scale past Ruby's 1024-descriptor limit, potentially to tens of thousands
+of connectors, with no significant impact on performance.
+
+(Another alternative which is very similar to epoll in principle is kqueue, supplied on BSD and its
+variants.)
+
+
+
+This note shows you how to use epoll in your programs.
+
+=== Compiling EventMachine to use epoll.
+
+You don't have to do anything to get epoll support in EventMachine.
+When you compile EventMachine on a platform that supports epoll, EM will
+automatically generate a Makefile that includes epoll. (At this writing, this will only work
+on Linux 2.6 kernels.) If you compile EM on a platform without epoll, then epoll support will
+be omitted from the Makefile, and EM will work just as it always has.
+
+=== Using epoll in your programs.
+
+First, you need to tell EventMachine to use epoll instead of select (but see below, as this requirement
+will be removed in a future EventMachine version). Second, you need to prepare your program to use
+more than 1024 descriptors, an operation that generally requires superuser privileges. Third, you will probably
+want your process to drop the superuser privileges after you increase your process's descriptor limit.
+
+=== Using EventMachine#epoll
+
+Call the method EventMachine#epoll anytime before you call EventMachine#run, and your program will
+automatically use epoll, if available. It's safe to call EventMachine#epoll on any platform because
+
+it compiles to a no-op on platforms that don't support epoll.
+
+  require 'rubygems'
+  require 'eventmachine'
+
+  EM.epoll
+  EM.run {
+    ...
+  }
+
+
+EventMachine#epoll was included in this initial release only to avoid changing the behavior of existing
+programs. However, it's expected that a future release of EM will convert EventMachine#epoll to a no-op,
+and run epoll by default on platforms that support it.
+
+=== Using EventMachine#set_descriptor_table_size
+
+In Linux (as in every Unix-like platform), every process has a internal table that determines the maximum
+number of file and socket descriptors you may have open at any given time. The size of this table is
+generally fixed at 1024, although it may be increased within certain system-defined hard and soft limits.
+
+If you want your EventMachine program to support more than 1024 total descriptors, you must use 
+EventMachine#set_descriptor_table_size, as follows:
+
+  require 'rubygems'
+  require 'eventmachine'
+
+  new_size = EM.set_descriptor_table_size( 60000 )
+  $>.puts "New descriptor-table size is #{new_size}"
+
+  EM.run {
+    ...
+  }
+
+If successful, this example will increase the maximum number of descriptors that epoll can use to 60,000.
+Call EventMachine#set_descriptor_table_size without an argument at any time to find out the current
+size of the descriptor table.
+
+Using EventMachine#set_descriptor_table_size ONLY affects the number of descriptors that can be used
+by epoll. It has no useful effect on platforms that don't support epoll, and it does NOT increase the
+number of descriptors that Ruby's own I/O functions can use.
+
+#set_descriptor_table_size can fail if your process is not running as superuser, or if you try to set a
+table size that exceeds the hard limits imposed by your system. In the latter case, try a smaller number.
+
+
+=== Using EventMachine#set_effective_user
+
+In general, you must run your program with elevated or superuser privileges if you want to increase
+your descriptor-table size beyond 1024 descriptors. This is easy enough to verify. Try running the
+sample program given above, that increases the descriptor limit to 60,000. You will probably find that
+the table size will not be increased if you don't run your program as root or with elevated privileges.
+
+But of course network servers, especially long-running ones, should not run with elevated privileges.
+You will want to drop superuser privileges as soon as possible after initialization. To do this,
+use EventMachine#set_effective_user:
+
+  require 'rubygems'
+  require 'eventmachine'
+
+  # (Here, program is running as superuser)
+
+  EM.set_descriptor_table_size( 60000 )
+  EM.set_effective_user( "nobody" )
+  # (Here, program is running as nobody)
+
+  EM.run {
+    ...
+  }
+
+Of course, you will need to replace "nobody" in the example with the name of an unprivileged user
+that is valid on your system. What if you want to drop privileges after opening a server socket
+on a privileged (low-numbered) port? Easy, just call #set_effective_user after opening your sockets:
+
+  require 'rubygems'
+  require 'eventmachine'
+
+  # (Here, program is running as superuser)
+
+  EM.set_descriptor_table_size( 60000 )
+
+  EM.run {
+    EM.start_server( "0.0.0.0", 80, MyHttpServer )
+    EM.start_server( "0.0.0.0", 443, MyEncryptedHttpServer )
+
+    EM.set_effective_user( "nobody" )
+    # (Here, program is running as nobody)
+
+    ...
+  }
+
+
+Because EventMachine#set_effective_user is used to enforce security
+requirements, it has no nonfatal errors. If you try to set a nonexistent or invalid effective user,
+#set_effective_user will abort your program, rather than continue to run with elevated privileges.
+
+EventMachine#set_effective_user is a silent no-op on platforms that don't support it, such as Windows.
+
+

data/RELEASE_NOTES

@@ -1,7 +1,20 @@
-$Id: RELEASE_NOTES 286 2006-11-28 03:18:18Z blackhedd $
+$Id: RELEASE_NOTES 388 2007-06-25 02:06:05Z blackhedd $
 
 RUBY/EventMachine RELEASE NOTES
 
+--------------------------------------------------
+Version: 0.9.0, released xxXXX07
+Added Erlang-like distributed-computing features
+
+--------------------------------------------------
+Version: 0.8.0, released 23Jun07
+Added an epoll implementation for Linux 2.6 kernels.
+Added evented #popen.
+
+--------------------------------------------------
+Version: 0.7.3, released 22May07
+Added a large variety of small features. See the ChangeLog.
+
 --------------------------------------------------
 Version: 0.7.1, released xxNov06
 Added protocol handlers for line-oriented protocols.

data/ext/cmain.cpp

@@ -1,6 +1,6 @@
 /*****************************************************************************
 
-$Id: cmain.cpp 318 2007-05-22 22:06:24Z blackhedd $
+$Id: cmain.cpp 377 2007-06-13 01:05:56Z blackhedd $
 
 File:     cmain.cpp
 Date:     06Apr06
@@ -21,6 +21,7 @@ See the file COPYING for complete licensing information.
 
 
 static EventMachine_t *EventMachine;
+static int bUseEpoll = 0;
 
 
 /***********************
@@ -35,6 +36,8 @@ extern "C" void evma_initialize_library (void(*cb)(const char*, int, const char*
 	if (EventMachine)
 		throw std::runtime_error ("already initialized");
 	EventMachine = new EventMachine_t (cb);
+	if (bUseEpoll)
+		EventMachine->_UseEpoll();
 }
 
 
@@ -210,6 +213,22 @@ extern "C" int evma_get_peername (const char *binding, struct sockaddr *sa)
 		return 0;
 }
 
+/***********************
+evma_get_subprocess_pid
+***********************/
+
+extern "C" int evma_get_subprocess_pid (const char *binding, pid_t *pid)
+{
+	if (!EventMachine)
+		throw std::runtime_error ("not initialized");
+	EventableDescriptor *ed = dynamic_cast <EventableDescriptor*> (Bindable_t::GetObject (binding));
+	if (ed) {
+		return ed->GetSubprocessPid (pid) ? 1 : 0;
+	}
+	else
+		return 0;
+}
+
 
 /*********************
 evma_signal_loopbreak
@@ -290,3 +309,53 @@ extern "C" void evma_setuid_string (const char *username)
     EventMachine_t::SetuidString (username);
 }
 
+
+/**********
+evma_popen
+**********/
+
+extern "C" const char *evma_popen (char * const*cmd_strings)
+{
+	if (!EventMachine)
+		throw std::runtime_error ("not initialized");
+	return EventMachine->Socketpair (cmd_strings);
+}
+
+
+/***************************
+evma_get_outbound_data_size
+***************************/
+
+extern "C" int evma_get_outbound_data_size (const char *binding)
+{
+	if (!EventMachine)
+		throw std::runtime_error ("not initialized");
+	EventableDescriptor *ed = dynamic_cast <EventableDescriptor*> (Bindable_t::GetObject (binding));
+	return ed ? ed->GetOutboundDataSize() : 0;
+}
+
+
+/***********
+evma__epoll
+***********/
+
+extern "C" void evma__epoll()
+{
+	bUseEpoll = 1;
+}
+
+
+/**********************
+evma_set_rlimit_nofile
+**********************/
+
+extern "C" int evma_set_rlimit_nofile (int nofiles)
+{
+	return EventMachine_t::SetRlimitNofile (nofiles);
+}
+
+
+
+
+
+

data/ext/ed.cpp

@@ -1,6 +1,6 @@
 /*****************************************************************************
 
-$Id: ed.cpp 318 2007-05-22 22:06:24Z blackhedd $
+$Id: ed.cpp 400 2007-07-11 12:10:33Z blackhedd $
 
 File:     ed.cpp
 Date:     06Apr06
@@ -43,13 +43,15 @@ bool SetSocketNonblocking (SOCKET sd)
 EventableDescriptor::EventableDescriptor
 ****************************************/
 
-EventableDescriptor::EventableDescriptor (int sd):
+EventableDescriptor::EventableDescriptor (int sd, EventMachine_t *em):
+	bCloseNow (false),
+	bCloseAfterWriting (false),
+	MySocket (sd),
 	EventCallback (NULL),
 	LastRead (0),
 	LastWritten (0),
-	MySocket (sd),
-	bCloseNow (false),
-	bCloseAfterWriting (false)
+	bCallbackUnbind (true),
+	MyEventMachine (em)
 {
 	/* There are three ways to close a socket, all of which should
 	 * automatically signal to the event machine that this object
@@ -74,7 +76,13 @@ EventableDescriptor::EventableDescriptor (int sd):
 
 	if (sd == INVALID_SOCKET)
 		throw std::runtime_error ("bad eventable descriptor");
+	if (MyEventMachine == NULL)
+		throw std::runtime_error ("bad em in eventable descriptor");
 	CreatedAt = gCurrentLoopTime;
+
+	#ifdef HAVE_EPOLL
+	EpollEvent.data.ptr = this;
+	#endif
 }
 
 
@@ -84,8 +92,8 @@ EventableDescriptor::~EventableDescriptor
 
 EventableDescriptor::~EventableDescriptor()
 {
-	if (EventCallback)
-		(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_UNBOUND, NULL, 0);
+	if (EventCallback && bCallbackUnbind)
+		(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_UNBOUND, NULL, 0);
 	Close();
 }
 
@@ -140,6 +148,7 @@ EventableDescriptor::ScheduleClose
 
 void EventableDescriptor::ScheduleClose (bool after_writing)
 {
+	// KEEP THIS SYNCHRONIZED WITH ::IsCloseScheduled.
 	if (after_writing)
 		bCloseAfterWriting = true;
 	else
@@ -147,12 +156,23 @@ void EventableDescriptor::ScheduleClose (bool after_writing)
 }
 
 
+/*************************************
+EventableDescriptor::IsCloseScheduled
+*************************************/
+
+bool EventableDescriptor::IsCloseScheduled()
+{
+	// KEEP THIS SYNCHRONIZED WITH ::ScheduleClose.
+	return (bCloseNow || bCloseAfterWriting);
+}
+
+
 /******************************************
 ConnectionDescriptor::ConnectionDescriptor
 ******************************************/
 
-ConnectionDescriptor::ConnectionDescriptor (int sd):
-	EventableDescriptor (sd),
+ConnectionDescriptor::ConnectionDescriptor (int sd, EventMachine_t *em):
+	EventableDescriptor (sd, em),
 	bConnectPending (false),
 	bReadAttemptedAfterClose (false),
 	OutboundDataSize (0),
@@ -163,6 +183,9 @@ ConnectionDescriptor::ConnectionDescriptor (int sd):
 	LastIo (gCurrentLoopTime),
 	InactivityTimeout (0)
 {
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = EPOLLOUT;
+	#endif
 }
 
 
@@ -190,12 +213,19 @@ STATIC: ConnectionDescriptor::SendDataToConnection
 int ConnectionDescriptor::SendDataToConnection (const char *binding, const char *data, int data_length)
 {
 	// TODO: This is something of a hack, or at least it's a static method of the wrong class.
+	// TODO: Poor polymorphism here. We should be calling one virtual method
+	// instead of hacking out the runtime information of the target object.
 	ConnectionDescriptor *cd = dynamic_cast <ConnectionDescriptor*> (Bindable_t::GetObject (binding));
 	if (cd)
 		return cd->SendOutboundData (data, data_length);
 	DatagramDescriptor *ds = dynamic_cast <DatagramDescriptor*> (Bindable_t::GetObject (binding));
 	if (ds)
 		return ds->SendOutboundData (data, data_length);
+	#ifdef OS_UNIX
+	PipeDescriptor *ps = dynamic_cast <PipeDescriptor*> (Bindable_t::GetObject (binding));
+	if (ps)
+		return ps->SendOutboundData (data, data_length);
+	#endif
 	return -1;
 }
 
@@ -257,7 +287,8 @@ int ConnectionDescriptor::_SendRawOutboundData (const char *data, int length)
 	// and not the whole process), and no coalescing of small pages.
 	// (Well, not so bad, small pages are coalesced in ::Write)
 
-	if (bCloseNow || bCloseAfterWriting)
+	if (IsCloseScheduled())
+	//if (bCloseNow || bCloseAfterWriting)
 		return 0;
 
 	if (!data && (length > 0))
@@ -269,6 +300,11 @@ int ConnectionDescriptor::_SendRawOutboundData (const char *data, int length)
 	buffer [length] = 0;
 	OutboundPages.push_back (OutboundPage (buffer, length));
 	OutboundDataSize += length;
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = (EPOLLIN | EPOLLOUT);
+	assert (MyEventMachine);
+	MyEventMachine->Modify (this);
+	#endif
 	return length;
 }
 
@@ -399,7 +435,8 @@ void ConnectionDescriptor::Read()
 	if (total_bytes_read == 0) {
 		// If we read no data on a socket that selected readable,
 		// it generally means the other end closed the connection gracefully.
-		bCloseNow = true;
+		ScheduleClose (false);
+		//bCloseNow = true;
 	}
 
 }
@@ -421,20 +458,20 @@ void ConnectionDescriptor::_DispatchInboundData (const char *buffer, int size)
 		while ((s = SslBox->GetPlaintext (B, sizeof(B) - 1)) > 0) {
 			B [s] = 0;
 			if (EventCallback)
-				(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_READ, B, s);
+				(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_READ, B, s);
 		}
 		// INCOMPLETE, s may indicate an SSL error that would force the connection down.
 		_DispatchCiphertext();
 	}
 	else {
 			if (EventCallback)
-				(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_READ, buffer, size);
+				(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_READ, buffer, size);
 	}
 	#endif
 
 	#ifdef WITHOUT_SSL
 	if (EventCallback)
-		(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_READ, buffer, size);
+		(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_READ, buffer, size);
 	#endif
 }
 
@@ -465,18 +502,23 @@ void ConnectionDescriptor::Write()
 		socklen_t len;
 		len = sizeof(error);
 		#ifdef OS_UNIX
-		int o = getsockopt (MySocket, SOL_SOCKET, SO_ERROR, &error, &len);
+		int o = getsockopt (GetSocket(), SOL_SOCKET, SO_ERROR, &error, &len);
 		#endif
 		#ifdef OS_WIN32
-		int o = getsockopt (MySocket, SOL_SOCKET, SO_ERROR, (char*)&error, &len);
+		int o = getsockopt (GetSocket(), SOL_SOCKET, SO_ERROR, (char*)&error, &len);
 		#endif
 		if ((o == 0) && (error == 0)) {
 			if (EventCallback)
-				(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_COMPLETED, "", 0);
+				(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_COMPLETED, "", 0);
 			bConnectPending = false;
+			#ifdef HAVE_EPOLL
+			// The callback may have scheduled outbound data.
+			EpollEvent.events = EPOLLIN | (SelectForWrite() ? EPOLLOUT : 0);
+			#endif
 		}
 		else
-			bCloseNow = true;
+			ScheduleClose (false);
+			//bCloseNow = true;
 	}
 	else {
 		_WriteOutboundData();
@@ -540,6 +582,12 @@ void ConnectionDescriptor::_WriteOutboundData()
 			buffer [len] = 0;
 			OutboundPages.push_front (OutboundPage (buffer, len));
 		}
+
+		#ifdef HAVE_EPOLL
+		EpollEvent.events = (EPOLLIN | (SelectForWrite() ? EPOLLOUT : 0));
+		assert (MyEventMachine);
+		MyEventMachine->Modify (this);
+		#endif
 	}
 	else {
 		#ifdef OS_UNIX
@@ -653,30 +701,71 @@ void ConnectionDescriptor::Heartbeat()
 
 	if (bConnectPending) {
 		if ((gCurrentLoopTime - CreatedAt) >= PendingConnectTimeout)
-			bCloseNow = true;
+			ScheduleClose (false);
+			//bCloseNow = true;
 	}
 	else {
 		if (InactivityTimeout && ((gCurrentLoopTime - LastIo) >= InactivityTimeout))
-			bCloseNow = true;
+			ScheduleClose (false);
+			//bCloseNow = true;
 	}
 }
 
 
+/****************************************
+LoopbreakDescriptor::LoopbreakDescriptor
+****************************************/
+
+LoopbreakDescriptor::LoopbreakDescriptor (int sd, EventMachine_t *parent_em):
+	EventableDescriptor (sd, parent_em)
+{
+	/* This is really bad and ugly. Change someday if possible.
+	 * We have to know about an event-machine (probably the one that owns us),
+	 * so we can pass newly-created connections to it.
+	 */
+
+	bCallbackUnbind = false;
+
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = EPOLLIN;
+	#endif
+}
+
+
+
+
+/*************************
+LoopbreakDescriptor::Read
+*************************/
+
+void LoopbreakDescriptor::Read()
+{
+	// TODO, refactor, this code is probably in the wrong place.
+	assert (MyEventMachine);
+	MyEventMachine->_ReadLoopBreaker();
+}
+
+
+/**************************
+LoopbreakDescriptor::Write
+**************************/
+
+void LoopbreakDescriptor::Write()
+{
+  // Why are we here?
+  throw std::runtime_error ("bad code path in loopbreak");
+}
+
 /**************************************
 AcceptorDescriptor::AcceptorDescriptor
 **************************************/
 
-AcceptorDescriptor::AcceptorDescriptor (EventMachine_t *parent_em, int sd):
-  EventableDescriptor (sd),
-  MyEventMachine (parent_em)
+AcceptorDescriptor::AcceptorDescriptor (int sd, EventMachine_t *parent_em):
+	EventableDescriptor (sd, parent_em)
 {
-  /* This is really bad and ugly. Change someday if possible.
-   * We have to know about an event-machine (probably the one that owns us),
-   * so we can pass newly-created connections to it.
-   */
-
-  if (!MyEventMachine)
-    throw std::runtime_error ("bad event-machine passed to acceptor");
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = EPOLLIN;
+	#endif
 }
 
 
@@ -739,13 +828,16 @@ void AcceptorDescriptor::Read()
 		setsockopt (sd, IPPROTO_TCP, TCP_NODELAY, (char*) &one, sizeof(one));
 
 
-		ConnectionDescriptor *cd = new ConnectionDescriptor (sd);
+		ConnectionDescriptor *cd = new ConnectionDescriptor (sd, MyEventMachine);
 		if (!cd)
 			throw std::runtime_error ("no newly accepted connection");
 		cd->SetServerMode();
 		if (EventCallback) {
-			(*EventCallback) (GetBinding().c_str(), EventMachine_t::CONNECTION_ACCEPTED, cd->GetBinding().c_str(), cd->GetBinding().size());
+			(*EventCallback) (GetBinding().c_str(), EM_CONNECTION_ACCEPTED, cd->GetBinding().c_str(), cd->GetBinding().size());
 		}
+		#ifdef HAVE_EPOLL
+		cd->GetEpollEvent()->events = EPOLLIN | (cd->SelectForWrite() ? EPOLLOUT : 0);
+		#endif
 		assert (MyEventMachine);
 		MyEventMachine->Add (cd);
 	}
@@ -780,13 +872,17 @@ void AcceptorDescriptor::Heartbeat()
 DatagramDescriptor::DatagramDescriptor
 **************************************/
 
-DatagramDescriptor::DatagramDescriptor (int sd):
-	EventableDescriptor (sd),
+DatagramDescriptor::DatagramDescriptor (int sd, EventMachine_t *parent_em):
+	EventableDescriptor (sd, parent_em),
 	OutboundDataSize (0),
 	LastIo (gCurrentLoopTime),
 	InactivityTimeout (0)
 {
 	memset (&ReturnAddress, 0, sizeof(ReturnAddress));
+
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = EPOLLIN;
+	#endif
 }
 
 
@@ -810,8 +906,9 @@ void DatagramDescriptor::Heartbeat()
 {
 	// Close it if its inactivity timer has expired.
 
-  if (InactivityTimeout && ((gCurrentLoopTime - LastIo) >= InactivityTimeout))
-		bCloseNow = true;
+	if (InactivityTimeout && ((gCurrentLoopTime - LastIo) >= InactivityTimeout))
+		ScheduleClose (false);
+		//bCloseNow = true;
 }
 
 
@@ -869,7 +966,7 @@ void DatagramDescriptor::Read()
 			memcpy (&ReturnAddress, &sin, slen);
 
 			if (EventCallback)
-				(*EventCallback)(GetBinding().c_str(), EventMachine_t::CONNECTION_READ, readbuffer, r);
+				(*EventCallback)(GetBinding().c_str(), EM_CONNECTION_READ, readbuffer, r);
 
 		}
 		else {
@@ -932,6 +1029,12 @@ void DatagramDescriptor::Write()
 			}
 		}
 	}
+
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = (EPOLLIN | (SelectForWrite() ? EPOLLOUT : 0));
+	assert (MyEventMachine);
+	MyEventMachine->Modify (this);
+	#endif
 }
 
 
@@ -954,7 +1057,8 @@ int DatagramDescriptor::SendOutboundData (const char *data, int length)
 	// This is an exact clone of ConnectionDescriptor::SendOutboundData.
 	// That means it needs to move to a common ancestor.
 
-	if (bCloseNow || bCloseAfterWriting)
+	if (IsCloseScheduled())
+	//if (bCloseNow || bCloseAfterWriting)
 		return 0;
 
 	if (!data && (length > 0))
@@ -966,6 +1070,11 @@ int DatagramDescriptor::SendOutboundData (const char *data, int length)
 	buffer [length] = 0;
 	OutboundPages.push_back (OutboundPage (buffer, length, ReturnAddress));
 	OutboundDataSize += length;
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = (EPOLLIN | EPOLLOUT);
+	assert (MyEventMachine);
+	MyEventMachine->Modify (this);
+	#endif
 	return length;
 }
 
@@ -980,7 +1089,8 @@ int DatagramDescriptor::SendOutboundDatagram (const char *data, int length, cons
 	// That means it needs to move to a common ancestor.
 	// TODO: Refactor this so there's no overlap with SendOutboundData.
 
-	if (bCloseNow || bCloseAfterWriting)
+	if (IsCloseScheduled())
+	//if (bCloseNow || bCloseAfterWriting)
 		return 0;
 
 	if (!address || !*address || !port)
@@ -1014,6 +1124,11 @@ int DatagramDescriptor::SendOutboundDatagram (const char *data, int length, cons
 	buffer [length] = 0;
 	OutboundPages.push_back (OutboundPage (buffer, length, pin));
 	OutboundDataSize += length;
+	#ifdef HAVE_EPOLL
+	EpollEvent.events = (EPOLLIN | EPOLLOUT);
+	assert (MyEventMachine);
+	MyEventMachine->Modify (this);
+	#endif
 	return length;
 }
 
@@ -1055,14 +1170,14 @@ ConnectionDescriptor::GetCommInactivityTimeout
 
 int ConnectionDescriptor::GetCommInactivityTimeout (int *value)
 {
-  if (value) {
-    *value = InactivityTimeout;
-    return 1;
-  }
-  else {
-    // TODO, extended logging, got bad parameter.
-    return 0;  
-  }
+	if (value) {
+		*value = InactivityTimeout;
+		return 1;
+	}
+	else {
+		// TODO, extended logging, got bad parameter.
+		return 0;  
+	}
 }
 
 
@@ -1072,25 +1187,25 @@ ConnectionDescriptor::SetCommInactivityTimeout
 
 int ConnectionDescriptor::SetCommInactivityTimeout (int *value)
 {
-  int out = 0;
-
-  if (value) {
-    if ((*value==0) || (*value >= 2)) {
-      // Replace the value and send the old one back to the caller.
-      int v = *value;
-      *value = InactivityTimeout;
-      InactivityTimeout = v;
-      out = 1;
-    }
-    else {
-      // TODO, extended logging, got bad value.
-    }
-  }
-  else {
-    // TODO, extended logging, got bad parameter.
-  }
+	int out = 0;
+
+	if (value) {
+		if ((*value==0) || (*value >= 2)) {
+			// Replace the value and send the old one back to the caller.
+			int v = *value;
+			*value = InactivityTimeout;
+			InactivityTimeout = v;
+			out = 1;
+		}
+		else {
+			// TODO, extended logging, got bad value.
+		}
+	}
+	else {
+		// TODO, extended logging, got bad parameter.
+	}
 
-  return out;
+	return out;
 }
 
 /*******************************