ftpd 0.4.0 → 0.5.0

data/Changelog.md

@@ -1,3 +1,36 @@
+### 0.5.0
+
+Bug fixes
+
+* Replies are sent with the correct line ending ("\r\n" instead of
+  "\n")
+* Do not hang on out-of-band commands.
+* When data connection disconnects, send "426 Connection closed"
+  response instead of ending the session.
+
+Enhancements
+
+* Now unconditionally compliant
+* Configurable session timeout (see {Ftpd::FtpServer#session_timeout}).
+  Defaults to 5 minutes.
+* Disable Nagle algorithm on control connection to decrease latency.
+  This makes the tests run much faster.
+* Support STAT (server status).
+* Example has --timeout option for session idle timeout.
+* Write log to Logger (see {Ftpd::FtpServer#log}).
+* Disallow active-mode connections to privileged ports (configurable).
+  See RFC 2577 section 3.
+* Added benchmarks.
+* Support telnet sequences.
+
+API Changes
+
+* Added {Ftpd::FtpServer#server_name}
+* Added {Ftpd::FtpServer#server_version}
+* Removed #debug and #debug_path from Ftpd::FtpServer.  They have been
+  replaced with #log
+# Added {Ftpd::FtpServer#allow_low_data_ports}
+
 ### 0.4.0
 
 Enhancements

data/Gemfile

@@ -5,6 +5,7 @@ gem 'memoizer', '~> 1.0.1'
 group :development do
   gem 'cucumber'
   gem 'double-bag-ftps', :git => 'git@github.com:wconrad/double-bag-ftps.git'
+  gem 'fuubar-cucumber'
   gem 'jeweler'
   gem 'rake'
   gem 'redcarpet'

data/Gemfile.lock

@@ -14,6 +14,9 @@ GEM
       gherkin (~> 2.11.0)
       json (>= 1.4.6)
     diff-lcs (1.2.1)
+    fuubar-cucumber (0.0.18)
+      cucumber (~> 1.2.0)
+      ruby-progressbar (~> 1.0.0)
     gherkin (2.11.6)
       json (>= 1.7.6)
     git (1.2.5)
@@ -36,6 +39,7 @@ GEM
     rspec-expectations (2.13.0)
       diff-lcs (>= 1.1.3, < 2.0)
     rspec-mocks (2.13.0)
+    ruby-progressbar (1.0.2)
     timecop (0.5.9.2)
     yard (0.8.5.2)
 
@@ -45,6 +49,7 @@ PLATFORMS
 DEPENDENCIES
   cucumber
   double-bag-ftps!
+  fuubar-cucumber
   jeweler
   memoizer (~> 1.0.1)
   rake

data/README.md

@@ -1,8 +1,8 @@
 # FTPD
 
 ftpd is a pure Ruby FTP server library.  It supports implicit and
-explicit TLS, passive and active mode, and most of the commands
-specified in RFC 969.  It an be used as part of a test fixture or
+explicit TLS, passive and active mode, and is unconditionally
+complaint per RFC-1123.  It an be used as part of a test fixture or
 embedded in a program.
 
 ## A note about this README
@@ -10,7 +10,7 @@ embedded in a program.
 This readme, and the other files, contains Yardoc markup, especially
 for links to the API docs; those links don't display properly on
 github.  You'll find a properly rendered version
-{http://rubydoc.info/gems/ftpd on rubydoc.info}
+[on rubydoc.info](http://rubydoc.info/gems/ftpd)
 
 ## HELLO WORLD
 
@@ -83,14 +83,17 @@ file system.
 Here are the methods a file system may expose:
 
 * {Ftpd::DiskFileSystem::Accessors#accessible? accessible?}
-* {Ftpd::DiskFileSystem::Accessors#exists? exists?}
 * {Ftpd::DiskFileSystem::Accessors#directory? directory?}
-* {Ftpd::DiskFileSystem::Write#write write}
-* {Ftpd::DiskFileSystem::Mkdir#mkdir mkdir}
-* {Ftpd::DiskFileSystem::Rmdir#rmdir rmdir}
-* {Ftpd::DiskFileSystem::List#file_info file_info}
+* {Ftpd::DiskFileSystem::Accessors#exists? exists?}
+* {Ftpd::DiskFileSystem::Append#append append}
+* {Ftpd::DiskFileSystem::Delete#delete delete}
 * {Ftpd::DiskFileSystem::List#dir dir}
+* {Ftpd::DiskFileSystem::List#file_info file_info}
+* {Ftpd::DiskFileSystem::Mkdir#mkdir mkdir}
+* {Ftpd::DiskFileSystem::Read#read read}
 * {Ftpd::DiskFileSystem::Rename#rename rename}
+* {Ftpd::DiskFileSystem::Rmdir#rmdir rmdir}
+* {Ftpd::DiskFileSystem::Write#write write}
 
 ### DiskFileSystem
 
@@ -113,6 +116,7 @@ file system, then use {Ftpd::ReadOnlyDiskFileSystem} instead.
 The DiskFileSystem is composed out of modules:
 
 * {Ftpd::DiskFileSystem::Base Base} - You will need this
+* {Ftpd::DiskFileSystem::Append Append} - File appending
 * {Ftpd::DiskFileSystem::Delete Delete} - File deletion
 * {Ftpd::DiskFileSystem::List List} - Directory listing
 * {Ftpd::DiskFileSystem::Mkdir Mkdir} - Directory creation
@@ -148,7 +152,7 @@ By default, the LIST command uses Unix "ls -l" formatting:
     -rw-r--r-- 1 user     group        1234 Mar  3 08:38 foo
 
 To switch to
-{http://cr.yp.to/ftp/list/eplf.html Easily Parsed LIST format (EPLF)}
+[Easily Parsed LIST format (EPLF)](http://cr.yp.to/ftp/list/eplf.html)
 format:
 
     ftp_server.list_formatter = Ftpd::ListFormat::Eplf
@@ -181,27 +185,15 @@ Debug output can also be enabled by setting the environment variable
 FTPD_DEBUG to a non-zero value.  This is a convenient way to get debug
 output without having to change any code.
 
-## LIMITATIONS
-
-Ftpd is not fully RFC compliant.  It does most of RFC969, and enough
-TLS to get by.  {file:doc/rfc.md Here} is a list of RFCs, indicating
-how much of each Ftpd complies with.
-
-RFC does not meet the following
-[RFC-1123](http://tools.ietf.org/rfc/rfc1123.txt) "MUST" requrements.
-If FTPD met these requirements, but did not meet the "SHOULD"
-requirements, it would be "conditionally compliant":
+## STANDARDS COMPLIANCE
 
-* Server-FTP handle Telnet options
-* Support command STAT
+Ftpd is fully unconditionally compliant per [RFC-1123 Requirements for
+Internet Hosts](http://tools.ietf.org/rfc/rfc1123.txt).
 
-RFC does not meet the following
-[RFC-1123](http://tools.ietf.org/rfc/rfc1123.txt)
-"SHOULD" requrements.  If FTPD met both the "MUST" and the "SHOULD"
-requirements, it would be "unconditionally compliant":
+Ftpd implements enough of [RFC-4217 Securing FTP with
+TLS](http://tools.ietf.org/rfc/rfc4217.txt) to get by.
 
-* Idle timeout in server-FTP
-* Configurable idle timeout
+See [RFC Compliance](doc/rfc-compliance.md) for details
 
 ## RUBY COMPATABILITY
 
@@ -231,6 +223,12 @@ or just:
 
     $ rake
 
+To force features to write the server log to stdout:
+
+    $ FTPD_DEBUG=1 rake test:features
+
+### Example
+
 To run the stand-alone example:
 
     $ examples/example.rb
@@ -256,6 +254,7 @@ and granted permission to donate it to the community.
 
 ## See also
 
-* {file:Changelog.md}
-* {file:doc/rfc-compliance.md RFC compliance}
-* {file:doc/references.md}
+* [Changelog](Changelog.md)
+* [RFC compliance](doc/rfc-compliance.md)
+* [References](doc/references.md)
+* [Benchmarks](doc/benchmarks.md)

data/VERSION

@@ -1 +1 @@
-0.4.0
+0.5.0

data/config/cucumber.yml

@@ -0,0 +1,2 @@
+---
+default: --format Cucumber::Formatter::Fuubar

data/doc/benchmarks.md

@@ -0,0 +1,92 @@
+# Benchmarks
+
+Benchmarks using pyftpd's [benchmark
+script](http://pyftpdlib.googlecode.com/svn/trunk/test/bench.py) and
+[procedures](http://code.google.com/p/pyftpdlib/wiki/Benchmarks).
+
+## Results
+
+### ftpd 0.4.0
+
+    (starting with 13.6M of RSS memory being used)
+    STOR (client -> server)                              845.55 MB/sec  13.6M RSS
+    RETR (server -> client)                             1234.80 MB/sec  859.5M RSS
+    200 concurrent clients (connect, login)                0.14 secs    863.0M RSS
+    STOR (1 file with 200 idle clients)                  848.56 MB/sec  863.0M RSS
+    RETR (1 file with 200 idle clients)                 1227.46 MB/sec  866.4M RSS
+    200 concurrent clients (RETR 10.0M file)               5.36 secs    2.0G RSS
+    200 concurrent clients (STOR 10.0M file)               4.65 secs    2.0G RSS
+    200 concurrent clients (QUIT)                          0.01 secs
+
+### pyftpdlib 1.0.1
+
+    (starting with 8.9M of RSS memory being used)
+    STOR (client -> server)                               81.73 MB/sec  8.9M RSS
+    RETR (server -> client)                             1001.05 MB/sec  8.9M RSS
+    200 concurrent clients (connect, login)                2.58 secs    9.6M RSS
+    STOR (1 file with 200 idle clients)                  108.46 MB/sec  9.6M RSS
+    RETR (1 file with 200 idle clients)                 1134.18 MB/sec  9.6M RSS
+    200 concurrent clients (RETR 10.0M file)               2.32 secs    10.3M RSS
+    200 concurrent clients (STOR 10.0M file)               4.12 secs    10.4M RSS
+    200 concurrent clients (QUIT)                          0.02 secs
+
+### proftpd 1.3.4a-3
+
+    (starting with 2.3M of RSS memory being used)
+    STOR (client -> server)                               93.06 MB/sec  6.6M RSS
+    RETR (server -> client)                             1267.63 MB/sec  6.6M RSS
+    200 concurrent clients (connect, login)               14.21 secs    868.0M RSS
+    STOR (1 file with 200 idle clients)                   76.04 MB/sec  872.3M RSS
+    RETR (1 file with 200 idle clients)                 1289.75 MB/sec  872.3M RSS
+    200 concurrent clients (RETR 10.0M file)               2.04 secs    868.0M RSS
+    200 concurrent clients (STOR 10.0M file)               4.51 secs    868.2M RSS
+    200 concurrent clients (QUIT)                          0.00 secs
+
+### Discussion
+
+ftpd's STOR results seen anomalous.  I suspect that proftpd and
+pyftpdlib aren't getting a fair shake here.  proftpd and pyftpdlib are
+serving my home directory, whereas ftpd is serving a temporary
+directory, but I don't know what difference that could make.
+
+Ftpd is a memory hog.  During a STOR or RETR, it loads the entire
+contents of a file into memory.  This limits the number of concurrent
+file transfers it can handle.  The pyftpd team uses -n 300 (300
+concurrent connections) when benchmarking, but Ftpd can't handle that
+many at the moment.
+
+Ftpd's fast time on the login test, compared to proftpd and pyftpdlib,
+is a result of it not doing PAM authentication.  It is an unfair
+comparison and should be disregarded.
+
+pyftpd's memory footprint is impressive.
+
+ftpd is less performant with many concurrent connections than either
+proftpd or pyftpdlib.
+
+## Setup
+
+### Machine
+
+* Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz (4 cores)
+* Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
+* ruby 2.0.0p0 (2013-02-24 revision 39474) [i686-linux]
+* Linux 3.0.0-1-686-pae #1 SMP Sat Aug 27 16:41:03 UTC 2011 i686 GNU/Linux
+
+### Benchmark command
+
+    $ python bench.py  -u <USER> -p <PASS> -H localhost -p <PORT> -b all -n 200 -k <PID>
+
+### Proftpd
+
+/etc/proftpd/proftpd.conf:
+
+    MaxInstances        2000
+
+### Ftpd
+
+    $ bundle exec examples/example.rb -p 2222 -U bench -P bench
+
+### Pyftpd
+
+    pyftpdlib-1.0.0/demo$ sudo python unix_daemon.py 2>/dev/null

data/doc/references.md

@@ -9,15 +9,38 @@ Copyright (c) 2008 James Healy_
 There are a range of RFCs that together specify the FTP protocol. In
 chronological order, the more useful ones are:
 
-* {http://tools.ietf.org/rfc/rfc959.txt RFC-959} - File Transfer Protocol
-* {http://tools.ietf.org/rfc/rfc1123.txt RFC-1123} - Requirements for Internet Hosts
-* {http://tools.ietf.org/rfc/rfc2228.txt RFC-2228} - FTP Security Extensions
-* {http://tools.ietf.org/rfc/rfc2389.txt RFC-2389} - Feature negotiation mechanism for the File Transfer Protocol
-* {http://tools.ietf.org/rfc/rfc2428.txt RFC-2428} - FTP Extensions for IPv6 and NATs
-* {http://tools.ietf.org/rfc/rfc2577.txt RFC-2577} - FTP Security Considerations
-* {http://tools.ietf.org/rfc/rfc2640.txt RFC-2640} - Internationalization of the File Transfer Protocol
-* {http://tools.ietf.org/rfc/rfc3659.txt RFC-3659} - Extensions to FTP
-* {http://tools.ietf.org/rfc/rfc4217.txt RFC-4217} - Internationalization of the File Transfer Protocol
+* [RFC-854](http://tools.ietf.org/rfc/rfc854.txt) - Telnet Protocol
+  Specification
+
+* [RFC-959](http://tools.ietf.org/rfc/rfc959.txt) - File Transfer
+  Protocol
+
+* [RFC-1123](http://tools.ietf.org/rfc/rfc1123.txt) - Requirements for
+  Internet Hosts
+
+* [RFC-1143](http://tools.ietf.org/rfc/rfc1143.txt) - The Q Method of
+  Implementing TELNET Option Negotation
+
+* [RFC-2228](http://tools.ietf.org/rfc/rfc2228.txt) - FTP Security
+  Extensions
+
+* [RFC-2389](http://tools.ietf.org/rfc/rfc2389.txt) - Feature
+  negotiation mechanism for the File Transfer Protocol
+
+* [RFC-2428](http://tools.ietf.org/rfc/rfc2428.txt) - FTP Extensions
+  for IPv6 and NATs
+
+* [RFC-2577](http://tools.ietf.org/rfc/rfc2577.txt) - FTP Security
+  Considerations
+
+* [RFC-2640](http://tools.ietf.org/rfc/rfc2640.txt) -
+  Internationalization of the File Transfer Protocol
+
+* [RFC-3659](http://tools.ietf.org/rfc/rfc3659.txt) - Extensions to
+  FTP
+
+* [RFC-4217](http://tools.ietf.org/rfc/rfc4217.txt) -
+  Internationalization of the File Transfer Protocol
 
 For an english summary that's somewhat more legible than the RFCs, and
 provides some commentary on what features are actually useful or
@@ -29,11 +52,15 @@ For a history lesson, check out Appendix III of RCF959. It lists the
 preceding (obsolete) RFC documents that relate to file transfers,
 including the ye old RFC114 from 1971, "A File Transfer Protocol"
 
-There is a {http://secureftp-test.com public test server} which is
+There is a [public test server](http://secureftp-test.com) which is
 very handy for checking out clients, and seeing how at least one
 server behaves.
 
+## How to reliably close a socket (and not lose data)
+
+[Why is my TCP not reliable](http://ia600609.us.archive.org/22/items/TheUltimateSo_lingerPageOrWhyIsMyTcpNotReliable/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable.html) by Bert Hubert
+
 ## LIST output format
 
-* {http://www.gnu.org/software/coreutils/manual/html_node/What-information-is-listed.html#What-information-is-listed GNU docs for ls}
-* {http://cr.yp.to/ftp/list/eplf.html Easily Parsed LIST format (EPLF)}
+* [GNU docs for ls](http://www.gnu.org/software/coreutils/manual/html_node/What-information-is-listed.html#What-information-is-listed)
+* [Easily Parsed LIST format (EPLF)](http://cr.yp.to/ftp/list/eplf.html)

data/doc/rfc-compliance.md

@@ -50,7 +50,7 @@ Commands supported:
     RNTO    Yes    0.2.1    Rename file (to)    
     SITE    No      ---     Site specific commands
     SMNT    No      ---     Structure Mount    
-    STAT    No      ---     Server status    
+    STAT    Yes    0.5.0    Server status    
     STOR    Yes    0.1.0    Store file    
     STOU    Yes    0.2.2    Store with unique name    
     STRU    Yes    0.1.0    Set file structure
@@ -100,10 +100,10 @@ Server-FTP send only correct reply format  |4.1.2.11       |x| | | | |  C
 Server-FTP use defined reply code if poss. |4.1.2.11       | |x| | | |  C
   New reply code following Section 4.2     |4.1.2.11       | | |x| | |  E
 Default data port same IP addr as ctl conn |4.1.2.12       |x| | | | |  C
-Server-FTP handle Telnet options           |4.1.2.12       |x| | | | |
+Server-FTP handle Telnet options           |4.1.2.12       |x| | | | |  C
 Handle "Experimental" directory cmds       |4.1.3.1        | |x| | | |  C
-Idle timeout in server-FTP                 |4.1.3.2        | |x| | | |
-    Configurable idle timeout              |4.1.3.2        | |x| | | |
+Idle timeout in server-FTP                 |4.1.3.2        | |x| | | |  C
+    Configurable idle timeout              |4.1.3.2        | |x| | | |  C
 Receiver checkpoint data at Restart Marker |4.1.3.4        | |x| | | |
 Sender assume 110 replies are synchronous  |4.1.3.4        | | | | |x|
                                            |               | | | | | |
@@ -157,7 +157,7 @@ Support commands:                          |               | | | | | |
   LIST                                     |4.1.2.13       |x| | | | |  C
   NLST                                     |4.1.2.13       |x| | | | |  C
   SITE                                     |4.1.2.8        | | |x| | |  E
-  STAT                                     |4.1.2.13       |x| | | | |
+  STAT                                     |4.1.2.13       |x| | | | |  C
   SYST                                     |4.1.2.13       |x| | | | |  C
   HELP                                     |4.1.2.13       |x| | | | |  C
   NOOP                                     |4.1.2.13       |x| | | | |  C
@@ -230,7 +230,7 @@ attempts and third-party "proxy FTP" transfers, which can be used in
 
 <pre>
 FTP bounce protection
-Restrict PASV/PORT to non-priv. ports     No      ---
+Restrict PASV/PORT to non-priv. ports     Yes    0.5.0
 Disconnect after so many wrong auths.     No      ---
 Delay on invalid password                 No      ---
 Per-source IP limit                       No      ---

data/examples/example.rb

@@ -15,11 +15,13 @@ module Example
 
     attr_reader :account
     attr_reader :auth_level
+    attr_reader :debug
     attr_reader :eplf
     attr_reader :interface
     attr_reader :password
     attr_reader :port
     attr_reader :read_only
+    attr_reader :session_timeout
     attr_reader :tls
     attr_reader :user
 
@@ -31,6 +33,8 @@ module Example
       @user = ENV['LOGNAME']
       @password = ''
       @account = ''
+      @session_timeout = default_session_timeout
+      @log = nil
       op = option_parser
       op.parse!(argv)
     rescue OptionParser::ParseError => e
@@ -76,9 +80,20 @@ module Example
               'defaults to empty string') do |t|
           @account = t
         end
+        op.on('--timeout SEC', Integer, 'Session idle timeout',
+              "defaults to #{default_session_timeout}") do |t|
+          @session_timeout = t
+        end
+        op.on('-d', '--debug', 'Write server debug log to stdout') do |t|
+          @debug = t
+        end
       end
     end
 
+    def default_session_timeout
+      Ftpd::FtpServer::DEFAULT_SESSION_TIMEOUT
+    end
+
   end
 end
 
@@ -157,6 +172,8 @@ module Example
         @server.list_formatter = Ftpd::ListFormat::Eplf
       end
       @server.auth_level = auth_level
+      @server.session_timeout = @args.session_timeout
+      @server.log = make_log
       @server.start
       display_connection_info
       create_connection_script
@@ -231,6 +248,10 @@ module Example
       @args.account
     end
 
+    def make_log
+      @args.debug && Logger.new($stdout)
+    end
+
   end
 end