sup 0.0.2 → 0.0.3

data/History.txt

@@ -1,3 +1,14 @@
+== 0.0.3 / 2007-01-02
+
+* Major speed increase for index views (inbox, search results), which
+  are now loaded completely from the IR index. The only time the
+  original sources need to be touched is when viewing a thread. This
+  is important for slow sources like IMAP and mbox+ssh.
+* Remote mbox support with mbox+ssh URIs.
+* IMAP now actually works.
+* sup-import uses HighLine and is generally much improved.
+* Multitudinous minor bug fixes and improvements.
+
 == 0.0.2 / 2006-12-10
 
 * IMAP support

data/Manifest.txt

@@ -1,54 +1,57 @@
 History.txt
+LICENSE
 Manifest.txt
 README.txt
-LICENSE
 Rakefile
+bin/sup
+bin/sup-import
+bin/sup-recover-sources
 doc/FAQ.txt
 doc/Philosophy.txt
 doc/TODO
-bin/sup
-bin/sup-import
+lib/sup.rb
+lib/sup/account.rb
+lib/sup/buffer.rb
+lib/sup/colormap.rb
+lib/sup/contact.rb
+lib/sup/draft.rb
+lib/sup/imap.rb
+lib/sup/index.rb
+lib/sup/keymap.rb
+lib/sup/label.rb
+lib/sup/logger.rb
+lib/sup/mbox.rb
 lib/sup/mbox/loader.rb
-lib/sup/modes/line-cursor-mode.rb
-lib/sup/modes/reply-mode.rb
-lib/sup/modes/scroll-mode.rb
-lib/sup/modes/resume-mode.rb
+lib/sup/mbox/ssh-file.rb
+lib/sup/mbox/ssh-loader.rb
+lib/sup/message.rb
+lib/sup/mode.rb
+lib/sup/modes/buffer-list-mode.rb
+lib/sup/modes/compose-mode.rb
 lib/sup/modes/contact-list-mode.rb
+lib/sup/modes/edit-message-mode.rb
 lib/sup/modes/forward-mode.rb
+lib/sup/modes/help-mode.rb
+lib/sup/modes/inbox-mode.rb
+lib/sup/modes/label-list-mode.rb
 lib/sup/modes/label-search-results-mode.rb
-lib/sup/modes/search-results-mode.rb
-lib/sup/modes/compose-mode.rb
+lib/sup/modes/line-cursor-mode.rb
+lib/sup/modes/log-mode.rb
+lib/sup/modes/person-search-results-mode.rb
 lib/sup/modes/poll-mode.rb
-lib/sup/modes/edit-message-mode.rb
+lib/sup/modes/reply-mode.rb
+lib/sup/modes/resume-mode.rb
+lib/sup/modes/scroll-mode.rb
+lib/sup/modes/search-results-mode.rb
+lib/sup/modes/text-mode.rb
 lib/sup/modes/thread-index-mode.rb
-lib/sup/modes/person-search-results-mode.rb
-lib/sup/modes/inbox-mode.rb
 lib/sup/modes/thread-view-mode.rb
-lib/sup/modes/log-mode.rb
-lib/sup/modes/buffer-list-mode.rb
-lib/sup/modes/text-mode.rb
-lib/sup/modes/label-list-mode.rb
-lib/sup/modes/help-mode.rb
-lib/sup/logger.rb
-lib/sup/util.rb
-lib/sup/update.rb
-lib/sup/label.rb
-lib/sup/message.rb
-lib/sup/mode.rb
-lib/sup/keymap.rb
-lib/sup/textfield.rb
-lib/sup/contact.rb
-lib/sup/account.rb
-lib/sup/draft.rb
-lib/sup/mbox.rb
-lib/sup/poll.rb
 lib/sup/person.rb
-lib/sup/index.rb
-lib/sup/thread.rb
-lib/sup/buffer.rb
+lib/sup/poll.rb
 lib/sup/sent.rb
-lib/sup/tagger.rb
-lib/sup/colormap.rb
 lib/sup/source.rb
-lib/sup/imap.rb
-lib/sup.rb
+lib/sup/tagger.rb
+lib/sup/textfield.rb
+lib/sup/thread.rb
+lib/sup/update.rb
+lib/sup/util.rb

data/README.txt

@@ -4,13 +4,30 @@ sup
 
 == DESCRIPTION:
 
-Sup is an attempt to take the UI innovations of web-based email
-readers (ok, really just GMail) and to combine them with the
-traditional wholesome goodness of a console-based email client.
+Sup is a console-based email client that combines the best
+features of GMail, mutt, and emacs. Sup matches the power of GMail
+with the speed and simplicity of a console interface.
 
-Sup is designed to work with massive amounts of email, potentially
-spread out across different mbox files, IMAP folders, and GMail
-accounts, and to pull them all together into a single interface.
+Sup makes it easy to:
+- Handle massive amounts of email.
+
+- Mix email from different sources: mbox files (even across
+  different machines), IMAP folders, POP accounts, and GMail
+  accounts.
+
+- Instantaneously search over your entire email collection. Search
+  over body text, or use a query language to combine search
+  predicates in any way.
+
+- Handle multiple accounts. Replying to email sent to a particular
+  account will use the correct SMTP server, signature, and from
+  address.
+
+- Add custom code to handle certain types of messages or to handle
+  certain types of text within messages.
+
+- Organize email with user-defined labels, automatically track
+  recent contacts, and much more!
 
 The goal of Sup is to become the email client of choice for nerds
 everywhere.
@@ -21,11 +38,10 @@ Features:
 
 - Scalability to massive amounts of email. Immediate startup and
   operability, regardless of how much amount of email you have.
-  (At least, once everything's been indexed.)
 
-- Immediate full-text search of your entire email archive, using
-  the full Ferret query langauge. Search over message bodies, labels,
-  from: and to: fields, or any combination thereof.
+- Immediate full-text search of your entire email archive, using the
+  Ferret query langauge. Search over message bodies, labels, from: and
+  to: fields, or any combination thereof.
 
 - Thread-centrism. Operations are performed at the thread, not the
   message level. Entire threads are manipulated and viewed (with
@@ -34,10 +50,11 @@ Features:
 - Labels instead of folders. Drop that tired old metaphor and you'll
   see how much easier it is to organize email.
 
-- GMail-style thread management.  Archive a thread, and it will
-  disappear from your inbox until someone replies. Kill a thread, and
-  it will never come back to your inbox. (But it will still show up in
-  searches, of course.)
+- GMail-style thread management (but better!). Archive a thread, and
+  it will disappear from your inbox until someone replies. Kill a
+  thread, and it will never come back to your inbox (but will still
+  show up in searches.) Mark a thread as spam and you'll never again
+  see it unless explicitly searching for spam.
 
 - Console based interface. No mouse clicking required!
 
@@ -53,8 +70,8 @@ Features:
 
 Current limitations which will be fixed:
 
-- Support for mbox and IMAP only at this point. No support for POP, mh,
-  or GMail mailstores.
+- Support for mbox, remote mbox, and IMAP only at this point. No
+  support for POP, mh, or GMail mailstores.
 
 - No internationalization support. No wide characters, no subject
   demangling. 
@@ -71,40 +88,47 @@ Current limitations which will be fixed:
   2. sup
   3. edit ~/.sup/config.yaml for the (very few) settings sup has
 
-  Where <source> is a filename (for mbox files), or an imap or imaps
-  url. In the case of imap, don't put the username and password in
-  the URI (which is a terrible, terrible idea). You will be prompted
-  for them.
+  Where <source> is a filename (for mbox files), an imap or imaps URI,
+  or a mbox+ssh URI (for remote mbox files). You will be prompted for
+  a username and password if required.
 
   sup-import has several options which control whether you want
-  messages from particular mailboxes not to be added to the inbox,
-  or not to be marked as new, so run it with -h for help.
+  messages from particular mailboxes not to be added to the inbox, or
+  not to be marked as new, so run it with -h for help.
 
   Note that Sup never changes the contents of any mailboxes; it only
   indexes in to them. So it shouldn't ever corrupt your mail. The flip
   side is that if you change a mailbox (e.g. delete messages, or, in
-  the case of mbox files, read an unread message) then Sup may crash,
-  and will tell you to run sup-import --rebuild to recalculate the
-  offsets within the mailbox.
+  the case of mbox files, read an unread message) then Sup will be
+  unable to load messages from that source and will ask you to run
+  sup-import --rebuild.
 
 == REQUIREMENTS:
 
 * ferret >= 0.10.13
-* ncurses >= 0.9.1
-* rmail >= 0.17
+* ncurses
+* rmail
+* highline
 
 == INSTALL:
 
 * gem install sup -y
-* Then, in rmail, change line 159 of multipart.rb to:
+
+== KNOWN BUGS IN OTHER PACKAGES:
+* If you get an error about frozen strings in RubyMail when importing
+  certain messages with attachments, in rmail, change line 159 of
+  multipart.rb to:
     chunk = chunk[0..start]
-  (Sorry; it's an unsupported package.) You might be able to get away
-  without doing this but if you get frozen string exceptions when
-  reading in multipart messages, this is what you need to change.
+* Occasionally Ferret produces something the Ruby GC doesn't like
+  (particularly when importing messages from very large sources).
+  No worries, just re-run sup-import. (This is unresolved atm.)
+* There are a couple other Ferret issues with outstanding patches but
+  they are pretty rare.
+
 
 == LICENSE:
 
-Copyright (c) 2006 William Morgan.
+Copyright (c) 2006, 2007 William Morgan.
 
 This program is free software; you can redistribute it and/or
 modify it under the terms of the GNU General Public License

data/Rakefile

@@ -12,7 +12,7 @@ Hoe.new('sup', Redwood::VERSION) do |p|
   p.url = p.paragraphs_of('README.txt', 0).first.split(/\n/)[2].gsub(/^\s+/, "")
   p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
   p.email = "wmorgan-sup@masanjin.net"
-  p.extra_deps = [['ferret', '>= 0.10.13'], ['ncurses', '>= 0.9.1'], ['rmail', '>= 0.17']]
+  p.extra_deps = [['ferret', '>= 0.10.13'], ['ncurses', '>= 0.9.1'], ['rmail', '>= 0.17'], 'highline']
 end
 
 rule 'ss?.png' => 'ss?-small.png' do |t|

data/bin/sup

@@ -4,20 +4,9 @@ require 'rubygems'
 require 'ncurses'
 require "sup"
 
-module Redwood
+Thread.abort_on_exception = true # make debugging possible
 
-$exception = nil
-def reporting_thread
-  ::Thread.new do
-    begin
-      yield
-    rescue Exception => e
-      $exception ||= e
-      raise
-    end
-  end
-end
-module_function :reporting_thread
+module Redwood
 
 global_keymap = Keymap.new do |k|
   k.add :quit, "Quit Redwood", 'q'
@@ -32,6 +21,7 @@ global_keymap = Keymap.new do |k|
   k.add :list_labels, "List labels", 'L'
   k.add :poll, "Poll for new messages", 'P'
   k.add :compose, "Compose new message", 'm'
+  k.add :recall_draft, "Edit most recent draft message", 'R'
 end
 
 def start_cursing
@@ -50,16 +40,8 @@ def stop_cursing
 end
 module_function :start_cursing, :stop_cursing
 
-Redwood::SentManager.new Redwood::SENT_FN
-Redwood::ContactManager.new Redwood::CONTACT_FN
-Redwood::LabelManager.new Redwood::LABEL_FN
-Redwood::AccountManager.new $config[:accounts]
-Redwood::DraftManager.new Redwood::DRAFT_DIR
-Redwood::UpdateManager.new
-Redwood::PollManager.new
-
+Redwood::start
 Index.new.load
-log "loaded #{Index.size} messages from index"
 
 if(s = Index.source_for DraftManager.source_name)
   DraftManager.source = s
@@ -72,7 +54,7 @@ if(s = Index.source_for SentManager.source_name)
 else
   Index.add_source SentManager.new_source
 end
-  
+
 begin
   log "starting curses"
   start_cursing
@@ -120,7 +102,8 @@ begin
   bm.draw_screen
   imode.load_more_threads ibuf.content_height
 
-  reporting_thread { sleep 3; PollManager.poll }
+  reporting_thread { sleep 5; PollManager.poll }
+  PollManager.start_thread
 
   until $exception
     bm.draw_screen
@@ -147,77 +130,73 @@ begin
         when :list_buffers
           bm.spawn_unless_exists("Buffer List") { BufferListMode.new }
         when :list_contacts
-          mode = ContactListMode.new 
-          bm.spawn "compose to contacts", mode
+          bm.spawn_unless_exists("Contact List") { ContactListMode.new }
         when :search
           text = bm.ask :search, "query: "
           next unless text && text !~ /^\s*$/
-          mode = SearchResultsMode.new text
-          short_text = 
-            if text.length < 20
-              text
-            else
-              text[0 ... 20] + "..."
-            end
-          bm.spawn "search: \"#{short_text}\"", mode
-          bm.draw_screen
-          mode.load_more_threads mode.buffer.content_height
-        when :list_labels
-          b = BufferManager.spawn_unless_exists("all labels") do
-            LabelListMode.new
+
+          begin
+            qobj = Index.parse_user_query_string text
+            short_text = text.length < 20 ? text : text[0 ... 20] + "..."
+            log "built query from #{text.inspect}: #{qobj}"
+            mode = SearchResultsMode.new qobj
+            bm.spawn "search: \"#{short_text}\"", mode
+            mode.load_more_threads mode.buffer.content_height
+          rescue Ferret::QueryParser::QueryParseException => e
+            bm.flash "Couldn't parse query."
           end
+
+        when :list_labels
+          b = bm.spawn_unless_exists("Label List") { LabelListMode.new }
           b.mode.load_in_background
         when :compose
           mode = ComposeMode.new
-          bm.spawn "new message", mode
+          bm.spawn "New Message", mode
           mode.edit
         when :poll
-          BufferManager.raise_to_front PollManager.buffer
-          PollManager.poll
+          bm.raise_to_front PollManager.buffer
+          reporting_thread { PollManager.poll }
+        when :recall_draft
+          case Index.num_results_for :label => :draft
+          when 0
+            bm.flash "No draft messages."
+          when 1
+            m = nil
+            Index.each_id_by_date(:label => :draft) { |mid, builder| m = builder.call }
+            BufferManager.spawn "Edit message", ResumeMode.new(m)
+          else
+            b = BufferManager.spawn_unless_exists(:draft) do
+              mode = LabelSearchResultsMode.new [:draft]
+            end
+            b.mode.load_more_threads b.content_height
+          end
         when :nothing
         when :redraw
           bm.completely_redraw_screen
         else
-          BufferManager.flash "Unknown key press '#{c.to_character}' for #{bm.focus_buf.mode.name}."
+          bm.flash "Unknown key press '#{c.to_character}' for #{bm.focus_buf.mode.name}."
         end
       end
     end
   end
   bm.kill_all_buffers
-  Redwood::LabelManager.save
-  Redwood::ContactManager.save
 rescue Exception => e
   $exception ||= e
 ensure
+  Redwood::finish
   stop_cursing
 end
 
 Index.save unless $exception # TODO: think about this
 
 if $exception 
-  case $exception
-  when IndexError
-    $stderr.puts <<EOS
-An error occurred while parsing a message from source:
-   #{$exception.source}.
-Typically, this means that the source has been modified in some
-way which has rendered the messages invalid. For example, if it's
-an mbox file, you may have read or deleted messages using another
-mail client.
-
-You must rebuild the index for this source. Please run:
-  sup-import --rebuild #{$exception.source}
-to correct this error.
-EOS
-#' stupid ruby-mode
-  else
-    $stderr.puts <<EOS
+  $stderr.puts <<EOS
 ----------------------------------------------------------------
 I'm very sorry, but it seems that an error occurred in Sup. 
 Please accept my sincere apologies. If you don't mind, please
 send the backtrace below and a brief report of the circumstances
-to user wmorgan-sup at site masanjin dot net so that I might
-address this problem. Thank you!
+to wmorgan-sup at masanjin dot nets so that I might address this
+problem. Thank you!
 
 Sincerely,
 William
@@ -226,7 +205,6 @@ William
 The problem was: #{$exception.message} (error type #{$exception.class.name})
 A backtrace follows:
 EOS
-  end
   raise $exception
 end
 

data/bin/sup-import

@@ -1,7 +1,13 @@
 #!/usr/bin/env ruby
 
+require 'uri'
+require 'rubygems'
+require 'highline/import'
 require "sup"
 
+
+Thread.abort_on_exception = true # make debugging possible
+
 class Float
   def to_s; sprintf '%.2f', self; end
 end
@@ -57,6 +63,39 @@ EOS
   exit
 end
 
+## for sources that require login information, prompt the user for
+## that. also provide a list of previously-defined login info to
+## choose from, if any.
+def get_login_info uri, sources
+  uri = URI(uri)
+  accounts = sources.map do |s|
+    next unless s.respond_to?(:username)
+    suri = URI(s.uri)
+    [suri.host, s.username, s.password]
+  end.compact.uniq.sort_by { |h, u, p| h == uri.host ? 0 : 1 }
+
+  username, password = nil, nil
+  unless accounts.empty?
+    say "Would you like to use the same account as for a previous source?"
+    choose do |menu|
+      accounts.each do |host, olduser, oldpw|
+        menu.choice("Use the account info for #{olduser}@#{host}") { username, password = olduser, oldpw }
+      end
+      menu.choice("Use a new account") { }
+      menu.prompt = "Account selection? "
+    end
+  end
+
+  unless username && password
+    username = ask("Username for #{uri.host}: ");
+    password = ask("Password for #{uri.host}: ") { |q| q.echo = false }
+    puts # why?
+  end
+
+  [username, password]
+end
+
+
 educate_user if ARGV.member? '--help'
 
 archive = ARGV.delete "--archive"
@@ -79,25 +118,27 @@ if(o = ARGV.find { |x| x =~ /^--/ })
   educate_user
 end
 
-puts "loading index..."
+$terminal.wrap_at = :auto
+Redwood::start
 index = Redwood::Index.new
 index.load
-puts "loaded index of #{index.size} messages"
 
-sources = ARGV.map do |fn|
-  fn = "mbox://#{fn}" unless fn =~ %r!://!
-  source = index.source_for fn
+sources = ARGV.map do |uri|
+  uri = "mbox://#{uri}" unless uri =~ %r!://!
+  source = index.source_for uri
   unless source
     source = 
-      case fn
+      case uri
+      when %r!^mbox\+ssh://!
+        say "For SSH connections, if you will use public key authentication, you may leave the username and password blank."
+        say "\n"
+        username, password = get_login_info uri, index.sources
+        Redwood::MBox::SSHLoader.new(uri, username, password, nil, !unusual, !!archive)
       when %r!^imaps?://!
-        print "Username for #{fn}: "
-        username = $stdin.gets.chomp
-        print "Password for #{fn} (warning: cleartext): "
-        password = $stdin.gets.chomp
-        Redwood::IMAP.new(fn, username, password, nil, !unusual, !!archive)
+        username, password = get_login_info uri, index.sources
+        Redwood::IMAP.new(uri, username, password, nil, !unusual, !!archive)
       else
-        Redwood::MBox::Loader.new(fn, nil, !unusual, !!archive)
+        Redwood::MBox::Loader.new(uri, nil, !unusual, !!archive)
       end
     index.add_source source
   end
@@ -118,7 +159,7 @@ start = Time.now
 begin
   sources.each do |source|
     if source.broken?
-      puts "error loading messages from #{source}: #{source.broken_msg}"
+      $stderr.puts "error loading messages from #{source}: #{source.broken_msg}"
       next
     end
     next if source.done?
@@ -137,8 +178,9 @@ begin
         else
           found[m.id] = true
         end
+
         m.remove_label :unread if m.status == "RO" unless force_read
-        puts "# message at #{offset}, labels: #{labels * ', '}" unless rebuild || force_rebuild
+        puts "# message at #{offset}, labels: #{labels * ', '}"
         if (rebuild || force_rebuild) && 
             (docid, entry = index.load_entry_for_id(m.id)) && entry
           if force_rebuild || entry[:source_info].to_i != offset
@@ -162,7 +204,9 @@ begin
     puts "loaded #{num} messages" unless num == 0
   end
 ensure
+  $stderr.puts "saving index and sources..."
   index.save
+  Redwood::finish
 end
 
 if rebuild || force_rebuild