sup 0.0.7 → 0.0.8

data/doc/TODO

@@ -1,29 +1,53 @@
 for 0.0.8
 ---------
-create attachments
-forward attachments
-warnings: top-posting, missing attachment
-maildir
+x nice little startup config program
+x bugfix: triggering a pageup when cursor scrolling up jumps to the
+   bottom of the page rather than the next line
+x bugfix: final logging messages to stdout?
+x bugfix: mbox directory shouldn't generate an exception, just an error
+x bugfix: m in thread-view-mode when a person is not selected should open up a
+   blank compose-mode rather than do nothing
+x bugfix: stars on messages with blue backgrounds still have green bgs
+x ferret upgrade script (dump & restore)
+x bugfix: mark messages as read immediately when t-v-m is opened
+x compose in thread-view-mode auto-fills in person
+x bugfix: 'N' in thread-view-mode (expand only new messages) crashes
+x bugfix: detect source corruption at startup
+x maildir
+x bugfix: single-line messages come empty upon reply
 
-for 0.0.9
+next release
+------------
+_ bugfix: when one new message comes into an imap folder, we don't
+   catch it until a reload (sometimes?)
+_ bugfix: add new message counts until keypress
+_ bugfix: attachment filenames sometimes not detected (filename=)
+_ split out threading & message chunk parsing to a separate library
+
+near future
 ---------
-select all, starred, to me, etc
-undo
-use Net::SMTP
-gmail
+_ Net::SMTP support (cuz I'm going to need it soon)
+_ create attachments
+_ forward attachments
+_ select all, starred, to me, etc
+_ undo
+_ gmail
+_ warnings: top-posting, missing attachment, ruby-talk:XXXX detection
+_ mboxz, mboxbz
 
 future
 ------
+search results: highlight relevant snippets and open to relevant portion of thread
+email address to name mapping needs some work. automatic email addresses (noreply@...) are often assigned to something screwy.
 decode RFC 2047 ("encoded word") headers
   - see: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/101949, http://dev.rubyonrails.org/ticket/6807
 swappable keymappings
 within-buffer search
 bugfix: when returning from a shelling out, ncurses is crazy 
-bugfix: miscellaneous weirdnesses in buffer line editing
+more control character support in buffer line editing
 wide character support
 i18n support
 batch deletion
-support for message-content modules such as ruby-talk:XXXXX detection
 tab completion on labels, contacts
 contact selector in edit-message-mode
 maybe: filters
@@ -33,6 +57,7 @@ pop
 be able to mark individual messages as spam in thread-view-mode
 toggle wrapping
 maybe: de-archived messages auto-added to inbox
+prune old entries from contacts.txt so that it doesn't arbitrarily
 
 done
 ----
@@ -82,3 +107,4 @@ x highlighting/different color stuff
 x config: your email, sendmail, etc
 x status: to/from_you, cc_you_others
 x status: new/not, important
+x bugfix: miscellaneous weirdnesses in buffer line editing

data/doc/UserGuide.txt

@@ -26,12 +26,21 @@ for messages or view your inbox, Sup talks only to the index (stored
 locally on disk). When you view a thread, Sup requests the full
 content of all the messages from the source.
 
-So let's add some sources to Sup's source list. Run 'sup-add' with
-a URI pointing to an email source. The URI should be of the form:
-- mbox://path/to/a/filename, for an mbox file on disk. (You can also
-  just provide the filename).
-- imap://imap.server/folder or imaps://secure.imap.server/folder for
-  an IMAP folder. (Leave the folder blank for INBOX.)
+The easiest way to set up all your sources is to run "sup-config".
+This will interactively walk you through some basic configuration,
+prompt you for all the sources you need, and optionally import
+messages from them.
+
+sup-config works by calling sup-add and sup-sync with the right
+arguments. If you have a non-trivial setup, you may need to run
+sup-add and sup-sync manually.
+
+You can manually add a source to Sup's source list by running
+'sup-add' with a URI pointing to it. The URI should be of the form:
+- mbox://path/to/a/filename, for an mbox file on disk.
+- maildir://path/to/a/filename, for an maildir directory on disk.
+- imap://imap.server/folder for an unsecure IMAP folder.
+- imaps://secure.imap.server/folder for a secure IMAP folder.
 - mbox+ssh://remote.machine/path/to/a/filename for a remote mbox file.
 
 Before you add the source, you need make two decisions. The first is
@@ -46,39 +55,33 @@ inbox, but will be found when you search. (Your inbox in Sup is, by
 definition, the set of all all non-archived messages). Specify
 --archive to automatically archive all messages from the source. This
 is useful for sources that contain, for example, high-traffic mailing
-lists that you don't want "polluting" your inbox.
+lists that you don't want polluting your inbox.
 
 If Sup requires account information, e.g. for IMAP servers and remote
 mbox files, sup-add will ask for it.
 
 Now that you've added the source, let's import all the current
-messages from it, by running sup-import with the source URI. You can
+messages from it, by running sup-sync with the source URI. You can
 specify --archive to automatically archive all messages in this
 import; typically you'll want to specify this for every source you
 import except your actual inbox. You can also specify --read to mark
 all imported messages as read; the default is to preserve the
 read/unread status from the source.
 
-sup-import will now load all the messages from the source into the
+sup-sync will now load all the messages from the source into the
 index. Depending on the size of the source, this may take a
 while. Don't panic! It's a one-time process.
 
-We're almost ready. But before we run 'sup' again, take a moment to
-edit your ~/.sup/config.yaml file. Replace "Your Name Here" with your
-name, "your.email.here@domain.tld" with your email address, and fill
-in your .signature file if you choose. You can also set the default
-editor.
-
 Ok, now run 'sup'. You should see the most recent unarchived messages
 appear in your inbox. Congratulations, you've got Sup working!
 
-If you're coming from the world of traditional email, there are a
+If you're coming from the world of traditional MUAs, there are a
 couple differences you should be aware of at this point. First, Sup
 does not use folders. Instead, you organize and find messages by a
 combination of search and labels (knowns as 'tags' everywhere else in
-the world). I like to say that 95% of the functionality of folders is
+the world). I claim that 95% of the functionality of folders is
 superceded by a quick, easy-to-access, and powerful search mechanism,
-and the other 5% by labels. (The Sup statement of philosophy has a
+and the other 5% by labels. (The Sup philosophical treatise has a
 little more on this.)
 
 Search and labels are an integral part of Sup because in Sup, rather
@@ -96,18 +99,18 @@ life can be easier than that if you just trust the search engine, and
 use labels judiciously for things that are too hard to find with
 search.
 
-But enough chit-chat! Let's take a look at your inbox. You'll see that
-Sup groups messages together into threads: each line in the inbox is a
-thread, and the number in parentheses is the number of messages in
-that thread. (If there's no number in parens, it means there's just
-one message.) In Sup, you rarely operate on an individual message. The
-idea is that you rarely want to operate on a message independent of
-its context; you typically want to view, archive, kill, or label all
-the messages in a thread at one time.
+Now let's take a look at your inbox. You'll see that Sup groups
+messages together into threads: each line in the inbox is a thread,
+and the number in parentheses is the number of messages in that
+thread. (If there's no number, there's just one message.) In Sup, most
+operations are on threads, not individual messages. The idea is that
+you rarely want to operate on a message independent of its context.
+You typically want to view, archive, kill, or label all the messages
+in a thread at one time.
 
 Use the up and down arrows to highlight a thread. ('j' and 'k' do the
 same thing, and 'J' and 'K' will scroll the whole window.  Even the
-left and right arrow keys work!) By default, Sup only loads as many
+left and right arrow keys work.) By default, Sup only loads as many
 threads as it takes to fill the window; if you'd like to load more,
 press 'M'. You can hit tab to cycle between only threads with new
 messages.
@@ -122,19 +125,18 @@ to expand or collapse all messages or 'N' to expand only the new
 messages. You'll also notice that Sup hides quoted text and
 signatures. If you highlight a particular hidden chunk, you can press
 enter to expand it, or you can press 'o' to toggle every hidden chunk
-in a particular message. (Don't worry about remembering all these
-things---you can hit '?' to see the full list of keyboard commands at
-any point.)
+in a particular message. (You can hit '?' to see the full list of
+keyboard commands at any point.)
 
 A few other useful commands while viewing a thread. Press 'd' to
 toggle a detailed header fpr a message. If you've scrolled too far to
 the right, press '[' to jump all the way to the left. Finally, you can
 press 'n' and 'p' to jump forward and backward between open
-messages. (I find that very useful!)
+messages, aligning the display as necessary.
 
 Now press 'x' to kill the thread view buffer. You should see the inbox
 again. If you don't, you can cycle through the buffers by pressing
-'b', or you can press 'A' to see a list of all buffers and simply
+'b', or you can press 'B' to see a list of all buffers and simply
 select the inbox.
 
 There are many operations you can perform on threads beyond viewing
@@ -190,11 +192,11 @@ details on more sophisticated queries (date ranges, "within n words",
 etc.)
 
 At this point, you're well on your way to figuring out all the cool
-things Sup can do. By repeated applications of the '?' key, see if you
+things Sup can do. By repeated application of the '?' key, see if you
 can figure out how to:
  - List some recent contacts
  - Easily search for all mail from a recent contact
- - Easily search for all mail from several recent contacts!
+ - Easily search for all mail from several recent contacts
  - Add someone to your address book
  - Postpone a message (i.e., save a draft)
  - Quickly re-edit a just-saved draft message
@@ -203,11 +205,10 @@ can figure out how to:
 
 There's one last thing to be aware of when using Sup: how it interacts
 with other email programs. Sup stores data about messages in the
-index---information necessary for searching, and message state---but
-doesn't duplicate the message contents themselves. The messages remain
-on the source. If the index and the source every fall out of sync,
-e.g. due to another email client modifying the source, then Sup will
-be unable to operate on that source.
+index, but doesn't duplicate the message contents themselves. The
+messages remain on the source. If the index and the source every fall
+out of sync, e.g. due to another email client modifying the source,
+then Sup will be unable to operate on that source.
 
 For example, for mbox files, Sup stores a byte offset into the file
 for each message. If a message deleted from that file by another
@@ -216,8 +217,8 @@ offsets will be wrong.
 
 That's the bad news. The good news is that Sup is fairly good at being
 able to detect this type of situation, and fixing it is just a matter
-of running sup-import --rebuild on the source. Sup will even tell you
-how to invoke sup-import when it detects a problem. This is a
+of running sup-sync --changed on the source. Sup will even tell you
+how to invoke sup-sync when it detects a problem. This is a
 complication you will almost certainly run in to if you use both Sup
 and another MUA on the same source, so it's good to be aware of it.
 

data/lib/sup.rb

@@ -13,7 +13,7 @@ class Object
 end
 
 module Redwood
-  VERSION = "0.0.7"
+  VERSION = "0.0.8"
 
   BASE_DIR   = ENV["SUP_BASE"] || File.join(ENV["HOME"], ".sup")
   CONFIG_FN  = File.join(BASE_DIR, "config.yaml")
@@ -93,9 +93,52 @@ module Redwood
     Redwood::LabelManager.save
     Redwood::ContactManager.save
     Redwood::PersonManager.save
+    Redwood::BufferManager.deinstantiate!
   end
 
-  module_function :register_yaml, :save_yaml_obj, :load_yaml_obj, :start, :finish
+  ## not really a good place for this, so I'll just dump it here.
+  def report_broken_sources opts={}
+    return unless BufferManager.instantiated?
+
+    broken_sources = Index.usual_sources.select { |s| s.error.is_a? FatalSourceError }
+    unless broken_sources.empty?
+      BufferManager.spawn "Broken source notification", TextMode.new(<<EOM), opts
+Source error notification
+-------------------------
+
+Hi there. It looks like one or more message sources is reporting
+errors. Until this is corrected, messages from these sources cannot
+be viewed, and new messages will not be detected.
+
+#{broken_sources.map { |s| "Source: " + s.to_s + "\n Error: " + s.error.message.wrap(70).join("\n        ")}.join('\n\n')}
+EOM
+#' stupid ruby-mode
+    end
+
+    desynced_sources = Index.usual_sources.select { |s| s.error.is_a? OutOfSyncSourceError }
+    unless desynced_sources.empty?
+      BufferManager.spawn "Out-of-sync source notification", TextMode.new(<<EOM), opts
+Out-of-sync source notification
+-------------------------------
+
+Hi there. It looks like one or more sources has fallen out of sync
+with my index. This can happen when you modify these sources with
+other email clients. (Sorry, I don't play well with others.)
+
+Until this is corrected, messages from these sources cannot be viewed,
+and new messages will not be detected. Luckily, this is easy to correct!
+
+#{desynced_sources.map do |s|
+  "Source: " + s.to_s + 
+   "\n Error: " + s.error.message.wrap(70).join("\n        ") + 
+   "\n   Fix: sup-sync --changed #{s.to_s}"
+  end}
+EOM
+#' stupid ruby-mode
+    end
+  end
+
+  module_function :register_yaml, :save_yaml_obj, :load_yaml_obj, :start, :finish, :report_broken_sources
 end
 
 ## set up default configuration file
@@ -105,8 +148,8 @@ else
   $config = {
     :accounts => {
       :default => {
-        :name => "Your Name Here",
-        :email => "your.email.here@domain.tld",
+        :name => "Sup Rocks",
+        :email => "sup-rocks@reading-my-emails",
         :alternates => [],
         :sendmail => "/usr/sbin/sendmail -oem -ti",
         :signature => File.join(ENV["HOME"], ".signature")
@@ -127,6 +170,7 @@ require "sup/update"
 require "sup/message"
 require "sup/source"
 require "sup/mbox"
+require "sup/maildir"
 require "sup/imap"
 require "sup/person"
 require "sup/account"

data/lib/sup/buffer.rb

@@ -50,6 +50,7 @@ module Redwood
 class Buffer
   attr_reader :mode, :x, :y, :width, :height, :title
   bool_reader :dirty
+  bool_accessor :force_to_top
 
   def initialize window, mode, width, height, opts={}
     @w = window
@@ -57,6 +58,7 @@ class Buffer
     @dirty = true
     @focus = false
     @title = opts[:title] || ""
+    @force_to_top = opts[:force_to_top] || false
     @x, @y, @width, @height = 0, 0, width, height
   end
 
@@ -156,18 +158,33 @@ class BufferManager
 
   def raise_to_front buf
     raise ArgumentError, "buffer not on stack: #{buf.inspect}" unless @buffers.member? buf
+
     @buffers.delete buf
-    @buffers.push buf
-    focus_on buf
+    if @buffers.length > 0 && @buffers.last.force_to_top?
+      @buffers.insert(-2, buf)
+    else
+      @buffers.push buf
+      focus_on buf
+    end
     @dirty = true
   end
 
+  ## we reset force_to_top when rolling buffers. this is so that the
+  ## human can actually still move buffers around, while still
+  ## programmatically being able to pop stuff up in the middle of
+  ## drawing a window without worrying about covering it up.
+  ##
+  ## if we ever start calling roll_buffers programmatically, we will
+  ## have to change this. but it's not clear that we will ever actually
+  ## do that.
   def roll_buffers
+    @buffers.last.force_to_top = false
     raise_to_front @buffers.first
   end
 
   def roll_buffers_backwards
     return unless @buffers.length > 1
+    @buffers.last.force_to_top = false
     raise_to_front @buffers[@buffers.length - 2]
   end
 
@@ -258,14 +275,14 @@ class BufferManager
     ## w = Ncurses::WINDOW.new(height, width, (opts[:top] || 0),
     ## (opts[:left] || 0))
     w = Ncurses.stdscr
-    b = Buffer.new w, mode, width, height, :title => realtitle
+    b = Buffer.new w, mode, width, height, :title => realtitle, :force_to_top => (opts[:force_to_top] || false)
     mode.buffer = b
     @name_map[realtitle] = b
+
+    @buffers.unshift b
     if opts[:hidden]
-      @buffers.unshift b
       focus_on b unless @focus_buf
     else
-      @buffers.push b
       raise_to_front b
     end
     b

data/lib/sup/draft.rb

@@ -12,7 +12,7 @@ class DraftManager
 
   def self.source_name; "sup://drafts"; end
   def self.source_id; 9999; end
-  def new_source; @source = DraftLoader.new; end
+  def new_source; @source = Recoverable.new DraftLoader.new; end
 
   def write_draft
     offset = @source.gen_offset
@@ -22,7 +22,7 @@ class DraftManager
     my_message = nil
     @source.each do |thisoffset, theselabels|
       m = Message.new :source => @source, :source_info => thisoffset, :labels => theselabels
-      Index.add_message m
+      Index.sync_message m
       UpdateManager.relay self, :add, m
       my_message = m if thisoffset == offset
     end

data/lib/sup/imap.rb

@@ -2,6 +2,7 @@ require 'uri'
 require 'net/imap'
 require 'stringio'
 require 'time'
+require 'rmail'
 
 ## fucking imap fucking sucks. what the FUCK kind of committee of
 ## dunces designed this shit.
@@ -35,7 +36,6 @@ class IMAP < Source
   ## upon these errors we'll try to rereconnect a few times
   RECOVERABLE_ERRORS = [ Errno::EPIPE, Errno::ETIMEDOUT ]
 
-  attr_reader_cloned :labels
   attr_accessor :username, :password
 
   def initialize uri, username, password, last_idate=nil, usual=true, archived=false, id=nil
@@ -52,7 +52,6 @@ class IMAP < Source
     @ids = []
     @last_scan = nil
     @labels = [:unread]
-    @labels << :inbox unless archived?
     @labels << mailbox.intern unless mailbox =~ /inbox/i
     @mutex = Mutex.new
   end
@@ -64,6 +63,18 @@ class IMAP < Source
     x.nil? || x.empty? ? 'INBOX' : x
   end
   def ssl?; @parsed_uri.scheme == 'imaps' end
+
+  def check
+    ids = 
+      @mutex.synchronize do
+        unsynchronized_scan_mailbox
+        @ids
+      end
+
+    start = ids.index(cur_offset || start_offset) or raise OutOfSyncSourceError, "Unknown message id #{cur_offset || start_offset}."
+  end
+
+  ## is this necessary? TODO: remove maybe
   def == o; o.is_a?(IMAP) && o.uri == self.uri && o.username == self.username; end
 
   def load_header id
@@ -89,7 +100,7 @@ class IMAP < Source
   synchronized :raw_full_message
 
   def connect
-    return if broken? || @imap
+    return if @imap
     safely { } # do nothing!
   end
   synchronized :connect
@@ -121,12 +132,12 @@ class IMAP < Source
         @ids
       end
 
-    start = ids.index(cur_offset || start_offset) or die_from "Unknown message id #{cur_offset || start_offset}.", :suggest_rebuild => true # couldn't find the most recent email
+    start = ids.index(cur_offset || start_offset) or raise OutOfSyncSourceError, "Unknown message id #{cur_offset || start_offset}."
 
     start.upto(ids.length - 1) do |i|         
       id = ids[i]
       self.cur_offset = id
-      yield id, labels
+      yield id, @labels.clone
     end
   end
 
@@ -196,26 +207,6 @@ private
     @say_id = nil
   end
 
-  def die_from e, opts={}
-    @imap = nil
-
-    message =
-      case e
-      when Exception
-        "Error while #{opts[:while]}: #{e.message.chomp} (#{e.class.name})."
-      when String
-        e
-      end
-
-    message += " It is likely that messages have been deleted from this IMAP mailbox. Please run sup-import --rebuild #{to_s} to correct this problem." if opts[:suggest_rebuild]
-
-    self.broken_msg = message
-    Redwood::log message
-    BufferManager.flash "Error communicating with IMAP server. See log for details." if BufferManager.instantiated?
-    raise SourceError, message
-  end
-  
-  ## build a fake unique id
   def make_id imap_stuff
     # use 7 digits for the size. why 7? seems nice.
     msize, mdate = imap_stuff.attr['RFC822.SIZE'] % 10000000, Time.parse(imap_stuff.attr["INTERNALDATE"])
@@ -223,13 +214,12 @@ private
   end
 
   def get_imap_fields id, *fields
-    raise SourceError, broken_msg if broken?
-    imap_id = @imap_ids[id] or die_from "Unknown message id #{id}.", :suggest_rebuild => true
+    imap_id = @imap_ids[id] or raise OutOfSyncSourceError, "Unknown message id #{id}"
 
     retried = false
     results = safely { @imap.fetch imap_id, (fields + ['RFC822.SIZE', 'INTERNALDATE']).uniq }.first
     got_id = make_id results
-    die_from "IMAP message mismatch: requested #{id}, got #{got_id}.", :suggest_rebuild => true unless got_id == id
+    raise OutOfSyncSourceError, "IMAP message mismatch: requested #{id}, got #{got_id}." unless got_id == id
 
     fields.map { |f| results.attr[f] }
   end
@@ -250,8 +240,8 @@ private
         end
         raise
       end
-    rescue Net, SocketError, Net::IMAP::Error, SystemCallError => e
-      die_from e, :while => "communicating with IMAP server"
+    rescue SocketError, Net::IMAP::Error, SystemCallError, IOError => e
+      raise FatalSourceError, "While communicating with IMAP server: #{e.message}"
     end
   end