sup 0.7 → 0.8

data/CONTRIBUTORS

@@ -1,17 +1,22 @@
-William Morgan <wmorgan-sup at the masanjin dot nets>
-Ismo Puustinen <ismo at the iki dot fis>
 Nicolas Pouillard <nicolas.pouillard at the gmail dot coms>
+Mike Stipicevic <stipim at the rpi dot edus>
 Marcus Williams <marcus-sup at the bar-coded dot nets>
 Lionel Ott <white.magic at the gmx dot des>
+Ingmar Vanhassel <ingmar at the exherbo dot orgs>
 Mark Alexander <marka at the pobox dot coms>
 Christopher Warrington <chrisw at the rice dot edus>
-Marc Hartstein <marc.hartstein at the alum.vassar dot edus>
+Richard Brown <rbrown at the exherbo dot orgs>
 Ben Walton <bwalton at the artsci.utoronto dot cas>
+Marc Hartstein <marc.hartstein at the alum.vassar dot edus>
 Grant Hollingworth <grant at the antiflux dot orgs>
 Steve Goldman <sgoldman at the tower-research dot coms>
 Decklin Foster <decklin at the red-bean dot coms>
+Ismo Puustinen <ismo at the iki dot fis>
 Jeff Balogh <its.jeff.balogh at the gmail dot coms>
+Alex Vandiver <alexmv at the mit dot edus>
 Giorgio Lando <patroclo7 at the gmail dot coms>
 Israel Herraiz <israel.herraiz at the gmail dot coms>
 Ian Taylor <ian at the lorf dot orgs>
+Stefan Lundström <lundst at the snabb.(none)>
 Rich Lane <rlane at the club.cc.cmu dot edus>
+Kirill Smelkov <kirr at the landau.phys.spbu dot rus>

data/History.txt

@@ -1,3 +1,22 @@
+== 0.8 / 2009-06-05
+* Undo support on many operations. Yay!
+* Mbox splitting fixes. No more "From "-line problems.
+* Mail parsing speedups.
+* Many utf8 and widechar fixes. Display of crazy characters should be pretty
+  close.
+* Outgoing email with non-ASCII headers is now properly encoded.
+* Email addresses are no longer permanently attached to names. This was
+  causing problems with automated email systems that used different names
+  with the same address.
+* Curses background now retains the terminal default color. This also makes
+  Sup work better on transparent terminals.
+* Improve dynamic loading of setlocale for Cygwin and BSD systems.
+* Labels can now be removed from multiple tagged threads.
+* Applying operations to tagged threads is now invoked with '='.
+* Buffer list is betterified and is now invoked with ';'.
+* Zsh autocompletion support.
+* As always, many bugfixes and tweaks.
+
 == 0.7 / 2009-03-16
 * Ferret index corruption issues fixed (hopefully!)
 * Text entry now scrolls to the right on overflow, i.e. is actually usable

data/README.txt

@@ -13,22 +13,20 @@ Sup makes it easy to:
 - Handle massive amounts of email.
 
 - Mix email from different sources: mbox files (even across different
-  machines), Maildir directories, IMAP folders, POP accounts, and
-  GMail accounts.
+  machines), Maildir directories, IMAP folders, and GMail accounts.
 
-- Instantaneously search over your entire email collection. Search
-  over body text, or use a query language to combine search
-  predicates in any way.
+- Instantaneously search over your entire email collection. Search over
+  body text, or use a query language to combine search predicates in any
+  way.
 
 - Handle multiple accounts. Replying to email sent to a particular
-  account will use the correct SMTP server, signature, and from
-  address.
+  account will use the correct SMTP server, signature, and from address.
 
-- Add custom code to handle certain types of messages or to handle
-  certain types of text within messages.
+- Add custom code to customize Sup to whatever particular and bizarre
+  needs you may have.
 
-- Organize email with user-defined labels, automatically track
-  recent contacts, and much more!
+- Organize email with user-defined labels, automatically track recent
+  contacts, and much more!
 
 The goal of Sup is to become the email client of choice for nerds
 everywhere.
@@ -48,40 +46,41 @@ Features:
   message level. Entire threads are manipulated and viewed (with
   redundancies removed) at a time.
 
-- Labels instead of folders. Drop that tired old metaphor and you'll
-  see how much easier it is to organize email.
+- Labels instead of folders. Drop that tired old metaphor and you'll see
+  how much easier it is to organize email.
 
-- GMail-style thread management (but better!). Archive a thread, and
-  it will disappear from your inbox until someone replies. Kill a
-  thread, and it will never come back to your inbox (but will still
-  show up in searches.) Mark a thread as spam and you'll never again
-  see it unless explicitly searching for spam.
+- GMail-style thread management. Archive a thread, and it will disappear
+  from your inbox until someone replies. Kill a thread, and it will
+  never come back to your inbox (but will still show up in searches.)
+  Mark a thread as spam and you'll never again see it unless explicitly
+  searching for spam.
 
 - Console based interface. No mouse clicking required!
 
-- Programmability. It's in Ruby. The code is good. It's easy to
-  extend.
+- Programmability. It's in Ruby. The code is good. It has an extensive
+  hook system that makes it easy to extend and customize.
 
-- Multiple buffer support. Why be limited to viewing one thread at a
+- Multiple buffer support. Why be limited to viewing one thing at a
   time?
 
-- Tons of other little features, like automatic context-sensitive
-  help, multi-message operations, MIME attachment viewing, recent
-  contact list generation, etc.
+- Tons of other little features, like automatic context-sensitive help,
+  multi-message operations, MIME attachment viewing, recent contact list
+  generation, etc.
 
 Current limitations which will be fixed:
 
-- Support for mbox, remote mbox, and IMAP only at this point. No
-  support for POP, mh, or GMail mailstores.
+- Sup doesn't play nicely with other mail clients. If you alter a mail
+  source (read, move, delete, etc) with another client Sup will punish
+  you with a lengthy reindexing process.
 
-- No internationalization support. No wide characters, no subject
-  demangling. 
+- Support for mbox, Maildir, and IMAP only at this point. No support for
+  POP or mh.
 
-- Unix-centrism in MIME attachment handling and in sendmail
-  invocation.
+- IMAP support is very slow due mostly to Ruby's IMAP library.  You may
+  consider something like offlineimap to mirror your IMAP folders with
+  local Maildir ones.
 
-- Several obvious missing features, like undo, filters / saved
-  searches, message annotations, etc.
+- Unix-centrism in MIME attachment handling and in sendmail invocation.
 
 == SYNOPSYS:
 
@@ -90,21 +89,23 @@ Current limitations which will be fixed:
 
   Note that Sup never changes the contents of any mailboxes; it only
   indexes in to them. So it shouldn't ever corrupt your mail. The flip
-  side is that if you change a mailbox (e.g. delete messages, or, in
-  the case of mbox files, read an unread message) then Sup will be
-  unable to load messages from that source and will ask you to run
-  sup-sync --changed.
+  side is that if you change a mailbox (e.g. delete messages, or, in the
+  case of mbox files, read an unread message) then Sup will be unable to
+  load messages from that source and will ask you to run sup-sync
+  --changed.
 
 == REQUIREMENTS:
 
-* ferret >= 0.10.13
-* ncurses
-* rmail
-* highline
-* net-ssh
-* trollop >= 1.7
-* lockfile
-* mime-types
+ - ferret >= 0.11.6
+ - ncurses >= 0.9.1
+ - rmail >= 0.17
+ - highline
+ - net-ssh
+ - trollop >= 1.12
+ - lockfile
+ - mime-types
+ - gettext
+ - fastthread
 
 == INSTALL:
 

data/ReleaseNotes

@@ -1,3 +1,9 @@
+Release 0.8:
+
+The big wins are undo support, mbox splitting fixes, and the various UI
+speedups and bugfixes. Parsing new email should also be faster, although
+IMAP remains tragically slow, as usual.
+
 Release 0.7:
 
 The big win in this release is that Ferret index corruption issues should now

data/bin/sup

@@ -8,7 +8,7 @@ require 'trollop'
 require 'fastthread'
 require "sup"
 
-BIN_VERSION = "0.7"
+BIN_VERSION = "0.8"
 
 unless Redwood::VERSION == BIN_VERSION
   $stderr.puts <<EOS
@@ -67,9 +67,9 @@ global_keymap = Keymap.new do |k|
   k.add :quit_now, "Quit Sup immediately", 'Q'
   k.add :help, "Show help", '?'
   k.add :roll_buffers, "Switch to next buffer", 'b'
-#  k.add :roll_buffers_backwards, "Switch to previous buffer", 'B'
+  k.add :roll_buffers_backwards, "Switch to previous buffer", 'B'
   k.add :kill_buffer, "Kill the current buffer", 'x'
-  k.add :list_buffers, "List all buffers", 'B'
+  k.add :list_buffers, "List all buffers", ';'
   k.add :list_contacts, "List contacts", 'C'
   k.add :redraw, "Redraw screen", :ctrl_l
   k.add :search, "Search all messages", '\\', 'F'
@@ -81,11 +81,42 @@ global_keymap = Keymap.new do |k|
   k.add :recall_draft, "Edit most recent draft message", 'R'
 end
 
+## the following magic enables wide characters when used with a ruby
+## ncurses.so that's been compiled against libncursesw. (note the w.) why
+## this works, i have no idea. much like pretty much every aspect of
+## dealing with curses.  cargo cult programming at its best.
+##
+## BSD users: if libc.so.6 is not found, try installing compat6x.
+require 'dl/import'
+module LibC
+  extend DL::Importable
+  setlocale_lib = case Config::CONFIG['arch']
+    when /darwin/; "libc.dylib"
+    when /cygwin/; "cygwin1.dll"
+    else; "libc.so.6"
+  end
+
+  Redwood::log "dynamically loading setlocale() from #{setlocale_lib}"
+  begin
+    dlload setlocale_lib
+    extern "void setlocale(int, const char *)"
+    Redwood::log "setting locale..."
+    LibC.setlocale(6, "")  # LC_ALL == 6
+  rescue RuntimeError => e
+    Redwood::log "cannot dlload setlocale(); ncurses wide character support probably broken."
+    Redwood::log "dlload error was #{e.class}: #{e.message}"
+    if Config::CONFIG['arch'] =~ /bsd/
+      Redwood::log "BSD variant detected. You may have to install a compat6x package to acquire libc."
+    end
+  end
+end
+
 def start_cursing
   Ncurses.initscr
   Ncurses.noecho
   Ncurses.cbreak
   Ncurses.stdscr.keypad 1
+  Ncurses.use_default_colors
   Ncurses.curs_set 0
   Ncurses.start_color
   $cursing = true
@@ -229,7 +260,7 @@ begin
     when :kill_buffer
       bm.kill_buffer_safely bm.focus_buf
     when :list_buffers
-      bm.spawn_unless_exists("Buffer List") { BufferListMode.new }
+      bm.spawn_unless_exists("buffer list", :system => true) { BufferListMode.new }
     when :list_contacts
       b, new = bm.spawn_unless_exists("Contact List") { ContactListMode.new }
       b.mode.load_in_background if new
@@ -240,7 +271,7 @@ begin
     when :search_unread
       SearchResultsMode.spawn_from_query "is:unread"
     when :list_labels
-      labels = LabelManager.listable_labels.map { |l| LabelManager.string_for l }
+      labels = LabelManager.all_labels.map { |l| LabelManager.string_for l }
       user_label = bm.ask_with_completions :label, "Show threads with label (enter for listing): ", labels
       unless user_label.nil?
         if user_label.empty?

data/bin/sup-recover-sources

@@ -72,18 +72,14 @@ ARGV.each do |fn|
   source_ids = {}
   count = 0
   source.each do |offset, labels|
-    begin
-      m = Redwood::Message.new :source => source, :source_info => offset
-      docid, entry = index.load_entry_for_id m.id
-      next unless entry
-      #puts "# #{source} #{offset} #{entry[:source_id]}"
-
-      source_ids[entry[:source_id]] = (source_ids[entry[:source_id]] || 0) + 1
-      count += 1
-      break if count == $opts[:scan_num]
-    rescue Redwood::MessageFormatError => e
-      puts "# #{e.message}"
-    end
+    m = Redwood::Message.new :source => source, :source_info => offset
+    docid, entry = index.load_entry_for_id m.id
+    next unless entry
+    #puts "# #{source} #{offset} #{entry[:source_id]}"
+
+    source_ids[entry[:source_id]] = (source_ids[entry[:source_id]] || 0) + 1
+    count += 1
+    break if count == $opts[:scan_num]
   end
 
   if source_ids.size == 1

data/bin/sup-sync

@@ -5,6 +5,8 @@ require 'rubygems'
 require 'trollop'
 require "sup"
 
+PROGRESS_UPDATE_INTERVAL = 15 # seconds
+
 class Float
   def to_s; sprintf '%.2f', self; end
    def to_time_s
@@ -100,7 +102,7 @@ restored_state =
     IO.foreach opts[:restore] do |l|
       l =~ /^(\S+) \((.*?)\)$/ or raise "Can't read dump line: #{l.inspect}"
       mid, labels = $1, $2
-      dump[mid] = labels.split(" ").map { |x| x.intern }
+      dump[mid] = labels.symbolistize
     end
     $stderr.puts "Read #{dump.size} entries from dump file."
     dump
@@ -122,7 +124,9 @@ begin
 
   unless target == :new
     if opts[:start_at]
-      sources.each { |s| s.seek_to! opts[:start_at] }
+      Trollop::die :start_at, "can only be used on one source" unless sources.size == 1
+      sources.first.seek_to! opts[:start_at]
+      sources.first.correct_offset! if sources.first.respond_to?(:correct_offset!)
     else
       sources.each { |s| s.reset! }
     end
@@ -137,18 +141,22 @@ begin
       num_scanned += 1
       seen[m.id] = true
 
+      if Time.now - last_info_time > PROGRESS_UPDATE_INTERVAL
+        last_info_time = Time.now
+        elapsed = last_info_time - start_time
+        start = opts[:start_at] || source.start_offset
+        pctdone = 100.0 * (source.cur_offset - start).to_f / (source.end_offset - start).to_f
+        remaining = (100.0 - pctdone) * (elapsed.to_f / pctdone)
+        $stderr.printf "## read %dm (about %.0f%%) @ %.1fm/s. %s elapsed, about %s remaining\n", num_scanned, pctdone, num_scanned / elapsed, elapsed.to_time_s, remaining.to_time_s
+      end
+
       ## skip if we're operating only on changed messages, the message
       ## is in the index, and it's unchanged from what the source is
       ## reporting.
       next if target == :changed && entry && entry[:source_id].to_i == source.id && entry[:source_info].to_i == offset
 
       ## get the state currently in the index
-      index_state =
-        if entry
-          entry[:label].split(/\s+/).map { |x| x.intern }
-        else
-          nil
-        end
+      index_state = entry[:label].symbolistize if entry
 
       ## skip if we're operating on restored messages, and this one
       ## ain't.
@@ -158,12 +166,12 @@ begin
       ## to default source state modification flags.
       m.labels -= [:inbox] if opts[:archive]
       m.labels -= [:unread] if opts[:read]
-      m.labels += opts[:extra_labels].split(/\s*,\s*/).map { |x| x.intern } if opts[:extra_labels]
+      m.labels += opts[:extra_labels].strip.split(/\s*,\s*/).map { |x| x.intern } if opts[:extra_labels]
 
       ## assign message labels based on the operation we're performing
       case op
       when :asis
-        m.labels = index_state if index_state
+        m.labels = ((m.labels - [:unread, :inbox]) + index_state).uniq if index_state
       when :restore
         ## if the entry exists on disk
         if restored_state[m.id]
@@ -176,16 +184,16 @@ begin
         ## nothin! use default source labels
       end
 
-      if Time.now - last_info_time > 60
+      if Time.now - last_info_time > PROGRESS_UPDATE_INTERVAL
         last_info_time = Time.now
         elapsed = last_info_time - start_time
         pctdone = source.respond_to?(:pct_done) ? source.pct_done : 100.0 * (source.cur_offset.to_f - source.start_offset).to_f / (source.end_offset - source.start_offset).to_f
         remaining = (100.0 - pctdone) * (elapsed.to_f / pctdone)
-        $stderr.puts "## #{num_scanned} (#{pctdone}%) read; #{elapsed.to_time_s} elapsed; #{remaining.to_time_s} remaining"
+        $stderr.printf "## read %dm (about %.0f%%) @ %.1fm/s. %s elapsed, about %s remaining\n", num_scanned, pctdone, num_scanned / elapsed, elapsed.to_time_s, remaining.to_time_s
       end
 
       if index_state.nil?
-        puts "Adding message #{source}##{offset} with state {#{m.labels * ', '}}" if opts[:verbose]
+        puts "Adding message #{source}##{offset} from #{m.from} with state {#{m.labels * ', '}}" if opts[:verbose]
         num_added += 1
       else
         puts "Updating message #{source}##{offset}, source #{entry[:source_id]} => #{source.id}, offset #{entry[:source_info]} => #{offset}, state {#{index_state * ', '}} => {#{m.labels * ', '}}" if opts[:verbose]
@@ -205,9 +213,7 @@ begin
   ## API.
   ##
   ## TODO: move this to Index, i suppose.
-
-
-  if target == :all || target == :changed
+  if (target == :all || target == :changed) && !opts[:start_at]
     $stderr.puts "Deleting missing messages from the index..."
     num_del, num_scanned = 0, 0
     sources.each do |source|

data/bin/sup-sync-back

@@ -109,7 +109,7 @@ EOS
       num_scanned += 1
 
       if entry
-        labels = entry[:label].split.map { |x| x.intern }.to_boolean_h
+        labels = entry[:label].symbolistize.to_boolean_h
 
         if labels.member? :deleted
           if opts[:drop_deleted]

data/bin/sup-tweak-labels

@@ -46,10 +46,12 @@ EOS
 Other options:
 EOS
   opt :verbose, "Print message ids as they're processed."
+  opt :very_verbose, "Print message names and subjects as they're processed."
   opt :all_sources, "Scan over all sources.", :short => :none
   opt :dry_run, "Don't actually modify the index. Probably only useful with --verbose.", :short => "-n"
   opt :version, "Show version information", :short => :none
 end
+opts[:verbose] = true if opts[:very_verbose]
 
 add_labels = (opts[:add] || "").split(",").map { |l| l.intern }.uniq
 remove_labels = (opts[:remove] || "").split(",").map { |l| l.intern }.uniq
@@ -71,7 +73,7 @@ begin
   end.map { |s| s.id }
   Trollop::die "nothing to do: no sources" if source_ids.empty?
 
-  query = "+(" + source_ids.map { |id| "source_id:#{id}" }.join(" ") + ")"
+  query = "+(" + source_ids.map { |id| "source_id:#{id}" }.join(" OR ") + ")"
   if add_labels.empty?
     ## if all we're doing is removing labels, we can further restrict the
     ## query to only messages with those labels
@@ -79,19 +81,15 @@ begin
   end
   query += ' ' + opts[:query] if opts[:query]
 
-  qobj, opts = Redwood::Index.parse_user_query_string query
-  query = Redwood::Index.build_query opts.merge(:qobj => qobj)
-
-  results = index.ferret.search query, :limit => :all
-  num_total = results.total_hits
+  docs = Redwood::Index.run_query query
+  num_total = docs.size
 
   $stderr.puts "Found #{num_total} documents across #{source_ids.length} sources. Scanning..."
 
   num_changed = num_scanned = 0
   last_info_time = start_time = Time.now
-  results.hits.each do |hit|
+  docs.each do |id|
     num_scanned += 1
-    id = hit.doc
 
     m = index.build_message id
     old_labels = m.labels.clone
@@ -102,7 +100,9 @@ begin
 
     unless m.labels.sort_by { |s| s.to_s } == old_labels.sort_by { |s| s.to_s }
       num_changed += 1
+      puts "From #{m.from}, subject: #{m.subj}" if opts[:very_verbose]
       puts "#{m.id}: {#{old_labels.join ','}} => {#{m.labels.join ','}}" if opts[:verbose]
+      puts if opts[:very_verbose]
       index.sync_message m unless opts[:dry_run]
     end