sup 0.14.1.1 → 0.15.0

data/CONTRIBUTORS

@@ -3,7 +3,9 @@ Rich Lane <rlane at the club.cc.cmu dot edus>
 Gaute Hope <eg at the gaute.vetsj dot coms>
 Whyme Lyu <callme5long at the gmail dot coms>
 Hamish Downer <dmishd at the gmail dot coms>
+Damien Leone <damien.leone at the fensalir dot frs>
 Sascha Silbe <sascha-pgp at the silbe dot orgs>
+Eric Weikl <eric.weikl at the gmx dot nets>
 Ismo Puustinen <ismo at the iki dot fis>
 Nicolas Pouillard <nicolas.pouillard at the gmail dot coms>
 Michael Stapelberg <michael at the stapelberg dot des>
@@ -15,28 +17,27 @@ Clint Byrum <clint at the ubuntu dot coms>
 Marcus Williams <marcus-sup at the bar-coded dot nets>
 Lionel Ott <white.magic at the gmx dot des>
 Gaudenz Steinlin <gaudenz at the soziologie dot chs>
-Damien Leone <damien.leone at the fensalir dot frs>
 Ingmar Vanhassel <ingmar at the exherbo dot orgs>
 Mark Alexander <marka at the pobox dot coms>
-Eric Weikl <eric.weikl at the gmx dot nets>
+Edward Z. Yang <ezyang at the mit dot edus>
+Matthieu Rakotojaona <matthieu.rakotojaona at the gmail dot coms>
 Christopher Warrington <chrisw at the rice dot edus>
 W. Trevor King <wking at the drexel dot edus>
 Richard Brown <rbrown at the exherbo dot orgs>
 Anthony Martinez <pi+sup at the pihost dot uss>
 Marc Hartstein <marc.hartstein at the alum.vassar dot edus>
 Israel Herraiz <israel.herraiz at the gmail dot coms>
-Matthieu Rakotojaona <matthieu.rakotojaona at the gmail dot coms>
 Bo Borgerson <gigabo at the gmail dot coms>
 Michael Hamann <michael at the content-space dot des>
-Jonathan Lassoff <jof at the thejof dot coms>
 William Erik Baxter <web at the superscript dot coms>
+Jonathan Lassoff <jof at the thejof dot coms>
 Grant Hollingworth <grant at the antiflux dot orgs>
+Adeodato Simó <dato at the net.com.org dot ess>
 Ico Doornekamp <ico at the pruts dot nls>
 Markus Klinik <markus.klinik at the gmx dot des>
-Adeodato Simó <dato at the net.com.org dot ess>
 Daniel Schoepe <daniel.schoepe at the googlemail dot coms>
+James Taylor <james at the jamestaylor dot orgs>
 Jason Petsod <jason at the petsod dot orgs>
-Edward Z. Yang <edwardzyang at the thewritingpot dot coms>
 Steve Goldman <sgoldman at the tower-research dot coms>
 Robin Burchell <viroteck at the viroteck dot nets>
 Peter Harkins <ph at the malaprop dot orgs>
@@ -44,25 +45,27 @@ Decklin Foster <decklin at the red-bean dot coms>
 Cameron Matheson <cam+sup at the cammunism dot orgs>
 Carl Worth <cworth at the cworth dot orgs>
 Alex Vandiver <alex at the chmrr dot nets>
-Jeff Balogh <its.jeff.balogh at the gmail dot coms>
 Andrew Pimlott <andrew at the pimlott dot nets>
+Jeff Balogh <its.jeff.balogh at the gmail dot coms>
 Matías Aguirre <matiasaguirre at the gmail dot coms>
 Kornilios Kourtis <kkourt at the cslab.ece.ntua dot grs>
-Kevin Riggle <kevinr at the free-dissociation dot coms>
 Giorgio Lando <patroclo7 at the gmail dot coms>
+Kevin Riggle <kevinr at the free-dissociation dot coms>
 Benoît PIERRE <benoit.pierre at the gmail dot coms>
 Alvaro Herrera <alvherre at the alvh.no-ip dot orgs>
 Steven Lawrance <stl at the koffein dot nets>
 Jonah <Jonah at the GoodCoffee dot cas>
 ian <itaylor at the uark dot edus>
 Adam Lloyd <adam at the alloy-d dot nets>
-Todd Eisenberger <teisenbe at the andrew.cmu dot edus>
 Gregor Hoffleit <gregor at the sam.mediasupervision dot des>
-MichaelRevell <mikearevell at the gmail dot coms>
+0xACE <0xACE at the users.noreply.github dot coms>
 Per Andersson <avtobiff at the gmail dot coms>
+MichaelRevell <mikearevell at the gmail dot coms>
+Todd Eisenberger <teisenbe at the andrew.cmu dot edus>
 Steven Walter <swalter at the monarch.(none)>
 Matthias Vallentin <vallentin at the icir dot orgs>
+akojo <atte.kojo at the gmail dot coms>
 Jon M. Dugan <jdugan at the es dot nets>
-Stefan Lundström <lundst at the snabb.(none)>
 Horacio Sanson <horacio at the skillupjapan.co dot jps>
+Stefan Lundström <lundst at the snabb.(none)>
 Kirill Smelkov <kirr at the landau.phys.spbu dot rus>

data/History.txt

@@ -1,3 +1,47 @@
+== 0.15.0 / 2013-11-07
+
+* Maildir Syncback has now been merged into main sup! This is a
+  long-time waiting feature initially developed by Damien Leone,
+  then picked up by Edward Z. Yang who continued development. Additionally
+  several others have been contributing.
+
+  Eventually, recently, Eric Weikl has picked up this branch, modernized
+  it to current sup, maintained it and gotten it ready for release.
+
+  Main authors:
+
+  Damien Leone
+  Edward Z. Yang
+  Eric Weikl
+
+  Not all of the features initially proposed have been included. This is
+  to maintain compatibility with more operating systems and wait with
+  the more daring features to make sure sup is stable-ish.
+
+  This is a big change since sup now can modify your mail (!), please
+  back up your mail and your configuration before using the maildir
+  syncback feature. For instructions on how to migrate an existing
+  maildir source or how to set up a new one, refer to the wiki:
+
+  https://github.com/sup-heliotrope/sup/wiki/Using-sup-with-other-clients
+
+  It is possible to both disable maildir syncback globally (default:
+  disabled) and per-source (default: enabled).
+
+* Sup on Ruby 2.0.0 now works - but beware, this has not been very throughly
+  tested. Patches are welcome.
+
+* We are now using our own rmail-sup gem with fixes for Ruby 2.0.0 and
+  various warnings fixed.
+
+* sup-sync-back has been renamed to sup-sync-back-mbox to conform with
+  the other sync-back scripts.
+
+* You can now save attachments to directories without specifying the full
+  filename (default filename is used).
+
+* Various encoding fixes and minor bug fixes
+
 == 0.14.1.1 / 2013-10-29
 
 * SBU1: security release

data/README.md

@@ -18,12 +18,16 @@ Features:
 * [Ruby-programmable hooks][hooks]
 * Automatically tracking recent contacts
 
-Current limitations which will be fixed:
+Current limitations:
 
-* [Doesn't run on Ruby 2.0][ruby20]
+* [Ruby 2.0 support][ruby20] is very fresh, consider it experimental. Patches
+  are welcome
 
-* Sup doesn't play nicely with other mail clients. Changes in Sup won't be
-  synced back to mail source.
+* Sup does in general not play nicely with other mail clients, not all
+  changes can be synced back to the mail source. Refer to [Maildir Syncback][maildir-syncback]
+  in the wiki for this recently included feature. Maildir Syncback
+  allows you to sync back flag changes in messages and to write messages
+  to maildir sources.
 
 * Unix-centrism in MIME attachment handling and in sendmail invocation.
 
@@ -68,3 +72,4 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
 [ruby20]: https://github.com/sup-heliotrope/sup/wiki/Development#sup-014
 [sup-talk]: http://rubyforge.org/mailman/listinfo/sup-talk
 [sup-devel]: http://rubyforge.org/mailman/listinfo/sup-devel
+[maildir-syncback]: https://github.com/sup-heliotrope/sup/wiki/Using-sup-with-other-clients

data/ReleaseNotes

@@ -1,3 +1,11 @@
+Release 0.15.0:
+
+Maildir Syncback has been included. Refer to the wiki for more information on
+how to set it up.
+
+sup-sync-back has been moved to sup-sync-back-mbox, please make sure
+you make any needed changes.
+
 Release 0.14.1.1:
 
 See 0.13.2.1.

data/bin/sup

@@ -153,6 +153,7 @@ Index.lock_interactively or exit
 begin
   Redwood::start
   Index.load
+  Redwood::check_syncback_settings
   Index.start_sync_worker unless $opts[:no_threads]
 
   $die = false

data/bin/sup-add

@@ -30,6 +30,7 @@ Options are:
 EOS
   opt :archive, "Automatically archive all new messages from these sources."
   opt :unusual, "Do not automatically poll these sources for new messages."
+  opt :sync_back, "Synchronize status flags back into messages, defaults to true (Maildir sources only).", :default => true
   opt :labels, "A comma-separated set of labels to apply to all messages from this source", :type => String
   opt :force_new, "Create a new account for this source, even if one already exists."
   opt :force_account, "Reuse previously defined account user@hostname.", :type => String
@@ -99,7 +100,7 @@ begin
     source =
       case parsed_uri.scheme
       when "maildir"
-        Redwood::Maildir.new uri, !$opts[:unusual], $opts[:archive], nil, labels
+        Redwood::Maildir.new uri, !$opts[:unusual], $opts[:archive], $opts[:sync_back], nil, labels
       when "mbox"
         Redwood::MBox.new uri, !$opts[:unusual], $opts[:archive], nil, labels
       when nil

data/bin/sup-config

@@ -88,6 +88,8 @@ def add_source
     usual = axe_yes "Does this source ever receive new messages?", "y"
     archive = usual ? axe_yes("Should new messages be automatically archived? (I.e. not appear in your inbox, though still be accessible via search.)") : false
 
+    sync_back = (type == :maildir) ? axe_yes("Should the original Maildir messages be modified to reflect changes like read status, starred messages, etc.?", "y") : false
+
     labels_str = axe("Enter any labels to be automatically added to all messages from this source, separated by spaces (or 'none')", default_labels.join(","))
 
     labels = if labels_str =~ /^\s*none\s*$/i
@@ -99,6 +101,7 @@ def add_source
     cmd = build_cmd "sup-add"
     cmd += " --unusual" unless usual
     cmd += " --archive" if archive
+    cmd += " --no-sync-back" unless sync_back
     cmd += " --labels=#{labels.join(',')}" if labels && !labels.empty?
     cmd += " #{uri}"
 

data/bin/sup-sync-back-maildir

@@ -0,0 +1,127 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+$:.unshift File.join(File.dirname(__FILE__), *%w[.. lib])
+
+require 'rubygems'
+require 'trollop'
+require "sup"
+
+opts = Trollop::options do
+  version "sup-sync-back-maildir (sup #{Redwood::VERSION})"
+  banner <<EOS
+Export Xapian entries to Maildir sources on disk.
+
+This script parses the Xapian entries for a given Maildir source and renames
+(changes maildir flags) e-mail files on disk according to the labels stored in
+the index. It will export all the changes you made in Sup to your
+Maildirs so that they can be propagated to your IMAP server with e.g. offlineimap.
+
+The script also merges some Maildir flags into Sup such
+as R (replied) and P (passed, forwarded), for instance suppose you
+have an e-mail file like this: foo_bar:2,FRS (flags are favorite,
+replied, seen) and its Xapian entry has labels 'starred', the merge
+operation will add the 'replied' label to the Xapian entry.
+
+If you choose not to merge (-m) you will lose information ('replied'), and in
+the previous example the file will be renamed to foo_bar:2,FS.
+
+Running this script is *strongly* recommended when setting the
+"sync_back_to_maildir" option from false to true in config.yaml or changing the
+"sync_back" flag to true for a source in sources.yaml.
+
+Usage:
+  sup-sync-back-maildir [options] <source>*
+
+where <source>* is source URIs. If no source is given, the default behavior is
+to sync back all Maildir sources marked as usual and that have not disabled
+sync back using the configuration parameter sync_back = false in sources.yaml.
+
+Options include:
+EOS
+  opt :no_confirm, "Don't ask for confirmation before synchronizing", :default => false, :short => "n"
+  opt :no_merge, "Don't merge new supported Maildir flags (R and P)", :default => false, :short => "m"
+  opt :list_sources, "List your Maildir sources and exit", :default => false, :short => "l"
+  opt :unusual_sources_too, "Sync unusual sources too if no specific source information is given", :default => false, :short => "u"
+end
+
+def die msg
+  $stderr.puts "Error: #{msg}"
+  exit(-1)
+end
+
+Redwood::start true
+index = Redwood::Index.init
+index.lock_interactively or exit
+index.load
+
+## Force sync_back_to_maildir option otherwise nothing will happen
+$config[:sync_back_to_maildir] = true
+
+begin
+  sync_performed = []
+  sync_performed = File.readlines(Redwood::SYNC_OK_FN).collect { |e| e.strip }.find_all { |e| not e.empty? } if File.exists? Redwood::SYNC_OK_FN
+  sources = []
+
+  ## Try to find out sources given in parameters
+  sources = ARGV.map do |uri|
+    s = Redwood::SourceManager.source_for(uri) or die "unknown source: #{uri}. Did you add it with sup-add first?"
+    s.is_a?(Redwood::Maildir) or die "#{uri} is not a Maildir source."
+    s.sync_back_enabled? or die "#{uri} has disabled sync back - check your configuration."
+    s
+  end unless opts[:list_sources]
+
+  ## Otherwise, check all sources in sources.yaml
+  if sources.empty? or opts[:list_sources] == true
+    if opts[:unusual_sources_too]
+      sources = Redwood::SourceManager.sources.select do |s|
+        s.is_a? Redwood::Maildir and s.sync_back_enabled?
+      end
+    else
+      sources = Redwood::SourceManager.usual_sources.select do |s|
+        s.is_a? Redwood::Maildir and s.sync_back_enabled?
+      end
+    end
+  end
+
+  if opts[:list_sources] == true
+    sources.each do |s|
+      puts "id: #{s.id}, uri: #{s.uri}"
+    end
+  else
+    sources.each do |s|
+      if opts[:no_confirm] == false
+        print "Are you sure you want to synchronize '#{s.uri}'? (Y/n) "
+        next if STDIN.gets.chomp.downcase == 'n'
+      end
+
+      infos = Enumerator.new(index, :each_source_info, s.id).to_a
+      counter = 0
+      infos.each do |info|
+        print "\rSynchronizing '#{s.uri}'... #{((counter += 1)/infos.size.to_f*100).to_i}%"
+        index.each_message({:location => [s.id, info]}, false) do |m|
+          if opts[:no_merge] == false
+            m.merge_labels_from_locations [:replied, :forwarded]
+          end
+
+          if Redwood::Index.message_joining_killed? m
+            m.labels += [:killed]
+          end
+
+          index.save_message m
+        end
+      end
+      print "\n"
+      sync_performed << s.uri
+    end
+    ## Write a flag file to tell sup that the synchronization has been performed
+    File.open(Redwood::SYNC_OK_FN, 'w') {|f| f.write(sync_performed.join("\n")) }
+  end
+rescue Exception => e
+  File.open("sup-exception-log.txt", "w") { |f| f.puts e.backtrace }
+  raise
+ensure
+  index.save_index
+  Redwood::finish
+  index.unlock
+end

data/bin/{sup-sync-back → sup-sync-back-mbox}

@@ -24,7 +24,7 @@ def has_any_from_source_with_label? index, source, label
 end
 
 opts = Trollop::options do
-  version "sup-sync-back (sup #{Redwood::VERSION})"
+  version "sup-sync-back-mbox (sup #{Redwood::VERSION})"
   banner <<EOS
 Drop or move messages from Sup sources that are marked as deleted or
 spam in the Sup index.
@@ -32,7 +32,7 @@ spam in the Sup index.
 Currently only works with mbox sources.
 
 Usage:
-  sup-sync-back [options] <source>*
+  sup-sync-back-mbox [options] <source>*
 
 where <source>* is zero or more source URIs. If no sources are given,
 sync back all usual sources.
@@ -112,7 +112,7 @@ EOS
     source.reset!
     num_dropped = num_moved = num_scanned = 0
 
-    out_fp = Tempfile.new "sup-sync-back-#{source.id}"
+    out_fp = Tempfile.new "sup-sync-back-mbox-#{source.id}"
     Redwood::PollManager.each_message_from source do |m|
       num_scanned += 1
 

data/lib/sup.rb

@@ -59,10 +59,12 @@ module Redwood
   HOOK_DIR   = File.join(BASE_DIR, "hooks")
   SEARCH_FN  = File.join(BASE_DIR, "searches.txt")
   LOG_FN     = File.join(BASE_DIR, "log")
+  SYNC_OK_FN = File.join(BASE_DIR, "sync-back-ok")
 
   YAML_DOMAIN = "supmua.org"
   LEGACY_YAML_DOMAIN = "masanjin.net"
   YAML_DATE = "2006-10-01"
+  MAILDIR_SYNC_CHECK_SKIPPED = 'SKIPPED'
 
   ## record exceptions thrown in threads nicely
   @exceptions = []
@@ -157,7 +159,7 @@ module Redwood
     SourceManager SearchManager IdleManager).map { |x| Redwood.const_get x.to_sym }
   end
 
-  def start
+  def start bypass_sync_check = false
     managers.each { |x| fail "#{x} already instantiated" if x.instantiated? }
 
     FileUtils.mkdir_p Redwood::BASE_DIR
@@ -173,6 +175,74 @@ module Redwood
     Redwood::SearchManager.init Redwood::SEARCH_FN
 
     managers.each { |x| x.init unless x.instantiated? }
+
+    return if bypass_sync_check
+
+    if $config[:sync_back_to_maildir]
+      if not File.exists? Redwood::SYNC_OK_FN
+        Redwood.warn_syncback <<EOS
+It appears that the "sync_back_to_maildir" option has been changed
+from false to true since the last execution of sup.
+EOS
+        $stderr.puts <<EOS
+
+Should I complain about this again? (Y/n)
+EOS
+        File.open(Redwood::SYNC_OK_FN, 'w') {|f| f.write(Redwood::MAILDIR_SYNC_CHECK_SKIPPED) } if STDIN.gets.chomp.downcase == 'n'
+      end
+    elsif not $config[:sync_back_to_maildir] and File.exists? Redwood::SYNC_OK_FN
+      File.delete(Redwood::SYNC_OK_FN)
+    end
+  end
+
+  def check_syncback_settings
+    # don't check if syncback was never performed
+    return unless File.exists? Redwood::SYNC_OK_FN
+    active_sync_sources = File.readlines(Redwood::SYNC_OK_FN).collect { |e| e.strip }.find_all { |e| not e.empty? }
+    return if active_sync_sources.length == 1 and active_sync_sources[0] == Redwood::MAILDIR_SYNC_CHECK_SKIPPED
+    sources = SourceManager.sources
+    newly_synced = sources.select { |s| s.is_a? Maildir and s.sync_back_enabled? and not active_sync_sources.include? s.uri }
+    unless newly_synced.empty?
+
+      details =<<EOS
+It appears that the option "sync_back" of the following source(s)
+has been changed from false to true since the last execution of
+sup:
+
+EOS
+      newly_synced.each do |s|
+        details += "#{s} (usual: #{s.usual})\n"
+      end
+
+      Redwood.warn_syncback details
+    end
+  end
+
+  def self.warn_syncback details
+    $stderr.puts <<EOS
+WARNING
+-------
+
+#{details}
+
+It is *strongly* recommended that you run "sup-sync-back-maildir"
+before continuing, otherwise you might lose changes you have made in sup
+to your Xapian index.
+
+This script should be run each time you change the
+"sync_back_to_maildir" flag in config.yaml from false to true or
+the "sync_back" flag is changed to true for a source in sources.yaml.
+
+Please run "sup-sync-back-maildir -h" for more information and why this
+is needed.
+
+Note that if you have any sources that are not marked as 'ususal' in
+sources.yaml you need to manually specify them when running  the
+sup-sync-back-maildir script.
+
+Are you really sure you want to continue? (y/N)
+EOS
+    abort "Aborted" unless STDIN.gets.chomp.downcase == 'y'
   end
 
   def finish
@@ -262,7 +332,8 @@ EOM
       :wrap_width => 0,
       :slip_rows => 0,
       :col_jump => 2,
-      :stem_language => "english"
+      :stem_language => "english",
+      :sync_back_to_maildir => false
     }
     if File.exists? filename
       config = Redwood::load_yaml_obj filename
@@ -303,7 +374,8 @@ EOM
   end
 
   module_function :save_yaml_obj, :load_yaml_obj, :start, :finish,
-                  :report_broken_sources, :load_config, :managers
+                  :report_broken_sources, :load_config, :managers,
+                  :check_syncback_settings
 end
 
 require 'sup/version'