imap.rb 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8be67e0de4db977ee6ba474e7355fadae929bd267132fd867d64f96caa3c7a19
-  data.tar.gz: e930fe80e9f8cffac1837670688e22f8826ce50391a36b5f5cc8474f0c89e9fb
+  metadata.gz: f9f5b0c607f3b3e66d7284613916d45c077de9a0e97f50f960fa8e63fd7b5586
+  data.tar.gz: 4529940cbd85a56bfbd4ef9d472536234852cd4cabf6349f90763644905ab953
 SHA512:
-  metadata.gz: 3b8f36bc1285a4f379624038e387cd70e825c07b617f07a60ebd5ba11ee62a775126eb1a0263107739bea073b44b2d2cc86c4c3f05a5533eb8ea848f5f11b0f0
-  data.tar.gz: 36f1969380582aef53570369140fdffddd920fabc04a157faca0accd49f38286f8563934b3db3c049e0769074d355339cc234ccfe65fa63f53d8a61eebc096fa
+  metadata.gz: 7279ae1bb4fd2456fa2acf57cad565a0af28d2be5795e1e6bf9924b2b9f6583acf2e4a71735944c8bf3f3dc56d403d3abba5943eb87ea15054f12c718665bab9
+  data.tar.gz: 060c6a1b974e983979130cf62d03e0630ef548b64b9e80a522c6064c031126385de84f762cb1a158fd50b03711902f1430038b8aa0b3f63864c7dc4e85366548

data/CHANGELOG

@@ -2,6 +2,30 @@
 
 ## 20260917
 
+0.7.0: ~ Imap::Message: + #uid, #flags, #seen?, #size, #received_at, #sent_at, #cc, #from_address, #from_domain, #attachment?, #attachment_filenames, #source, .decode; + Imap#mailboxes
+
+1. + Imap::Message#uid, #flags, #seen?, #size, #received_at, #sent_at, #cc, #from_address, #from_domain, #attachment?, #attachment_filenames, #source. Everything a search had to go around the gem to Net::IMAP for.
+2. ~ Imap::Message: + FETCH_ATTRIBUTES, UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE, which .search asks for in the one fetch it already made. The round trip costs the same whether it asks for one of these or for all six, so it asks for all six.
+3. ~ Imap::Message: keeps the Net::IMAP::FetchData rather than its attr hash, + Imap::Message#data for the old #attrs. FetchData reads every one of these attributes itself, so #uid, #flags, #size, #envelope and #bodystructure delegate rather than index, #body and #source become #text and #message, and #received_at is Net::IMAP.decode_time's work rather than Time.parse's, the INTERNALDATE being IMAP's own date-time and not RFC 5322's. #sent_at keeps Time.parse, the envelope carrying the header as the sender wrote it. Half a dozen accessors of net-imap's were being reimplemented here a little worse.
+4. ~ Imap::Message#envelope: reads FETCH_ATTRIBUTES where a caller built the message, rather than the ENVELOPE alone, so that a message built by hand answers the same questions as a searched one for the same one round trip.
+5. + Imap::Message.decode: RFC 2047, =?charset?B?...?= and =?charset?Q?...?=, the whitespace between adjacent encoded words dropped as the RFC says. A subject or a display name in anything but ASCII arrived as its encoding and was unreadable and unsearchable both. The text as it stands where the charset is not one Ruby knows.
+6. ~ Imap::Message#subject: decoded, and the folding newlines made spaces. #from and #to likewise decode the display name.
+7. + Imap::Message#source: BODY.PEEK[] for the whole message, headers and all, for whatever wants to parse the MIME. The gem does not: that is the mail gem's work and not IMAP's, and this is the one line it needs.
+8. + Imap#mailboxes: LIST '' '*', the \Noselect containers left out, since there is no searching everywhere without first knowing where everywhere is.
+9. ~ test/test_helper.rb: MockIMAP answers with a Net::IMAP::FetchData as a server's response parses to, rather than an OpenStruct wearing seqno and attr, and answers the six attributes and LIST as a server does. Message 2 carries the encoded words and names its attachment in the content type, message 1 in the disposition, message 99 an envelope with neither to nor cc.
+10. + 20 tests: the attributes off the one fetch, both dates as Times, the decoder upon each encoding and upon a charset it cannot know, both ways an attachment is named, and the selectable mailboxes.
+11. ~ Imap::VERSION: /0.6.0/0.7.0/
+
+0.6.0: One fetch for a slice of messages rather than one per attribute per message.
+
+1. ~ Imap::Message.search: fetches the ENVELOPE for a slice of message ids and hands each Imap::Message its own, + SLICE = 200. A hundred messages cost a round trip apiece for the subject and another apiece for the from; they cost one for the hundred now, which at any real latency is the whole cost of a search.
+2. ~ Imap::Message#initialize: + attrs, which search() supplies and a caller need not. + Imap::Message#envelope, which reads what was prefetched or fetches where there is none, so a message built by hand behaves as before.
+3. ~ Imap::Message#subject, #from, #to: read from the envelope rather than fetching a header apiece. #from answers 'name <mailbox@host>' where the envelope names one, and the bare address where it does not, having answered the raw From header.
+4. ~ Imap::Message#body: unchanged and still lazy, a body being the one thing worth not fetching until it is wanted.
+5. ~ test/test_helper.rb: MockIMAP#fetch takes a list as a server does, answers with a seqno apiece, and counts its calls. Its envelope carries a subject and a from; the two BODY[HEADER.FIELDS] cases went with the code that asked for them.
+6. + tests: that a search of three fetches once, that the prefetched envelope is not fetched again, and that a message built by hand still fetches.
+7. ~ Imap::VERSION: /0.5.0/0.6.0/
+
 0.5.0: + OR and NOT to the criteria, and HEADER given its field
 
 1. + Imap::Search#any_of: or: [{from: 'a@x'}, {to: 'b@y'}]. OR takes two keys and no more, so three or more nest, OR OR a b c, as the server reads them. Each side is a criteria hash of its own and may nest further.

data/lib/Imap/Message.rb

@@ -10,38 +10,133 @@
 # 6. Imap::Message.new(message_id, imap_client).subject
 # 7. Imap::Message.new(message_id, imap_client).from
 # 8. Imap::Message.new(message_id, imap_client).to
+# 9. Imap::Message.new(message_id, imap_client).received_at
+# 10. Imap::Message.new(message_id, imap_client).attachment_filenames
+# 11. Imap::Message.new(message_id, imap_client).source
+
+require 'time'
 
 require_relative './Search'
 
 class Imap
   class Message
+
+    # What one fetch brings back, since the round trip costs the same whether it
+    # asks for one of these or for all of them.
+    FETCH_ATTRIBUTES = %w{UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE}
+    SLICE = 200
+    ENCODED_WORD = /=\?([^?]+)\?([BbQq])\?([^?]*)\?=/
+
     class << self
 
+      # One fetch for a slice of messages rather than one per attribute per
+      # message.  A hundred messages cost a round trip apiece for the subject and
+      # another apiece for the from; they cost one for the hundred now.
       def search(imap_client, **search_criteria)
         message_ids = Imap::Search.new(imap_client, search_criteria).message_ids
-        message_ids.collect{|message_id| Imap::Message.new(message_id, imap_client)}
+        message_ids.each_slice(SLICE).flat_map do |slice|
+          imap_client.imap.fetch(slice, FETCH_ATTRIBUTES).collect do |data|
+            Imap::Message.new(data.seqno, imap_client, data)
+          end
+        end
       end
       alias_method :find, :search
 
+      # RFC 2047: =?charset?B?base64?= and =?charset?Q?quoted-printable?=, which
+      # headers carry wherever the subject or a name is not ASCII.
+      def decode(value)
+        return nil if value.nil?
+        value.to_s.gsub(/(?<=\?=)\s+(?==\?)/, '').gsub(ENCODED_WORD){decode_word($1, $2, $3)}
+      end
+
+      private
+
+      def decode_word(charset, encoding, text)
+        bytes =
+          if encoding.upcase == 'B'
+            text.unpack1('m')
+          else
+            text.tr('_', ' ').gsub(/=([0-9A-Fa-f]{2})/){$1.hex.chr}
+          end
+        bytes.force_encoding(charset.split('*').first).encode('UTF-8', invalid: :replace, undef: :replace)
+      rescue StandardError
+        text
+      end
+
     end # class << self
 
     attr_accessor :imap_client
     attr_accessor :message_id
 
-    def body
-      @body ||= fetch_data('BODY.PEEK[TEXT]').first.attr['BODY[TEXT]']
+    def uid
+      data.uid
+    end
+
+    def flags
+      data.flags || []
+    end
+
+    def seen?
+      flags.include?(:Seen)
+    end
+
+    def size
+      data.size.to_i
+    end
+
+    # When the server took delivery, which is what a mailbox is ordered by.
+    def received_at
+      @received_at ||= data.internaldate
+    end
+
+    # When the sender says they sent it, which is the Date header and is theirs
+    # to get wrong.  Time.parse and not Net::IMAP.decode_time: the envelope
+    # carries the RFC 5322 header, not IMAP's own date-time.
+    def sent_at
+      @sent_at ||= to_time(envelope && envelope.date)
     end
 
     def subject
-      @subject ||= fetch_data('BODY[HEADER.FIELDS (SUBJECT)]').first.attr['BODY[HEADER.FIELDS (SUBJECT)]'].sub(/^Subject: /, '').strip
+      @subject ||= Message.decode(envelope && envelope.subject).to_s.tr("\r\n", ' ').strip
     end
 
     def from
-      @from ||= fetch_data('BODY[HEADER.FIELDS (FROM)]').first.attr['BODY[HEADER.FIELDS (FROM)]'].sub(/^From: /, '').strip
+      @from ||= address_to_s(envelope.from && envelope.from.first)
+    end
+
+    def from_address
+      @from_address ||= address_only(envelope.from && envelope.from.first)
+    end
+
+    def from_domain
+      @from_domain ||= from_address && from_address.split('@').last.to_s.downcase
     end
 
     def to
-      @to ||= (fetch_data('ENVELOPE').first.attr['ENVELOPE'].to || []).map{|addr| addr.mailbox + '@' + addr.host}
+      @to ||= addresses(envelope.to)
+    end
+
+    def cc
+      @cc ||= addresses(envelope.cc)
+    end
+
+    # IMAP cannot search upon an attachment, but the bodystructure names them and
+    # arrives with the rest.
+    def attachment?
+      !attachment_filenames.empty?
+    end
+
+    def attachment_filenames
+      @attachment_filenames ||= filenames_in(data.bodystructure)
+    end
+
+    def body
+      @body ||= fetch_data('BODY.PEEK[TEXT]').first.text
+    end
+
+    # The whole message, headers and all, for whatever wants to parse MIME.
+    def source
+      @source ||= fetch_data('BODY.PEEK[]').first.message
     end
 
     def urls
@@ -55,13 +150,71 @@ class Imap
 
     private
 
-    def initialize(message_id = nil, imap_client = nil)
+    def initialize(message_id = nil, imap_client = nil, data = nil)
       @message_id = message_id
       @imap_client = imap_client
+      @data = data
+    end
+
+    # The Net::IMAP::FetchData itself rather than its attr hash, since it reads
+    # every one of these attributes already and decodes the internaldate to a
+    # Time along the way.  Prefetched where search() built this, fetched where
+    # the caller built it.
+    def data
+      @data ||= fetch_data(*FETCH_ATTRIBUTES).first
+    end
+
+    def envelope
+      data.envelope
+    end
+
+    def to_time(value)
+      return value if value.is_a?(Time)
+      value.nil? || value.to_s.empty? ? nil : Time.parse(value.to_s)
+    rescue ArgumentError
+      nil
+    end
+
+    def addresses(list)
+      (list || []).collect{|address| address_to_s(address)}.compact
+    end
+
+    def address_only(address)
+      address && [address.mailbox, address.host].compact.join('@')
+    end
+
+    def address_to_s(address)
+      return nil unless address
+      email = address_only(address)
+      name = Message.decode(address.name).to_s.strip
+      name.empty? ? email : "#{name} <#{email}>"
+    end
+
+    def filenames_in(structure)
+      return [] unless structure
+      return structure.parts.flat_map{|part| filenames_in(part)} if structure.respond_to?(:parts) && structure.parts
+      return filenames_in(structure.body) if structure.respond_to?(:body) && structure.body
+      [filename_of(structure)].compact
+    end
+
+    # The disposition names it where the sender set one, the content type's NAME
+    # where they did not.
+    def filename_of(part)
+      name = disposition_filename(part) || parameter_filename(part)
+      name.nil? || name.to_s.strip.empty? ? nil : Message.decode(name).strip
+    end
+
+    def disposition_filename(part)
+      disposition = part.respond_to?(:disposition) ? part.disposition : nil
+      disposition && disposition.param ? disposition.param['FILENAME'] : nil
+    end
+
+    def parameter_filename(part)
+      part.respond_to?(:param) && part.param ? part.param['NAME'] : nil
     end
 
-    def fetch_data(*attrs)
-      @fetch_data = imap_client.imap.fetch(message_id, [*attrs])
+    def fetch_data(*fetch_attributes)
+      @fetch_data = imap_client.imap.fetch(message_id, [*fetch_attributes])
     end
   end
 end

data/lib/Imap/VERSION.rb

@@ -1,3 +1,3 @@
 class Imap
-  VERSION = '0.5.0'
+  VERSION = '0.7.0'
 end

data/lib/imap.rb

@@ -44,6 +44,11 @@ class Imap
     imap.select(mailbox)
   end
 
+  # Every mailbox which can be selected, the \Noselect containers left out.
+  def mailboxes
+    (imap.list('', '*') || []).reject{|mailbox| mailbox.attr.include?(:Noselect)}.collect(&:name)
+  end
+
   def search(criteria = {})
     Imap::Message.search(self, **criteria)
   end

data/test/Imap/Message_test.rb

@@ -30,6 +30,30 @@ describe Imap::Message do
     end
   end
 
+  describe 'the batched fetch' do
+    it 'fetches once for the whole slice rather than once per message' do
+      messages = Imap::Message.search(client, from: 'test@example.com')
+      _(messages.size).must_equal 3
+      _(client.imap.fetch_count).must_equal 1
+    end
+
+    it 'reads the prefetched envelope rather than fetching again' do
+      message = Imap::Message.search(client, from: 'test@example.com').first
+      before = client.imap.fetch_count
+      message.subject
+      message.from
+      message.to
+      _(client.imap.fetch_count).must_equal before
+    end
+
+    it 'still fetches for a message the caller built' do
+      message = Imap::Message.new(1, client)
+      before = client.imap.fetch_count.to_i
+      message.subject
+      _(client.imap.fetch_count).must_equal before + 1
+    end
+  end
+
   describe '#body' do
     it 'returns the message body' do
       _(imap_message.body).must_match(/Mock body/)
@@ -85,6 +109,111 @@ describe Imap::Message do
     end
   end
 
+  describe '#uid, #flags, #seen? and #size' do
+    it 'reads what the one fetch brought back' do
+      _(imap_message.uid).must_equal 1001
+      _(imap_message.flags).must_equal []
+      _(imap_message.seen?).must_equal false
+      _(imap_message.size).must_equal 2048
+    end
+
+    it 'is seen where the flags say so' do
+      _(Imap::Message.new(2, client).seen?).must_equal true
+    end
+
+    it 'asks for them in the same fetch as the envelope' do
+      imap_message.uid
+      imap_message.subject
+      _(client.imap.fetch_count).must_equal 1
+      _(client.imap.fetched_attrs).must_equal Imap::Message::FETCH_ATTRIBUTES
+    end
+  end
+
+  describe '#received_at and #sent_at' do
+    it 'returns the internaldate as a Time, zone and all' do
+      _(imap_message.received_at).must_be_instance_of Time
+      _(imap_message.received_at.year).must_equal 2026
+      _(imap_message.received_at.utc_offset).must_equal 10 * 3600
+    end
+
+    it 'returns the envelope date as a Time' do
+      _(imap_message.sent_at).must_be_instance_of Time
+      _(imap_message.sent_at.min).must_equal 5
+    end
+  end
+
+  describe '#cc' do
+    it 'returns an array of addresses' do
+      _(imap_message.cc).must_equal ['copied@example.com']
+    end
+
+    it 'is empty where the envelope carries no cc' do
+      _(Imap::Message.new(99, client).cc).must_equal []
+    end
+  end
+
+  describe '#from_address and #from_domain' do
+    it 'drops the display name and downcases the domain' do
+      message = Imap::Message.new(2, client)
+      _(message.from).must_equal 'Café <sender@Example.COM>'
+      _(message.from_address).must_equal 'sender@Example.COM'
+      _(message.from_domain).must_equal 'example.com'
+    end
+  end
+
+  describe '.decode' do
+    it 'decodes a base64 encoded word' do
+      _(Imap::Message.decode('=?UTF-8?B?UsOpc3Vtw6k=?=')).must_equal 'Résumé'
+    end
+
+    it 'decodes a quoted-printable encoded word, underscore for space' do
+      _(Imap::Message.decode('=?UTF-8?Q?Caf=C3=A9_au_lait?=')).must_equal 'Café au lait'
+    end
+
+    it 'drops the whitespace between adjacent encoded words' do
+      _(Imap::Message.decode('=?UTF-8?Q?Caf=C3=A9?= =?UTF-8?Q?_au_lait?=')).must_equal 'Café au lait'
+    end
+
+    it 'leaves plain text alone and nil nil' do
+      _(Imap::Message.decode('Plain subject')).must_equal 'Plain subject'
+      _(Imap::Message.decode(nil)).must_be_nil
+    end
+
+    it 'returns the text where the charset is not one Ruby knows' do
+      _(Imap::Message.decode('=?NOSUCHSET?B?UsOpc3Vtw6k=?=')).must_equal 'UsOpc3Vtw6k='
+    end
+  end
+
+  describe '#subject' do
+    it 'decodes the encoded word' do
+      _(Imap::Message.new(2, client).subject).must_equal 'Résumé for review'
+    end
+  end
+
+  describe '#attachment? and #attachment_filenames' do
+    it 'names the attachment from the disposition' do
+      _(imap_message.attachment?).must_equal true
+      _(imap_message.attachment_filenames).must_equal ['invoice.pdf']
+    end
+
+    it 'falls back to the content type NAME and decodes it' do
+      _(Imap::Message.new(2, client).attachment_filenames).must_equal ['résumé.pdf']
+    end
+
+    it 'has none where the bodystructure carries none' do
+      message = Imap::Message.new(3, client)
+      _(message.attachment_filenames).must_equal []
+      _(message.attachment?).must_equal false
+    end
+  end
+
+  describe '#source' do
+    it 'peeks at the whole message, headers and all' do
+      _(imap_message.source).must_match(/\ASubject: /)
+      _(client.imap.fetched_attrs).must_include 'BODY.PEEK[]'
+    end
+  end
+
   describe '#urls' do
     it 'extracts http URLs from body' do
       imap_message.stub(:body, 'Check out http://example.com and https://test.com') do

data/test/Imap_test.rb

@@ -70,6 +70,16 @@ describe Imap do
     end
   end
 
+  describe "#mailboxes" do
+    it 'names every mailbox which can be selected' do
+      _(client.mailboxes).must_equal ['INBOX', 'Archive/2026']
+    end
+
+    it 'leaves out the \\Noselect containers' do
+      _(client.mailboxes).wont_include 'Archive'
+    end
+  end
+
   describe "#search" do
     it 'returns an array of Message objects' do
       Net::IMAP.stub(:new, MockIMAP.new('imap.example.com', ssl: true)) do

data/test/test_helper.rb

@@ -31,11 +31,25 @@ class MockIMAP
     [1, 2, 3]
   end
 
+  attr_reader :list_called
   attr_reader :fetched_attrs
+  attr_reader :fetch_count
 
-  def fetch(message_id, attrs)
+  def fetch(message_ids, attrs)
     @fetched_attrs = attrs
-    [OpenStruct.new(attr: mock_fetch_attrs(message_id, attrs))]
+    @fetch_count = (@fetch_count || 0) + 1
+    Array(message_ids).collect do |message_id|
+      Net::IMAP::FetchData.new(message_id, mock_fetch_attrs(message_id, attrs))
+    end
+  end
+
+  def list(reference, pattern)
+    @list_called = true
+    [
+      OpenStruct.new(attr: [:Hasnochildren], name: 'INBOX'),
+      OpenStruct.new(attr: [:Noselect, :Haschildren], name: 'Archive'),
+      OpenStruct.new(attr: [:Hasnochildren], name: 'Archive/2026')
+    ]
   end
 
   def store(message_id, flags, values); end
@@ -63,16 +77,63 @@ class MockIMAP
       case attr
       when 'BODY[TEXT]', 'BODY.PEEK[TEXT]'
         result['BODY[TEXT]'] = "Mock body for message #{message_id}"
-      when 'BODY[HEADER.FIELDS (SUBJECT)]'
-        result['BODY[HEADER.FIELDS (SUBJECT)]'] = "Subject: Mock Subject #{message_id}\r\n"
-      when 'BODY[HEADER.FIELDS (FROM)]'
-        result['BODY[HEADER.FIELDS (FROM)]'] = "From: sender@example.com\r\n"
+      when 'BODY[]', 'BODY.PEEK[]'
+        result['BODY[]'] = "Subject: Mock Subject #{message_id}\r\n\r\nMock body for message #{message_id}"
+      when 'UID'
+        result['UID'] = 1000 + message_id
+      when 'FLAGS'
+        result['FLAGS'] = message_id == 2 ? [:Seen] : []
+      when 'INTERNALDATE'
+        result['INTERNALDATE'] = '17-Sep-2026 09:15:00 +1000'
+      when 'RFC822.SIZE'
+        result['RFC822.SIZE'] = 2048
+      when 'BODYSTRUCTURE'
+        result['BODYSTRUCTURE'] = mock_bodystructure(message_id)
       when 'ENVELOPE'
-        result['ENVELOPE'] = OpenStruct.new(
-          to: message_id == 99 ? nil : [OpenStruct.new(mailbox: 'user', host: 'example.com')]
-        )
+        result['ENVELOPE'] = mock_envelope(message_id)
       end
     end
     result
   end
+
+  # Message 2 carries the encoded words, message 99 the empty envelope.
+  def mock_envelope(message_id)
+    OpenStruct.new(
+      date: 'Thu, 17 Sep 2026 09:05:00 +1000',
+      subject: message_id == 2 ? '=?UTF-8?B?UsOpc3Vtw6kgZm9yIHJldmlldw==?=' : "Mock Subject #{message_id}",
+      from: [
+        message_id == 2 ?
+          OpenStruct.new(name: '=?UTF-8?Q?Caf=C3=A9?=', mailbox: 'sender', host: 'Example.COM') :
+          OpenStruct.new(mailbox: 'sender', host: 'example.com')
+      ],
+      to: message_id == 99 ? nil : [OpenStruct.new(mailbox: 'user', host: 'example.com')],
+      cc: message_id == 99 ? nil : [OpenStruct.new(mailbox: 'copied', host: 'example.com')]
+    )
+  end
+
+  # Message 1 names its attachment in the disposition, message 2 in the content
+  # type's NAME and in an encoded word, the rest carry none.
+  def mock_bodystructure(message_id)
+    text = OpenStruct.new(media_type: 'TEXT', subtype: 'PLAIN', param: nil, disposition: nil)
+    case message_id
+    when 1
+      attachment = OpenStruct.new(
+        media_type: 'APPLICATION',
+        subtype: 'PDF',
+        param: {'NAME' => 'ignored.pdf'},
+        disposition: OpenStruct.new(dsp_type: 'ATTACHMENT', param: {'FILENAME' => 'invoice.pdf'})
+      )
+      OpenStruct.new(media_type: 'MULTIPART', subtype: 'MIXED', parts: [text, attachment])
+    when 2
+      attachment = OpenStruct.new(
+        media_type: 'APPLICATION',
+        subtype: 'PDF',
+        param: {'NAME' => '=?UTF-8?B?csOpc3Vtw6kucGRm?='},
+        disposition: nil
+      )
+      OpenStruct.new(media_type: 'MULTIPART', subtype: 'MIXED', parts: [text, attachment])
+    else
+      text
+    end
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: imap.rb
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.7.0
 platform: ruby
 authors:
 - thoran