mms2r 1.0.7 → 1.1.0

data/History.txt

@@ -1,3 +1,21 @@
+== 1.1.0 / 2007-06-08 (Toki Wartooth)
+
+* get_body to return body text (Luke Francl)
+* get_subject returns "" for default subjects now
+* default subjects listed in yaml by carrier in conf directory
+* added granularity to Cingular, Sprint, and Verizon carrier services (Will Jessup)
+* refactored Sprint instance to process all media (Will Jessup + Mike)
+* optimized text transformations (Will Jessup)
+* properly handle ISO-8859-1 and UTF-8 text (Will Jessup)
+* autotest powers activate! (ZenTest autotest discovery enabled)
+* configuration file ignores, transforms, and subjects all store Regexp's
+* Put vendor Text::Format & TMail::Mail as an external subversion dependency 
+  to the 1.2 stable branch of Rails ActionMailer
+* added get_number method to return the phone number associated with this MMS
+* get_media and get_text attachment_fu helper return the largest piece of media
+  of that type if the more than one exits in the media (Luke Francl)
+* added block support to process() method (Shane Vitarana)
+
 == 1.0.7 / 2007-04-27 (Senator Stampingston)
 
 * patch submitted by Luke Francl
@@ -20,7 +38,7 @@
 * made ignore_media? start its text check from the start of the file (Luke)
 * added new text transform for Verizon messages (Luke)
 * updated Nextel ignore conf (Luke)
-* added addtional samples and tests for T-Mobile & Verizon (Luke)
+* added additional samples and tests for T-Mobile & Verizon (Luke)
 * cleaned up MMS2R::Media documentation
 * added Sprint broken image test for when media goes stale on their content server
 * fixed teardown typo in lots of plases

data/Manifest.txt

@@ -2,22 +2,31 @@ History.txt
 Manifest.txt
 README.txt
 Rakefile
-conf/mms2r_cingularmedia_transform.yml
-conf/mms2r_nextelmedia_ignore.yml
-conf/mms2r_sprintmedia_ignore.yml
-conf/mms2r_tmobilemedia_ignore.yml
-conf/mms2r_verizonmedia_ignore.yml
-conf/mms2r_verizonmedia_transform.yml
+conf/mms2r_cingular_me_media_subject.yml
+conf/mms2r_cingular_me_media_transform.yml
+conf/mms2r_dobson_media_ignore.yml
+conf/mms2r_media_ignore.yml
+conf/mms2r_media_subject.yml
+conf/mms2r_my_cingular_media_subject.yml
+conf/mms2r_nextel_media_ignore.yml
+conf/mms2r_sprint_media_ignore.yml
+conf/mms2r_sprint_media_subject.yml
+conf/mms2r_t_mobile_media_ignore.yml
+conf/mms2r_verizon_media_ignore.yml
+conf/mms2r_verizon_media_transform.yml
+dev_tools/debug_sprint_hpricot_parsing.rb
 lib/mms2r.rb
-lib/mms2r/cingular_media.rb
+lib/mms2r/cingular_me_media.rb
 lib/mms2r/dobson_media.rb
+lib/mms2r/m_mode_media.rb
 lib/mms2r/media.rb
-lib/mms2r/mmode_media.rb
+lib/mms2r/my_cingular_media.rb
 lib/mms2r/nextel_media.rb
 lib/mms2r/sprint_media.rb
-lib/mms2r/tmobile_media.rb
+lib/mms2r/sprint_pcs_media.rb
+lib/mms2r/t_mobile_media.rb
 lib/mms2r/verizon_media.rb
-lib/mms2r/version.rb
+lib/mms2r/vtext_media.rb
 lib/vendor/text/format.rb
 lib/vendor/tmail.rb
 lib/vendor/tmail/address.rb
@@ -42,23 +51,28 @@ lib/vendor/tmail/scanner_r.rb
 lib/vendor/tmail/stringio.rb
 lib/vendor/tmail/tmail.rb
 lib/vendor/tmail/utils.rb
-test/files/cingular-image-01.mail
 test/files/cingularme-text-01.mail
+test/files/cingularme-text-02.mail
 test/files/dobson-image-01.mail
 test/files/dot.jpg
+test/files/hello_world_empty_text.mail
 test/files/hello_world_mail_multipart.mail
 test/files/hello_world_mail_plain_no_content_type.mail
 test/files/hello_world_mail_plain_with_content_type.mail
 test/files/mmode-image-01.mail
+test/files/mycingular-image-01.mail
 test/files/nextel-image-01.mail
 test/files/nextel-image-02.mail
 test/files/nextel-image-03.mail
 test/files/nextel-image-04.mail
+test/files/simple-with-two-images-two-texts.mail
 test/files/simple_image.mail
 test/files/simple_multipart_alternative.mail
+test/files/sprint-broken-image-01.mail
 test/files/sprint-image-01.mail
-test/files/sprint-image-02.mail
+test/files/sprint-pcs-text-01.mail
 test/files/sprint-text-01.mail
+test/files/sprint-two-images-01.mail
 test/files/sprint-video-01.mail
 test/files/sprint.mov
 test/files/tmobile-image-01.mail
@@ -69,11 +83,16 @@ test/files/verizon-text-01.mail
 test/files/verizon-video-01.mail
 test/files/vtext-text-01.mail
 test/test_helper.rb
-test/test_mms2r_cingular.rb
-test/test_mms2r_dobson.rb
+test/test_mms2r_cingular_me_media.rb
+test/test_mms2r_dobson_media.rb
+test/test_mms2r_m_mode_media.rb
 test/test_mms2r_media.rb
-test/test_mms2r_mmode.rb
-test/test_mms2r_nextel.rb
-test/test_mms2r_sprint.rb
-test/test_mms2r_tmobile.rb
-test/test_mms2r_verizon.rb
+test/test_mms2r_my_cingular_media.rb
+test/test_mms2r_nextel_media.rb
+test/test_mms2r_sprint_media.rb
+test/test_mms2r_sprint_pcs_media.rb
+test/test_mms2r_t_mobile_media.rb
+test/test_mms2r_verizon_media.rb
+test/test_mms2r_vtext_media.rb
+vendor/plugins/mms2r/lib/autotest/discover.rb
+vendor/plugins/mms2r/lib/autotest/mms2r.rb

data/README.txt

@@ -8,7 +8,7 @@ MMS2R is a library that decodes the parts of an MMS message to disk while
 stripping out advertising injected by the cellphone carriers.  MMS messages are 
 multipart email and the carriers often inject branding into these messages.  Use
 MMS2R if you want to get at the real user generated content from a MMS without
-having to deal with the garbage from the carriers.
+having to deal with the cruft from the carriers.
 
 If MMS2R is not aware of a particular carrier no extra processing is done 
 to the MMS other than decoding and consolidating its media.
@@ -31,15 +31,18 @@ Corpus of carriers currently processed by MMS2R:
 
 == FEATURES
 
-* TMail from 1.3.1 of ActionMailer is shipped as a vendor library with MMS2R
 * get_media and get_text methods return a File that can be used in attachment_fu 
+
 == SYNOPSIS:
 
+  # required to use the MMS2R gem proper
   require 'rubygems'
   require 'mms2r'
-  require 'mms2r/media'
+
+  # required for the example
   require 'tmail'
   require 'fileutils'
+  require 'logger'
 
   # TMail::Mail.parse is what ActionMailer::Base.receive(email) does, see:
   # http://wiki.rubyonrails.com/rails/pages/HowToReceiveEmailsWithActionMailer
@@ -50,14 +53,17 @@ Corpus of carriers currently processed by MMS2R:
   # writes the user generated media to disk in a temporary subdirectory
   mms.process
 
-  puts "MMS has default carrier subject!" if mms.get_subject.nil?
+  puts "MMS has default carrier subject!" if mms.get_subject.eql?("")
+
+  # access the senders phone number
+  puts "MMS was from phone #{mms.get_number}"
 
-  # most MMS are either image or video, get_media will blindly return
-  # the first video or image found
+  # most MMS are either image or video, get_media will return the largest
+  # (non-advertising) video or image found
   file = mms.get_media
   puts "MMS had a media: #{file.inspect}" unless file.nil?
 
-  # get_text will blindly return the first (non-advertising) text found
+  # get_text return the largest (non-advertising) text found
   file = mms.get_text
   puts "MMS had some text: #{file.inspect}" unless file.nil?
 
@@ -80,6 +86,14 @@ Corpus of carriers currently processed by MMS2R:
   #remove all the media that was put to temporary disk
   mms.purge
 
+  # Block support, process and receive all media types of video.
+  # Purge is called at the conclusion of the block so be sure
+  # to do something with the bits you are looking for
+  mms.process do |media_type, files|
+    # assumes a Clip model
+    Clip.create(:uploaded_data => files.first, :title => "From phone") if media_type =~ /video/
+  end
+
 == REQUIREMENTS:
 
 * Hpricot
@@ -88,15 +102,33 @@ Corpus of carriers currently processed by MMS2R:
 
 * sudo gem install mms2r
 
+== CONTRIBUTE:
+
+If you contribute a patch that we accept then generally we'll
+give you developer rights for the project on RubyForge.  Please
+ensure your work includes 100% test converage.  Your text 
+coverage can be verified with the rcov rake task.  The library
+is ZenTest autotest discovery enabled so running autotest in the
+root of the project is very helpful during development.
+
 == Authors
 
 Copyright (c) 2007 by Mike Mondragon (blog[http://blog.mondragon.cc/])
 
+MMS2R's Flickr page[http://www.flickr.com/photos/8627919@N05/]
+
+== Contributors
+
+* Luke Francl (blog[http://railspikes.com/])
+* Will Jessup (blog[http://www.willjessup.com/])
+* Shane Vitarana (blog[http://www.shanesbrain.net/])
+
 == LICENSE:
 
 (The MIT License)
 
-Copyright (c) 2007 Mike Mondragon.  All rights reserved.
+Copyright (c) 2007 Mike Mondragon (mikemondragon@gmail.com).
+All rights reserved.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/Rakefile

@@ -11,8 +11,8 @@ require 'mms2r/media'
 Hoe.new('mms2r', MMS2R::Media::VERSION) do |p|
   p.rubyforge_name = 'mms2r'
   p.author = 'Mike Mondragon'
-  p.email = 'mike@mondragon.cc'
-  p.summary = 'Extract media from MMS'
+  p.email = 'mikemondragon@gmail.com'
+  p.summary = 'Extract user media from MMS (and not carrier cruft)'
   p.description = p.paragraphs_of('README.txt', 2..6).join("\n\n")
   p.url = p.paragraphs_of('README.txt', 1).first.strip
   p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")

data/conf/mms2r_cingular_me_media_subject.yml

@@ -0,0 +1,3 @@
+---
+- !ruby/regexp /^Multimedia message$/
+

data/conf/{mms2r_cingularmedia_transform.yml → mms2r_cingular_me_media_transform.yml}

@@ -4,3 +4,4 @@ text/plain:
     ===============================================\n\
     Brought to you by, Cingular Wireless Messaging\n\
     http://www.CingularMe.COM/": ""
+

data/conf/mms2r_dobson_media_ignore.yml

@@ -0,0 +1,4 @@
+--- 
+application/smil: 
+- !ruby/regexp /^<smil><head><layout><root-layout width=/
+

data/conf/mms2r_media_ignore.yml

@@ -0,0 +1,4 @@
+---
+text/plain:
+- !ruby/regexp /^$/
+

data/conf/mms2r_media_subject.yml

@@ -0,0 +1,3 @@
+---
+- !ruby/regexp /^\(no subject\)$/
+

data/conf/mms2r_my_cingular_media_subject.yml

@@ -0,0 +1,3 @@
+---
+- !ruby/regexp /^Multimedia message$/
+

data/conf/mms2r_nextel_media_ignore.yml

@@ -0,0 +1,11 @@
+--- 
+text/plain: 
+- !ruby/regexp /^This is an MMS message. 20\d\d Sprint Nextel. All rights reserved./
+- !ruby/regexp /^This is an MMS message. Please go to http:\/\/mms.nextel.com\/mms\/anon\/do\/legacy\/viewDirect/
+- !ruby/regexp /^This is a multi-part message in MIME format./
+text/html: 
+- !ruby/regexp /^<html><body> This is an MMS message. <br><center> 20\d\d Sprint Nextel. All rights reserved <br><\/center> <\/body><\/html>/
+- !ruby/regexp /^<html><body> <center>This is an MMS message. <br><center> 20\d\d Sprint Nextel. All rights reserved<\/center><br><img/
+- !ruby/regexp /^<html><body> This is an MMS message. <br><center> 20\d\d Sprint Nextel. All rights reserved<\/center> <\/body><\/html>/
+- !ruby/regexp /^<html><body> This is an MMS message. Please go to <a href="http:\/\/mms.nextel.com\/mms\/anon\/do\/legacy\/viewDirect/
+

data/conf/mms2r_sprint_media_ignore.yml

@@ -0,0 +1,5 @@
+---
+text/plain:
+- !ruby/regexp /^You have new Picture Mail! Click Go/View to see now./
+- !ruby/regexp /^You have new Video Mail! Click Go/View to see now./
+

data/conf/mms2r_sprint_media_subject.yml

@@ -0,0 +1,3 @@
+---
+- !ruby/regexp /^You have new Picture Mail!$/
+

data/conf/{mms2r_tmobilemedia_ignore.yml → mms2r_t_mobile_media_ignore.yml}

@@ -1,9 +1,6 @@
 --- 
-text/html: 
-- |
-  <html>
-    <head>
-  		<title>T-Mobile</title> 
+text/html:
+- !ruby/regexp /^<html> <head> <title>T-Mobile</title>/
 image/gif: 
 - dottedline350.gif
 - dottedline600.gif

data/conf/mms2r_verizon_media_ignore.yml

@@ -0,0 +1,4 @@
+--- 
+text/plain: 
+- !ruby/regexp /^This message was sent using PIX-FLIX Messaging service from Verizon Wireless!.*$/
+

data/conf/mms2r_verizon_media_transform.yml

@@ -0,0 +1,6 @@
+--- 
+text/plain: 
+  ? !ruby/regexp /^(.+?)\s+This message was sent using PIX-FLIX Messaging service from Verizon Wireless!.*/m
+  : \1
+
+

data/dev_tools/debug_sprint_hpricot_parsing.rb

@@ -0,0 +1,82 @@
+require 'rubygems'
+require 'hpricot'
+require 'net/http'
+require 'rubygems'
+require 'hpricot'
+require 'pp'
+
+if ARGV[0].nil?
+  puts "please execute with a file path to a Sprint HTML file that was extracted from a MMS"
+  puts "ruby #{$0} MYFILE"
+  exit
+end
+
+doc = open(ARGV[0]) { |f| Hpricot(f) }
+
+puts "TITLE: #{doc.at('title').inner_html}"
+
+#phone number is tucked away in the comment in the head
+c = doc.search("/html/head/comment//").last
+t = c.content.gsub(/\s+/m," ").strip
+number = / name="MDN">(\d+)</.match(t)[1]
+puts "NUMBER: #{number}"
+
+#if there is a text message with the MMS its in the
+#inner html of the only pre on the page
+text = doc.search("/html/body//pre").first.inner_html
+puts "TEXT: #{text}"
+
+# just see what they say this MMS is it really doesn't
+# mean anything, the content is in paux image with 
+# a RECIPIENT in its URI path
+trs = doc.search("/html/body//tr")
+text = trs[3].search("/td/p/font/b/")
+case text.text
+when /You have a Video Mail from/
+  puts "it claims to be a video"
+when /You have a Picture Mail from /
+  puts "it claims to be an image"
+else
+  puts "what is it? #{text.text}"
+end
+
+# group all the images together
+srcs = Array.new
+imgs = doc.search("/html/body//img")
+imgs.each do |i|
+  src = i.attributes['src']
+  #next unless /pictures.sprintpcs.com\/+mmps\/RECIPIENT\//.match(src)
+  #we don't want to double fetch content and we only
+  #want to fetch media from the content server, you get
+  #a clue about that as there is a RECIPIENT in the URI path
+  next unless /mmps\/RECIPIENT\//.match(src)
+  next if srcs.detect{|s| s.eql?(src)}
+  srcs << src
+end
+
+# now fetch the media
+puts "there are #{srcs.size} sources to fetch"
+cnt = 0
+srcs.each do |src|
+  puts "--"
+  puts "FETCHING:\n  #{src}"
+
+   url = URI.parse(src)
+   #res = Net::HTTP.get_response(url)
+   agent = "Mozilla/5.0 (X11; U; Minix3 i686 (x86_64); en-US; rv:1.8.1.1) Gecko/20061208 Firefox/2.0.0.1"
+   res = Net::HTTP.start(url.host, url.port) { |http|
+     req = Net::HTTP::Get.new(url.request_uri, {'User-Agent' => agent})
+     http.request(req)
+   }
+
+  # prep and write a file
+  base = /\/RECIPIENT\/([^\/]+)\//.match(src)[1]
+  ext = /^[^\/]+\/(.+)/.match(res.content_type)[1]
+  file_name ="#{base}.#{cnt}.#{ext}"
+  puts "writing file: #{file_name}"
+  File.open(file_name,'w'){ |f| f.write(res.body) }
+  puts "file is sized #{File.size(file_name)}"
+  cnt = cnt + 1
+end
+
+puts "no images or video" if srcs.size == 0

data/lib/mms2r.rb

@@ -1,3 +1,64 @@
+#--
+# Copyright (c) 2007 by Mike Mondragon (mikemondragon@gmail.com)
+#
+# Please see the LICENSE file for licensing information.
+#++
+
 $:.unshift(File.dirname(__FILE__) + "/vendor/")
+require 'mms2r/media'
+require 'mms2r/cingular_me_media'
+require 'mms2r/dobson_media'
+require 'mms2r/m_mode_media'
+require 'mms2r/my_cingular_media'
+require 'mms2r/nextel_media'
+require 'mms2r/sprint_media'
+require 'mms2r/sprint_pcs_media'
+require 'mms2r/t_mobile_media'
+require 'mms2r/verizon_media'
+require 'mms2r/vtext_media'
+
 module MMS2R
+
+  ##
+  # A hash of carriers that MMS2r is currently aware of.
+  # The factory create method uses the hostname portion
+  # of an MMS's From header to select the correct type
+  # of MMS2R::Media product.  If a specific media product
+  # is not available MMS2R::Media should be used.
+
+  CARRIER_CLASSES = {
+    'cingularme.com' => MMS2R::CingularMeMedia,
+    'mms.dobson.net' => MMS2R::DobsonMedia,
+    'mmode.com' => MMS2R::MModeMedia,
+    'mms.mycingular.com' => MMS2R::MyCingularMedia,
+    'messaging.nextel.com' => MMS2R::NextelMedia,
+    'pm.sprint.com' => MMS2R::SprintMedia,
+    'messaging.sprintpcs.com' => MMS2R::SprintPcsMedia,
+    'tmomail.net' => MMS2R::TMobileMedia,
+    'vzwpix.com' => MMS2R::VerizonMedia,
+    'vtext.com' => MMS2R::VtextMedia
+  }
+
+  ##
+  # A hash of file extensions for common mimetypes
+
+  EXT = {
+    'text/plain' => 'txt',
+    'text/html' => 'html',
+    'image/png' => 'png',
+    'image/gif' => 'gif',
+    'image/jpeg' => 'jpg',
+    'video/quicktime' => 'mov',
+    'video/3gpp2' => '3g2'
+  }
+
+  class MMS2R::Media
+
+    ##
+    # MMS2R Library version
+
+    VERSION = '1.1.0'
+
+    end
+
 end