glued 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6525daf00b7018118ea699e561d5da854e6b2c6b
+  data.tar.gz: 2f88d6c7f82b4e132623840697f985d0dd02ce91
+SHA512:
+  metadata.gz: dfbbc52f1ca331675ec8e488cdd395876afc88bdc001d2623f24b48b7475d902a609cf999063663dac85051254913fffa7a0502b89fbf4a8bf3e5c27f80bed67
+  data.tar.gz: 71ac92849df89959ffe10b9191be2375845b32542e5be1de0405f08aa1abb5f6419b3d0c320caea76c22f4945230dd82181850f99d02ab18712c1a7941dbfc51

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,34 @@
+PATH
+  remote: .
+  specs:
+    glued (0.1.0)
+      curb
+      nokogiri
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    curb (0.8.5)
+    diff-lcs (1.2.5)
+    mini_portile (0.5.2)
+    nokogiri (1.6.1)
+      mini_portile (~> 0.5.0)
+    observr (1.0.5)
+    rake (0.9.6)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.5)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.5)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  glued!
+  observr
+  rake
+  rspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) MMXIV The original author or authors
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,76 @@
+# Glued
+
+[![Build Status](https://travis-ci.org/simongregory/glued.png?branch=master)](https://travis-ci.org/simongregory/glued)
+
+A Ruby client to download [HDS][hds] *recorded* fragments and glue them together. If it's live content you want then you need to start coding.  
+
+The primary aim is to download video on demand content. Encrypted content will download, however it will not playback unless your player accomodates it and a license is available for the media.  
+
+[HDS][hds] is designed for live media that dynamically adapts to network conditions and avoids buffering. It uses the HTTP protocol to benefit from existing network infastructre (ie [CDN][cdn]s). It is similar to [MPEG-DASH][mpeg-dash] and [HLS][hls].
+
+## References
+
+### Specification Documents
+
+* [Flash Media Manifest Specification][f4m-spec]
+* [Flash Video Format Specification][f4v-spec]
+
+### Code
+
+* [OSMF][] [httpstreaming][osmf-httpstreaming] (AS3)
+* [OSMF][] [f4mClasses][osmf-f4mclasses] with [F4MLoader][osmf-f4mloader] as the entry point. (AS3)
+* [HDS Union](https://github.com/AndyA/hds_union) (perl)
+* [K-S-V Scripts](https://github.com/K-S-V/Scripts/blob/master/AdobeHDS.php) (php)
+
+### Etc
+
+* [Getting started with HDS](http://www.thekuroko.com/http-dynamic-streaming-getting-started/)
+* [Fragmented MP4](http://technology-pedia.blogspot.co.uk/2012/09/fragmented-mp4-format-fmp4-f4f-adobe.html)
+
+* [flvtool2](https://github.com/unnu/flvtool2)
+* [get-flash-videos](https://github.com/monsieurvideo/get-flash-videos)
+* [rtmp-dump](http://rtmpdump.mplayerhq.hu)
+* [get_iplayer](https://github.com/dinkypumpkin/get_iplayer/)
+
+### Questions / Answers / Guesses
+
+A *box* is a convention within a stream of bytes. Each box starts with a header, which describes length and type, followed by the data in the box.
+
+### Abbreviations
+
+HDS - HTTP Dynamic Streaming  
+F4M - Flash Media Manfiest  
+F4F - Flash Media File fragment  
+F4X - Flash Media Index file  
+F4V - H.264/AAC based content  
+FLV - Other flash supported codecs  
+
+## How it works
+
+Load the f4m manifest  
+    - Bad url? go 💥  
+    - Detect any alternate bitrate manifests? go 💥  
+    - Detect a live stream? go 💥  
+    - Automatically picks the higest bitrate stream  
+Builds a list of fragment urls  
+Download each fragment  
+    - Fragment fails? go 💥  
+Glue it to the previous fragment  
+Repeat till done  
+
+## License
+
+[MIT][], see accomanying [LICENSE](LICENSE) document
+
+[ruby]: https://www.ruby-lang.org
+[hds]: http://www.adobe.com/uk/products/hds-dynamic-streaming.html "Adobe HTTP Dynamic Streaming"
+[cdn]: http://en.wikipedia.org/wiki/Content_delivery_network
+[hls]: http://en.wikipedia.org/wiki/HTTP_Live_Streaming
+[mpeg-dash]: http://en.wikipedia.org/wiki/MPEG_DASH
+[OSMF]: http://osmf.org/ "Open Source Media Framework"
+[osmf-httpstreaming]: http://opensource.adobe.com/svn/opensource/osmf/trunk/framework/OSMF/org/osmf/net/httpstreaming/
+[osmf-f4mclasses]: http://opensource.adobe.com/svn/opensource/osmf/trunk/framework/OSMF/org/osmf/elements/f4mClasses/
+[osmf-f4mloader]: http://opensource.adobe.com/svn/opensource/osmf/trunk/framework/OSMF/org/osmf/elements/F4MLoader.as
+[f4m-spec]: doc/adobe-media-manifest-specification.pdf 
+[f4v-spec]: doc/adobe-flash-video-file-format-spec.pdf
+[MIT]: http://opensource.org/licenses/MIT

data/Rakefile

@@ -0,0 +1,10 @@
+require 'rspec'
+require 'rspec/core/rake_task'
+
+desc "Run all examples"
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = %w[--color]
+end
+
+task :t => [:spec]
+task :default => [:spec]

data/bin/glued

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require 'glued'
+
+puts %{Ruby HDS Glue #{Glued::VERSION}
+
+Fetch it, glue it, watch it.
+
+}
+
+begin
+  Glue.new(ARGV[0])
+rescue Exception => e
+  abort "Glue failed because: #{e.message}"
+end

data/glued.watchr

@@ -0,0 +1,3 @@
+watch( '.*\.rb' ) do
+  system 'rspec --color'
+end

data/lib/glued.rb

@@ -0,0 +1,13 @@
+# encoding: utf-8
+
+require 'curb'
+require 'base64'
+require 'nokogiri'
+
+require 'glued/bootstrap'
+require 'glued/f4f'
+require 'glued/f4m'
+require 'glued/f4vio'
+require 'glued/glue'
+require 'glued/grabber'
+require 'glued/version'

data/lib/glued/bootstrap.rb

@@ -0,0 +1,196 @@
+# encoding: utf-8
+
+class Bootstrap
+  attr_reader :boxes
+
+  def initialize(data)
+    @reader = F4VIO.new(data)
+    @boxes = []
+    scan
+  end
+
+  #Top level
+  AFRA = 'afra' #Fragment random access for HTTP streaming
+  ABST = 'abst' #Bootstrap info for HTTP streaming
+  MOOV = 'moov' #Container for structural metadata
+  MOOF = 'moof' #Movie Fragment
+  MDAT = 'mdat' #Moovie data container
+
+  #Inside ABST
+  ASRT = 'asrt' #Segment run table box
+  AFRT = 'afrt' #Fragment runt table box
+
+  def segments
+    @boxes.first.segments
+  end
+
+  def fragments
+    @boxes.first.segment_run_tables.first.run_entry_table.first.fragments_per_segment
+  end
+
+  private
+
+  def scan
+    # Scan for 'boxes' in the stream see spec 1.3 F4V box format
+    until (@reader.eof?) do
+      box = get_box_info
+
+      case box.type
+      when ABST
+        @boxes << get_bootstrap_box(box)
+      when AFRA
+        @boxes << box
+      when MDAT
+        @boxes << box
+      else
+        break;
+      end
+    end
+
+    raise "Computer says no" if @boxes.empty?
+
+    @boxes
+  end
+
+  def get_box_info
+    pos = @reader.pos
+    size = @reader.int32
+    type = @reader.fourCC
+    size = @reader.int64 if size == 1 #For boxes over 4GB the size is moved here.
+
+    Header.new(pos, size, type)
+  end
+
+  def get_bootstrap_box(header)
+    # 2.11.2 Bootstrap Info box
+    b                        = BootstrapBox.new
+    b.header                 = header
+    b.version                = @reader.byte
+    b.flags                  = @reader.int24
+    b.bootstrap_info_version = @reader.int32
+
+    plu                      = @reader.byte
+    b.profile                = plu >> 6
+    b.live                   = (plu & 0x20) ? 1 : 0
+    b.update                 = (plu & 0x01) ? 1 : 0
+
+    b.time_scale             = @reader.int32
+    b.current_media_time     = @reader.int64
+    b.smpte_timecode_offset  = @reader.int64
+    b.movie_identifier       = @reader.string
+    b.servers                = @reader.byte_ar
+    b.quality                = @reader.byte_ar
+    b.drm_data               = @reader.string
+    b.metadata               = @reader.string
+    b.segments               = @reader.byte
+    b.segment_run_tables     = []
+    b.segments.times { b.segment_run_tables << get_asrt_box(get_box_info) }
+
+    raise "There should be at least one segment entry" if b.segment_run_tables.empty?
+
+    b.fragments               = @reader.byte
+    b.fragment_run_tables     = []
+    b.fragments.times { b.fragment_run_tables << get_afrt_box(get_box_info) }
+
+    raise "There should be at least one fragment entry" if b.fragment_run_tables.empty?
+
+    b
+  end
+
+  def get_asrt_box(header)
+    # 2.11.2.1 Segment Run Table box
+    raise "Unexpected segment run table box header '#{header.type}' instead of '#{ASRT}'" unless header.type == ASRT
+
+    b = RunTableBox.new
+    b.header                        = header
+    b.version                       = @reader.byte
+    b.flags                         = @reader.int24
+    b.quality_segment_url_modifiers = @reader.byte_ar
+
+    table = []
+    runs = @reader.int32
+
+    runs.times do
+      first_segment = @reader.int32
+      fragments_per_segment = @reader.int32
+
+      table << SegmentRunEntry.new(first_segment, fragments_per_segment)
+    end
+
+    b.run_entry_table = table
+    b
+  end
+
+  def get_afrt_box(header)
+    # 2.11.2.2 Fragment Run Table box
+    raise "Unexpected fragment run table box header '#{header.type}' instead of '#{AFRT}'" unless header.type == AFRT
+
+    b = RunTableBox.new
+    b.header                        = header
+    b.version                       = @reader.byte
+    b.flags                         = @reader.int24
+    b.time_scale                    = @reader.int32
+    b.quality_segment_url_modifiers = @reader.byte_ar
+
+    table = []
+    runs = @reader.int32
+
+    runs.times do
+      f = FragmentRunEntry.new
+      f.first_fragment = @reader.int32
+      f.first_fragment_timestamp = @reader.int64
+      f.fragment_duration = @reader.int32
+      f.discontinuity_indicator = @reader.byte if f.fragment_duration == 0
+
+      table << f
+    end
+
+    b.run_entry_table = table
+    b
+  end
+end
+
+class Header < Struct.new(:pos, :size, :type)
+  #pos, starting position within the byte stream
+  #size, number of bytes within the box
+  #type, descriptive type for the bytes stored in the box
+end
+
+class BootstrapBox < Struct.new(:header,
+                                :version,
+                                :flags,
+                                :bootstrap_info_version,
+                                :profile,
+                                :live,
+                                :update,
+                                :time_scale,
+                                :current_media_time,
+                                :smpte_timecode_offset,
+                                :movie_identifier,
+                                :servers,
+                                :quality,
+                                :drm_data,
+                                :metadata,
+                                :segments,
+                                :segment_run_tables,
+                                :fragments,
+                                :fragment_run_tables)
+end
+
+class SegmentRunEntry < Struct.new(:first_segment, :fragments_per_segment)
+end
+
+class FragmentRunEntry < Struct.new(:first_fragment,
+                                    :first_fragment_timestamp,
+                                    :fragment_duration,
+                                    :discontinuity_indicator)
+end
+
+#For Segment and Fragment boxes
+class RunTableBox < Struct.new(:header,
+                               :version,
+                               :flags,
+                               :time_scale,
+                               :quality_segment_url_modifiers,
+                               :run_entry_table)
+end

data/lib/glued/f4f.rb

@@ -0,0 +1,44 @@
+# encoding: utf-8
+
+class F4F
+  attr_reader :boxes
+
+  def initialize(reader)
+    @reader = reader
+    @boxes = []
+
+    # Scan for boxes in the stream see spec 1.3 F4V box format
+    until (@reader.pos >= @reader.size) do
+      box = next_box
+      @boxes << box
+      @reader.pos = box.pos + box.size
+    end
+  end
+
+  def ok?
+    # WARNING: There are rumours that "Some moronic servers add wrong
+    # boxSize in header causing fragment verification to fail so we
+    # have to fix the boxSize before processing the fragment."
+
+    (@boxes[0].type == Bootstrap::AFRA && @boxes[1].type == Bootstrap::MOOF && @boxes[2].type == Bootstrap::MDAT)
+  end
+
+  def next_box
+    pos = @reader.pos
+    size = @reader.int32
+    type = @reader.fourCC
+    size = @reader.int64 if size == 1 #For boxes over 4GB the size is moved here.
+
+    header_size = @reader.pos-pos
+    content_size = size - header_size
+
+    F4FHeader.new(pos, size, type, @reader.pos, content_size)
+  end
+
+end
+
+class F4FHeader < Struct.new(:pos, :size, :type, :content_start, :content_size)
+  #pos, starting position within the byte stream
+  #size, number of bytes within the box
+  #type, descriptive type for the bytes stored in the box
+end

data/lib/glued/f4m.rb

@@ -0,0 +1,44 @@
+# encoding: utf-8
+
+class F4M
+  NAMESPACE = 'http://ns.adobe.com/f4m/1.0'
+
+  attr_reader :duration,
+              :bootstrap_info,
+              :base_ref,
+              :media_filename,
+              :media
+
+  def initialize(url, data)
+    xml = Nokogiri::XML(data)
+
+    namespace = xml.root.namespace.href rescue 'not found'
+    raise "Invalid manifest namespace. It was #{namespace} but should have been #{NAMESPACE}" unless namespace == NAMESPACE
+
+    xml.remove_namespaces!
+
+    stream_type = xml.xpath("//streamType").first.text rescue 'unknown'
+    raise "Only recorded streams are supported." unless stream_type == 'recorded'
+
+    @duration = xml.xpath("//duration").first.text.to_i
+
+    b64_bootstrap_info = xml.xpath("//bootstrapInfo").first.text
+    @bootstrap_info = Base64.strict_decode64(b64_bootstrap_info)
+
+    media_nodes = xml.xpath("/manifest[1]/media")
+    @media = find_highest_bitrate(media_nodes)
+    @media_filename = @media.at_css('@url').value
+
+    @base_ref = url.split('/').slice(0...-1).join('/') #can be specified in xml
+  end
+
+  def find_highest_bitrate(list)
+    rates = []
+    list.each do |media|
+      br = media.at_css("@bitrate").value.to_i
+      rates[br] = media
+    end
+
+    rates.reject {|e| e.nil? }.pop
+  end
+end

data/lib/glued/f4vio.rb

@@ -0,0 +1,47 @@
+# encoding: utf-8
+
+class F4VIO < StringIO
+
+  def byte
+    self.read(1).unpack('C').first
+  end
+
+  def int16
+    self.read(2).unpack('n').first
+  end
+
+  def int24
+    "\x00#{self.read(3)}".unpack('N').first
+  end
+
+  def int32
+    self.read(4).unpack('N').first
+  end
+
+  def int64
+    hi = int32
+    lo = int32
+    (hi * 4294967296) + lo
+  end
+
+  def fourCC
+    self.read(4).unpack('A4').first
+  end
+
+  def string
+    o, p = self.pos, 0
+    p += 1 while (self.read(1) != "\x00")
+
+    self.pos = o
+    str = self.read(p)
+    self.pos += 1
+
+    str
+  end
+
+  def byte_ar
+    ar = []
+    byte.times { ar << byte }
+    ar
+  end
+end

data/lib/glued/glue.rb

@@ -0,0 +1,16 @@
+# encoding: utf-8
+
+class Glue
+
+  def initialize(url)
+    raise "Invalid manifest url '#{url}' (it should end with .f4m)" unless url.to_s =~ /\.f4m$/ #Only by convention
+
+    xml = Curl::Easy.perform(url).body
+    manifest = F4M.new(url, xml)
+    bootstrap = Bootstrap.new(manifest.bootstrap_info)
+    grabber = Grabber.new(manifest, bootstrap)
+
+    puts "\rComplete                                                                "
+  end
+
+end

data/lib/glued/grabber.rb

@@ -0,0 +1,91 @@
+# encoding: utf-8
+
+class Grabber
+  attr_reader :urls
+
+  def initialize(manifest, bootstrap, io=nil)
+    raise "Only one segment can be handled" if bootstrap.segments != 1 #As we've hardcoded 1 below
+    raise "Not enough fragments" if bootstrap.fragments < 1
+    raise "Too many fragments" if bootstrap.fragments > 10000 #not sure what this limit should be
+
+    @uri = "#{manifest.media_filename}.flv"
+    @url = "#{manifest.base_ref}/#{manifest.media_filename}Seg1-Frag"
+    @total_fragments = bootstrap.fragments
+    @urls = []
+    @downloaded_fragments = []
+    @fragments_downloaded = 0
+
+    #TODO: Track how much has already been downloaded and append from that point
+    raise "Aborting as the download target file '#{@uri}' already exists" if File.exist? @uri
+
+    @out = io ||= File.new(@uri, 'ab')
+    @out.write(flv_header(1,1))
+
+    build
+  end
+
+  private
+
+  #TODO: Inspect first fragment, test for audio and video, write header accordingly
+  def flv_header(audio, video)
+    #Audio and video need to be a 1 or 0
+    flv_header = ["464c5601050000000900000000"].pack("H*")
+
+    result = audio << 2 | video
+    flv_header[4] = [result].pack('C')
+
+    flv_header
+  end
+
+  def build
+    #TODO, correctly set the fragment start
+    fragment = 1
+    while fragment <= @total_fragments
+      @urls << "#{@url}#{fragment}"
+      fragment += 1
+    end
+
+    @urls.each { |url| download url }
+  end
+
+  def download url
+    file_name = url.split('/').pop
+
+    dl = fetch_and_report(url)
+
+    raise "Invalid content type '#{dl.content_type}' for fragment #{file_name}." unless dl.content_type == 'video/f4f'
+
+    reader = F4VIO.new(dl.body)
+    f4f = F4F.new(reader)
+
+    raise "Fragment did not verify" unless f4f.ok?
+
+    f4f.boxes.each do |box|
+      if box.type == 'mdat'
+        reader.pos = box.content_start
+        @out.write(reader.read(box.content_size))
+      end
+    end
+
+    @fragments_downloaded += 1
+  end
+
+  def fetch_and_report(url)
+    start_time = Time.now
+
+    dl = Curl::Easy.perform(url)
+
+    report(dl.body.length, Time.now-start_time)
+
+    dl
+  end
+
+  def report(number_of_bytes, in_seconds)
+    bps = (number_of_bytes*8) / in_seconds
+    kbps = bps/1000
+    mbps = bps/1000000
+
+    print "\rDownloading #{@fragments_downloaded+1}/#{@total_fragments} at #{mbps.round(2)} Mbps"
+  end
+end
+

data/lib/glued/version.rb

@@ -0,0 +1,7 @@
+# encoding: utf-8
+
+module Glued
+  VERSION = '0.1.0'
+  NAME = 'glued'
+  USER_AGENT = "#{NAME}-#{VERSION}"
+end

data/spec/bootstrap_spec.rb

@@ -0,0 +1,89 @@
+# encoding: utf-8
+
+require File.join(File.dirname(__FILE__), 'spec_helper')
+
+describe Bootstrap, "parsing a bootstrap" do
+
+  def info
+    Base64.strict_decode64 'AAAAjWFic3QAAAAAAAACFAAAAAPoAAAAAAA+QYAAAAAAAAAAAEFXAAAAAAABAAAAGWFzcnQAAAAAAAAAAAEAAAABAAACFAEAAABGYWZydAAAAAAAAAPoAAAAAAMAAAABAAAAAAAAAAAAAB4AAAACFAAAAAAAPjoAAAAHgAAAAAAAAAAAAAAAAAAAAAAA'
+  end
+
+  it "decodes bootstrap info (abst) boxes" do
+    boot = Bootstrap.new(info)
+    box = boot.boxes.first
+
+    box.version.should eq(0)
+    box.flags.should eq(0)
+    box.bootstrap_info_version.should eq(532)
+    box.current_media_time.should eq(4080000)
+    box.smpte_timecode_offset.should eq(0)
+    box.movie_identifier.should eq('AW')
+    box.servers.should eq([])
+    box.quality.should eq([])
+    box.drm_data.should eq('')
+    box.metadata.should eq('')
+  end
+
+  it "errors when data is corrupt" do
+    expect { Bootstrap.new('Papa+LAZAR0U') }.to raise_error "Computer says no"
+  end
+
+  it "decodes segment run table (asrt) boxes" do
+    boot = Bootstrap.new(info)
+    box = boot.boxes.first
+
+    srt = box.segment_run_tables.first
+    sre = srt.run_entry_table.first
+
+    sre.first_segment.should eq(1)
+    sre.fragments_per_segment.should eq(532)
+  end
+
+  it "determines the number of segments" do
+    boot = Bootstrap.new(info)
+
+    boot.segments.should eq(1)
+  end
+
+  it "determines the number of fragments" do
+    boot = Bootstrap.new(info)
+
+    boot.fragments.should eq(532)
+  end
+
+
+  it "decodes fragment run table (afra) boxes" do
+    boot = Bootstrap.new(info)
+    box = boot.boxes.first
+
+    box.fragments.should eq(1)
+
+    box.fragment_run_tables.length.should eq(1)
+
+    frt = box.fragment_run_tables.first
+    fre = frt.run_entry_table.first
+
+    frt.run_entry_table.length.should eq(3)
+
+    fre.first_fragment.should eq(1)
+  end
+
+  it "errors when unexpected fragment boxes are found" do
+    bad_info = Base64.strict_decode64 "AAAAjWFic3QAAAAAAAACFAAAAAPoAAAAAAA+QYAAAAAAAAAAAEFXAAAAAAABAAAAGWFzcnQAAAAAAAAAAAEAAAABAAACFAEAAABGYWZ6dAAAAAAAAAPoAAAAAAMAAAABAAAAAAAAAAAAAB4AAAACFAAAAAAAPjoAAAAHgAAAAAAAAAAAAAAAAAAAAAAA"
+
+    expect {
+      boot = Bootstrap.new(bad_info)
+    }.to raise_error "Unexpected fragment run table box header 'afzt' instead of 'afrt'"
+  end
+
+  it "errors unless there are one or more segment table entries" do
+  end
+
+  it "errors when unexpected segment boxes are found" do
+    bad_info = Base64.strict_decode64 'AAAAjWFic3QAAAAAAAACFAAAAAPoAAAAAAA+QYAAAAAAAAAAAEFXAAAAAAABAAAAGWFzcHQAAAAAAAAAAAEAAAABAAACFAEAAABGYWZydAAAAAAAAAPoAAAAAAMAAAABAAAAAAAAAAAAAB4AAAACFAAAAAAAPjoAAAAHgAAAAAAAAAAAAAAAAAAAAAAA'
+
+    expect {
+      boot = Bootstrap.new(bad_info)
+    }.to raise_error "Unexpected segment run table box header 'aspt' instead of 'asrt'"
+  end
+end

data/spec/f4f_spec.rb

@@ -0,0 +1,4 @@
+# encoding: utf-8
+
+describe F4F, "Extracting Flash Media Fragments" do
+end

data/spec/f4m_spec.rb

@@ -0,0 +1,58 @@
+# encoding: utf-8
+
+describe F4M, "Loading manifests" do
+
+  describe "a recorded manifest" do
+
+    before(:each) do
+      @manifest = F4M.new('http://cloud.org/only/fools/vod.f4m',
+                          IO.read('spec/fixtures/vod.f4m'))
+    end
+
+    after(:each) do
+      @manifest = nil
+    end
+
+    it "creates a base url for fragments downloads" do
+      @manifest.base_ref.should eq('http://cloud.org/only/fools')
+    end
+
+    it "defines the total duration of the media in seconds" do
+      @manifest.duration.should eq(4080)
+    end
+
+    it "has bootstrap information" do
+      @manifest.bootstrap_info.should_not be_empty
+    end
+
+    it "defines the media filename" do
+      @manifest.media_filename.should eq('some-audio-video-')
+    end
+
+  end
+
+  describe "loading a live manifest" do
+    it "raises errors if a live manifest is used" do
+      expect {
+          F4M.new('http://cloud.org/and/horses/live.f4m',
+          IO.read('spec/fixtures/live.f4m'))
+      }.to raise_error "Only recorded streams are supported."
+    end
+  end
+
+  describe "loading manifests with unrecognised namespaces" do
+    it "raises errors with no namespaces" do
+      expect {
+          F4M.new('http://only.fools/and/horses/live.f4m',
+          IO.read('spec/fixtures/no_namespace.f4m'))
+      }.to raise_error "Invalid manifest namespace. It was not found but should have been http://ns.adobe.com/f4m/1.0"
+    end
+
+    it "raises errors for incompatible namespaces" do
+      expect {
+          F4M.new('http://only.fools/and/horses/live.f4m',
+          IO.read('spec/fixtures/bad_namespace.f4m'))
+      }.to raise_error "Invalid manifest namespace. It was http://ns.adobe.com/f4m/2.0 but should have been http://ns.adobe.com/f4m/1.0"
+    end
+  end
+end

data/spec/f4vio_spec.rb

@@ -0,0 +1,45 @@
+# encoding: utf-8
+
+require File.join(File.dirname(__FILE__), 'spec_helper')
+
+describe F4VIO, "Flash video binary decoding (big endian)" do
+
+  it "reads sequences of Unicode 8-bit characters (UTF-8), terminated with 0x00 (unless otherwise specified)" do
+    scanner = F4VIO.new "abc\x00xyz\x00"
+
+    scanner.string.should eq('abc')
+    scanner.string.should eq('xyz')
+  end
+
+  it "reads bytes" do
+    scanner = F4VIO.new 'hello'
+
+    scanner.byte.should eq(104)
+    scanner.byte.should eq(101)
+    scanner.byte.should eq(108)
+    scanner.byte.should eq(108)
+    scanner.byte.should eq(111)
+  end
+
+  it "reads 16 bit integers" do
+    # eg = {}
+  end
+
+  it "reads 24 bit integers" do
+    # eg = {}
+  end
+
+  it "reads 32 bit integers" do
+    # eg = {}
+  end
+
+  it "reads 64 bit integers" do
+    # eg = {}
+  end
+
+  it "reads 4CC, Four-character ASCII code, such as 'moov', encoded as UI32" do
+    scanner = F4VIO.new 'moov'
+    scanner.fourCC.should eq('moov')
+  end
+
+end

data/spec/fixtures/bad_namespace.f4m

@@ -0,0 +1,3 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns="http://ns.adobe.com/f4m/2.0">
+</manifest>

data/spec/fixtures/live.f4m

@@ -0,0 +1,13 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns="http://ns.adobe.com/f4m/1.0" version="3.0">
+    <id>myvideo</id>
+    <duration>253</duration>
+    <mimeType>video/x-flv</mimeType>
+    <streamType>live</streamType>
+    <baseURL>http://comedy</baseURL>
+    <drmAdditionalHeader url="http://drm.com/mydrmadditionalheader"/>
+    <bootstrapInfo profile="named" url="/mybootstrapinfo" fragmentDuration="4"/>
+    <media url="/horses/low" bitrate="408" width="640" height="480"/>
+    <media url="/horses/med" bitrate="908" width="800" height="600"/>
+    <media url="/horses/hi" bitrate="1708" width="1920" height="1080"/>
+</manifest>

data/spec/fixtures/no_namespace.f4m

@@ -0,0 +1,3 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest>
+</manifest>

data/spec/fixtures/vod.f4m

@@ -0,0 +1,28 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest
+  xmlns="http://ns.adobe.com/f4m/1.0">
+  <id>jan2014_p40</id>
+  <startTime>2006-07-24T07:15:00+01:00</startTime>
+  <duration>4080</duration>
+  <mimeType>video/mp4</mimeType>
+  <streamType>recorded</streamType>
+  <deliveryType>streaming</deliveryType>
+  <!-- -audio_eng=128000-video=4947245 -->
+  <bootstrapInfo
+    id="boot1"
+    profile="named">AAAAjWFic3QAAAAAAAACFAAAAAPoAAAAAAA+QYAAAAAAAAAAAEFXAAAAAAABAAAAGWFzcnQAAAAAAAAAAAEAAAABAAACFAEAAABGYWZydAAAAAAAAAPoAAAAAAMAAAABAAAAAAAAAAAAAB4AAAACFAAAAAAAPjoAAAAHgAAAAAAAAAAAAAAAAAAAAAAA</bootstrapInfo>
+  <media
+    url="some-audio-video-"
+    bitrate="5075"
+    bootstrapInfoId="boot1"
+    width="1280"
+    height="720">
+    <metadata>AgAKb25NZXRhRGF0YQgAAAAAAA9tZXRhZGF0YWNyZWF0b3ICACVDb2RlU2hvcCdzIFVuaWZpZWQgU3RyZWFtaW5nIFBsYXRmb3JtAAhoYXNBdWRpbwEBAAhoYXNWaWRlbwEBAAhkdXJhdGlvbgBAr+AAAAAAAAAPYXVkaW9zYW1wbGVyYXRlAEDncAAAAAAAAA1hdWRpb2RhdGFyYXRlAEBgAAAAAAAAAAxhdWRpb2NvZGVjaWQCAARtcDRhAAZhYWNhb3QAQAAAAAAAAAAABXdpZHRoAECUAAAAAAAAAAZoZWlnaHQAQIaAAAAAAAAADXZpZGVvZGF0YXJhdGUAQLNTPrhR64UADHZpZGVvY29kZWNpZAIABEFWQzEACmF2Y3Byb2ZpbGUAQFkAAAAAAAAACGF2Y2xldmVsAEBAAAAAAAAAAAAJ</metadata>
+  </media>
+
+
+  <media url="/horses/low" bitrate="408" width="640" height="480"/>
+  <media url="/horses/med" bitrate="908" width="800" height="600"/>
+  <media url="/horses/hi" bitrate="1708" width="1920" height="1080"/>
+
+</manifest>

data/spec/glue_spec.rb

@@ -0,0 +1,15 @@
+# encoding: utf-8
+
+require File.join(File.dirname(__FILE__), 'spec_helper')
+
+describe "Glue" do
+  it "downloads hds fragments and glues them together" do
+    # one day it just might
+  end
+
+  it "raises errors when the manifest url looks doubtful" do
+    expect {
+      Glue.new('/steptoe/and/son.f5m')
+    }.to raise_error "Invalid manifest url '/steptoe/and/son.f5m' (it should end with .f4m)"
+  end
+end

data/spec/grabber_spec.rb

@@ -0,0 +1,43 @@
+# encoding: utf-8
+
+require File.join(File.dirname(__FILE__), 'spec_helper')
+
+describe "grabbing fragments" do
+
+  before(:each) do
+    @bs = double(Bootstrap)
+    @f4m = double(F4M)
+    @io = StringIO.new
+  end
+
+  after(:each) do
+    @bs = nil
+    @f4m = nil
+    @io = nil
+  end
+
+  it "builds a list of url fragments" do
+    @f4m.stub(:base_ref).and_return('http://the.young.ones')
+    @f4m.stub(:media_filename).and_return('some-audio-video-')
+    @bs.stub(:segments).and_return(1)
+    @bs.stub(:fragments).and_return(20)
+
+    grabber = Grabber.new(@f4m, @bs, @io)
+
+    expect(grabber.urls.first).to eq('http://the.young.ones/some-audio-video-Seg1-Frag1')
+    expect(grabber.urls.last).to eq('http://the.young.ones/some-audio-video-Seg1-Frag20')
+  end
+
+  it "fails without the correct number of segments" do
+    @bs.stub(:segments).and_return(0)
+    expect { Grabber.new(@f4m, @bs) }.to raise_error "Only one segment can be handled"
+
+    @bs.stub(:segments).and_return(1)
+    @bs.stub(:fragments).and_return(0)
+
+    expect { Grabber.new(@f4m, @bs) }.to raise_error "Not enough fragments"
+  end
+
+  it "downloads the list of fragments" do
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,9 @@
+# encoding: utf-8
+
+$:.push File.join(File.dirname(__FILE__), "..", "lib")
+$:.push File.dirname(__FILE__)
+
+require 'glued'
+
+require 'rspec'
+require 'rspec/autorun'

metadata

@@ -0,0 +1,161 @@
+--- !ruby/object:Gem::Specification
+name: glued
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Simon Gregory
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-02-03 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: curb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: observr
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: HDS Downloader
+email: simon@helvector.org
+executables:
+- glued
+extensions: []
+extra_rdoc_files: []
+files:
+- bin/glued
+- Gemfile
+- Gemfile.lock
+- glued.watchr
+- lib/glued/bootstrap.rb
+- lib/glued/f4f.rb
+- lib/glued/f4m.rb
+- lib/glued/f4vio.rb
+- lib/glued/glue.rb
+- lib/glued/grabber.rb
+- lib/glued/version.rb
+- lib/glued.rb
+- LICENSE
+- Rakefile
+- README.md
+- spec/bootstrap_spec.rb
+- spec/f4f_spec.rb
+- spec/f4m_spec.rb
+- spec/f4vio_spec.rb
+- spec/fixtures/bad_namespace.f4m
+- spec/fixtures/live.f4m
+- spec/fixtures/no_namespace.f4m
+- spec/fixtures/vod.f4m
+- spec/glue_spec.rb
+- spec/grabber_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/simongregory/glued
+licenses:
+- MIT
+metadata: {}
+post_install_message: |
+  Welcome to Glued
+  ================
+
+  To download HDS delivered media pass a f4m manifest to the
+  tool, ie:
+
+    glued http://royston.vasey.org/papa/lazarou.f4m
+
+  The flv file should then play in a media player such as VLC.
+  DRM is not removed. If the file has DRM it will only be
+  watchable with a Flash based player and a valid license.
+
+  Functionality is basic. The tool assumes you want the highest
+  quality media. Live content cannot be recorded. Multilevel
+  manifests are not supported, and bootstrap info has to be
+  embedded within the manifest.
+
+  If you try a manifest that doesn't work please open an issue
+  at https://github.com/simongregory/glue/issues and leave the
+  details.
+rdoc_options: []
+require_paths:
+- lib
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: Downloads and joins HTTP Dynamic Stream fragments into a single media file
+test_files: []