addressabler 0.0.7 → 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 843009d16f40ee1495e8df0b7cdebc38be93dfc9
+  data.tar.gz: e07d20c2fe95ce66837dc8136ef460bb57a15b84
+SHA512:
+  metadata.gz: 72479f1ded77aebcd346504fea4071b29047b93bff9b636cdcdca4193eedc6b43caf0452c7d4062243885e05f88d512137756b9293bf93c2bd5d2329b6804b66
+  data.tar.gz: 27043e8cdf72fb5f19dbb2a709400faf1d79bf010c5a41640d9eb01207b258c4e4c1f92f06387f2fc39fc34acc6a8863de4a463faff84209ab0f3b783ec231b0

data/CHANGELOG.md

@@ -1,3 +1,6 @@
+## 0.0.7
+* Updated to latest TLD list from [publicsuffix.org][]
+
 ## 0.0.6
 * Merged [pull request from Jean Mertz][] adding custom TLD support
 
@@ -9,3 +12,4 @@
 * Updated to latest Addressable spec
 
   [pull request from Jean Mertz]: https://github.com/flipsasser/addressabler/pull/2
+  [publicsuffix.org]: http://publicsuffix.org/

data/README.md

@@ -7,14 +7,11 @@ setting of nested hashes to query strings.
 
 ## Install
 
-Install using Rubygems:
+Add Addressabler to your `Gemfile`:
 
-  gem install addressabler
-
-Then:
-
-  require 'rubygems'
-  require 'addressabler'
+```ruby
+gem "addressabler", ">= 0.1"
+```
 
 Addressabler will automatically require `addressable/uri`.
 
@@ -58,12 +55,22 @@ TLDs, as well:
 @uri.tld #=> "co.uk"
 ```
 
+#### Private / Public TLDs
+
+By default, Addressabler knows about ICANN public TLDs. There are, however, lots and lots of private TLDs that companies have registered. For example, as Dom Hodgson [points out](https://github.com/flipsasser/addressabler/issues/3), "blogspot.com" is a TLD by private, non-ICANN standards which are applied by the Mozilla foundation to the TLD list.
+
+As such, Addressabler defaults to parsing the ICANN public TLDS (`Addressabler.public_tlds`) but can easily be instructed to look at private TLDs like so:
+
+```ruby
+Addressabler.use_private_tlds = true
+```
+
 #### Custom TLDs
 You can specify custom TLDs - which aren't actually working TLD's on the
 internet - for internal usage. One example would be a custom development TLD:
 
 ```ruby
-Addressable::URI.custom_tlds = {
+Addressabler.custom_tlds = {
   'dev' => {},              # mydomain.dev
   'bar' => { 'foo' => {} }  # mydomain.foo.bar
 }
@@ -90,7 +97,7 @@ Frankly, I don't disagree with anything he has to say on the issue, but it is a
 problem many people have experienced.
 
 *As such,* since Rack already supports building nested hashes "the Rails Way"
-(shudder), I added support for assigning nested hashes to `URI`s **only if Rack
+(shudder), I added support for assigning nested hashes to `Addressable::URI`s **only if Rack
 is available.** Addressabler will attempt to load `Rack::Utils` and, if it finds
 it, you can assign a nested hash in the `query_hash=` method like so:
 
@@ -99,11 +106,18 @@ it, you can assign a nested hash in the `query_hash=` method like so:
 @uri.to_s #=> "http://www.google.co.uk/?foo[bar]=baz"
 ```
 
-**HANDLE WITH CARE!** As [Bob explains in the discussion]
-(https://github.com/sporkmonger/addressable/issues/77#issuecomment-8534480),
+**HANDLE WITH CARE!** As [Bob explains in the discussion](https://github.com/sporkmonger/addressable/issues/77#issuecomment-8534480),
 there's a better alternative to nested hashes in query strings, so try that
 before you install this library.
 
 That's it. Enjoy.
 
-#### Copyright © 2013 Flip Sasser
+## Contributors THANKS GUYS
+
+Super special thankses to
+
+* [Jean Mertz](https://github.com/jeanmertz) for the custom [TLD implementation](https://github.com/flipsasser/addressabler/pull/2)
+* [Paul Dix](https://github.com/pauldix) for the [Domainatrix](https://github.com/pauldix/domainatrix) parsing code
+* [Bob Aman](https://github.com/sporkmonger) for maintaining [Addressable](https://github.com/sporkmonger/addressable)
+
+#### Copyright © 2013 Flip Sasser

data/lib/addressabler/uri.rb

@@ -0,0 +1,62 @@
+require 'addressabler'
+require 'addressabler/query'
+require 'addressable/uri'
+
+module Addressabler
+  module URI
+    def domain
+      return nil unless host
+      parse_domain_parts[:domain]
+    end
+
+    def domain=(new_domain)
+      self.host = "#{subdomain}.#{new_domain}.#{tld}"
+    end
+
+    def query_hash
+      @query_hash ||= query_hash_for(query_values || {})
+    end
+
+    def query_hash=(new_query_hash)
+      @query_hash = query_hash_for(new_query_hash || {})
+    end
+
+    def subdomain
+      return nil unless host
+      parse_domain_parts[:subdomain]
+    end
+
+    def subdomain=(new_subdomain)
+      self.host = "#{new_subdomain}.#{domain}.#{tld}"
+    end
+
+    def tld
+      return nil unless host
+      Addressabler.parse_tld(host)
+    end
+
+    def tld=(new_tld)
+      self.host = "#{subdomain}.#{domain}.#{new_tld}"
+    end
+
+    private
+    def query_hash_for(contents)
+      hash = Addressabler::Query[contents]
+      hash.uri = self
+      hash
+    end
+
+    def parse_domain_parts
+      tld = self.tld
+      subdomain_parts = host.to_s.gsub(/\.#{tld}$/, '').split('.')
+      @domain_parts = {
+        :domain => subdomain_parts.pop,
+        :subdomain => subdomain_parts.join('.'),
+        :tld => tld
+      }
+    end
+  end
+
+end
+
+Addressable::URI.send :include, Addressabler::URI

data/lib/addressabler/uri_deprecation.rb

@@ -0,0 +1,23 @@
+require 'addressabler'
+require 'addressable/uri'
+
+module Addressabler
+  module URIDeprecation
+    def custom_tlds
+      deprecate_custom_tlds
+      Addressabler.custom_tlds
+    end
+
+    def custom_tlds=(new_custom_tlds)
+      deprecate_custom_tlds
+      Addressabler.custom_tlds = new_custom_tlds
+    end
+
+    private
+    def deprecate_custom_tlds
+      Addressabler.deprecation_warning('Addressable::URI.custom_tlds will be replaced with Addressabler.custom_tlds in Addressable 1.0. Please note the deprecated accessor is on "Addressable::URI" and the new accessor is on "Addressable*r*")')
+    end
+  end
+end
+
+Addressable::URI.extend Addressabler::URIDeprecation

data/lib/addressabler.rb

@@ -1,88 +1,68 @@
 require 'rubygems'
-require 'addressable/uri'
-require 'addressabler/query'
-
-module Addressable
-  class URI
-    class << self; attr_accessor :custom_tlds; end
-    @custom_tlds = {}
-  end
-end
+require 'addressabler/uri'
+require 'addressabler/uri_deprecation'
 
 module Addressabler
-  module ClassMethods
-    def parse_tld(host)
-      host = host.to_s.split('.')
-      tlds = []
-      sub_hash = Addressabler::TLDS.merge(@custom_tlds)
-      while sub_hash = sub_hash[tld = host.pop]
-        tlds.unshift(tld)
-        if sub_hash.has_key? '*'
-          tlds.unshift(host.pop)
-        end
+  def self.all_tlds
+    @all_tlds ||= {}.tap do |tlds|
+      with_tld_file do |line|
+        add_line_to_tlds(line, tlds)
       end
-      tlds.join('.')
     end
   end
 
-  module InstanceMethods
-    def domain
-      return nil unless host
-      parse_domain_parts[:domain]
-    end
-
-    def domain=(new_domain)
-      self.host = "#{subdomain}.#{new_domain}.#{tld}"
-    end
-
-    def query_hash
-      @query_hash ||= query_hash_for(query_values || {})
-    end
-
-    def query_hash=(new_query_hash)
-      @query_hash = query_hash_for(new_query_hash || {})
-    end
+  def self.custom_tlds
+    @custom_tlds ||= {}
+  end
 
-    def subdomain
-      return nil unless host
-      parse_domain_parts[:subdomain]
-    end
+  def self.custom_tlds=(new_custom_tlds)
+    @custom_tlds = new_custom_tlds
+  end
 
-    def subdomain=(new_subdomain)
-      self.host = "#{new_subdomain}.#{domain}.#{tld}"
-    end
+  def self.parse_tld(host)
+    host = host.to_s.split('.')
+    tlds = []
+    tlds = look_for_tld_in(private_tlds, host.dup) if use_private_tlds?
+    tlds = look_for_tld_in(public_tlds.merge(custom_tlds), host.dup) if tlds.empty?
+    tlds.join('.')
+  end
 
-    def tld
-      return nil unless host
-      self.class.parse_tld(host)
+  def self.private_tlds
+    @private_tlds ||= {}.tap do |tlds|
+      in_private_tlds = false
+      with_tld_file do |line|
+        if in_private_tlds
+          add_line_to_tlds(line, tlds)
+        elsif line =~ /BEGIN PRIVATE/
+          in_private_tlds = true
+        end
+      end
     end
+  end
 
-    def tld=(new_tld)
-      self.host = "#{subdomain}.#{domain}.#{new_tld}"
+  def self.public_tlds
+    @public_tlds ||= {}.tap do |tlds|
+      with_tld_file do |line|
+        break if line =~ /END ICANN/
+        add_line_to_tlds(line, tlds)
+      end
     end
+  end
 
-    private
-    def query_hash_for(contents)
-      hash = Addressabler::Query[contents]
-      hash.uri = self
-      hash
-    end
+  def self.use_private_tlds=(new_use_private_tlds)
+    @use_private_tlds = new_use_private_tlds
+  end
 
-    def parse_domain_parts
-      tld = self.tld
-      subdomain_parts = host.to_s.gsub(/\.#{tld}$/, '').split('.')
-      @domain_parts = {
-        :domain => subdomain_parts.pop,
-        :subdomain => subdomain_parts.join('.'),
-        :tld => tld
-      }
-    end
+  def self.use_private_tlds?
+    return @use_private_tlds if defined? @use_private_tlds
+    @use_private_tlds = false
   end
 
+  private # More for my own reference than anything else
+
   # Thanks Domainatrix for the parsing logic!
-  tlds = {}
-  File.readlines(File.join(File.dirname(__FILE__), 'tlds')).each do |line|
-    line.strip!
+  def self.add_line_to_tlds(line, tlds)
+    line = line.to_s.strip
     unless line == '' || line =~ /^\//
       parts = line.split(".").reverse
       sub_hash = tlds
@@ -91,8 +71,40 @@ module Addressabler
       end
     end
   end
-  TLDS = tlds
-end
 
-Addressable::URI.extend Addressabler::ClassMethods
-Addressable::URI.send :include, Addressabler::InstanceMethods
+  def self.const_missing(name)
+    if name.to_s == 'TLDS'
+      deprecation_warning "Addressabler::TLDS will be replaced by Addressabler::public_tlds and Addressabler::private_tlds in Addressable 1.0"
+      public_tlds
+    else
+      super
+    end
+  end
+
+  def self.deprecation_warning(message)
+    puts "ADDRESSABLER DEPRECATION WARNING: #{message}"
+    non_addressable_caller = caller.find do |line|
+      line !~ /#{File.expand_path(File.join(File.dirname(__FILE__), '..'))}/
+    end
+    if non_addressable_caller
+      puts "  Called from:\n    #{non_addressable_caller}"
+    end
+  end
+
+  def self.look_for_tld_in(sub_hash, host_parts)
+    tlds = []
+    while sub_hash = sub_hash[tld = host_parts.pop]
+      tlds.unshift(tld)
+      if sub_hash.has_key? '*'
+        tlds.unshift(host_parts.pop)
+      end
+    end
+    tlds
+  end
+
+  def self.with_tld_file
+    File.readlines(File.join(File.dirname(__FILE__), 'tlds')).each do |line|
+      yield line
+    end
+  end
+end

data/spec/addressable/idna_spec.rb

@@ -0,0 +1,238 @@
+# coding: utf-8
+# Copyright (C) 2006-2013 Bob Aman
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+
+require "spec_helper"
+
+# Have to use RubyGems to load the idn gem.
+require "rubygems"
+
+require "addressable/idna"
+
+shared_examples_for "converting from unicode to ASCII" do
+  it "should convert 'www.google.com' correctly" do
+    Addressable::IDNA.to_ascii("www.google.com").should == "www.google.com"
+  end
+
+  it "should convert 'www.詹姆斯.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "www.詹姆斯.com"
+    ).should == "www.xn--8ws00zhy3a.com"
+  end
+
+  it "should convert 'www.Iñtërnâtiônàlizætiøn.com' correctly" do
+    "www.Iñtërnâtiônàlizætiøn.com"
+    Addressable::IDNA.to_ascii(
+      "www.I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4" +
+      "n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n.com"
+    ).should == "www.xn--itrntinliztin-vdb0a5exd8ewcye.com"
+  end
+
+  it "should convert 'www.Iñtërnâtiônàlizætiøn.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "www.In\xCC\x83te\xCC\x88rna\xCC\x82tio\xCC\x82n" +
+      "a\xCC\x80liz\xC3\xA6ti\xC3\xB8n.com"
+    ).should == "www.xn--itrntinliztin-vdb0a5exd8ewcye.com"
+  end
+
+  it "should convert " +
+      "'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
+      "correctly" do
+    Addressable::IDNA.to_ascii(
+      "www.\343\201\273\343\202\223\343\201\250\343\201\206\343\201\253\343" +
+      "\201\252\343\201\214\343\201\204\343\202\217\343\201\221\343\201\256" +
+      "\343\202\217\343\201\213\343\202\211\343\201\252\343\201\204\343\201" +
+      "\251\343\202\201\343\201\204\343\202\223\343\202\201\343\201\204\343" +
+      "\201\256\343\202\211\343\201\271\343\202\213\343\201\276\343\201\240" +
+      "\343\201\252\343\201\214\343\201\217\343\201\227\343\201\252\343\201" +
+      "\204\343\201\250\343\201\237\343\202\212\343\201\252\343\201\204." +
+      "w3.mag.keio.ac.jp"
+    ).should ==
+      "www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
+      "fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
+  end
+
+  it "should convert " +
+      "'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
+      "correctly" do
+    Addressable::IDNA.to_ascii(
+      "www.\343\201\273\343\202\223\343\201\250\343\201\206\343\201\253\343" +
+      "\201\252\343\201\213\343\202\231\343\201\204\343\202\217\343\201\221" +
+      "\343\201\256\343\202\217\343\201\213\343\202\211\343\201\252\343\201" +
+      "\204\343\201\250\343\202\231\343\202\201\343\201\204\343\202\223\343" +
+      "\202\201\343\201\204\343\201\256\343\202\211\343\201\270\343\202\231" +
+      "\343\202\213\343\201\276\343\201\237\343\202\231\343\201\252\343\201" +
+      "\213\343\202\231\343\201\217\343\201\227\343\201\252\343\201\204\343" +
+      "\201\250\343\201\237\343\202\212\343\201\252\343\201\204." +
+      "w3.mag.keio.ac.jp"
+    ).should ==
+      "www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
+      "fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
+  end
+
+  it "should convert '点心和烤鸭.w3.mag.keio.ac.jp' correctly" do
+    Addressable::IDNA.to_ascii(
+      "点心和烤鸭.w3.mag.keio.ac.jp"
+    ).should == "xn--0trv4xfvn8el34t.w3.mag.keio.ac.jp"
+  end
+
+  it "should convert '가각갂갃간갅갆갇갈갉힢힣.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "가각갂갃간갅갆갇갈갉힢힣.com"
+    ).should == "xn--o39acdefghijk5883jma.com"
+  end
+
+  it "should convert " +
+      "'\347\242\274\346\250\231\346\272\226\350" +
+      "\220\254\345\234\213\347\242\274.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "\347\242\274\346\250\231\346\272\226\350" +
+      "\220\254\345\234\213\347\242\274.com"
+    ).should == "xn--9cs565brid46mda086o.com"
+  end
+
+  it "should convert 'ﾘ宠퐱〹.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "\357\276\230\345\256\240\355\220\261\343\200\271.com"
+    ).should == "xn--eek174hoxfpr4k.com"
+  end
+
+  it "should convert 'リ宠퐱卄.com' correctly" do
+    Addressable::IDNA.to_ascii(
+      "\343\203\252\345\256\240\355\220\261\345\215\204.com"
+    ).should == "xn--eek174hoxfpr4k.com"
+  end
+
+  it "should convert 'ᆵ' correctly" do
+    Addressable::IDNA.to_ascii(
+      "\341\206\265"
+    ).should == "xn--4ud"
+  end
+
+  it "should convert 'ﾯ' correctly" do
+    Addressable::IDNA.to_ascii(
+      "\357\276\257"
+    ).should == "xn--4ud"
+  end
+end
+
+shared_examples_for "converting from ASCII to unicode" do
+  it "should convert 'www.google.com' correctly" do
+    Addressable::IDNA.to_unicode("www.google.com").should == "www.google.com"
+  end
+
+  it "should convert 'www.詹姆斯.com' correctly" do
+    Addressable::IDNA.to_unicode(
+      "www.xn--8ws00zhy3a.com"
+    ).should == "www.詹姆斯.com"
+  end
+
+  it "should convert 'www.iñtërnâtiônàlizætiøn.com' correctly" do
+    Addressable::IDNA.to_unicode(
+      "www.xn--itrntinliztin-vdb0a5exd8ewcye.com"
+    ).should == "www.iñtërnâtiônàlizætiøn.com"
+  end
+
+  it "should convert " +
+      "'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
+      "correctly" do
+    Addressable::IDNA.to_unicode(
+      "www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
+      "fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
+    ).should ==
+      "www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp"
+  end
+
+  it "should convert '点心和烤鸭.w3.mag.keio.ac.jp' correctly" do
+    Addressable::IDNA.to_unicode(
+      "xn--0trv4xfvn8el34t.w3.mag.keio.ac.jp"
+    ).should == "点心和烤鸭.w3.mag.keio.ac.jp"
+  end
+
+  it "should convert '가각갂갃간갅갆갇갈갉힢힣.com' correctly" do
+    Addressable::IDNA.to_unicode(
+      "xn--o39acdefghijk5883jma.com"
+    ).should == "가각갂갃간갅갆갇갈갉힢힣.com"
+  end
+
+  it "should convert " +
+      "'\347\242\274\346\250\231\346\272\226\350" +
+      "\220\254\345\234\213\347\242\274.com' correctly" do
+    Addressable::IDNA.to_unicode(
+      "xn--9cs565brid46mda086o.com"
+    ).should ==
+      "\347\242\274\346\250\231\346\272\226\350" +
+      "\220\254\345\234\213\347\242\274.com"
+  end
+
+  it "should convert 'リ宠퐱卄.com' correctly" do
+    Addressable::IDNA.to_unicode(
+      "xn--eek174hoxfpr4k.com"
+    ).should == "\343\203\252\345\256\240\355\220\261\345\215\204.com"
+  end
+
+  it "should convert 'ﾯ' correctly" do
+    Addressable::IDNA.to_unicode(
+      "xn--4ud"
+    ).should == "\341\206\265"
+  end
+
+  it "should normalize 'string' correctly" do
+    Addressable::IDNA.unicode_normalize_kc(:'string').should == "string"
+    Addressable::IDNA.unicode_normalize_kc("string").should == "string"
+  end
+end
+
+describe Addressable::IDNA, "when using the pure-Ruby implementation" do
+  before do
+    Addressable.send(:remove_const, :IDNA)
+    load "addressable/idna/pure.rb"
+  end
+
+  it_should_behave_like "converting from unicode to ASCII"
+  it_should_behave_like "converting from ASCII to unicode"
+
+  begin
+    require "fiber"
+
+    it "should not blow up inside fibers" do
+      f = Fiber.new do
+        Addressable.send(:remove_const, :IDNA)
+        load "addressable/idna/pure.rb"
+      end
+      f.resume
+    end
+  rescue LoadError
+    # Fibers aren't supported in this version of Ruby, skip this test.
+    warn('Fibers unsupported.')
+  end
+end
+
+begin
+  require "idn"
+
+  describe Addressable::IDNA, "when using the native-code implementation" do
+    before do
+      Addressable.send(:remove_const, :IDNA)
+      load "addressable/idna/native.rb"
+    end
+
+    it_should_behave_like "converting from unicode to ASCII"
+    it_should_behave_like "converting from ASCII to unicode"
+  end
+rescue LoadError
+  # Cannot test the native implementation without libidn support.
+  warn('Could not load native IDN implementation.')
+end

data/spec/addressable/net_http_compat_spec.rb

@@ -0,0 +1,28 @@
+# coding: utf-8
+# Copyright (C) 2006-2013 Bob Aman
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+
+require "spec_helper"
+
+require "addressable/uri"
+require "net/http"
+
+describe Net::HTTP do
+  it "should be compatible with Addressable" do
+    response_body =
+      Net::HTTP.get(Addressable::URI.parse('http://www.google.com/'))
+    response_body.should_not be_nil
+  end
+end