craig 0.0.1.pre → 0.0.1

data/README.md

@@ -1,51 +1,188 @@
 # Craig
 
-A robust little Craigslist scraper.
+A Craigslist listings scraper.
+
+* Inspired by [gregstallings/craigslist](https://github.com/gregstallings/craigslist).
+* A just-for-fun self-educational exercise.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+    $ gem install craig
 
-    gem 'craig'
+## Usage
 
-And then execute:
+~~~ ruby
+require "craig"
+~~~
 
-    $ bundle
+Pass in the city and category:
 
-Or install it yourself as:
+~~~ ruby
+listings = Craig.query(:austin, :for_sale)
+~~~
 
-    $ gem install craig
+Or narrow down your results with a subcategory:
+ 
+~~~ ruby
+listings = Craig.query(:austin, :for_sale => :computers)
+~~~
 
-## Usage
+Returns up to 100 `Listing`s.
 
-    require "craig"
+## Examples
 
-Let's check out some cars within our budget (that have images).
+Let's check out some cars within our budget.
     
-    vehicles = Craig.query(:austin, :for_sale => :cars_and_trucks)
-    vehicles.select { |v| 
-      v.price.between?(6000, 8000) && 
-      v.image? &&
-      v.seller == :owner 
-    }
-    
-    #=> [<Listing>, <Listing>, <Listing>]
+~~~ ruby
+vehicles = Craig.query(:austin, :for_sale => :cars_trucks)
+vehicles.select { |v| 
+  v.price.between?(6000, 8000) && 
+  v.image? &&
+  v.seller == :owner 
+}
+
+#=> [<Listing>, <Listing>, <Listing>]
+~~~
     
 I'm about to move to Birmingham. I wonder where the kids my age are living.
     
-    listings = Craig.query(:birmingham, :personals => :strictly_platonic)
-    listings.select { |p| 
-      p.age < 30 && 
-    }.map(&:location)
-   
-    #= ["Highland Ave", "Highland Ave", "Highland Ave"]
+~~~ ruby
+listings = Craig.query(:birmingham, :personals => :strictly_platonic)
+listings.select { |p| 
+  p.age < 30
+}.map(&:location)
+
+#= ["Highland Ave", "Highland Ave", "Highland Ave"]
+~~~
 
 Oh wait, I left my trombone downtown last night. I wonder if anyone found it.
     
-    listings = Craig.query(:austin, :community => :lost_and_found
-    listings.any? { |listing| listing.title =~ /trombone/i }
-    
-    #=> true
+~~~ ruby
+listings = Craig.query(:austin, :community => :lost_found
+listings.any? { |listing| listing.title =~ /trombone/i }
+
+#=> true
+~~~
+
+Let's get the price of the most expensive Macbook that's sold by an owner 
+(instead of a dealer) on or near UT campus.
+
+~~~ ruby
+Craig.query(:austin, :for_sale => :computer)
+  .select { |c| 
+    c.seller == :owner && 
+    c.location[/campus/i] &&
+    c.title[/macbook/i]
+  }.sort_by { |c| 
+    -c.price 
+  }.first
+
+# Output
+
+#=> {
+  :url => "http://austin.craigslist.org/sys/3723418912.html"
+  :title => "Late 2011 Macbook Pro"
+  :seller => :owner
+  :price => 1050
+  :posted_at => #<Date: 2013-01-23 ((2456316j,0s,0n),+0s,2299161j)>
+  :location => "West campus"
+  :has_map? => false
+  :has_image? => true
+  :id => 3723418912
+}
+~~~
+
+## Available Listing methods
+
+The easiest way to see what listings respond to is to just send them `Listing#to_hash`:
+
+~~~ ruby
+Craig.query(:new_york_city, :jobs).map(&:to_hash)
+~~~
+
+Or just look at console output (`Listing#to_s` outputs the hash in string form):
+
+~~~ ruby
+Craig.query(:new_york_city, :jobs)
+~~~
+
+Or you can look in `listing.rb` to see the type of nodes that Listings have, then find the method each node defines in `node.rb`.
+
+Since I rarely use Ruby Modules, this gem started out as an self-educational scraper concept where Nodes are composed of Nodes are composed of Nodes but I didn't get very far. So now the internal API is a little spread out. TODO...
+
+## Available cities and categories
+
+The Craigslist website itself serves as documentation for this
+gem's API.
+
+* Cities: http://www.craigslist.org/about/sites/
+* Categories: http://austin.craigslist.org/
+
+Just downcase Craigslist's name for it and replace spaces and punctuation with underscores.
+
+Here, let me translate some for you to be clear:
+
+<table>
+  <tr>
+    <th colspan="2">Cities</th>
+  </tr>
+  <tr>
+    <th>How it appears on Craigslist</th><th>The symbol you pass into Craig</th>
+  </tr>
+  <tr>
+    <td>beaumont / port arthur</td>
+    <td>:beaumont_port_arthur</td>
+  </tr>
+  <tr>
+    <td>cambridge, UK</td>
+    <td>:cambridge_uk</td>
+  </tr>
+  <tr>
+    <td>nice / cote d'azur</td>
+    <td>:nice_cote_d_azur</td>
+  </tr>
+  <tr>
+    <th colspan="2">Categories/Subcategories</th>
+  </tr>
+  <tr>
+    <th>How it appears on Craigslist</th><th>The symbol you pass into Craig</th>
+  </tr>
+  <tr>
+    <td>for sale</td>
+    <td>:for_sale</td>
+  </tr>
+  <tr>
+    <td>cds/dvd/vhs </td>
+    <td>:cds_dvd_vhs</td>
+  </tr>
+  <tr>
+    <td>accounting+finance </td>
+    <td>:accounting_finance</td>
+  </tr>
+</table>
+
+~~~ ruby
+Craig.query(:nice_cote_d_azur, :for_sale => :cds_dvd_vhs)
+~~~
+
+If you're still unsure, just check out Craig's internal lookup table.
+
+## Diverges from Craigslist homepage
+
+While I've mostly modeled Craig's API after Craigslist's website, I've 
+diverged from the homepage where I felt it made sense. 
+
+For example, I prefer the "Show All" view where possible.
+
+    :community => :events goes to /eee instead of /eve.
+
+    This way you see all events instead of just events with 
+    date ranges.
+
+    :for_sale => :cars_trucks goes to the owners+dealers listings
+
+    instead of the splash page that lets you choose
+    TODO: Add search API or expose API to specify this stuff.
 
 ## Contributing
 

data/Rakefile

@@ -7,3 +7,10 @@ Rake::TestTask.new do |t|
 end
 desc "Run tests"
 task :default => :test
+
+namespace :cities do
+  desc "Update Craig::CITIES using Craigslists' city directory"
+  task :update do
+     system("ruby lib/craig/tasks/update_cities.rb")
+  end
+end

data/craig.gemspec

@@ -8,8 +8,8 @@ Gem::Specification.new do |gem|
   gem.version       = Craig::VERSION
   gem.authors       = ["Dan Neumann"]
   gem.email         = ["danrodneu@gmail.com"]
-  gem.description   = %q{a robust little Craigslist scraper that fetches listings.}
-  gem.summary       = %q{a robust little Craigslist scraper.}
+  gem.description   = %q{A Craigslist listings scraper.}
+  gem.summary       = %q{A Craigslist listings scraper.}
   gem.homepage      = "http://github.com/danneu/craig"
 
   gem.files         = `git ls-files`.split($/)

data/lib/craig/categories.rb

@@ -0,0 +1,169 @@
+module Craig
+  CATEGORIES = {
+    'community' => {
+      'path' => 'ccc',
+      'children' => {
+        'activities' => 'act',
+        'artists' => 'ats',
+        'childcare' => 'kid',
+        'general' => 'com',
+        'groups' => 'grp',
+        'pets' => 'pet',
+        'events' => 'eee', # Diverges. Original: "eve"
+        'lost_found' => 'laf',
+        'musicians' => 'muc',
+        'local news' => 'vnn',
+        'politics' => 'pol',
+        'rideshare' => 'rid',
+        'volunteers' => 'vol',
+        'classes' => 'cls'
+      }
+    },
+    'personals' => {
+      'children' => {
+        'strictly_platonic' => 'stp',
+        'women_seek_women' => 'w4w',
+        'women_seeking_men' => 'w4m',
+        'men_seeking_women' => 'm4w',
+        'men_seeking_men' => 'm4m',
+        'misc_romance' => 'msr',
+        'casual_encounters' => 'cas',
+        'missed_connections' => 'mis',
+        'rants_and_raves' => 'rnr'
+      }
+    },
+    'housing' => {
+      'path' => 'hhh',
+      'children' => {
+        'apts_housing' => 'apa',
+        'rooms_shared' => 'roo',
+        'sublets_temporary' => 'sub',
+        'housing_wanted' => 'hsw',
+        'housing_swap' => 'swp',
+        'vacation_rentals' => 'vac',
+        'parking_storage' => 'prk',
+        'office_commercial' => 'off',
+        'real_estate_for_sale' => 'rea'
+      }
+    },
+    'for_sale' => {
+      'path' => 'sss',
+      'children' => {
+        'appliances' => 'ppa',
+        'antiques' => 'ata',
+        'barter' => 'bar',
+        'bikes' => 'bia',
+        'boats' => 'boo',
+        'books' => 'bka',
+        'business' => 'bfa',
+        'computer' => 'sya',
+        'free' => 'zip',
+        'furniture' => 'fua',
+        'general' => 'foa',
+        'jewelry' => 'jwa',
+        'materials' => 'maa',
+        'rvs' => 'rva',
+        'sporting' => 'sga',
+        'tickets' => 'tia',
+        'tools' => 'tla',
+        'wanted' => 'wan',
+        'arts_crafts' => 'ara',
+        'auto_parts' => 'pta',
+        'baby_kids' => 'baa',
+        'beauty_hlth' => 'haa',
+        'cars_trucks' => 'cta', # Diverges. Original: "/i/autos"
+        'cds_dvd_vhs' => 'ema',
+        'cell_phones' => 'moa',
+        'clothes_acc' => 'cla',
+        'collectibles' => 'cba',
+        'electronics' => 'ela',
+        'farm_garden' => 'gra',
+        'garage_sale' => 'gms',
+        'household' => 'hsa',
+        'motorcycles' => 'mca',
+        'music_instr' => 'msa',
+        'photo_video' => 'pha',
+        'toys_games' => 'taa',
+        'video_gaming' => 'vga'
+      }
+    },
+    'services' => {
+      'path' => 'bbb',
+      'children' => {
+        'beauty' => 'brs',
+        'creative' => 'crs',
+        'computer' => 'cps',
+        'cycle' => 'cys',
+        'event' => 'evs',
+        'financial' => 'fns',
+        'legal' => 'lgs',
+        'lessons' => 'lss',
+        'marine' => 'mas',
+        'pet' => 'pas',
+        'automotive' => 'aos',
+        'farm_garden' => 'fgs',
+        'household' => 'hss',
+        'labor_move' => 'lbs',
+        'skilld_trade' => 'sks',
+        'real_estate' => 'rts',
+        'sm_biz_ads' => 'biz',
+        'therapeutic' => 'thp',
+        'travel_vac' => 'trv',
+        'write_ed_tr8' => 'wet'
+      }
+    },
+    'jobs' => {
+      'path' => 'jjj',
+      'children' => {
+        'accounting_finance' => 'acc',
+        'admin_office' => 'ofc',
+        'arch_engineering' => 'egr',
+        'art_media_design' => 'med',
+        'biotech_science' => 'sci',
+        'business_mgmt' => 'bus',
+        'customer_service' => 'csr',
+        'education' => 'edu',
+        'food_bev_hosp' => 'fbh',
+        'general_labor' => 'lab',
+        'government' => 'gov',
+        'human_resources' => 'hum',
+        'internet_engineers' => 'eng',
+        'legal_paralegal' => 'lgl',
+        'manufacturing' => 'mnu',
+        'marketing_pr_ad' => 'mar',
+        'medical_health' => 'hea',
+        'nonprofit_sector' => 'npo',
+        'real_estate' => 'rej',
+        'retail_wholesale' => 'ret',
+        'sales_biz_dev' => 'sls',
+        'salon_spa_fitness' => 'spa',
+        'security' => 'sec',
+        'skilled_trade_craft' => 'trd',
+        'software_qa_dba' => 'sof',
+        'systems_network' => 'sad',
+        'technical_support' => 'tch',
+        'transport' => 'trp',
+        'tv_film_video' => 'tfr',
+        'web_info_design' => 'web',
+        'writing_editing' => 'wri',
+        'etc' => 'etc'
+      }
+    },
+    'gigs' => {
+      'path' => 'ggg',
+      'children' => {
+        'crew' => 'cwg',
+        'event' => 'evg',
+        'labor' => 'lbg',
+        'talent' => 'tlg',
+        'computer' => 'cpg',
+        'creative' => 'crg',
+        'domestic' => 'dmg',
+        'writing' => 'wrg'
+      }
+    },
+    'resumes' => {
+      'path' => 'res'
+    }
+  }
+end

data/lib/craig/cities.rb

@@ -0,0 +1,262 @@
+# Generated by update_cities.rb
+# Date: 22 Jan 2013
+#
+# Total Craigslist cities: 460
+# Hash size:               252
+
+module Craig
+  CITIES = {
+    "birmingham" => "bham",
+    "florence_muscle_shoals" => "shoals",
+    "gadsden_anniston" => "gadsden",
+    "huntsville_decatur" => "huntsville",
+    "anchorage_mat_su" => "anchorage",
+    "kenai_peninsula" => "kenai",
+    "southeast_alaska" => "juneau",
+    "flagstaff_sedona" => "flagstaff",
+    "mohave_county" => "mohave",
+    "fayetteville" => "fayar",
+    "fresno_madera" => "fresno",
+    "hanford_corcoran" => "hanford",
+    "humboldt_county" => "humboldt",
+    "imperial_county" => "imperial",
+    "mendocino_county" => "mendocino",
+    "monterey_bay" => "monterey",
+    "san_francisco_bay_area" => "sfbay",
+    "san_luis_obispo" => "slo",
+    "siskiyou_county" => "siskiyou",
+    "ventura_county" => "ventura",
+    "visalia_tulare" => "visalia",
+    "colorado_springs" => "cosprings",
+    "eastern_co" => "eastco",
+    "fort_collins_north_co" => "fortcollins",
+    "high_rockies" => "rockies",
+    "western_slope" => "westslope",
+    "eastern_ct" => "newlondon",
+    "northwest_ct" => "nwct",
+    "washington" => "washingtondc",
+    "daytona_beach" => "daytona",
+    "florida_keys" => "keys",
+    "ft_myers_sw_florida" => "fortmyers",
+    "heartland_florida" => "cfl",
+    "north_central_fl" => "lakecity",
+    "okaloosa_walton" => "okaloosa",
+    "sarasota_bradenton" => "sarasota",
+    "south_florida" => "miami",
+    "tampa_bay_area" => "tampa",
+    "treasure_coast" => "treasure",
+    "albany" => "albanyga",
+    "athens" => "athensga",
+    "columbus" => "columbusga",
+    "macon_warner_robins" => "macon",
+    "northwest_ga" => "nwga",
+    "savannah_hinesville" => "savannah",
+    "hawaii" => "honolulu",
+    "lewiston_clarkston" => "lewiston",
+    "bloomington_normal" => "bn",
+    "champaign_urbana" => "chambana",
+    "la_salle_co" => "lasalle",
+    "mattoon_charleston" => "mattoon",
+    "southern_illinois" => "carbondale",
+    "springfield" => "springfieldil",
+    "western_il" => "quincy",
+    "lafayette_west_lafayette" => "tippecanoe",
+    "muncie_anderson" => "muncie",
+    "richmond" => "richmondin",
+    "south_bend_michiana" => "southbend",
+    "southeast_ia" => "ottumwa",
+    "waterloo_cedar_falls" => "waterloo",
+    "manhattan" => "ksu",
+    "northwest_ks" => "nwks",
+    "southeast_ks" => "seks",
+    "southwest_ks" => "swks",
+    "bowling_green" => "bgky",
+    "eastern_kentucky" => "eastky",
+    "western_ky" => "westky",
+    "central_louisiana" => "cenla",
+    "southern_maryland" => "smd",
+    "western_maryland" => "westmd",
+    "cape_cod_islands" => "capecod",
+    "western_massachusetts" => "westernmass",
+    "worcester_central_ma" => "worcester",
+    "central_michigan" => "centralmich",
+    "detroit_metro" => "detroit",
+    "jackson" => "jxn",
+    "monroe" => "monroemi",
+    "northern_michigan" => "nmi",
+    "saginaw_midland_baycity" => "saginaw",
+    "southwest_michigan" => "swmi",
+    "the_thumb" => "thumb",
+    "upper_peninsula" => "up",
+    "duluth_superior" => "duluth",
+    "minneapolis_st_paul" => "minneapolis",
+    "rochester" => "rmn",
+    "southwest_mn" => "marshall",
+    "gulfport_biloxi" => "gulfport",
+    "north_mississippi" => "northmiss",
+    "southwest_ms" => "natchez",
+    "columbia_jeff_city" => "columbiamo",
+    "lake_of_the_ozarks" => "loz",
+    "southeast_missouri" => "semo",
+    "montana_old" => "montana",
+    "omaha_council_bluffs" => "omaha",
+    "scottsbluff_panhandle" => "scottsbluff",
+    "reno_tahoe" => "reno",
+    "new_hampshire" => "nh",
+    "central_nj" => "cnj",
+    "north_jersey" => "newjersey",
+    "clovis_portales" => "clovis",
+    "roswell_carlsbad" => "roswell",
+    "santa_fe_taos" => "santafe",
+    "elmira_corning" => "elmira",
+    "new_york_city" => "newyork",
+    "plattsburgh_adirondacks" => "plattsburgh",
+    "potsdam_canton_massena" => "potsdam",
+    "twin_tiers_ny_pa" => "twintiers",
+    "utica_rome_oneida" => "utica",
+    "eastern_nc" => "eastnc",
+    "hickory_lenoir" => "hickory",
+    "jacksonville" => "onslow",
+    "raleigh_durham_ch" => "raleigh",
+    "fargo_moorhead" => "fargo",
+    "north_dakota" => "nd",
+    "athens" => "athensohio",
+    "dayton_springfield" => "dayton",
+    "lima_findlay" => "limaohio",
+    "tuscarawas_co" => "tuscarawas",
+    "zanesville_cambridge" => "zanesville",
+    "northwest_ok" => "enid",
+    "corvallis_albany" => "corvallis",
+    "klamath_falls" => "klamath",
+    "medford_ashland" => "medford",
+    "altoona_johnstown" => "altoona",
+    "cumberland_valley" => "chambersburg",
+    "lehigh_valley" => "allentown",
+    "scranton_wilkes_barre" => "scranton",
+    "state_college" => "pennstate",
+    "rhode_island" => "providence",
+    "florence" => "florencesc",
+    "greenville_upstate" => "greenville",
+    "northeast_sd" => "nesd",
+    "pierre_central_sd" => "csd",
+    "rapid_city_west_sd" => "rapidcity",
+    "sioux_falls_se_sd" => "siouxfalls",
+    "south_dakota" => "sd",
+    "jackson" => "jacksontn",
+    "beaumont_port_arthur" => "beaumont",
+    "dallas_fort_worth" => "dallas",
+    "deep_east_texas" => "nacogdoches",
+    "del_rio_eagle_pass" => "delrio",
+    "killeen_temple_ft_hood" => "killeen",
+    "mcallen_edinburg" => "mcallen",
+    "odessa_midland" => "odessa",
+    "southwest_tx" => "bigbend",
+    "tyler_east_tx" => "easttexas",
+    "victoria" => "victoriatx",
+    "ogden_clearfield" => "ogden",
+    "provo_orem" => "provo",
+    "vermont" => "burlington",
+    "hampton_roads" => "norfolk",
+    "new_river_valley" => "blacksburg",
+    "southwest_va" => "swva",
+    "kennewick_pasco_richland" => "kpr",
+    "olympic_peninsula" => "olympic",
+    "pullman_moscow" => "pullman",
+    "seattle_tacoma" => "seattle",
+    "skagit_island_sji" => "skagit",
+    "spokane_coeur_d_alene" => "spokane",
+    "charleston" => "charlestonwv",
+    "eastern_panhandle" => "martinsburg",
+    "huntington_ashland" => "huntington",
+    "northern_panhandle" => "wheeling",
+    "parkersburg_marietta" => "parkersburg",
+    "southern_wv" => "swv",
+    "west_virginia_old" => "wv",
+    "appleton_oshkosh_fdl" => "appleton",
+    "kenosha_racine" => "racine",
+    "guam_micronesia" => "micronesia",
+    "u_s_virgin_islands" => "virgin",
+    "medicine_hat" => "hat",
+    "peace_river_country" => "peace",
+    "fraser_valley" => "abbotsford",
+    "kelowna_okanagan" => "kelowna",
+    "kootenays" => "cranbrook",
+    "skeena_bulkley" => "skeena",
+    "sunshine_coast" => "sunshine",
+    "st_john_s" => "newfoundland",
+    "brantford_woodstock" => "brantford",
+    "chatham_kent" => "chatham",
+    "hamilton_burlington" => "hamilton",
+    "kitchener_waterloo_cambridge" => "kitchener",
+    "london" => "londonon",
+    "niagara_region" => "niagara",
+    "ottawa_hull_gatineau" => "ottawa",
+    "sault_ste_marie" => "soo",
+    "prince_edward_island" => "pei",
+    "quebec_city" => "quebec",
+    "belgium" => "brussels",
+    "croatia" => "zagreb",
+    "finland" => "helsinki",
+    "brittany" => "rennes",
+    "loire_valley" => "loire",
+    "marseille" => "marseilles",
+    "nice_cote_d_azur" => "cotedazur",
+    "normandy" => "rouen",
+    "essen_ruhr" => "essen",
+    "greece" => "athens",
+    "florence_tuscany" => "florence",
+    "napoli_campania" => "naples",
+    "sicilia" => "sicily",
+    "venice_veneto" => "venice",
+    "amsterdam_randstad" => "amsterdam",
+    "norway" => "oslo",
+    "poland" => "warsaw",
+    "faro_algarve" => "faro",
+    "romania" => "bucharest",
+    "sweden" => "stockholm",
+    "turkey" => "istanbul",
+    "birmingham_west_mids" => "birmingham",
+    "cambridge_uk" => "cambridge",
+    "cardiff_wales" => "cardiff",
+    "devon_cornwall" => "devon",
+    "east_anglia" => "norwich",
+    "east_midlands" => "eastmids",
+    "newcastle_ne_england" => "newcastle",
+    "guam_micronesia" => "micronesia",
+    "chennai_madras" => "chennai",
+    "kolkata_calcutta" => "kolkata",
+    "surat_surat" => "surat",
+    "indonesia" => "jakarta",
+    "iran" => "tehran",
+    "iraq" => "baghdad",
+    "west_bank" => "ramallah",
+    "osaka_kobe_kyoto" => "osaka",
+    "beirut_lebanon" => "beirut",
+    "bicol_region" => "naga",
+    "cagayan_de_oro" => "cdo",
+    "taiwan" => "taipei",
+    "thailand" => "bangkok",
+    "united_arab_emirates" => "dubai",
+    "newcastle_nsw" => "ntl",
+    "tasmania" => "hobart",
+    "caribbean_islands" => "caribbean",
+    "bolivia" => "lapaz",
+    "rio_de_janeiro" => "rio",
+    "salvador_bahia" => "salvador",
+    "chile" => "santiago",
+    "dominican_republic" => "santodomingo",
+    "ecuador" => "quito",
+    "baja_california_sur" => "bajasur",
+    "ciudad_juarez" => "juarez",
+    "puerto_vallarta" => "pv",
+    "nicaragua" => "managua",
+    "peru" => "lima",
+    "venezuela" => "caracas",
+    "virgin_islands" => "virgin",
+    "egypt" => "cairo",
+    "ethiopia" => "addisababa",
+    "ghana" => "accra",
+    "morocco" => "casablanca",
+    "tunisia" => "tunis"
+  }
+end