purdie 0.0.8 → 0.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1defd8c81ee1d21b6f6a39584ca1836d4f4d281b
-  data.tar.gz: 05a2275ab518862ebd272c0725b75056d533f614
+  metadata.gz: d177170f97f8a450e4d71facbf39f758da4b21ef
+  data.tar.gz: ea6e3b001078141fd3c36ac313245b3b8a658e17
 SHA512:
-  metadata.gz: 4612b24c608b32764d7c4ce8a3b42e8ed080dbabce1726065ec85394209a6edfe561a8c87fd2492e8ce59c862840ba121e37e31a40e9fb233fcea66d37bd3151
-  data.tar.gz: a0937e230214dfb2ee3033513c72a386f28560b84db2a9fce6426191b1818e4d44ae3e4d4a78dd343c9b420a6ebf5933dba2eabf9fb93e07e0804888b0f0d0bc
+  metadata.gz: 7e614308d4a9467a0dda0f69b38d9ee425ce8584132c0f06cec0bdcd046868585cfc32160480f567b4372baaddbd76041a0c71cc03c69ad2bd72fd55e2f75318
+  data.tar.gz: 9fe6faca8d5736f14647826b88927ef1c389744b6e220cc4b8db0b91c1fd55d5127c53a52742e7580460d544f3cb6631c625c9cd76aa8b4d86c923076aede7d8

data/README.md

@@ -10,7 +10,7 @@
 
 We have music on SoundCloud, photos on Flickr, and videos on Vimeo, and we want to feature them on [http://rawfunkmaharishi.uk/](http://rawfunkmaharishi.uk/). Up until now, this has been managed by curating, by hand (or [very shonky scripts](https://github.com/rawfunkmaharishi/rawfunkmaharishi.github.io/blob/master/flickriser.rb)), [bits](https://github.com/rawfunkmaharishi/rawfunkmaharishi.github.io/blob/master/_data/sounds.yml) [of](https://github.com/rawfunkmaharishi/rawfunkmaharishi.github.io/blob/master/_data/pictures.yml) [YAML](https://github.com/rawfunkmaharishi/rawfunkmaharishi.github.io/blob/master/_data/videos.yml) to feed into Jekyll, but this gets old quickly, especially when you run into things like SoundCloud's decision to only expose the track ID deep inside the embeddable iframe code.
 
-But you know what, It's 2015 and everything has an API, so let's build a robot to do this stuff properly!
+But this is dumb. It's 2015 and everything has an API, so let's build a robot to do this stuff properly!
 
 ##The great metadata shift
 
@@ -24,19 +24,70 @@ Am I going to regret these decisions? Almost certainly.
 
 ##Using it
 
-You need to set up a *_sources* directory in your Jekyll project, containing files with one-URL-per-line, like this:
+You need to create a *_sources* directory in your Jekyll project, containing files with one-URL-per-line, like this:
 
     https://soundcloud.com/rawfunkmaharishi/hexaflexagon-1
     https://soundcloud.com/rawfunkmaharishi/junalbandi-3
 
-and a *.env* with API keys in it, and then you can run
+(you can alternatively specify a source file on the command line with the `-f` flag).
+ 
+You also need a *.env* file with API secrets in it:
+
+    FLICKR_API_KEY: this_a_key
+    FLICKR_SECRET: this_a_secret
+
+    SOUNDCLOUD_CLIENT_ID: this_a_client_is
+    SOUNDCLOUD_USER_ID: this_a_user_id
+
+    VIMEO_BEARER_TOKEN: this_bearer_token
+    
+(get those keys from [Flickr](https://www.flickr.com/services/apps/create/apply), [SoundCloud](http://soundcloud.com/you/apps/new) and [Vimeo](https://developer.vimeo.com/apps/new))
+
+And then you can run
 
     purdie fetch
 
 and it will dump out YAML files into *_data* ready for Jekyll to consume.
 
-For now, though, please don't. None of my metadata hacks are documented, it's got some very Raw Funk Maharishi-specific lookups nailed into it, the output filenames are hardcoded (so it may well overwrite stuff you care about), there are flickering tests, you'll have a bad time. But soon...
+###Customisation
+
+You can supply your own _config/purdie.yaml_ file to specify a few things:
+
+    # Flickr photos are happy to have a null title
+    default_title: Raw Funk Maharishi 
+
+    # Map Flickr users to better names
+    photographer_lookups:
+      pikesley: sam
+      
+    # Specify output files per-service
+    services:
+      Flickr:
+        output_file: "_outfiles/photos.yaml"
+        
+(see [this](https://github.com/rawfunkmaharishi/purdie/blob/master/_config/defaults.yaml) for some other things you can tweak)
+
+Tread carefully for now, because my metadata hacks aren't fully documented. 
+
+##What next?
+
+There's no reason I couldn't support other services - YouTube springs immediately to mind, and others should be fairly simple. There's some introspection magic at the heart of all of this which means that as long as each service is represented by a class that:
+
+* includes the `Purdie::Ingester` module, and
+* sports a `#configure` method which sets at least a 
+  * `@matcher` string which will pick a URL out of an input file, and an
+  * `@output_file`, and
+* has a `#distill` method which takes a URL representing an item on the service and returns a hash of metadata, see e.g.
+  * [Flickr](https://github.com/rawfunkmaharishi/purdie/blob/master/lib/purdie/services/flickr.rb#L26-L48)
+  * [SoundCloud](https://github.com/rawfunkmaharishi/purdie/blob/master/lib/purdie/services/soundcloud.rb#L30-L41)
+  * [Vimeo](https://github.com/rawfunkmaharishi/purdie/blob/master/lib/purdie/services/vimeo.rb#L27-L37)
+
+then this should all Just Work. There's definitely a blog post in this, because Ruby introspection and metaprogramming is just mind-bogglingly powerful (and dangerous).
 
+More prosaically, I think it will be fairly simple to support Flickr sets and SoundCloud albums as input data.
+  
+And I can definitely rationalise these [horrible license lookups](https://github.com/rawfunkmaharishi/purdie/blob/master/_config/defaults.yaml#L5-L13) into a module or even a gem of their own.
+  
 ##Why Purdie?
 
-Because Bernard Purdie is [amazing](https://www.youtube.com/watch?v=E9E0WxLbqVA&list=PLuPLM2FI60-OIgFTc9YCrGgH5XWGT6znV&index=6)
+Because Bernard Purdie is [even more amazing that Ruby introspection](https://www.youtube.com/watch?v=E9E0WxLbqVA&list=PLuPLM2FI60-OIgFTc9YCrGgH5XWGT6znV&index=6).

data/_config/defaults.yaml

@@ -1,13 +1,14 @@
-source-dir:    _sources
-output-dir:    _data
-default-title: Bernard Purdie, the World's most-recorded drummer
+source_dir:    _sources
+output_dir:    _data
+default_title: Bernard Purdie, the World's most-recorded drummer
 
-license-lookups:
+# those unwieldy lookups are because each service calls the CC licenses by a slightly different name and defaults to a different version of the license. I think I need to abstract all of this out into a modlue or even a gem of its own
+license_lookups:
   cc-by-nc-sa:
-    short-name: BY-NC-SA
-    full-name:  Attribution-NonCommercial-ShareAlike
+    short_name: BY-NC-SA
+    full_name:  Attribution-NonCommercial-ShareAlike
     url:        http://creativecommons.org/licenses/by-nc-sa/4.0/
   by-nc-sa:
-    short-name: BY-NC-SA
-    full-name:  Attribution-NonCommercial-ShareAlike
+    short_name: BY-NC-SA
+    full_name:  Attribution-NonCommercial-ShareAlike
     url:        http://creativecommons.org/licenses/by-nc-sa/3.0/

data/features/flickr-alternative-outfile.feature

@@ -0,0 +1,29 @@
+@vcr
+Feature: Flickr
+
+  Scenario: Generate Flickr YAML
+    Given a file named "_sources/flickr.csv" with:
+    """
+    https://www.flickr.com/photos/rawfunkmaharishi/15631479625/
+    """
+    And a file named "_config/purdie.yaml" with:
+    """
+    services:
+      Flickr:
+        output_file: "_outfiles/photos.yaml"
+    """
+    When I successfully run `purdie fetch`
+    Then a file named "_outfiles/photos.yaml" should exist
+    And the file "_outfiles/photos.yaml" should contain:
+    """
+    - title: The Comedy, October 2014
+      date: '2014-10-22'
+      photo_page: https://www.flickr.com/photos/rawfunkmaharishi/15631479625/
+      photo_url: https://farm4.staticflickr.com/3933/15631479625_b6168ee903_m.jpg
+      license: Attribution-NonCommercial-ShareAlike
+      license_url: https://creativecommons.org/licenses/by-nc-sa/2.0/
+      photographer: kim
+    """
+    And a file named "_data/flickr.yaml" should not exist
+    And a file named "_data/soundcloud.yaml" should not exist
+    And a file named "_data/vimeo.yaml" should not exist

data/features/support/fixtures/_config/purdie.yaml

@@ -1,4 +1,4 @@
-default-title: Raw Funk Maharishi
+default_title: Raw Funk Maharishi
 
-photographer-lookups:
+photographer_lookups:
   cluttercup: jane

data/features/support/fixtures/vcr/Vimeo/Generate_Vimeo_YAML.yml

@@ -17,7 +17,7 @@ http_interactions:
       message: OK
     headers:
       Date:
-      - Tue, 03 Mar 2015 17:44:01 GMT
+      - Wed, 04 Mar 2015 22:02:49 GMT
       Server:
       - Apache
       Vary:
@@ -25,7 +25,7 @@ http_interactions:
       Cache-Control:
       - no-cache, max-age=315360000
       Expires:
-      - Fri, 28 Feb 2025 17:44:01 GMT
+      - Sat, 01 Mar 2025 22:02:49 GMT
       Content-Length:
       - '3688'
       Content-Type:
@@ -37,5 +37,5 @@ http_interactions:
       string: '{"uri":"/videos/117102891","name":"Bernard","description":null,"link":"https://vimeo.com/117102891","duration":239,"width":1280,"language":null,"height":720,"created_time":"2015-01-18T17:26:10+00:00","modified_time":"2015-02-25T21:11:06+00:00","content_rating":["unrated"],"license":"by-nc-sa","privacy":{"view":"anybody","embed":"public","download":true,"add":true,"comments":"anybody"},"pictures":{"uri":"/videos/117102891/pictures/503683214","active":true,"sizes":[{"width":100,"height":75,"link":"https://i.vimeocdn.com/video/503683214_100x75.jpg"},{"width":200,"height":150,"link":"https://i.vimeocdn.com/video/503683214_200x150.jpg"},{"width":295,"height":166,"link":"https://i.vimeocdn.com/video/503683214_295x166.jpg"},{"width":640,"height":360,"link":"https://i.vimeocdn.com/video/503683214_640x360.jpg"},{"width":960,"height":540,"link":"https://i.vimeocdn.com/video/503683214_960x540.jpg"},{"width":1280,"height":720,"link":"https://i.vimeocdn.com/video/503683214_1280x720.jpg"}]},"tags":[],"stats":{"plays":18},"embed":{"buttons":{"like":true,"watchlater":true,"share":true,"embed":true,"hd":false,"fullscreen":true,"scaling":true},"logos":{"vimeo":true,"custom":{"active":false,"link":null,"sticky":false}},"title":{"name":"user","owner":"user","portrait":"user"},"playbar":true,"volume":true,"color":"00adef"},"metadata":{"connections":{"comments":{"uri":"/videos/117102891/comments","options":["GET","POST"],"total":0},"credits":{"uri":"/videos/117102891/credits","options":["GET","POST"],"total":1},"likes":{"uri":"/videos/117102891/likes","options":["GET"],"total":0},"pictures":{"uri":"/videos/117102891/pictures","options":["GET","POST"],"total":1},"texttracks":{"uri":"/videos/117102891/texttracks","options":["GET","POST"],"total":0}},"interactions":{"watchlater":{"added":false,"added_time":null,"uri":"/users/33772623/watchlater/117102891"}}},"embed_presets":null,"user":{"uri":"/users/33772623","name":"Raw
         Funk Maharishi","link":"https://vimeo.com/rawfunkmaharishi","location":null,"bio":null,"created_time":"2014-10-27T12:06:25+00:00","account":"basic","pictures":{"uri":"/users/33772623/pictures/8588995","active":true,"sizes":[{"width":30,"height":30,"link":"https://i.vimeocdn.com/portrait/8588995_30x30.jpg"},{"width":75,"height":75,"link":"https://i.vimeocdn.com/portrait/8588995_75x75.jpg"},{"width":100,"height":100,"link":"https://i.vimeocdn.com/portrait/8588995_100x100.jpg"},{"width":300,"height":300,"link":"https://i.vimeocdn.com/portrait/8588995_300x300.jpg"}]},"websites":[{"name":"http://rawfunkmaharishi.uk/","link":"http://rawfunkmaharishi.uk/","description":null}],"metadata":{"connections":{"activities":{"uri":"/users/33772623/activities","options":["GET"]},"albums":{"uri":"/users/33772623/albums","options":["GET"],"total":0},"channels":{"uri":"/users/33772623/channels","options":["GET"],"total":0},"feed":{"uri":"/users/33772623/feed","options":["GET"]},"followers":{"uri":"/users/33772623/followers","options":["GET"],"total":0},"following":{"uri":"/users/33772623/following","options":["GET"],"total":0},"groups":{"uri":"/users/33772623/groups","options":["GET"],"total":0},"likes":{"uri":"/users/33772623/likes","options":["GET"],"total":0},"portfolios":{"uri":"/users/33772623/portfolios","options":["GET"],"total":0},"videos":{"uri":"/users/33772623/videos","options":["GET"],"total":4},"watchlater":{"uri":"/users/33772623/watchlater","options":["GET"],"total":0},"shared":{"uri":"/users/33772623/shared/videos","options":["GET"],"total":0},"pictures":{"uri":"/users/33772623/pictures","options":["GET","POST"],"total":1}}},"content_filter":["language","drugs","violence","nudity","safe","unrated"]},"app":null,"status":"available"}'
     http_version: 
-  recorded_at: Tue, 03 Mar 2015 17:44:01 GMT
+  recorded_at: Wed, 04 Mar 2015 22:02:49 GMT
 recorded_with: VCR 2.9.3

data/lib/purdie/bernard.rb

@@ -7,8 +7,8 @@ module Purdie
     def initialize
       @config = Config.new
       begin
-        @sources = Dir.entries(@config['source-dir']).select { |e| e !~ /^\./ }
-        @sources.map! { |s| "#{@config['source-dir']}/#{s}"}
+        @sources = Dir.entries(@config['source_dir']).select { |e| e !~ /^\./ }
+        @sources.map! { |s| "#{@config['source_dir']}/#{s}"}
       rescue Errno::ENOENT
         @sources = nil
       end

data/lib/purdie/config.rb

@@ -14,5 +14,9 @@ module Purdie
     def [] key
       @conf[key]
     end
+
+    def []= key, value
+      @conf[key] = value
+    end
   end
 end

data/lib/purdie/ingester.rb

@@ -21,6 +21,17 @@ module Purdie
       configure
     end
 
+    def configure
+      if @config['services']
+        if @config['services'][Purdie.basename self]
+          if @config['services'][Purdie.basename self]['output_file']
+            @config['output_dir'] = File.dirname(@config['services'][Purdie.basename self]['output_file'])
+            @output_file = File.basename(@config['services'][Purdie.basename self]['output_file'])
+          end
+        end
+      end
+    end
+
     def ingest url
       @items.push distill url
     end
@@ -39,9 +50,8 @@ module Purdie
 
     def write
       if self.has_items?
-        FileUtils.mkdir_p @config['output-dir']
-
-        File.open "#{@config['output-dir']}/#{@output_file}", 'w' do |f|
+        FileUtils.mkdir_p @config['output_dir']
+        File.open "#{@config['output_dir']}/#{@output_file}", 'w' do |f|
           f.write self.to_yaml
         end
       end

data/lib/purdie/services/flickr.rb

@@ -13,6 +13,8 @@ module Purdie
       def configure
         @matcher = 'flickr.com'
         @output_file = 'flickr.yaml'
+
+        super
       end
 
       def licenses
@@ -28,7 +30,7 @@ module Purdie
         results = {}
 
         results['title'] = photo['title']
-        results['title'] = @config['default-title'] if photo['title'] == ''
+        results['title'] = @config['default_title'] if photo['title'] == ''
         results['date'] = photo['dates']['taken'].split(' ')[0]
         results['photo_page'] = photo['urls'][0]['_content']
         results['photo_url'] = FlickRaw.url_m(photo)
@@ -41,7 +43,7 @@ module Purdie
           snapper = photo['tags'].select { |t| t['raw'] =~ /photographer/ }[0]
           results['photographer'] = snapper['raw'].split(':')[1]
         rescue NoMethodError
-          results['photographer'] = @config['photographer-lookups'][photo['owner']['username']]
+          results['photographer'] = @config['photographer_lookups'][photo['owner']['username']]
           unless results['photographer']
             results['photographer'] = photo['owner']['username']
           end

data/lib/purdie/services/soundcloud.rb

@@ -11,6 +11,8 @@ module Purdie
         @host = 'https://api.soundcloud.com'
         @matcher = 'soundcloud.com'
         @output_file = 'soundcloud.yaml'
+
+        super
       end
 
       def all_tracks
@@ -34,8 +36,8 @@ module Purdie
         results['id'] = track['id']
         results['location'] = track['description']
         results['date'] = "%4d-%02d-%02d" % [ track['release_year'], track['release_month'], track['release_day'] ]
-        results['license'] = @config['license-lookups'][track['license']]['full-name']
-        results['license_url'] = @config['license-lookups'][track['license']]['url']
+        results['license'] = @config['license_lookups'][track['license']]['full_name']
+        results['license_url'] = @config['license_lookups'][track['license']]['url']
 
         results
       end

data/lib/purdie/services/vimeo.rb

@@ -9,6 +9,8 @@ module Purdie
         @host = 'https://api.vimeo.com'
         @matcher = 'vimeo.com'
         @output_file = 'vimeo.yaml'
+
+        super
       end
 
       def get url
@@ -30,8 +32,8 @@ module Purdie
 
         results['title'] = video['name']
         results['id'] = @id
-        results['license'] = @config['license-lookups'][video['license']]['full-name']
-        results['license_url'] = @config['license-lookups'][video['license']]['url']
+        results['license'] = @config['license_lookups'][video['license']]['full_name']
+        results['license_url'] = @config['license_lookups'][video['license']]['url']
 
         results
       end

data/lib/purdie/version.rb

@@ -1,3 +1,3 @@
 module Purdie
-  VERSION = "0.0.8"
+  VERSION = "0.0.9"
 end

data/spec/config_spec.rb

@@ -4,12 +4,12 @@ module Purdie
   describe Config do
     it 'has defaults' do
       c = Config.new
-      expect(c['output-dir']).to eq '_data'
+      expect(c['output_dir']).to eq '_data'
     end
 
     it 'reads a local config file' do
       c = Config.new
-      expect(c['default-title']).to eq 'Raw Funk Maharishi'
+      expect(c['default_title']).to eq 'Raw Funk Maharishi'
     end
   end
 end

data/spec/purdie_spec.rb

@@ -4,7 +4,7 @@ module Purdie
   describe Bernard do
     it 'has config' do
       b = Bernard.new
-      expect(b.config['output-dir']).to eq '_data'
+      expect(b.config['output_dir']).to eq '_data'
     end
 
     it 'does not throw a fit when initialized with no _sources dir' do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: purdie
 version: !ruby/object:Gem::Version
-  version: 0.0.8
+  version: 0.0.9
 platform: ruby
 authors:
 - pikesley
@@ -284,6 +284,7 @@ files:
 - cruft.rb
 - features/bad-url.feature
 - features/comments.feature
+- features/flickr-alternative-outfile.feature
 - features/flickr.feature
 - features/no-sources.feature
 - features/purdie.feature
@@ -364,6 +365,7 @@ summary: Capture metadata from Flickr, SoundCloud et al for when we're building
 test_files:
 - features/bad-url.feature
 - features/comments.feature
+- features/flickr-alternative-outfile.feature
 - features/flickr.feature
 - features/no-sources.feature
 - features/purdie.feature