pinterest-dl 1.0.3 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 73cae48cd87360325168ee188b8a67b25944402f33857a61dee4ea3a13457f91
-  data.tar.gz: 313a17fd00772aa8980ee8389dc5bd934b8217225f4e4b11f130ad4882e629c0
+  metadata.gz: b4d94f0a5bbac63a983308d541b5e50a6a13c1d5bee422c356e56395c686ebcd
+  data.tar.gz: 148473c309ec313a3e0d7ddb0864f0388d22a2bf9edee39e819e54ccaab10745
 SHA512:
-  metadata.gz: e08fc0e08bec2a47a91659d10b1ba3a87e6d2c70e81fc27134366768108ebf3e1a5f10bfea2dc3391c63fa038eb9daba58594ffd8c5997338b240dd21fcaa558
-  data.tar.gz: 7bcfe71875c82f31ddaf9e27a0f648b646bb6381045830d03d6ee0a8fb561283e8a0eb4e85036915a6d71e6f5865e05f0aa4a5e0658f601b593f8e5a6f42f714
+  metadata.gz: 4e26a226009b81da00d41f46f9ccfcdca2f058aa7ff93210b488e1f2082c27f927103f5a0329dcea921ea34a62c49c3ad70c77d411d4a1b569b4b97fbba8c4cf
+  data.tar.gz: 66b3ddf6d10555f9e018198a3de376578b22fc504dfe386d8a49f06bdcc213519e0b273c5f123d34bcb33eb3217da51af7609712a2f1a0a3b67004be6858eb9b

data/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+All notable changes to this project are documented here.
+This project follows [Semantic Versioning](https://semver.org/).
+
+## [2.0.0]
+
+### Added
+- `PinterestDL.download` / `PinterestDL.download_batch` for real file downloads,
+  the latter with per-file progress callbacks and a `PinterestDL::ProgressBar` helper.
+- `PinterestDL.board` for fetching pins from a board or board section, with pagination.
+- `PinterestDL.configure` for global settings: cookies/session, user agent,
+  timeouts, retry count, request rate limiting, and default image quality.
+- `PinterestDL.get_media` to fetch image + video URLs for a pin in one request.
+- Pagination support in `PinterestDL.search` (`pages:` option).
+- `creator` field on pin metadata returned by `search` and `board`.
+- Meaningful exception hierarchy under `PinterestDL::Error` (`NetworkError`,
+  `NotFoundError`, `RateLimitError`, `AuthenticationError`, `InvalidURLError`,
+  `DownloadError`) replacing silent `nil` returns and `puts`-based error reporting.
+- Real CLI: `pinterest-dl image|video|search|board ...`.
+- Configurable timeouts and retry/backoff on all HTTP requests.
+- Small in-memory response cache and optional request rate limiting.
+- Test suite (Minitest) and GitHub Actions CI matrix (Ruby 3.0–3.3).
+- RuboCop configuration.
+
+### Changed
+- User-Agent updated to a current Chrome UA string.
+- All `puts` debug/status output removed from library code in favor of a
+  configurable `Logger` (`PinterestDL.config.logger`).
+- Internals reorganized into `lib/pinterest_dl/{client,configuration,errors,
+  extractors,search,board,downloader,progress_bar,api}.rb`; `PinterestDL::VERSION`
+  is now the single source of truth for the gem version.
+
+### Fixed
+- The gem no longer prints anythign on `require`.
+
+## [1.0.3]
+
+- Initial public release: `get_image_url`, `get_video_url`, `search`,
+  `search_json`, `search_images`, `search_videos`.

data/LICENSE

@@ -0,0 +1,13 @@
+DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+                    Version 2, December 2004
+
+ Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
+
+ Everyone is permitted to copy and distribute verbatim or modified
+ copies of this license document, and changing it is allowed as long
+ as the name is changed.
+
+            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. You just DO WHAT THE FUCK YOU WANT TO.

data/README.md

@@ -1,24 +1,30 @@
-<div align="center">
 
 # pinterest-dl
+**Get direct image/video URLs from Pinterest, search pins, fetch board contents, and download media — as a Ruby library or a CLI.**
 
----
-
-[![Gem Downloads](https://img.shields.io/gem/dt/pinterest-dl)](https://rubygems.org/gems/pinterest-dl)
-[![License](https://img.shields.io/badge/License-MIT-green)](LICENSE)
-[![Ruby](https://img.shields.io/badge/Ruby-2.5+-red)](https://www.ruby-lang.org/)
-
+[![Gem Version](https://img.shields.io/gem/v/pinterest-dl)](https://rubygems.org/gems/pinterest-dl)  
+[![Gem Downloads](https://img.shields.io/gem/dt/pinterest-dl)](https://rubygems.org/gems/pinterest-dl)  
+[![CI](https://github.com/monji024/pinterest-dl/actions/workflows/ci.yml/badge.svg)](https://github.com/monji024/pinterest-dl/actions/workflows/ci.yml)  
+[![License](https://img.shields.io/badge/License-WTFPL-blue)](LICENSE)  
+[![Ruby](https://img.shields.io/badge/Ruby-2.7+-red)](https://www.ruby-lang.org/)  
 <img src="https://img.icons8.com/color/96/000000/pinterest--v1.png" width="80">
 
-</div>
-
 ---
 
-Get direct image and video URLs from Pinterest.
+> **Note:** Users in some regions (e.g. Iran) may need a fu*king VPN to reach Pinterest.  
+> (Yeah, because even in 2026 Pinterest still gives you the middle finger on clean APIs.)
+
+Pinterest doesn't give you a clean, official API for getting a direct media URL out of a pin, walking a board, or downloading a batch of images. `pinterest-dl` does that: point it at a pin, a search query, or a board, and get back real `i.pinimg.com` / `v.pinimg.com` URLs — or the fucking files themselves, already saved to disk.
 
-![Demo](https://github.com/monji024/pinterest-dl/raw/main/test.gif)
+## What it does
 
-> **Note:** Users in Iran may need a VPN to access Pinterest.
+- Pulls the **direct image or video URL** out of a single pin page.  
+- **Searches** Pinterest and returns pin metadata (id, url, title, description, creator, image/video URL), with pagination.  
+- Fetches all pins from a **board** (or a specific board section).  
+- **Downloads** a URL — or a whole batch of them — straight to disk, with progress reporting and per-file error handling.  
+- Supports **authenticated requests** via cookies, for boards/pins that require a logged-in session.  
+- Raises **specific, catchable exceptions** (`NotFoundError`, `RateLimitError`, `NetworkError`, ...) instead of returning `nil` or printing to stdout.  
+- Ships a **CLI** (`pinterest-dl`) for one-off use from the shell.
 
 ## Installation
 
@@ -26,51 +32,156 @@ Get direct image and video URLs from Pinterest.
 gem install pinterest-dl
 ```
 
-## Usage
+Or add it to your Gemfile:
 
 ```ruby
-require 'pinterest-dl'
+gem 'pinterest-dl'
 ```
 
-### Get Image URL
+## Quick start
 
 ```ruby
-url = PinterestDL.get_image_url('https://www.pinterest.com/pin/example/')
+require 'pinterest-dl'
+# Direct image URL
+PinterestDL.get_image_url('https://www.pinterest.com/pin/1234567890123456789/') # => "https://i.pinimg.com/originals/aa/bb/cc/aabbcc...jpg"
+# Direct video URL (raises PinterestDL::NotFoundError if the pin has no video)
+PinterestDL.get_video_url('https://www.pinterest.com/pin/1234567890123456789/') # => "https://v.pinimg.com/videos/mc/720p/aa/bb/cc/aabbcc....mp4"
+# Download it
+PinterestDL.download(
+  PinterestDL.get_image_url('https://www.pinterest.com/pin/1234567890123456789/'),
+  path: 'downloads/pin.jpg'
+)
 ```
 
-### Get Video URL
+## Usage
+
+### Get image / video URLs
 
 ```ruby
+url = PinterestDL.get_image_url('https://www.pinterest.com/pin/example/') # Pick a specific size bucket
+url = PinterestDL.get_image_url('https://www.pinterest.com/pin/example/', quality: :"474x") # quality can be :originals, :"736x", :"474x", :"236x"
 url = PinterestDL.get_video_url('https://www.pinterest.com/pin/example/')
+# Both at once
+media = PinterestDL.get_media('https://www.pinterest.com/pin/example/') # => { image_url: "https://i.pinimg.com/...", video_url: nil }
 ```
 
-### Search Pins
+### Search
 
 ```ruby
 # Search for pins with a specific query
-results = PinterestDL.search('nature landscape', limit: 10)
+results = PinterestDL.search('nature landscape', limit: 10) # => [{ id: "...", pin_url: "...", title: "...", description: "...", creator: "...", image_url: "...", video_url: nil }, ...]
+# Fetch more than one page of results (25 pins/page from Pinterest)
+results = PinterestDL.search('nature landscape', limit: 100, pages: 10)
+# Get only image URLs from search results
+images = PinterestDL.search_images('sunset', limit: 5) # => ["https://i.pinimg.com/originals/...", ...]
+# Get only video URLs from search results
+videos = PinterestDL.search_videos('tutorial', limit: 3) # => ["https://v.pinimg.com/videos/...", ...]
+# Get search results as a JSON string
+json_results = PinterestDL.search_json('anime art', limit: 20)
+```
 
-# Returns an array of pin objects with:
-# id, pin_url, title, description, image_url, video_url
+### Boards & sections
 
-# Get only image URLs from search results
-images = PinterestDL.search_images('sunset', limit: 5)
-# => ["https://i.pinimg.com/originals/...", ...]
+```ruby
+# All pins on a board
+pins = PinterestDL.board('https://www.pinterest.com/username/board-name/', limit: 50)
+# A specific section within a board
+pins = PinterestDL.board('https://www.pinterest.com/username/board-name/section-slug/', limit: 50)
+```
 
-# Get only video URLs from search results
-videos = PinterestDL.search_videos('tutorial', limit: 3)
-# => ["https://v.pinimg.com/videos/...", ...]
+### Downloading
 
-# Get search results as JSON
-json_results = PinterestDL.search_json('anime art', limit: 20)
-# => Returns formatted JSON string
+```ruby
+# Single file
+PinterestDL.download('https://i.pinimg.com/originals/aa/bb/cc/img.jpg', path: 'out/img.jpg')
+# Batch download (returns { succeeded: [...], failed: [{ url:, error: }, ...] })
+urls = PinterestDL.search_images('mountains', limit: 20)
+result = PinterestDL.download_batch(urls, dir: './downloads')
+# With progress reporting
+bar = PinterestDL::ProgressBar.new(total: urls.size)
+PinterestDL.download_batch(urls, dir: './downloads') do |done, total, url|
+  bar.update(done)
+end
+```
+
+### Cookies / authenticated sessions
+
+Some boards and pins require a logged-in session to view. Pass your browser's Pinterest cookies through global configuration:
+
+```ruby
+PinterestDL.configure do |c|
+  c.cookies    = 'sessionid=...; csrftoken=...' # yeah, steal your own fucking session if you have to
+  c.user_agent = 'Mozilla/5.0 ...' # optional override
+end
+```
+
+### Other configuration
+
+```ruby
+PinterestDL.configure do |c|
+  c.open_timeout        = 10   # seconds, connection open timeout
+  c.read_timeout         = 20   # seconds, response read timeout
+  c.max_retries          = 3    # retries on transient network failures
+  c.retry_wait           = 1.0  # base seconds between retries (backs off linearly)
+  c.rate_limit_interval   = 0.5  # minimum seconds between outgoing requests
+  c.quality               = :originals # default image quality for get_image_url
+  c.logger.level          = Logger::INFO # library uses this Logger instead of puts
+end
+```
+
+### Error handling
+
+Every method raises a subclass of `PinterestDL::Error` instead of returning `nil` or printing to the console:
+
+```ruby
+begin
+  PinterestDL.get_image_url('https://www.pinterest.com/pin/does-not-exist/')
+rescue PinterestDL::NotFoundError
+  # pin/board/search had nothing to extract
+rescue PinterestDL::RateLimitError => e
+  # Pinterest throttled us; e.retry_after may hold a hint
+rescue PinterestDL::AuthenticationError
+  # the resource needs c.cookies to be set
+rescue PinterestDL::NetworkError => e
+  # DNS/timeout/connection-level failure; e.cause_error has the original exception
+rescue PinterestDL::InvalidURLError
+  # the given string isn't a Pinterest pin/board URL
+rescue PinterestDL::Error
+  # catch-all for anything above
+end
+```
+
+### CLI
+
+```bash
+pinterest-dl image https://www.pinterest.com/pin/example/
+pinterest-dl image https://www.pinterest.com/pin/example/ -o pin.jpg
+pinterest-dl video https://www.pinterest.com/pin/example/ -o pin.mp4
+pinterest-dl search "cyberpunk city" -n 20
+pinterest-dl search "cyberpunk city" -n 20 -o ./downloads
+pinterest-dl board https://www.pinterest.com/username/board-name/ -n 50 -o ./downloads
+pinterest-dl --help
+```
+
+Flags: `-o/--output`, `-n/--limit`, `-q/--quality`, `-c/--cookies`, `-v/--verbose`.
+
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test     # run the test suite
+bundle exec rake rubocop  # lint
+bundle exec rake          # both (default task)
 ```
 
 ## Links
 
 - [RubyGems](https://rubygems.org/gems/pinterest-dl)
 - [GitHub](https://github.com/monji024/pinterest-dl)
+- [Changelog](CHANGELOG.md)
+- [Issue tracker](https://github.com/monji024/pinterest-dl/issues)
 
 ## License
 
-MIT © [monji024](https://github.com/monji024)
+WTFPL © [monji024](https://github.com/monji024)

data/bin/pinterest-dl

@@ -0,0 +1,107 @@
+#!/usr/bin/env ruby
+# Monji024
+require_relative '../lib/pinterest-dl'
+require 'optparse'
+
+def usage
+  <<~USAGE
+    pinterest-dl v#{PinterestDL::VERSION}
+
+    Usage:
+      pinterest-dl image URL [-o FILE]
+      pinterest-dl video URL [-o FILE]
+      pinterest-dl search QUERY [-n LIMIT] [-o DIR]
+      pinterest-dl board URL [-n LIMIT] [-o DIR]
+
+    Options:
+      -o, --output PATH   File (image/video) or directory (search/board) to save to
+      -n, --limit N       Max number of results (search/board), default 25
+      -q, --quality Q     Image quality: originals, 736x, 474x, 236x (default originals)
+      -c, --cookies STR   Cookie string for authenticated requests
+      -v, --verbose       Print debug logging
+      -h, --help          Show this help
+  USAGE
+end
+
+options = { limit: 25, quality: :originals }
+parser = OptionParser.new do |opts|
+  opts.on('-o PATH', '--output PATH') { |v| options[:output] = v }
+  opts.on('-n N', '--limit N', Integer) { |v| options[:limit] = v }
+  opts.on('-q Q', '--quality Q') { |v| options[:quality] = v.to_sym }
+  opts.on('-c STR', '--cookies STR') { |v| options[:cookies] = v }
+  opts.on('-v', '--verbose') { options[:verbose] = true }
+  opts.on('-h', '--help') do
+    puts usage
+    exit 0
+  end
+end
+
+begin
+  parser.parse!(ARGV)
+rescue OptionParser::ParseError => e
+  warn "pinterest-dl: #{e.message}"
+  exit 1
+end
+
+command = ARGV.shift
+target = ARGV.shift
+
+if command.nil? || target.nil?
+  puts usage
+  exit(command.nil? ? 0 : 1)
+end
+
+PinterestDL.configure do |c|
+  c.cookies = options[:cookies] if options[:cookies]
+  c.logger.level = Logger::DEBUG if options[:verbose]
+end
+
+def die(message)
+  warn "pinterest-dl: #{message}"
+  exit 1
+end
+
+begin
+  case command
+  when 'image'
+    url = PinterestDL.get_image_url(target, quality: options[:quality])
+    if options[:output]
+      PinterestDL.download(url, path: options[:output])
+      puts "Saved to #{options[:output]}"
+    else
+      puts url
+    end
+  when 'video'
+    url = PinterestDL.get_video_url(target)
+    if options[:output]
+      PinterestDL.download(url, path: options[:output])
+      puts "Saved to #{options[:output]}"
+    else
+      puts url
+    end
+  when 'search'
+    pins = PinterestDL.search(target, limit: options[:limit])
+    if options[:output]
+      urls = pins.filter_map { |p| p[:image_url] }
+      bar = PinterestDL::ProgressBar.new(total: urls.size)
+      result = PinterestDL.download_batch(urls, dir: options[:output]) { |done, total, _| bar.update(done) }
+      puts "Downloaded #{result[:succeeded].size}/#{urls.size} images to #{options[:output]}"
+    else
+      puts JSON.pretty_generate(pins)
+    end
+  when 'board'
+    pins = PinterestDL.board(target, limit: options[:limit])
+    if options[:output]
+      urls = pins.filter_map { |p| p[:image_url] }
+      bar = PinterestDL::ProgressBar.new(total: urls.size)
+      result = PinterestDL.download_batch(urls, dir: options[:output]) { |done, total, _| bar.update(done) }
+      puts "Downloaded #{result[:succeeded].size}/#{urls.size} images to #{options[:output]}"
+    else
+      puts JSON.pretty_generate(pins)
+    end
+  else
+    die "unknown command '#{command}'\n\n#{usage}"
+  end
+rescue PinterestDL::Error => e
+  die e.message
+end

data/lib/pinterest-dl.rb

@@ -1,193 +1,17 @@
-#!/usr/bin/env ruby
-# Pinterest downloader
-# creator: Monji 
-# ver: 1.0.3
-# github: https://github.com/monji024/pinterest-dl
-
-require 'net/http'
-require 'uri'
-require 'json'
-require 'openssl'
-
+# frozen_string_literal: true
+
+# Monji024
+require_relative 'pinterest_dl/version'
+require_relative 'pinterest_dl/errors'
+require_relative 'pinterest_dl/configuration'
+require_relative 'pinterest_dl/client'
+require_relative 'pinterest_dl/extractors/image'
+require_relative 'pinterest_dl/extractors/video'
+require_relative 'pinterest_dl/search'
+require_relative 'pinterest_dl/board'
+require_relative 'pinterest_dl/downloader'
+require_relative 'pinterest_dl/progress_bar'
+require_relative 'pinterest_dl/api'
 
 module PinterestDL
-  ver = '1.0.3'.freeze
-  github  = 'https://github.com/monji024/pinterest-dl'.freeze
-
-  puts "pinterest-dl v#{ver} (#{github})\n\n"
-
-  USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'.freeze
-
-  class << self
-    def get_image_url(pin_url)
-      html = fetch_html(pin_url)
-      return nil unless html
-
-      # 1 -> json-ld
-      url = extract_json_ld(html, 'ImageObject')
-      return url if url
-      # 2 -> open graph
-      match = html.match(%r{<meta[^>]*property="og:image"[^>]*content="([^"]+)"})
-      if match
-        url = match[1]
-        puts "image url: #{url}"
-        return url
-      end
-      # 3 -> direct i.pinimg.com patterns
-      match = html.match(%r{https://i\.pinimg\.com/(?:originals|474x|236x|736x)/[^"\s]+})
-      if match
-        url = match[0]
-        reset = "\e[0m"
-        green = "\e[32m"
-        puts "#{green}image url :#{reset} #{url}"
-        return url
-      end
-      puts 'No found:)'
-      nil
-    end
-
-
-    def get_video_url(pin_url)
-      html = fetch_html(pin_url)
-      return nil unless html
-
-      # 1 -> json-ld
-      url = extract_json_ld(html, 'VideoObject')
-      if url
-        url = url.gsub('\\u002F', '/').gsub('\\', '')
-        puts "ok : #{url}"
-        return url
-      end
-      # 2 -> Open Graph video
-      match = html.match(%r{<meta[^>]*property="og:video"[^>]*content="([^"]+)"})
-      if match
-        url = match[1].gsub('\\u002F', '/').gsub('\\', '')
-        puts "video url: #{url}"
-        return url
-      end
-      # 3 -> contentUrl in page source
-      match = html.match(/"contentUrl"\s*:\s*"([^"]+\.mp4[^"]*)"/)
-      if match
-        url = match[1].gsub('\\u002F', '/').gsub('\\', '')
-        green = "\e[32m"
-        reset = "\e[0m"
-        puts "#{green}video url :#{reset} #{url}"
-        return url
-      end
-
-      puts 'No found'
-      nil
-    end
-    def search(query, limit: 25)
-      source_path = "/search/pins/?q=#{URI.encode_www_form_component(query)}"
-      home_uri = URI("https://www.pinterest.com#{source_path}")
-      http = Net::HTTP.new(home_uri.host, home_uri.port)
-      http.use_ssl = true
-      http.open_timeout = 10
-      http.read_timeout = 20
-      home_req = Net::HTTP::Get.new(home_uri, 'User-Agent' => USER_AGENT)
-      home_resp = http.request(home_req)
-      cookies = (home_resp.get_fields('set-cookie') || []).map { |c| c.split(';').first }
-      cookie_header = cookies.join('; ')
-      csrf = cookies.find { |c| c.start_with?('csrftoken=') }&.split('=', 2)&.last || 'undefined'
-      data = {
-        options: { query: query, scope: 'pins', page_size: limit },
-        context: {}}.to_json
-      res_uri = URI('https://www.pinterest.com/resource/BaseSearchResource/get/')
-      res_uri.query = URI.encode_www_form('source_url' => source_path, 'data' => data)
-      req = Net::HTTP::Get.new(res_uri)
-      req['User-Agent'] = USER_AGENT
-      req['Accept'] = 'application/json, text/javascript, */*, q=0.01'
-      req['X-Requested-With'] = 'XMLHttpRequest'
-      req['X-Pinterest-PWS-Handler'] = 'www/search/[scope].js'
-      req['X-CSRFToken'] = csrf
-      req['Referer'] = home_uri.to_s
-      req['Cookie'] = cookie_header
-      resp = http.request(req)
-      unless resp.is_a?(Net::HTTPSuccess)
-        puts "\e[31msearch http err!:\e[0m #{resp.code}"
-        return []
-      end
-      json = begin
-        JSON.parse(resp.body)
-      rescue JSON::ParserError
-        nil
-      end
-      return [] unless json
-      raw_results = (json.dig('resource_response', 'data', 'results') || []).first(limit)
-      pins = raw_results.filter_map do |r|
-        id = r['id']
-        next nil unless id
-        orig_image = r.dig('images', 'orig', 'url')
-        fallback_image = r['images']&.values&.map { |v| v.is_a?(Hash) ? v['url'] : nil }&.compact&.last
-        image_url = orig_image || fallback_image
-        video_url = r.dig('videos', 'video_list')&.values&.first&.dig('url')
-        {
-          id: id,
-          pin_url: "https://www.pinterest.com/pin/#{id}/",
-          title: r['title'] || r['grid_title'],
-          description: r['description'],
-          image_url: image_url,
-          video_url: video_url
-        }
-      end
-      green = "\e[32m"
-      reset = "\e[0m"
-      puts "#{green}found #{pins.size} pin(s) for \"#{query}\"#{reset}"
-      pins
-    rescue StandardError => e
-      puts "\e[31msearch err!:\e[0m #{e.message}"
-      []
-    end
-    def search_json(query, limit: 25)
-      JSON.pretty_generate(search(query, limit: limit))
-    end
-    def search_images(query, limit: 25)
-      search(query, limit: limit).filter_map { |p| p[:image_url] }
-    end
-    def search_videos(query, limit: 25)
-      search(query, limit: limit).filter_map { |p| p[:video_url] }
-    end
-    private
-
-    def fetch_html(url)
-      uri = URI(url)
-      http = Net::HTTP.new(uri.host, uri.port)
-      http.use_ssl = (uri.scheme == 'https')
-      http.max_retries = 2
-      http.open_timeout = 10
-      http.read_timeout = 20
-      resp = http.request(Net::HTTP::Head.new(uri, 'User-Agent' => USER_AGENT))
-      5.times do
-        break unless resp.is_a?(Net::HTTPRedirection)
-        location = resp['location']
-        location = URI.join(uri.to_s, location) if location.start_with?('/')
-        uri = URI(location)
-        http = Net::HTTP.new(uri.host, uri.port)
-        http.use_ssl = (uri.scheme == 'https')
-        resp = http.request(Net::HTTP::Head.new(uri, 'User-Agent' => USER_AGENT))
-      end
-      # get final page
-      req = Net::HTTP::Get.new(uri, 'User-Agent' => USER_AGENT)
-      resp = http.request(req)
-      return resp.body if resp.is_a?(Net::HTTPSuccess)
-      nil
-    rescue StandardError => e
-      red = "\e[31m"
-      reset = "\e[0m"
-      puts "#{red}http err!:#{reset} #{e.message}"
-      nil
-    end
-    def extract_json_ld(html, type)
-      match = html.match(%r{<script[^>]*type="application/ld\+json"[^>]*>(.*?)</script>}m)
-      return nil unless match
-      begin
-        data = JSON.parse(match[1])
-        data = data[0] if data.is_a?(Array) && !data.empty?
-        return data['contentUrl'] if data['@type'] == type && data['contentUrl']
-      rescue JSON::ParserError
-      end
-      nil
-    end
-  end
-end
+end