bookmeter_scraper 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6c328c66bbd91ea36ee0471c01f4e16a69ffd347
+  data.tar.gz: d1d5fde5a9d223c0aada00c670d1feed0b5e9c3a
+SHA512:
+  metadata.gz: 9a2c3a6149faa92850aca03c455bd5ccd95f8eddaee005e4befd4daa328c340e171df5018803f9feb0ebb0f7fdf630f91d9b9259c52fb358f5b3ccf5570595e7
+  data.tar.gz: dcbf2db1efa63928b1c00a00d35af13ffba46345578c84b40b26a9b320ba69acd899c66c9319af5e15894a3d4e5ccc5fadbf64136fbbcbea804a0097f0729251

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+spec/examples.txt

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+rvm:
+  - 2.0.0
+  - 2.1.0
+  - 2.2.0
+  - 2.3.0
+before_install:
+  - gem update bundler
+env:
+  - TZ=Asia/Tokyo

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in bookmeter_scraper.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Kohei Yamamoto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.ja.md

@@ -0,0 +1,157 @@
+# Bookmeter Scraper [![Build Status](https://travis-ci.org/kymmt90/bookmeter_scraper.svg?branch=master)](https://travis-ci.org/kymmt90/bookmeter_scraper)
+
+[読書メーター](http://bookmeter.com)の情報をスクレイピングして Ruby で扱えるようにするための gem です。
+
+- 書籍情報
+  - 読んだ本
+  - 読んでる本
+  - 積読本
+  - 読みたい本
+- お気に入り / お気に入られユーザ
+- ユーザプロフィール
+
+を取得可能です。
+
+## 注意
+
+スクレイピングの頻度は常識の範囲内にとどめてください。読書メーターのサーバーへ故意に著しい負荷をかける行為は、利用規約の第 9 条で禁止されています。
+
+- [利用規約 - 読書メーター](http://bookmeter.com/terms.php)
+
+## 使いかた
+
+この gem を使うときは以下のコードが必要です。
+
+```ruby
+require 'bookmeter_scraper'
+```
+
+### ログイン
+
+書籍情報、お気に入り / お気に入られユーザ情報を取得するには、`Bookmeter.log_in` でログインしておく必要があります。
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.log_in('example@example.com', 'password')
+bookmeter.logged_in?    # true
+```
+
+`Bookmeter#log_in` でもログイン可能です。
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.new
+bookmeter.log_in('example@example.com', 'password')
+```
+
+### 書籍情報の取得
+
+以下の書籍情報
+
+- 読んだ本
+- 読んでる本
+- 積読本
+- 読みたい本
+
+を取得できます。取得には事前のログインが必要です。
+
+#### 読んだ本
+
+`Bookmeter#read_books` で「読んだ本」情報が取得できます。
+
+```ruby
+books = bookmeter.read_books        # ログインユーザの「読んだ本」を取得
+bookmeter.read_books('01010101')    # 他のユーザの ID を指定して、そのユーザの「読んだ本」を取得
+```
+
+書籍情報は書名 `name` と読了日（初読了日と再読日の両方）の配列 `read_dates` を属性として持つ `Struct` の配列として取得できます。
+
+```ruby
+books[0].name
+books[0].read_dates
+```
+
+さらに、`Bookmeter#read_books_in` で特定年月の「読んだ本」情報が取得できます。
+
+```ruby
+books = bookmeter.read_books_in(2016, 1)                # ログインユーザが 2016 年 1 月に「読んだ本」を取得
+books = bookmeter.read_books_in(2016, 1, '01010101')    # ID で指定した他のユーザが 2016 年 1 月に「読んだ本」を取得
+```
+
+#### 読んでる本 / 積読本 / 読みたい本
+
+「読んだ本」以外の書籍情報
+
+- 読んでる本
+- 積読本
+- 読みたい本
+
+も、それぞれ
+
+- `Bookmeter#reading_books`
+- `Bookmeter#tsundoku`
+- `Bookmeter#wish_list`
+
+で取得できます。
+
+```ruby
+books = bookmeter.reading_books    # ログインユーザの「読んでる本」を取得
+books[0].name
+books[0].read_dates    # 読了日の Array は空
+
+bookmeter.tsundoku     # ログインユーザの「積読本」を取得
+bookmeter.wish_list    # ログインユーザの「読みたい本」を取得
+```
+
+### お気に入り / お気に入られユーザ情報の取得
+
+`Bookmeter#followings` と `Bookmeter#followers` でログインユーザが参照できるお気に入り / お気に入られユーザの情報を取得できます。取得には事前のログインが必要です。
+
+```ruby
+following_users = bookmeter.followings    # 「お気に入り」ユーザの情報を取得
+followers = bookmeter.followers           # 「お気に入られ」ユーザの情報を取得
+```
+
+ユーザ情報はユーザ名 `name` とユーザ ID `id` を持つ `Struct` の配列として取得できます。
+
+```ruby
+following_users[0].name
+following_users[0].id
+followers[0].name
+followers[0].id
+```
+
+#### 注意
+
+**お気に入り / お気に入られのページにページネーションが存在する場合には未対応です。**
+
+### ユーザのプロフィールの取得
+
+`Bookmeter#profile` でユーザのプロフィールを取得できます。プロフィールはログインなしで閲覧できるため、ログインは不要です。
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.new
+user_id = '000000'
+profile = bookmeter.profile(user_id)    # 任意ユーザの ID を指定してプロフィールを取得可能
+```
+
+プロフィール情報は以下の属性を持つ `Struct` として取得できます。プロフィールで設定されていない属性は `nil` となります。
+
+```ruby
+profile.name                # ユーザ名
+profile.gender              # 性別
+profile.age                 # 年齢
+profile.blood_type          # 血液型
+profile.job                 # 職業
+profile.address             # 現住所
+profile.url                 # URL / ブログ
+profile.description         # 自己紹介
+profile.first_day           # 記録初日
+profile.elapsed_days        # 経過日数
+profile.read_books_count    # 読んだ本の数
+profile.read_pages_count    # 読んだページの数
+profile.reviews_count       # 感想/レビューの数
+profile.bookshelfs_count    # 本棚の数
+```
+
+## ライセンス
+
+[MIT License](http://opensource.org/licenses/MIT)

data/README.md

@@ -0,0 +1,163 @@
+# Bookmeter Scraper [![Build Status](https://travis-ci.org/kymmt90/bookmeter_scraper.svg?branch=master)](https://travis-ci.org/kymmt90/bookmeter_scraper)
+
+A library for scraping [Bookmeter](http://bookmeter.com).
+
+Japanese README is [here](https://github.com/kymmt90/bookmeter_scraper/blob/master/README.ja.md).
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'bookmeter_scraper'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install bookmeter_scraper
+
+
+## Usage
+
+Add this line to your code before using this library:
+
+```ruby
+require 'bookmeter_scraper'
+```
+
+### Log in
+
+You need to log in Bookmeter to get books and followings / followers information by `Bookmeter.log_in`:
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.log_in('example@example.com', 'password')
+bookmeter.logged_in?    # true
+```
+
+`Bookmeter#log_in` is also available:
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.new
+bookmeter.log_in('example@example.com', 'password')
+```
+
+### Get books information
+
+You can get books information:
+
+- read books
+- reading books
+- tsundoku (stockpile)
+- wish list
+
+You need to log in Bookmeter in advance to get these information.
+
+#### Read books
+
+You can get read books information by `Bookmeter#read_books`:
+
+```ruby
+books = bookmeter.read_books        # get read books of the logged in user
+bookmeter.read_books('01010101')    # get read books of a user specified by ID
+```
+
+Books infomation is an array of `Struct` which has `name` and `read_dates` as attributes.
+`read_dates` is an array of finished reading dates (first finished date and reread dates):
+
+```ruby
+books[0].name
+books[0].read_dates
+```
+
+To specify year-month for read books, you can use `Bookmeter#read_books_in`:
+
+```ruby
+books = bookmeter.read_books_in(2016, 1)                # get read books of the logged in user in 2016-01
+books = bookmeter.read_books_in(2016, 1, '01010101')    # get read books of a user in 2016-01
+```
+
+#### Reading books / Tsundoku / Wish list
+
+You can get other books information:
+
+- `Bookmeter#reading_books`
+- `Bookmeter#tsundoku`
+- `Bookmeter#wish_list`
+
+```ruby
+books = bookmeter.reading_books
+books[0].name
+books[0].read_dates    # this array is empty
+
+bookmeter.tsundoku
+bookmeter.wish_list
+```
+
+### Get followings users / followers information
+
+You can get following users (followings) and followers information by `Bookmeter#followings` and `Bookmeter#followers`:
+
+```ruby
+following_users = bookmeter.followings
+followers = bookmeter.followers
+```
+
+You need to log in Bookmeter in advance to get these information.
+
+Users information is an array of `Struct` which has `name` and `id` as attributes.
+
+```ruby
+following_users[0].name
+following_users[0].id
+followers[0].name
+followers[0].id
+```
+
+#### Notice
+
+**`Bookmeter#followings` and `Bookmeter#followers` have not supported paginated followings / followers pages yet.**
+
+### Get user profile
+
+You can get a user profile by `Bookmeter#profile`:
+
+```ruby
+bookmeter = BookmeterScraper::Bookmeter.new
+user_id = '000000'
+profile = bookmeter.profile(user_id)    # You can specify arbitrary user ID
+```
+
+You do not need to log in to get user profiles.
+Profile information is `Struct` which has these attributes:
+
+```ruby
+profile.name
+profile.gender
+profile.age
+profile.blood_type
+profile.job
+profile.address
+profile.url
+profile.description
+profile.first_day
+profile.elapsed_days
+profile.read_books_count
+profile.read_pages_count
+profile.reviews_count
+profile.bookshelfs_count
+```
+
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/kymmt90/bookmeter_scraper.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,5 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new("spec")
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "bookmeter_scraper"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/bookmeter_scraper.gemspec

@@ -0,0 +1,28 @@
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'bookmeter_scraper/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "bookmeter_scraper"
+  spec.version       = BookmeterScraper::VERSION
+  spec.authors       = ["Kohei Yamamoto"]
+  spec.email         = ["kymmt90@gmail.com"]
+
+  spec.summary       = %q{Bookmeter scraping library}
+  spec.description   = %q{Bookmeter scraping library}
+  spec.homepage      = "https://github.com/kymmt90/bookmeter_scraper"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.4"
+  spec.add_development_dependency "webmock", "~> 1.22"
+
+  spec.add_dependency "yasuri", "~> 0.0"
+  spec.add_dependency "mechanize", "~> 2.7"
+end

data/exe/bookmeter_scraper

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+require "bookmeter_scraper"

data/lib/bookmeter_scraper.rb

@@ -0,0 +1,2 @@
+require 'bookmeter_scraper/bookmeter'
+require 'bookmeter_scraper/version'

data/lib/bookmeter_scraper/bookmeter.rb

@@ -0,0 +1,414 @@
+require 'mechanize'
+require 'yasuri'
+
+module BookmeterScraper
+  class Bookmeter
+    ROOT_URI  = 'http://bookmeter.com'.freeze
+    LOGIN_URI = "#{ROOT_URI}/login".freeze
+
+    PROFILE_ATTRIBUTES = %i(name gender age blood_type job address url description first_day elapsed_days read_books_count read_pages_count reviews_count bookshelfs_count)
+    Profile = Struct.new(*PROFILE_ATTRIBUTES)
+
+    BOOK_ATTRIBUTES = %i(name read_dates)
+    Book = Struct.new(*BOOK_ATTRIBUTES)
+
+    USER_ATTRIBUTES = %i(name id)
+    User = Struct.new(*USER_ATTRIBUTES)
+
+    JP_ATTRIBUTE_NAMES = {
+      gender: '性別',
+      age: '年齢',
+      blood_type: '血液型',
+      job: '職業',
+      address: '現住所',
+      url: 'URL / ブログ',
+      description: '自己紹介',
+      first_day: '記録初日',
+      elapsed_days: '経過日数',
+      read_books_count: '読んだ本',
+      read_pages_count: '読んだページ',
+      reviews_count: '感想/レビュー',
+      bookshelfs_count: '本棚',
+    }
+
+    NUM_BOOKS_PER_PAGE = 40
+    NUM_USERS_PER_PAGE = 20
+
+    attr_reader :log_in_user_id
+
+    def self.mypage_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}"
+    end
+
+    def self.read_books_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/booklist"
+    end
+
+    def self.reading_books_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/booklistnow"
+    end
+
+    def self.tsundoku_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/booklisttun"
+    end
+
+    def self.wish_list_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/booklistpre"
+    end
+
+    def self.followings_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/favorite_user"
+    end
+
+    def self.followers_uri(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      "#{ROOT_URI}/u/#{user_id}/favorited_user"
+    end
+
+    def self.log_in(mail, password)
+      Bookmeter.new.tap do |bookmeter|
+        bookmeter.log_in(mail, password)
+      end
+    end
+
+
+    def initialize(agent = nil)
+      @agent = agent.nil? ? Bookmeter.new_agent : agent
+      @logged_in = false
+    end
+
+    def log_in(mail, password)
+      raise BookmeterError if @agent.nil?
+
+      next_page = nil
+      page = @agent.get(LOGIN_URI) do |page|
+        next_page = page.form_with(action: '/login') do |form|
+          form.field_with(name: 'mail').value = mail
+          form.field_with(name: 'password').value = password
+        end.submit
+      end
+      @logged_in = next_page.uri.to_s == ROOT_URI + '/'
+      return unless logged_in?
+
+      mypage = next_page.link_with(text: 'マイページ').click
+      @log_in_user_id = extract_user_id(mypage)
+    end
+
+    def logged_in?
+      @logged_in
+    end
+
+    def profile(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+
+      mypage = @agent.get(Bookmeter.mypage_uri(user_id))
+
+      profile_dl_tags = mypage.search('#side_left > div.inner > div.profile > dl')
+      jp_attribute_names = profile_dl_tags.map { |i| i.children[0].children.text }
+      attribute_values   = profile_dl_tags.map { |i| i.children[1].children.text }
+      jp_attributes = Hash[jp_attribute_names.zip(attribute_values)]
+      attributes = PROFILE_ATTRIBUTES.map do |attribute|
+        jp_attributes[JP_ATTRIBUTE_NAMES[attribute]]
+      end
+      attributes[0] = mypage.at_css('#side_left > div.inner > h3').text
+
+      Profile.new(*attributes)
+    end
+
+    def read_books(user_id = @log_in_user_id)
+      books = get_books(user_id, :read_books_uri)
+      books.each { |b| yield b } if block_given?
+      books
+    end
+
+    def read_books_in(year, month, user_id = @log_in_user_id)
+      date = Time.local(year, month)
+      books = get_read_books(user_id, date)
+      books.each { |b| yield b } if block_given?
+      books
+    end
+
+    def reading_books(user_id = @log_in_user_id)
+      books = get_books(user_id, :reading_books_uri)
+      books.each { |b| yield b } if block_given?
+      books
+    end
+
+    def tsundoku(user_id = @log_in_user_id)
+      books = get_books(user_id, :tsundoku_uri)
+      books.each { |b| yield b } if block_given?
+      books
+    end
+
+    def wish_list(user_id = @log_in_user_id)
+      books = get_books(user_id, :wish_list_uri)
+      books.each { |b| yield b } if block_given?
+      books
+    end
+
+    def followings(user_id = @log_in_user_id)
+      users = get_followings(user_id)
+    end
+
+    def followers(user_id = @log_in_user_id)
+      users = get_followers(user_id)
+    end
+
+    private
+
+    def self.new_agent
+      agent = Mechanize.new do |a|
+        a.user_agent_alias = 'Mac Safari'
+      end
+    end
+
+    def extract_user_id(page)
+      page.uri.to_s.match(/\/u\/(\d+)$/)[1]
+    end
+
+    def get_books(user_id, uri_method)
+      books = []
+      scraped_pages = scrape_book_pages(user_id, uri_method)
+      scraped_pages.each do |page|
+        books << get_book_structs(page)
+        books.flatten!
+      end
+      books
+    end
+
+    def get_read_books(user_id, target_ym)
+      result = []
+      scrape_book_pages(user_id, :read_books_uri).each do |page|
+        first_book_date = get_read_date(page['book_1_link'])
+        last_book_date  = get_last_book_date(page)
+
+        first_book_ym = Time.local(first_book_date['year'].to_i, first_book_date['month'].to_i)
+        last_book_ym  = Time.local(last_book_date['year'].to_i, last_book_date['month'].to_i)
+
+        if target_ym < last_book_ym
+          next
+        elsif target_ym == first_book_ym && target_ym > last_book_ym
+          result.concat(get_target_books(target_ym, page))
+          break
+        elsif target_ym < first_book_ym && target_ym > last_book_ym
+          result.concat(get_target_books(target_ym, page))
+          break
+        elsif target_ym <= first_book_ym && target_ym >= last_book_ym
+          result.concat(get_target_books(target_ym, page))
+        elsif target_ym > first_book_ym
+          break
+        end
+      end
+      result
+    end
+
+    def get_last_book_date(page)
+      NUM_BOOKS_PER_PAGE.downto(1) do |i|
+        link = page["book_#{i}_link"]
+        next if link.empty?
+        return get_read_date(link)
+      end
+    end
+
+    def get_target_books(target_ym, page)
+      target_books = []
+
+      1.upto(NUM_BOOKS_PER_PAGE) do |i|
+        next if page["book_#{i}_link"].empty?
+
+        read_yms = []
+        read_date = get_read_date(page["book_#{i}_link"])
+        read_dates = [Time.local(read_date['year'], read_date['month'], read_date['day'])]
+        read_yms << Time.local(read_date['year'], read_date['month'])
+
+        reread_dates = []
+        reread_dates << get_reread_date(page["book_#{i}_link"])
+        reread_dates.flatten!
+
+        unless reread_dates.empty?
+          reread_dates.each do |date|
+            read_yms << Time.local(date['reread_year'], date['reread_month'])
+          end
+        end
+
+        next unless read_yms.include?(target_ym)
+
+        unless reread_dates.empty?
+          reread_dates.each do |date|
+            read_dates << Time.local(date['reread_year'], date['reread_month'], date['reread_day'])
+          end
+        end
+        book_name = get_book_name(page["book_#{i}_link"])
+        book = Book.new(book_name, read_dates)
+        target_books << book
+      end
+
+      target_books
+    end
+
+    def scrape_book_pages(user_id, uri_method)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      raise ArgumentError unless Bookmeter.methods.include?(uri_method)
+      return [] unless logged_in?
+
+      books_page = @agent.get(Bookmeter.method(uri_method).call(user_id))
+
+      # if books are not found at all
+      return [] if books_page.search('#main_left > div > center > a').empty?
+
+      if books_page.search('span.now_page').empty?
+        books_root = Yasuri.struct_books '//*[@id="main_left"]/div' do
+          1.upto(NUM_BOOKS_PER_PAGE) do |i|
+            send("text_book_#{i}_name", "//*[@id=\"main_left\"]/div/div[#{i + 1}]/div[2]/a")
+            send("text_book_#{i}_link", "//*[@id=\"main_left\"]/div/div[#{i + 1}]/div[2]/a/@href")
+          end
+        end
+        return [books_root.inject(@agent, books_page)]
+      end
+
+      books_root = Yasuri.pages_root '//span[@class="now_page"]/following-sibling::span[1]/a' do
+        text_page_index '//span[@class="now_page"]/a'
+        1.upto(NUM_BOOKS_PER_PAGE) do |i|
+          send("text_book_#{i}_name", "//*[@id=\"main_left\"]/div/div[#{i + 1}]/div[2]/a")
+          send("text_book_#{i}_link", "//*[@id=\"main_left\"]/div/div[#{i + 1}]/div[2]/a/@href")
+        end
+      end
+      books_root.inject(@agent, books_page)
+    end
+
+    def get_book_name(book_link)
+      @agent.get(ROOT_URI + book_link).search('#title').text
+    end
+
+    def get_read_date(book_link)
+      book_page = @agent.get(ROOT_URI + book_link)
+      book_date = Yasuri.struct_date '//*[@id="book_edit_area"]/form[1]/div[2]' do
+        text_year  '//*[@id="read_date_y"]/option[1]', truncate: /\d+/, proc: :to_i
+        text_month '//*[@id="read_date_m"]/option[1]', truncate: /\d+/, proc: :to_i
+        text_day   '//*[@id="read_date_d"]/option[1]', truncate: /\d+/, proc: :to_i
+      end
+      book_date.inject(@agent, book_page)
+    end
+
+    def get_reread_date(book_link)
+      book_page = @agent.get(ROOT_URI + book_link)
+      book_reread_date = Yasuri.struct_reread_date '//*[@id="book_edit_area"]/div/form[1]/div[2]' do
+        text_reread_year  '//div[@class="reread_box"]/form[1]/div[2]/select[1]/option[1]', truncate: /\d+/, proc: :to_i
+        text_reread_month '//div[@class="reread_box"]/form[1]/div[2]/select[2]/option[1]', truncate: /\d+/, proc: :to_i
+        text_reread_day   '//div[@class="reread_box"]/form[1]/div[2]/select[3]/option[1]', truncate: /\d+/, proc: :to_i
+      end
+      book_reread_date.inject(@agent, book_page)
+    end
+
+    def get_book_structs(page)
+      books = []
+
+      1.upto(NUM_BOOKS_PER_PAGE) do |i|
+        break if page["book_#{i}_link"].empty?
+
+        read_dates = []
+        read_date = get_read_date(page["book_#{i}_link"])
+        unless read_date.empty?
+          read_dates << Time.local(read_date['year'], read_date['month'], read_date['day'])
+        end
+
+        reread_dates = []
+        reread_dates << get_reread_date(page["book_#{i}_link"])
+        reread_dates.flatten!
+
+        unless reread_dates.empty?
+          reread_dates.each do |date|
+            read_dates << Time.local(date['reread_year'], date['reread_month'], date['reread_day'])
+          end
+        end
+
+        book_name = get_book_name(page["book_#{i}_link"])
+        book = Book.new(book_name, read_dates)
+        books << book
+      end
+
+      books
+    end
+
+    def get_followings(user_id)
+      users = []
+      scraped_pages = user_id == @log_in_user_id ? scrape_followings_page(user_id)
+                                                 : scrape_others_followings_page(user_id)
+      scraped_pages.each do |page|
+        users << get_user_structs(page)
+        users.flatten!
+      end
+      users
+    end
+
+    def get_followers(user_id)
+      users = []
+      scraped_pages = scrape_followers_page(user_id)
+      scraped_pages.each do |page|
+        users << get_user_structs(page)
+        users.flatten!
+      end
+      users
+    end
+
+    def get_user_structs(page)
+      users = []
+
+      1.upto(NUM_USERS_PER_PAGE) do |i|
+        break if page["user_#{i}_name"].empty?
+
+        user_name = page["user_#{i}_name"]
+        user_id = page["user_#{i}_link"].match(/\/u\/(\d+)$/)[1]
+        user = User.new(user_name, user_id)
+        users << user
+      end
+
+      users
+    end
+
+    def scrape_followings_page(user_id)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      return [] unless logged_in?
+
+      followings_page = @agent.get(Bookmeter.followings_uri(user_id))
+      followings_root = Yasuri.struct_books '//*[@id="main_left"]/div' do
+        1.upto(NUM_USERS_PER_PAGE) do |i|
+          send("text_user_#{i}_name", "//*[@id=\"main_left\"]/div/div[#{i}]/a/@title")
+          send("text_user_#{i}_link", "//*[@id=\"main_left\"]/div/div[#{i}]/a/@href")
+        end
+      end
+      [followings_root.inject(@agent, followings_page)]
+    end
+
+    def scrape_others_followings_page(user_id)
+      scrape_users_listing_page(user_id, :followings_uri)
+    end
+
+    def scrape_followers_page(user_id)
+      scrape_users_listing_page(user_id, :followers_uri)
+    end
+
+    def scrape_users_listing_page(user_id, uri_method)
+      raise ArgumentError unless user_id =~ /^\d+$/
+      raise ArgumentError unless Bookmeter.methods.include?(uri_method)
+      return [] unless logged_in?
+
+      page = @agent.get(Bookmeter.method(uri_method).call(user_id))
+      root = Yasuri.struct_users '//*[@id="main_left"]/div' do
+        1.upto(NUM_USERS_PER_PAGE) do |i|
+          send("text_user_#{i}_name", "//*[@id=\"main_left\"]/div/div[#{i}]/div/div[2]/a/@title")
+          send("text_user_#{i}_link", "//*[@id=\"main_left\"]/div/div[#{i}]/div/div[2]/a/@href")
+        end
+      end
+      [root.inject(@agent, page)]
+    end
+  end
+
+  class BookmeterError < StandardError; end
+end

data/lib/bookmeter_scraper/version.rb

@@ -0,0 +1,3 @@
+module BookmeterScraper
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,144 @@
+--- !ruby/object:Gem::Specification
+name: bookmeter_scraper
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Kohei Yamamoto
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-02-26 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.22'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.22'
+- !ruby/object:Gem::Dependency
+  name: yasuri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+- !ruby/object:Gem::Dependency
+  name: mechanize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.7'
+description: Bookmeter scraping library
+email:
+- kymmt90@gmail.com
+executables:
+- bookmeter_scraper
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.ja.md
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- bookmeter_scraper.gemspec
+- exe/bookmeter_scraper
+- lib/bookmeter_scraper.rb
+- lib/bookmeter_scraper/bookmeter.rb
+- lib/bookmeter_scraper/version.rb
+homepage: https://github.com/kymmt90/bookmeter_scraper
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Bookmeter scraping library
+test_files: []