niwa_textream 0.1.0

data/.travis.yml

@@ -0,0 +1,3 @@
+language: ruby
+rvm:
+  - 2.3.1

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at niwatolli3@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in niwa_textream.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2016 niwatolli3
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,52 @@
+# NiwaTextream
+
+[![Build Status](https://travis-ci.org/niwatolli3/niwa_textream.svg?branch=master)](https://travis-ci.org/niwatolli3/niwa_textream)
+[![Code Climate](https://codeclimate.com/github/niwatolli3/niwa_textream/badges/gpa.svg)](https://codeclimate.com/github/niwatolli3/niwa_textream)
+[![Test Coverage](https://codeclimate.com/github/niwatolli3/niwa_textream/badges/coverage.svg)](https://codeclimate.com/github/niwatolli3/niwa_textream/coverage)
+[![Issue Count](https://codeclimate.com/github/niwatolli3/niwa_textream/badges/issue_count.svg)](https://codeclimate.com/github/niwatolli3/niwa_textream)
+
+NiwaTextream is a scraping library for Yahoo! Textream.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'niwa_textream'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install niwa_textream
+
+## Usage
+
+This gem is following Page Object Pattern.
+
+Get category list displayed on top page
+```ruby
+mecha = Mechanize.new
+NiwaTextream::TopPage.goTo(mecha)
+@topPage = NiwaTextream::TopPage.new(mecha)
+print(@topPage.categories)
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/niwa_textream. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+task :default => :spec
+
+begin
+  require 'rspec/core/rake_task'
+  RSpec::Core::RakeTask.new(:spec) do |spec|
+    spec.pattern = 'spec/**/*_spec.rb'
+  end
+rescue LoadError => e
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "niwa_textream"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/niwa_textream.rb

@@ -0,0 +1,13 @@
+require "niwa_textream/version"
+require "mechanize"
+
+%w[ models/category pages/top/top_page pages/category/category_page ].each do |file|
+  require "niwa_textream/#{ file }"
+end
+
+module NiwaTextream
+  def self.test
+    TopPage.new(Mechanize.new)
+    TopPage.url
+  end
+end

data/lib/niwa_textream/models/base_model.rb

@@ -0,0 +1,10 @@
+module NiwaTextream
+  # Contains Category object.
+  class BaseModel
+    @@elem = nil
+
+    # element of this model
+    attr_accessor :elem
+
+  end
+end

data/lib/niwa_textream/models/category.rb

@@ -0,0 +1,18 @@
+require 'niwa_textream/models/base_model'
+
+module NiwaTextream
+  # Contains Category object.
+  class Category < BaseModel
+    # category name
+    attr_accessor :name, :parent, :num_thread
+
+    @name = nil
+    # parent's category object. top category: Nil
+    @parent = nil
+
+    @num_thread = -1
+
+    def initialize
+    end
+  end
+end

data/lib/niwa_textream/models/message.rb

@@ -0,0 +1,16 @@
+require 'niwa_textream/models/base_model'
+
+module NiwaTextream
+  class Message < BaseModel
+    attr_accessor :message_number, :body, :posted_at
+    @message_number
+    @body
+    @posted_at
+
+    def initialize(elem, body, posted_at)
+      @elem = elem
+      @body = body
+      @posted_at = posted_at
+    end
+  end
+end

data/lib/niwa_textream/models/thread.rb

@@ -0,0 +1,17 @@
+require 'niwa_textream/models/base_model'
+
+module NiwaTextream
+  class Thread < BaseModel
+    attr_accessor :title, :num_comment, :last_updated
+    @title
+    @num_comment
+    @last_updated
+
+    def initialize(elem, title, num_comment, last_updated)
+      @elem = elem
+      @title = title
+      @num_comment = num_comment
+      @last_updated = last_updated
+    end
+  end
+end

data/lib/niwa_textream/pages/category/category_page.rb

@@ -0,0 +1,37 @@
+require 'niwa_textream/pages/main/main_page.rb'
+require 'niwa_textream/pages/thread/thread_page'
+
+module NiwaTextream
+  class CategoryPage < MainPage
+    @@url = "http://textream.yahoo.co.jp/category/%{category_id}"
+    @categories = nil
+    attr_accessor :categories
+
+    def initialize(mechanize)
+      super(mechanize)
+      setCategory
+      return self
+    end
+
+    # set category(its parent category is not set)
+    def setCategory
+      @categories = {}
+      @mechanize.page.search("//a[@class='cf']").each do |cat|
+        num_thread_with_bracket = cat.search('.//span')[0].inner_text
+        num_thread = num_thread_with_bracket.match('\((\d+)\)')[1]
+        catObj = NiwaTextream::Category.new
+        catObj.elem = cat
+        catObj.name = cat.inner_text.match('(.+?)\((.+?)\)')[1]
+        catObj.num_thread = num_thread
+        @categories[catObj.name] = catObj
+        puts("--#CategoryPage#--")
+        puts(catObj.name)
+      end
+    end
+
+    def clickCategory(name)
+      @mechanize.click(@categories[name].elem)
+      return NiwaTextream::ThreadPage.new(@mechanize)
+    end
+  end
+end

data/lib/niwa_textream/pages/main/main_page.rb

@@ -0,0 +1,28 @@
+module NiwaTextream
+  class MainPage
+    attr_accessor :url
+
+    @@url = nil
+    # mechanize object
+    @mechanize = nil
+    # header object
+    @header = nil
+    # body object
+    @body = nil
+
+    @page = nil
+
+    def initialize(mechanize)
+      @mechanize = mechanize
+    end
+
+    # go to page by using mechanize
+    def self.goTo(mechanize)
+      @page = mechanize.get(@@url)
+    end
+
+    def self.url
+      @@url
+    end
+  end
+end

data/lib/niwa_textream/pages/message/message_header.rb

@@ -0,0 +1,45 @@
+require 'niwa_textream/pages/main/main_page'
+require 'niwa_textream/pages/message/message_header'
+require 'niwa_textream/models/message'
+
+module NiwaTextream
+  class MessageHeader < MainPage
+    # @@url = "http://textream.yahoo.co.jp/category/%{category_id}"
+    attr_accessor :bar, :prevBtn, :nextBtn
+    @bar = nil
+    @prevBtn = nil
+    @nextBtn = nil
+
+    def initialize(mechanize)
+      super(mechanize)
+      setTopBg
+      return self
+    end
+
+    def setTopBg
+      @messages = []
+      @bar = @mechanize.page.search("//div[@id='toppg']")
+      @prevBtn = @bar.search(".//li[@class='prev']/a")[0]
+      @nextBtn = @bar.search(".//li[@class='next']/a")[0]
+    end
+
+    def prevPageAvail?
+      return @prevBtn.nil? == false
+    end
+
+    def nextPageAvail?
+      return @nextBtn.nil? == false
+    end
+
+    def clickPrevButton
+      @mechanize.click(@prevBtn)
+      return MessagePage.new(@mechanize)
+    end
+
+    def clickNextButton
+      @mechanize.click(@nextBtn)
+      return MessagePage.new(@mechanize)
+    end
+  end
+end
+

data/lib/niwa_textream/pages/message/message_page.rb

@@ -0,0 +1,34 @@
+require 'niwa_textream/pages/main/main_page.rb'
+require 'niwa_textream/pages/thread/thread_page'
+require 'niwa_textream/utils/TimeUtil'
+require 'niwa_textream/models/message'
+require 'niwa_textream/pages/message/message_header'
+
+module NiwaTextream
+  class MessagePage < MainPage
+    # @@url = "http://textream.yahoo.co.jp/category/%{category_id}"
+    @messages = nil
+    @message_header = nil
+    attr_accessor :messages, :message_header
+
+    def initialize(mechanize)
+      super(mechanize)
+      @message_header = MessageHeader.new(mechanize)
+      setMessages
+      return self
+    end
+
+    def setMessages
+      @messages = []
+      @mechanize.page.search("//ul[@class='commentList']//div[@class='comment']").each do |message|
+        message_id = message['data-comment']
+        body = message.search(".//p[@class='comText']")[0].inner_text()
+        posted_at_str = message.search(".//p[@class='comWriter']/span/a").inner_text()
+        posted_at = NiwaTextream::TimeUtil.getDateTime(posted_at_str)
+        messageObj = NiwaTextream::Message.new(message, body, posted_at)
+        @messages.push(messageObj)
+        puts("#{body}, #{posted_at}")
+      end
+    end
+  end
+end

data/lib/niwa_textream/pages/thread/thread_page.rb

@@ -0,0 +1,64 @@
+require 'niwa_textream/pages/main/main_page.rb'
+require 'niwa_textream/models/thread'
+require 'niwa_textream/pages/message/message_page'
+
+module NiwaTextream
+  # thread list
+  class ThreadPage < MainPage
+    @@url = "http://textream.yahoo.co.jp/thread/%{category_id}"
+    @threads = nil
+    @next_page_elem
+    @prev_page_elem
+    attr_accessor :threads, :prev_page_elem, :next_page_elem
+
+    def initialize(mechanize)
+      super(mechanize)
+      setThreads
+      setNextPageElem
+      setPrevPageElem
+      return self
+    end
+
+    def setThreads
+      @threads = []
+      @mechanize.page.search("//*[@id='trdlst']//dl[@class='cf']").each do |thread|
+        thread_title_elem = thread.search(".//a[@data-sec='trdlst']")[0]
+        last_updated = DateTime.parse(thread.search(".//li[@class='time bold']")[0].inner_text())
+        num_comment = thread.search("//*[@class='commentCount']").inner_text().to_i
+        @threads.push(NiwaTextream::Thread.new(thread_title_elem, thread_title_elem.inner_text(), num_comment, last_updated))
+        puts("#{thread_title_elem.inner_text()}, #{num_comment}, #{last_updated}")
+      end
+    end
+
+    def setNextPageElem
+      @next_page_elem = @mechanize.page.search("//*[@class='btnNext']/a")[0]
+    end
+
+    def setPrevPageElem
+      @prev_page_elem = @mechanize.page.search("//*[@class='btnPrev']/a")[0]
+    end
+
+    def prevPageAvail?
+      @prev_page_elem.nil? == false
+    end
+
+    def nextPageAvail?
+      @next_page_elem.nil? == false
+    end
+
+    def clickNextPage
+      @mechanize.click(@next_page_elem)
+      return NiwaTextream::ThreadPage.new(@mechanize)
+    end
+
+    def clickPrevPage
+      @mechanize.click(@prev_page_elem)
+      return NiwaTextream::ThreadPage.new(@mechanize)
+    end
+
+    def clickThread(thread)
+      @mechanize.click(thread.elem)
+      return MessagePage.new(@mechanize)
+    end
+  end
+end