zombie_writer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d2fc89617a2463cd8ee6a523de15d6b35d39669a
+  data.tar.gz: e5205c355f563885805cf3e0bfa36f1814a0725d
+SHA512:
+  metadata.gz: b74df2a53753e40d64772771d7d9e26791e2e1c9c545d495f736065ef2aff193fe6a0d2454dbaf6c6e6421d8505227c0b9eca4241ea406729c72292010e4a1f2
+  data.tar.gz: 4fc4596727922929a099110e0942969c136fd7f94a144c6d3074506497bdac6e267f5fbcdcf758d408adcdddaae84eb073cbcd3de7caf35b9a904568b8613d38

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/toy_zone/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.3.0
+before_install: gem install bundler -v 1.11.2

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in zombie.gemspec
+gemspec

data/LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Tariq Ali
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,157 @@
+# ZombieWriter
+
+![Logo](zombie_writer_logo.png)
+
+ZombieWriter is a Ruby gem that will enable users to generate news articles by aggregating paragraphs from other sources.
+
+While you have to provide the paragraphs, Zombie will arrange the paragraphs into different articles for you to use and edit to your heart's content. You may choose between Machine Learning (Latent Semantic Analysis and k-means clustering) or Randomization.
+
+## Installation
+
+### Command Line
+
+```
+gem install zombie_writer
+```
+
+### Gemfile
+
+```ruby
+gem 'zombie_writer'
+```
+
+## Usage
+First, decide whether you want to use ZombieWriter::MachineLearning or ZombieWriter::Randomization. ZombieWriter::MachineLearning uses Latent Semantic Analysis and k-means clustering to group your content into different articles, while Randomization simply pick random paragraphs to put in each article.
+
+ZombieWriter::MachineLearning has the potential of producing better-quality articles, but is slightly slower than ZombieWriter::Randomization.
+
+To create a MachineLearning zombie...
+```ruby
+zombie = ZombieWriter::MachineLearning.new
+```
+
+And to create a Randomization zombie...
+```ruby
+zombie = ZombieWriter::Randomization.new
+```
+
+Then, once you have your zombie, add your content.
+```ruby
+zombie.add_string(content: "Lorem ipsum dolor sit amet.",
+sourcetext: "Cicero's Great Speech On Ethics",
+sourceurl: "http://example.com/lorem-ipsum")
+```
+
+In the generated article, this content will appear in Markdown as:
+```markdown
+Lorem ipsum dolor sit amet.--[Cicero's Great Speech On Ethics](http://example.com/lorem-ipsum)
+```
+
+If your content is located in an external file, such as a CSV file, you can easily automate the process of adding strings to your zombie.
+
+```csv
+Content,SourceText,SourceUrl
+"Lorem ipsum dolor sit amet.","Cicero's Great Speech On Ethics","http://example.com/lorem-ipsum"
+"Leverage agile frameworks.","Corporate Ipsum","http://www.cipsum.com/"
+"Bacon ipsum dolor amet.","Bacon Ipsum","http://baconipsum.com/"
+"Pork belly seitan photo booth.","Hipster Ipsum","https://hipsum.co/"
+```
+
+```ruby
+require 'smarter_csv'
+
+array_of_paragraphs = SmarterCSV.process("ipsum_quotes.csv")
+
+array_of_paragraphs.each do |paragraph|
+  zombie.add_string(paragraph)
+end
+```
+
+Once you have finished giving your Zombie all the strings it needs, tell it to generate your articles. It will save them as an array, which you can then save elsewhere. Each article is numbered, starting from zero, and is also given an headline (which is the "most important sentence" in the article). All articles are formatted using Markdown.
+
+```ruby
+array = zombie.generate_articles
+
+File.open("articles.md", "w+") do |f|
+  array.each { |article| f.puts("#{article}<hr>\n") }
+end
+```
+
+Here's an example article that might be generated by Zombie:
+
+```markdown
+<h2>0 - Lorem ipsum dolor sit amet.</h2>
+Lorem ipsum dolor sit amet.---[Cicero's Great Speech On Ethics](http://example.com/lorem-ipsum)
+
+Bacon ipsum dolor amet.---[Bacon Ipsum](http://baconipsum.com/)
+
+Leverage agile frameworks.---[Corporate Ipsum](http://www.cipsum.com/)
+
+Pork belly seitan photo booth.---[Hipster Ipsum](https://hipsum.co/)
+<hr>
+```
+
+###Citation
+You do not need to provide sourcetext or sourceurl. If you exclude the sourceurl, the article will only display the sourcetext as citation (with no hyperlink).
+
+```ruby
+zombie.add_string(content: "This is some Lorem filler that my friend made up.",
+sourcetext: "tra38's anonymous friend")
+```
+
+```markdown
+This is some Lorem filler that my friend made up.---tra38's anonymous friend
+```
+
+If you exclude the sourcetext, the article will use the sourceurl, while providing a hyperlink as well.
+
+```ruby
+zombie.add_string(content: "Zombie ipsum reversus ab viral inferno.",
+sourceurl: "http://www.zombieipsum.com")
+```
+
+```markdown
+Zombie ipsum reversus ab viral inferno.---[http://www.zombieipsum.com](http://www.zombieipsum.com)
+```
+
+If you exclude both the sourcetext and the sourceurl, the article will display no citation. This is useful for situations where you don't need to provide any citation metadata (such as if you have handwritten the content).
+```ruby
+zombie.add_string(content: "This is filler text that I invented.")
+```
+
+```markdown
+This is filler text that I invented.
+```
+
+##Real-World Examples
+
+###NaNoGenMo articles
+The "National Novel Generation Month" competition has generated a lot of commentary on the Internet. Rather than hand-writing out new commentary, why not reuse existing ones?
+
+- [Articles generated using ZombieWriter::MachineLearning](https://gist.github.com/tra38/aa7e9c63708f6e21c32db5c3616162b5)
+- [Articles generated using ZombieWriter::Randomization](https://gist.github.com/tra38/a65408790642560498aa1d40a05be9fe)
+
+In both instances, we used [this CSV file](https://gist.github.com/tra38/805003ef51ff63093b3c2775f161ce3c) as source data.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. Run `bundle exec zombie_writer` to use the gem in this directory, ignoring other installed copies of this gem.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+1. Fork it ( https://github.com/tra38/zombie/fork )
+2. Create your feature branch (git checkout -b my-new-feature)
+3. Commit your changes (git commit -am 'Add some feature')
+4. Push to the branch (git push origin my-new-feature)
+5. Create a new Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+## Credits
+The name is inspired by the term "[Philosphical Zombie](https://en.wikipedia.org/wiki/Philosophical_zombie)". According to Wikipedia, a P-Zombie is "a hypothetical being that is indistinguishable from a normal human being except that it lacks conscious experience, qualia, or sentience". AI is the closest we can get to building a P-Zombie of our own.
+
+The logo for this project was generated using [MarkMaker](http://emblemmatic.org/markmaker/#/).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "zombie_writer"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/zombie_writer

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+require "zombie_writer"

data/lib/zombie_writer/redcarpet_configuration.rb

@@ -0,0 +1,8 @@
+require 'redcarpet'
+require 'redcarpet/render_strip'
+
+class CustomStripDownRender < Redcarpet::Render::StripDown
+  def link(link, title, content)
+    "#{content}"
+  end
+end

data/lib/zombie_writer/version.rb

@@ -0,0 +1,3 @@
+module ZombieWriter
+  VERSION = "0.1.0"
+end

data/lib/zombie_writer.rb

@@ -0,0 +1,132 @@
+require "zombie_writer/version"
+require "zombie_writer/redcarpet_configuration"
+require 'classifier-reborn'
+require 'kmeans-clusterer'
+
+module ZombieWriter
+
+  def self.citation_constructor(paragraph)
+    if (paragraph[:sourceurl] && paragraph[:sourcetext])
+      "---[#{paragraph[:sourcetext]}](#{paragraph[:sourceurl]})"
+    elsif paragraph[:sourcetext]
+      "---#{paragraph[:sourcetext]}"
+    elsif paragraph[:sourceurl]
+      "---[#{paragraph[:sourceurl]}](#{paragraph[:sourceurl]})"
+    else
+      ""
+    end
+  end
+
+  class MachineLearning
+    attr_reader :lsi, :labels, :paragraph_data, :renderer, :plain_to_markdown
+
+    def initialize
+      @lsi = ClassifierReborn::LSI.new
+      @labels = []
+      @paragraph_data = Hash.new
+      @plain_to_markdown = Hash.new
+      @renderer = Redcarpet::Markdown.new(CustomStripDownRender)
+    end
+
+    def add_string(paragraph)
+      content = paragraph[:content]
+
+      stripped_down_content = renderer.render(content)
+
+      plain_to_markdown[stripped_down_content] = content
+
+      paragraph_data[content] = ZombieWriter.citation_constructor(paragraph)
+
+      labels << stripped_down_content
+      lsi.add_item(stripped_down_content)
+    end
+
+    def generate_articles
+      number_of_articles = labels.length
+      clusters = determine_number_of_clusters(number_of_articles)
+      clusters = generate_clusters(clusters: clusters, runs: 10)
+      clusters.map do |cluster|
+        article_for_summarization = generate_article(cluster) do |point|
+          point.label
+        end
+
+        final_article = generate_article(cluster) do |point|
+          stripped_down_content = point.label
+          content = plain_to_markdown[stripped_down_content]
+          citation = paragraph_data[content]
+          "#{content}#{citation}"
+        end
+
+        generated_title = ClassifierReborn::Summarizer.summary(article_for_summarization, 1)
+        "<h2>#{cluster.id.to_s} - #{generated_title}</h2>\n#{final_article}\n"
+      end
+    end
+
+    private
+    def generate_clusters(clusters:, runs:)
+      string_data = lsi.instance_variable_get(:"@items")
+      data = labels.map do |string|
+        string_data[string].lsi_norm.to_a
+      end
+      kmeans = KMeansClusterer.run clusters, data, labels: labels, runs: runs
+      kmeans.clusters
+    end
+
+    def determine_number_of_clusters(number_of_articles)
+      [1, ((number_of_articles/5).to_f).floor].max
+    end
+
+    def generate_article(cluster, &block)
+      cluster.points.map do |point|
+        yield(point)
+      end.join("\n\n")
+    end
+  end
+
+  class Randomization
+    attr_reader :labels, :paragraph_data, :renderer
+
+    def initialize
+      @labels = []
+      @paragraph_data = Hash.new
+      @renderer = Redcarpet::Markdown.new(CustomStripDownRender)
+    end
+
+    def add_string(paragraph)
+      content = paragraph[:content]
+
+      paragraph_data[content] = ZombieWriter.citation_constructor(paragraph)
+
+      labels << content
+    end
+
+    def generate_articles
+      number_of_paragraphs = labels.length
+      possible_paragraphs = labels.shuffle
+
+      possible_paragraphs.each_slice(5).with_index.map do |cluster, index|
+        article_for_summarization = generate_article(cluster) do |content|
+          renderer.render(content)
+        end
+
+        final_article = generate_article(cluster) do |content|
+          citation = paragraph_data[content]
+          "#{content}#{citation}"
+        end
+
+        generated_title = ClassifierReborn::Summarizer.summary(article_for_summarization, 1)
+        "<h2>#{index} - #{generated_title}</h2>\n#{final_article}\n"
+      end
+    end
+
+    private
+    def generate_article(cluster, &block)
+      cluster.map do |content|
+        yield(content)
+      end.join("\n\n")
+    end
+
+  end
+
+
+end

data/zombie_writer.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'zombie_writer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "zombie_writer"
+  spec.version       = ZombieWriter::VERSION
+  spec.authors       = ["Tariq Ali"]
+  spec.email         = ["tra38@nau.edu"]
+
+  spec.summary       = %q{ZombieWriter is a Ruby gem that will enable users to generate news articles by aggregating paragraphs from other sources.}
+  spec.description   = %q{While you have to provide the paragraphs, ZombieWriter will arrange the paragraphs into different articles for you to use and edit to your heart's content. You may choose between Machine Learning (Latent Semantic Analysis and k-means clustering) or Randomization.}
+  spec.homepage      = "https://github.com/tra38/Zombie"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.11"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_runtime_dependency "classifier-reborn", "~> 2.1"
+  spec.add_runtime_dependency "kmeans-clusterer", "~> 0.11.4"
+  spec.add_runtime_dependency "redcarpet", "~> 3.4"
+end

metadata

@@ -0,0 +1,148 @@
+--- !ruby/object:Gem::Specification
+name: zombie_writer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Tariq Ali
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2017-02-21 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.11'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.11'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: classifier-reborn
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: kmeans-clusterer
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.11.4
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.11.4
+- !ruby/object:Gem::Dependency
+  name: redcarpet
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+description: While you have to provide the paragraphs, ZombieWriter will arrange the
+  paragraphs into different articles for you to use and edit to your heart's content.
+  You may choose between Machine Learning (Latent Semantic Analysis and k-means clustering)
+  or Randomization.
+email:
+- tra38@nau.edu
+executables:
+- zombie_writer
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE.md
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- exe/zombie_writer
+- lib/zombie_writer.rb
+- lib/zombie_writer/redcarpet_configuration.rb
+- lib/zombie_writer/version.rb
+- zombie_writer.gemspec
+- zombie_writer_logo.png
+homepage: https://github.com/tra38/Zombie
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.8
+signing_key: 
+specification_version: 4
+summary: ZombieWriter is a Ruby gem that will enable users to generate news articles
+  by aggregating paragraphs from other sources.
+test_files: []