rawbotz 0.1.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 81bcca5d73862da12d5eb4ed197377f0866bae8c
+  data.tar.gz: 903e12e32aade0a32acc87310d820b574e6edf0e
+SHA512:
+  metadata.gz: 081bc679e520975b5df4541200295bf9925798cacbad4246582b998c539a3bb1a3cc8a561bd6d36ce8b6e58955405094eb613fb82492de5229313fd4077888a7
+  data.tar.gz: c20fd92925a37221f111e5156ef9f9db0da466e4469f0002869a1422e8eed9924bfbe4dd23cc5d06f9052f7716e2cddc7264ce537ecf8652087499a8ee3dab7a

data/.gitignore

@@ -0,0 +1,12 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*config*yml
+rawbotz.conf
+*.swp

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,49 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at felix.wolfsteller@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+# Uncomment these for local development
+#gem 'magento_remote', path: "../magento_remote"
+#gem 'rawgento_models', path: "../rawgento_models"
+#gem 'rawgento_db',     path: "../rawgento_db"
+
+# Specify your gem's dependencies in rawbotz.gemspec
+gemspec

data/README.md

@@ -0,0 +1,185 @@
+# Rawbotz
+
+Rawbotz provides a Web-Interface to do certain management tasks for a particular Magento Web-Shop (RawLiving Germany).
+
+In this shop, products are supplied by different suppliers, of which one offers a magento-shop itself.
+
+Rawbotz couples the `rawgento_db`, `rawgento_models` and `magento_remote` gems to make it easy to order items from this particular other shop and provides further functionality, e.g. accessing stock and sales history information.
+
+Note that this git repository includes software components whose Copyright and License differ from the other parts.
+
+These are
+- Font Awesome (http://fontawesome.io/) in `lib/rawbotz/public/font-awesome-4.5.0`
+- Pure CSS (http://purecss.io) in `lib/rawbotz/public/pure-min.css`
+- Chart.JS (http://chartjs.org) in `lib/rawbotz/public/Chart.min.js`
+- jquery and jquery-ui (http://jquery.com, http://jqueryui.com) in `lib/rawbotz/public/jquery-2.2.0.min.js` and `lib/rawbotz/public/jui`
+
+The copyright and license information is contained in the respective file headers. All other files are Copyright 2016 Felix Wolfsteller and licensed under the AGPLv3 (or any later).
+
+## Installation
+
+While bundling rawbotz up and installing it as a gem should be possible, it is strongly advised to install it via git.
+
+Note that the dependencies should be handled with bundler if you are not running a development setup.
+
+Anyway, the application is so specific to RawLiving Germanys needs that you probably want to get in contact first.
+
+## Assumptions
+
+- supplier attribute
+- shelve attribute
+- packsize attribute
+- Ubuntu 14.04 installation, using rvm in an unprivileged users home.
+
+## Usage
+
+Until `rawgento_models`, `rawgento_db` and `magento_remote` (gems by same author) have settled, please adjust the `Gemfile` and checkout these gems in the parent folder.
+
+### Configuration
+
+Rawbotz combines the configuration files of `rawgento_db`, `rawgento_models` and `magento_remote` in a single file (default: `rawbotz.conf`).  In theory, the components configure themselves using their respective default configuration file paths.
+
+
+#### Unified appraoch
+
+Create a `rawbotz.conf` YAML-file with the unified keys needed.  Note that you can pass the path to this configuration file to the various executables in `exe/`.
+
+    # Rawbotz own database
+    default: &default
+      adapter: sqlite3
+      database: /home/rawbotz/database.sqlite
+      encoding: utf8
+      pool: 5
+      timeout: 5000
+    
+    development:
+      <<: *default
+    test:
+      <<: *default
+      database: db/rawgento_test.db
+    
+    # Local Magento MySQL database
+    host: 127.0.0.1
+    port: 3306
+    database: magento_shop_dbname
+    username: magento_db_username
+    password: magento_db_password
+    # Attributes needed
+    attribute_ids:
+      name: 11
+      supplier_name: 666
+      shelve_nr: 42
+      packsize: 1337
+    
+    # Remote Magento Web Interface (for our mech)
+    remote_shop:
+      base_uri: https://magentoshop.remote
+      user: mylogin@email.address
+      pass: whatnottobenamed
+    
+    supplier_name: MagentoShop Remote
+    
+    local_shop:
+      base_uri: https://magentoshop.mine
+    
+    mail:
+      to: your@email.address
+      from: senders@email.address
+      host: email.address
+      user: senders@email.user
+      pass: senders.email.password
+      port: 587
+
+Then, tell RawbotzApp to eat your config via `exe/rawbotz -c rawbotz.conf`.
+To have fun directly with rack instead use the environment variable `RAWBOTZ_CONFIG`, like in `RAWBOTZ_CONFIG=/home/rawbotz/rawbotz.conf rackup`.
+
+#### Reference to the configurable required components
+
+Look in the corresponding gems to check how their configuration has to be done:
+
+  * [rawgento_db s rawgento_db-config.yml](https://github.com/rawliving-germany/rawgento_db)
+  * [rawgento_models s db/config.yml](https://github.com/rawliving-germany/rawgento_models)
+  * [magento_remotes config](https://github.com/fwolfst/magento_remote)
+
+##### In Quick
+
+Configure your database in db/config.yml (Rails style, `rawgento_models`).
+
+Configure your magento mysql-connection in rawgento_db-config.yml (`rawgento_db`):
+    host: myshop.shop
+    port: 3306
+    database: magento_myshop
+    username: magento_myshop_dbuser
+    password: magento_myshop_dbpassword
+
+Finally, configure the remote shops credentials (`magento_remote`)
+
+## Setup
+
+Note that a script with basic support for maintenance-tasks is underway (exe/rawbotz_maintenance.sh).
+
+#### Setup the database
+
+Run `rake db:setup` (if `db/schema.rb` is present) or `rake db:migrate` to setup the database.
+Unfortunately, there is no way to pass in the config file, so for now you have to create `db/config.yml` (which can be nearly the same as `rawbotz.conf`) temporarily.
+
+#### Populate the database with local and remote products
+
+E.g. with `bundle exec exe/rawbotz_update_local_products -c rawbotz.conf` .
+This will query your magento MySQL database and create 'local' products, expect the command to run a while (minutes).
+
+E.g. with `bundle exec exe/rawbotz_update_remote_products -c rawbotz.conf`
+This will query the remote magento shop (scraping it via html GET requests) and create 'remote' products, expect the command to run a while longer (more minutes).  You might need to adapt parameters, depending on the remote shop.  `bundle exec exe/rawbotz_update_remote_products --help` gives you a hint on how to optimize your settings.
+
+#### Setup mailing
+
+Mails are sent via pony.
+
+Create following `rawbotz.conf` section:
+
+    mail:
+      to: your@email.address
+      from: senders@email.address
+      host: email.address
+      user: senders@email.user
+      pass: senders.email.password
+      port: 587
+
+.  Multiple receipients can be addressed like this:
+
+    mail:
+      to:
+        - your@email.address
+        - colleagues@mail.address
+.
+
+### Deployment or web-app startup
+
+You can run `(bundle exec) exe/rawbotz`, `rackup` or put rawbotz behind a phusion passenger.
+There is a `-c` option to pass in a config file.
+
+As usual, for `rackup` you can specify port (`-p`) and host (`-o`) parameters.  The path to config file has to be exposed as RAWBOTZ_CONFIG env var.
+
+### Stock History Update via cron
+
+Assuming an Ubuntu Server Setup, run `crontab -e` and add following line to fetch stock values every day at 03:00 am.
+
+### Other tools included
+
+#### rawbotz_update_local_products
+
+Updates Products from magento MySQL database.  See `--help` for more information.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment. Run `bundle exec rawbotz` to use the gem in this directory, ignoring other installed copies of this gem.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+You can `bundle console` to jump into a pre-setup irb, then call `RawgentoModels::establish_connection "rawbotz.conf"` to setup the database connection and deal with real world data.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/rawbotz. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+Feel free to get in contact with me.

data/Rakefile

@@ -0,0 +1,29 @@
+require "bundler/gem_tasks"
+require 'rawgento_models'
+
+require 'rake/testtask'
+
+task :default => :spec
+
+RawgentoModels.load_tasks
+
+# TODO Setup the environment by calling something like RawgentoModels::establish_connection "rawbotz.conf"` and reading the rest of the config file
+
+# These tasks should be solveable either via a rake task or via single executables
+
+# initial load
+
+# remote scrape
+
+# local load and update
+
+# stock update
+
+Rake::TestTask.new do |t|
+  t.pattern = "test/*test.rb"
+end
+#Rake::TestTask.new do |t|
+#  t.libs << "test"
+#  t.test_files = FileList['test/*test.rb']
+#  t.verbose = true
+#end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "rawbotz"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/config.ru

@@ -0,0 +1,9 @@
+# encoding: UTF-8
+require 'rawbotz'
+
+if ENV['RAWBOTZ_CONFIG']
+  Rawbotz.conf_file_path = ENV['RAWBOTZ_CONFIG']
+end
+
+require 'rawbotz/app'
+run RawbotzApp

data/exe/debug/test_mail

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require "rawbotz"
+require 'rawbotz/option_parser'
+
+optparse = Rawbotz::OptionParser.new do |opts, options|
+  opts.banner = "Usage: #{$PROGRAM_NAME} [OPTIONS] ORDER_ID"
+  opts.separator ""
+end
+optparse.parse!
+options = optparse.options
+
+mail = Rawbotz::mail("rawbotz test mail", "Success.\nHere the body.")
+
+exit 0

data/exe/rawbotz

@@ -0,0 +1,39 @@
+#!/usr/bin/env ruby
+
+require "rawbotz"
+require 'optparse'
+
+options = {}
+program_name = File.basename __FILE__
+$program_name = program_name
+
+optparse = OptionParser.new do |opts|
+  opts.banner = "Usage: #{program_name} [OPTIONS]"
+  opts.separator ""
+
+  opts.separator "Single rawbotz configuration file"
+  opts.on("-c", "--config FILE", 'file path to rawgento-db YAML config file.') do |c|
+    if !File.exist? c
+      STDERR.puts "Cannot load conf file #{c}"
+      exit 2
+    end
+
+    Rawbotz.conf_file_path = c
+  end
+
+  opts.separator ""
+  opts.separator "General"
+  opts.on_tail('--version', 'Show version.') do
+    puts "#{program_name} #{Rawbotz::VERSION}"
+    exit 0
+  end
+  opts.on('-h', '--help', 'Show this help.') do
+    puts opts
+    exit 0
+  end
+end
+optparse.parse!
+
+require "rawbotz/app"
+
+RawbotzApp.run!

data/exe/rawbotz_import_links

@@ -0,0 +1,74 @@
+#!/usr/bin/env ruby
+
+require "rawbotz"
+require "rawbotz/option_parser"
+
+require 'csv'
+
+program_name = File.basename __FILE__
+
+optparse = Rawbotz::OptionParser.new do |opts, options|
+  opts.banner = "Usage: #{program_name} [OPTIONS] <CSVFILE>"
+  opts.separator ""
+
+  opts.on("-d", "--dry-run", "Simulate, do not actually write to db") do |d|
+    options[:dry_run] = d
+  end
+end
+optparse.parse!
+
+logger = Logger.new(STDOUT)
+logger.level = optparse.options[:verbose] ? Logger::DEBUG : Logger::INFO
+
+logger.debug "#{program_name} #{Rawbotz::VERSION}"
+
+if ARGV.length != 1
+  STDERR.puts "Need to supply an argument!"
+  STDERR.puts optparse.banner
+  exit 2
+end
+
+logger.info("Read csv input file #{ARGV[0]}")
+
+if optparse.options[:dry_run]
+  logger.info("In dry-run-mode, will not actually write to database.")
+end
+
+begin
+  CSV.foreach(ARGV[0]) do |line|
+    local_pid = line[0]
+    remote_pid = line[1]
+
+    local_product = RawgentoModels::LocalProduct.find_by(product_id: local_pid)
+    remote_product = RawgentoModels::RemoteProduct.find_by(product_id: remote_pid)
+
+    if local_product.nil?
+      logger.error "Local Product with product-id #{local_pid} could not be found"
+      next
+    elsif remote_product.nil?
+      logger.error "Remote Product with product-id #{remote_pid} could not be found"
+      next
+    end
+    if local_product.remote_product == remote_product
+      logger.info "Link #{local_pid} -> #{remote_pid} already present"
+      next
+    end
+    if !options[:dry_run]
+      local_product.remote_product = remote_product
+      local_product.save
+    end
+    logger.info "Linked #{local_pid} -> #{remote_pid}"
+    logger.debug "Linked #{local_product.name} -> #{remote_product.name}"
+  end
+rescue Exception => e
+  logger.error "Could not read csv input file"
+  logger.error e.inspect
+  logger.error e.backtrace
+  exit 1
+end
+
+if optparse.options[:dry_run]
+  logger.info("In dry-run-mode, did not actually write to database.")
+end
+
+exit 0

data/exe/rawbotz_maintenance.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# Part of rawbotz, (c) Felix Wolfsteller 2016
+# Licensed under the GPLv3
+
+set -e
+
+command -v dialog >/dev/null 2>&1 || { 
+  echo >&2 "The program 'dialog' seems not to be installed, exiting."
+  exit 1
+}
+
+PROGRESS_WIDTH=60
+PROGRESS_HEIGHT=30
+
+RAWBOTZ_CONF="rawbotz.conf"
+
+menu_selection=$(mktemp /tmp/rawbotz_maintenance.XXXXXX)
+
+dialog --menu "Rawbotz Menu" 18 30 8 \
+  1 "Start rawbotz on port 9711" \
+  2 "Update local magento products" \
+  3 "Look for new remote products" \
+  4 "Save current stock data" \
+  5 "Trigger Order queue processing" \
+  2> tmp
+
+if [ "$?" = 0 ]
+then
+  _ret=$(cat tmp)
+  rm tmp
+  if [ "$_ret" = "1" ]
+  then
+    echo "starting rawbotz"
+    ./start_rb.sh
+  elif [ "$_ret" = "2" ]
+  then
+    bundle exec rawbotz_update_local_products -v -c "$RAWBOTZ_CONF" | sed -e 's/.* -- : //' | dialog --progressbox "Update local magento products" $PROGRESS_HEIGHT $PROGRESS_WIDTH
+  elif [ "$_ret" = "3" ]
+  then
+    # program "unbuffer?"
+    bundle exec rawbotz_update_remote_products -c "$RAWBOTZ_CONF" -n -w 0 | stdbuf -oL -eL sed -e 's/.* -- : //' | dialog --progressbox "Updating Remote Products" $PROGRESS_HEIGHT $PROGRESS_WIDTH
+  elif [ "$_ret" = "4" ]
+  then
+    rawbotz_stock_update -v -c "$RAWBOTZ_CONF" | dialog --progressbox "Saving current stock data" $PROGRESS_HEIGHT $PROGRESS_WIDTH
+  elif [ "$_ret" = "5" ]
+  then
+    rawbotz_process_order_queue -v -c "$RAWBOTZ_CONF" | dialog --progressbox "Processing Order Queue" $PROGRESS_HEIGHT $PROGRESS_WIDTH
+  else
+    echo "Bad choice."
+    exit 1
+  fi
+fi
+
+rm "$menu_selection"
+
+exit 0

data/exe/rawbotz_process_order

@@ -0,0 +1,86 @@
+#!/usr/bin/env ruby
+
+require 'date'
+
+require "rawbotz"
+require 'rawbotz/option_parser'
+
+optparse = Rawbotz::OptionParser.new do |opts, options|
+  opts.banner = "Usage: #{$PROGRAM_NAME} [OPTIONS] ORDER_ID"
+  opts.separator ""
+
+  opts.on("-m", "--[no]-mail", "Send mail with result") do |m|
+    options[:mail] = true
+  end
+
+  opts.on("-f", "--force", "Force order even if state is not queued") do |m|
+    options[:force] = true
+  end
+end
+optparse.parse!
+
+options = optparse.options
+Rawbotz::configure_logger options
+
+include RawgentoModels
+
+def main options
+  Rawbotz::logger.debug "#{$PROGRAM_NAME} #{Rawbotz::VERSION}"
+
+  if ARGV.length != 1
+    Rawbotz::logger.error("Need an argument (order_id)")
+    exit 4
+  end
+
+  Rawbotz::logger.info("Starting order")
+
+  order = RawgentoModels::Order.find(ARGV[0].to_i)
+  if !order.present?
+    Rawbotz::logger.error("No Order with #{ARGV[0].to_i} found.")
+    exit 5
+  end
+
+  Rawbotz::logger.info("#{order.order_items.length} items in Order")
+  if order.state != :new
+    Rawbotz::logger.warn("Order in state #{order.state}!")
+    if !options[:force]
+      Rawbotz::logger.warn("Run with -f,--force flag to ignore")
+      exit 6
+    end
+  end
+
+  order.update(state: :processing)
+
+  order_processor = Rawbotz::OrderProcessor.new(order, Rawbotz::logger)
+
+  order_processor.process!
+
+  Rawbotz::logger.info("Finished order")
+
+  order.update(state: :ordered)
+
+  if options[:mail]
+    settings = YAML::load_file(Rawbotz::conf_file_path)["mail"]
+    pony_via_options = { address: settings["host"],
+                         user_name: settings["user"],
+                         port: settings["port"].to_i,
+                         authentication: :login,
+                         password: settings["pass"]}
+    Rawbotz::logger.info("Attempting mail send to #{settings['to']}")
+    Pony.mail({
+      to: settings["to"],
+      from: settings["from"],
+      subject: "test",
+      body: "great",
+      via: :smtp,
+      via_options: pony_via_options
+    })
+  else
+    Rawbotz::logger.info("Not sending mail (invoke with -m to do so)")
+  end
+
+  diff = order_processor.check_against_cart
+end
+
+main options
+exit 0

data/exe/rawbotz_process_order_queue

@@ -0,0 +1,80 @@
+#!/usr/bin/env ruby
+
+require 'date'
+
+require 'terminal-table'
+
+require "rawbotz"
+require 'rawbotz/option_parser'
+require 'rawbotz/cli/order_result_table'
+
+
+optparse = Rawbotz::OptionParser.new do |opts, options|
+  opts.banner = "Usage: #{$PROGRAM_NAME} [OPTIONS] ORDER_ID"
+  opts.separator ""
+
+  opts.on("-m", "--[no-]mail", "Send mail with result") do |m|
+    options[:mail] = m
+  end
+end
+optparse.parse!
+options = optparse.options
+
+include RawgentoModels
+
+def main options
+  logger = Logger.new(STDOUT)
+  logger.level = options[:verbose] ? Logger::DEBUG : Logger::INFO
+
+  logger.debug "#{$PROGRAM_NAME} #{Rawbotz::VERSION}"
+
+  logger.info("Checking queue")
+
+  queue = RawgentoModels::Order.queued
+  logger.info("-> #{queue.count} orders to process")
+
+  if queue.count == 0
+    logger.info("Nothing to do, exiting.")
+    exit 0
+  end
+
+  order = queue.first
+
+  logger.info("#{order.order_items.length} items in Order")
+
+  if order.state != "queued"
+    logger.warn("Order in state #{order.state}!")
+  end
+  order.update(state: :processing)
+
+  logger.info ("#{order.order_items.processible.count} products to order.")
+
+  mech = Rawbotz.new_mech
+  mech.login
+
+  order_processor = Rawbotz::OrderProcessor.new(order, logger)
+  order_processor.process!
+
+  order.update(state: :ordered)
+
+  logger.info("Finished order")
+
+  logger.info("Checking current cart")
+  diffs = order_processor.check_against_cart
+
+  # Option to put to stdout
+  diff_table = Rawbotz::CLI::OrderResultTable.tables diffs
+  puts diff_table
+
+  if options[:mail]
+    logger.debug("Sending mail")
+    mail = Rawbotz::mail("rawbotz order finished",
+                  "Finished Order nr #{order.id}. Result: [tbd]\n\n#{diff_table}")
+    logger.debug("Mail sent: #{mail}")
+  else
+    logger.debug("not sending mail")
+  end
+end
+
+main options
+exit 0