tendersync 1.0.2

data/History.txt

@@ -0,0 +1,13 @@
+== 1.0.2 2009-08-15
+
+* Fixed some bugs in posting
+
+== 1.0.1 2009-07-17
+
+* Indexing working
+* More command line options
+* Tests and Documentation
+
+== 1.0.0 2009-06-11
+
+* First Release

data/Manifest

@@ -0,0 +1,27 @@
+bin/tendersync
+config/website.yml.sample
+History.txt
+lib/tendersync/document.rb
+lib/tendersync/runner.rb
+lib/tendersync/session.rb
+lib/tendersync/tendersync.rb
+lib/tendersync.rb
+Manifest
+Rakefile
+README.md
+script/console
+script/destroy
+script/generate
+script/txt2html
+spec/fixtures/passenger_restart_issues
+spec/spec.opts
+spec/spec_helper.rb
+spec/tendersync_document_spec.rb
+spec/tendersync_session_spec.rb
+spec/tendersync_spec.rb
+tasks/rspec.rake
+website/index.html
+website/index.txt
+website/javascripts/rounded_corners_lite.inc.js
+website/stylesheets/screen.css
+website/template.html.erb

data/README.md

@@ -0,0 +1,157 @@
+# Tendersync
+
+Authors: Markus Roberts and Bill Kayser
+
+Tendersync allows you to sync documents stored in the 
+[ENTP Tender](http://www.tenderapp.com)
+`faqs` section with a local filesystem, allowing you to manage your
+documents with git or subversion.
+
+It includes a command for creating an index document for any given
+section.
+
+Find out more about Tender by visiting [the Tender site](http://www.tenderapp.com).
+
+## Features
+
+* List remote sections and documents under the `/faqs` area
+* Pull single documents or entire tree from Tender site to local
+  filesystem
+* Push local changes back to Tender one at a time or en masse
+* Manage document meta-data, like keywords, in headers
+* Push changed versions to the server
+
+## Synopsis
+
+Create a working directory where you want to store the tender docs in a hierarchy
+and run tendersync from there.
+
+    sudo gem install tendersync
+    cd $workdir
+    tendersync -h
+
+## Using Tendersync
+
+To get started, you need to pass in your account information.  You
+only need to do this once.  A local file `.tendersync` is created with
+the configuration information.
+
+This will get you set up:
+
+    tendersync -u user@me.com -p password --docurl=http://company.tenderapp.com
+
+To verify it worked run the `ls` command:
+
+    tendersync ls
+
+Tender documents are organized into sections defined by you.  At New
+Relic, we have faqs, docs, and troubleshooting.  You can specify
+commands to apply to one or more sections by passing in section names
+with -s:
+
+    tendersync -s docs -s troubleshooting pull
+
+### Examples
+
+Start with:
+
+    tendersync -h
+
+Download all your docs:
+
+    tendersync pull 
+
+Download just the faq docs:
+
+    tendersync pull -s faqs
+
+Create a git repository and save all the documents:
+
+    git init
+    git add .
+    git commit -m "First version of docs on Tender"
+
+Upload docs to the server:
+
+    tendersync post faqs/sinatra_support
+    tendersync post docs/install-*
+    tendersync post -s docs
+
+Upload everything to the server (regardless of whether the content has
+changed or not):
+  
+    tendersync post
+
+### Using the `index` Command
+
+You can generate a table of contents for any section with the index
+command.  By default index will generate a single file named
+`SECTION_table_of_contents`.
+
+In this file will be a list of all the files in the given section with
+links to those files.  Under each file link will be a bullet list of
+the topmost sections in the document.  If these sections are preceeded
+by anchor links (A elements with the name attribute) then the bullets
+will have links to those sections.
+
+It will look something like this:
+
+<pre>
+  ## Installation and configuration
+  ### [Agent Installation (Ruby)](agent-installation)
+  * [Installing the Plug-in](agent-installation#Installing_the_Plug-in)
+  * [Installing the Gem](agent-installation#Installing_the_Gem)
+</pre>
+
+#### Customizing the amount of detail in the Index
+
+You can show sections deeper than one level in a particular document
+using the `-d` option.  The default is 1.
+
+    tendersync index -d 2
+
+#### Definiting TOC groups
+
+If you want to divide the table of contents into groups of related
+documents, you can pass in a title for a group and a regular expression
+to match against document titles that belong in that group. These group
+definitions will be saved so you only need to enter them once.
+
+Enter a group using the `-g` option passing in a title and regular
+expression separated by a semi-colon.
+
+    tendersync index -g "Page Details;/page/i"
+
+You can add multiple groups with additional -g options:
+
+    tendersync index -g "Page Details;/page/i" -g "Installation Info;/installation/i"
+
+If you want to remove a group definition, you need to remove it 
+manually from the `.tendersync` file.
+
+## THANKS
+
+All due regards, credit, thanks, etc., to the ENTP team for a great tool.
+
+## LICENSE
+
+Copyright (c) 2009 New Relic, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,40 @@
+require 'rubygems'
+require 'echoe'
+%w[rake rake/clean fileutils newgem rubigen].each { |f| require f }
+require File.dirname(__FILE__) + '/lib/tendersync'
+
+GEM_NAME = "tendersync"
+GEM_VERSION = Tendersync::VERSION
+AUTHOR = "Bill Kayser"
+EMAIL = "bkayser@newrelic.com"
+HOMEPAGE = "http://www.github.com/newrelic/tendersync"
+SUMMARY = "Utility for syncing and indexing files from ENTP's Tender site."
+DESCRIPTION = <<-EOF
+Tendersync is a utility for syncing files from ENTP's Tender site for managing customer facing documentation.  It can be used to pull and push documents to a local repository as well as create indexes for each documentation section.
+  EOF
+
+# Generate all the Rake tasks
+# Run 'rake -T' to see list of generated tasks (from gem root directory)
+Echoe.new(GEM_NAME, Tendersync::VERSION) do |p|
+  p.author = AUTHOR
+  p.email = EMAIL
+  p.summary = SUMMARY
+  p.url = HOMEPAGE
+  p.project = 'newrelic'
+  p.description = DESCRIPTION
+  p.version = Tendersync::VERSION
+  p.need_tar_gz = false
+  p.need_gem = true
+  p.bin_files = 'bin/tendersync'
+  p.runtime_dependencies = [
+     ['mechanize','>= 0.9.3'],
+  ]
+  p.development_dependencies = [
+    ['newgem', ">= #{::Newgem::VERSION}"]
+  ]
+  p.ignore_pattern = %w[docs/** general/** troubleshooting/**]
+  p.clean_pattern |= %w[**/.DS_Store tmp *.log]
+end
+
+Dir['tasks/**/*.rake'].each { |t| load t }
+

data/bin/tendersync

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2009-6-11.
+#  Copyright (c) 2009. All rights reserved.
+
+require File.expand_path(File.dirname(__FILE__) + "/../lib/tendersync")
+
+require "tendersync/runner"
+
+begin
+  Tendersync::Runner.new(ARGV.dup).run
+rescue Tendersync::Runner::Error => e
+  puts e.message
+end

data/config/website.yml.sample

@@ -0,0 +1,2 @@
+host: unknown@rubyforge.org
+remote_dir: /var/www/gforge-projects/tendersync

data/lib/tendersync/document.rb

@@ -0,0 +1,218 @@
+require 'fileutils'
+require 'set'
+require 'yaml'
+
+class Tendersync::Document
+  Properties = [:section, :document_id, :title, :permalink, :keywords, :body]
+  attr_accessor *Properties
+  
+  NUM_DASHES = 28 # the number of dashes in keyword fields
+  
+  class TOCEntry
+    attr_reader :name, :link, :level
+    attr_accessor :parent
+    def initialize name, link=nil, level=nil
+      @name = name
+      @link = link if link
+      @level = level if level
+    end
+    def children
+      @children ||= []
+    end
+    # Write this element and all children, recursively as bullet lists with links
+    def write_entries(io, depth=1, indent = 0, doc_link = parent.link)
+      io.write " " * 4 * indent # indentation
+      io.write "* " # bullet
+      if link
+        io.puts "[#{name}](#{doc_link}##{self.link})"
+      else
+        io.puts name
+      end
+      children.each { | child | child.write_entries(io, depth-1, indent+1, doc_link)} unless depth == 1
+    end
+    def add child
+      if !parent || child.level > self.level
+        children << child
+        child.parent = self
+      else
+        parent.add(child)
+      end
+      child
+    end
+  end
+  
+  class Group < TOCEntry
+    attr_reader :title_regex
+    def initialize(name, title_regex=//)
+      super(name, nil, nil)
+      @title_regex = title_regex
+    end
+    Default = Group.new('Other')
+
+  end
+  
+  def initialize(values={})
+    values.each do | prop, value |
+      self.send "#{prop}=", value
+    end
+  end
+  #
+  # Documents can be read from / written to a file
+  #
+  def to_s
+    io = StringIO.new
+    Properties.each do |field|
+      next unless value = self.send(field) 
+      io.write  "-" * NUM_DASHES
+      io.write " #{field} "
+      io.write "-" * NUM_DASHES
+      io.puts
+      io.puts value
+    end
+    io.string
+  end
+  
+  def save
+    FileUtils.mkdir_p section
+    File.open("#{section}/#{permalink}",'w') { |f| f.print self }
+    self
+  end
+  
+  def self.load(section, io)
+    values = { :section => section }
+    key = data = nil
+    while line = io.gets
+      line.chomp!
+      if line =~ /^----+ (.+) -----+$/
+        values[key] = data.join("\n") if data 
+        key = $1.intern
+        data = []
+      else
+        raise "keyword line not recognized: #{line}" unless data
+        data << line
+      end
+    end
+    values[key] = data.join("\n") if key
+    new values
+  end
+  
+  def self.read_from_file(file_name)
+    section = file_name.split('/')[-2] 
+    if !File.exists? file_name
+      raise Tendersync::Runner::Error, "Cannot read #{file_name}"
+    end
+    File.open(file_name) { |f| self.load(section, f) }
+  end
+  
+  #
+  # Can be scraped from a form
+  #
+  def self.from_form(section,form)
+    values = {
+      :document_id => form.action[%r{/faqs/(\d+)/edit},1],
+      :section => section
+    }
+    form.fields.each { |tf|
+      if field_name = tf.name[/faq\[(.*)\]/,1]
+        value = tf.value.map { |line| line.chomp }.join("\n")
+        values[field_name.intern] = value
+      end
+    }
+    new(values)
+  end
+  def to_form(form)
+    form.fields.each { |tf|
+      if field_name = tf.name[/faq\[(.*)\]/,1] and self.send(field_name.intern)
+        lines = []
+        self.send(field_name.intern).each_line {|line| lines << line.chomp }
+        tf.value = lines.join("\r\n")
+      end
+    }
+  end
+  
+  def self.each(section)
+    Dir.glob("#{section}/*").each { |f| yield Tendersync::Document.read_from_file(f) }
+  end
+  
+  def self.index_for(section_id, section_name)
+    new(:section => section_id,
+        :title => "#{section_name} Table of Contents",
+        :permalink => "#{section_id}-table-of-contents",
+        :keywords => "toc index")
+  end
+  
+  # Update this document body with an index of all the documents in
+  # this section.  Underneath the TOC entry for a document will be sub
+  # entries for each named A element.
+  #  
+  # group_map is an associative array of /regex/ to "Title String" of
+  # a group of documents.  It is used to divide up documents into
+  # groups within the table of contents.  A document is placed in a
+  # TOC group based on the first regex it matches in the group map.
+  #
+  # If group_map is empty then headings will be sorted alphabetically
+  # and not grouped.
+  #
+  # depth s the number of nested levels to descend into a document.
+  def refresh_index(groups=[], depth=2)
+    generate_index(create_toc(groups), depth)  
+  end
+  
+  private
+  
+  def create_toc(groups)
+    groups += groups + [Group::Default] # array of groups
+    link_root = {}
+    self.class.each(section) do |document|
+      next if document.permalink =~ /-table-of-contents$/
+      puts "indexing #{document.permalink}..."
+      title = document.title
+      group = groups.detect { | g | title =~ g.title_regex }
+      doc_entry = TOCEntry.new title, document.permalink, 0
+      group.add doc_entry
+      last = doc_entry
+      link = nil
+      document.body.scan(%r{<a name=(.*?)>|^(#+)\s*(.*?)\s*$}i) do
+        name = $1
+        heading_level = $2 && $2.length
+        text = $3
+        if name
+          # Record the link name for the next header
+          link = eval(name)
+        elsif heading_level == 1
+          last = last.add(TOCEntry.new(text, link, heading_level)) 
+        elsif heading_level >= 2  # level 2
+          last = last.add(TOCEntry.new(text, link, heading_level)) 
+          link = nil
+        end
+      end
+    end
+    groups
+  end
+  
+  def generate_index(groups, depth)
+    groups.reject! { | group | group.children.empty? }
+    # Now go through each group
+    io = StringIO.new
+    io.puts
+    groups.each do | group |
+      # Show the group heading unless there is only one group
+      io.puts "## #{group.name}" unless groups.size == 1
+      group.children.each do | doc_entry |
+        doc_link = doc_entry.link
+        io.puts "### [#{doc_entry.name}](#{doc_link})"
+        doc_entry.children.each do | doc_section |
+          doc_section.write_entries(io, depth)
+        end
+        io.puts 
+      end
+    end
+    if $dry_run
+      puts io.string
+    else
+      self.body = io.string
+    end
+  end
+  
+end
+

data/lib/tendersync/runner.rb

@@ -0,0 +1,211 @@
+require 'rubygems'
+require 'optparse'
+require 'tendersync/session'
+require 'tendersync/document'
+require 'mechanize'
+require 'yaml'
+
+class Tendersync::Runner
+  class Error < StandardError; end
+  
+  def initialize argv
+    @dry_run  = false
+    @sections = []
+    @groups = []
+    settings['groups'] ||= []
+    @parser = OptionParser.new do |op|
+      op.banner += " command\n"
+      op.on('-n',                                  "dry run" )       { @dry_run  = true }
+      op.on('-s', '--section', '=SECTION', String, "section, specify multiple separately" ) { |s| @sections << s }
+      op.on('-u', '--username','=EMAIL',   String, "* login e-mail" ) {|str| settings['username'] = str }
+      op.on('-p', '--password','=PASS',    String, "* password" )     {|str| settings['password'] = str }
+      op.on(      '--docurl',  '=URL',     String, "* tender site URL" ) { |dir| settings['docurl'] = dir }
+      op.separator ""
+      op.separator "Indexing Options:"
+      op.on('-g', '--group',   '=TITLE;regex', String, "*map of regex to group title for TOC groups") do | g |
+        pair = g.split(';')
+        settings['groups'] << [pair.first, pair.last]
+      end
+      op.on('-d', '--depth',   '=DEPTH', String, "*Number of levels to descend into a document being indexed") { | g | settings['depth'] = g.to_i }
+      
+      
+        %Q{
+    * saved in .tendersync file for subsequent default
+ 
+    Commands:
+
+        pull [URL, URL...]   -- download documents from tender; specify sections with -s, a page URL, or
+                                nothing to download all documents
+        index                -- create a master index of each section, writing to section/file; specify
+                                the sections with -s options; you can organize the TOC into groups by
+                                mapping document titles to groups via a regular expression with -g options
+        ls                   -- list files in specified session
+        post PATTERN         -- post the matching documents to tender; use /regexp/ or glob
+        irb                  -- drops you into IRB with a tender session & related classes (for hacking/
+                                one-time tasks).  Programmers only.
+        create PERMALINK     -- create a new tender document with the specified permalink in the section
+                                specified by --section=... (must be only one.)
+
+    }.split(/\n/).each {|line| op.separator line.chomp }
+    end
+    
+    begin
+      @command,*@args = *@parser.parse(argv)
+    rescue OptionParser::InvalidOption => e
+      raise Error, e.message
+    end
+    
+    @username = settings['username']
+    @password = settings['password']
+    @dochome = settings['docurl'] && settings['docurl'] =~ /^(http.*?)\/?$/ && $1
+    @root = settings['root']
+    
+    case
+    when ! @username
+      raise Error, "Please enter a username and password.  You only need to do this once."
+    when ! @password
+      raise Error, "Please enter a password.  You only need to do this once."
+    when ! @dochome
+      raise Error, "Please enter a --docurl indicating the home page URL of your Tender docs.\n" +
+           "You only need to do this once."
+    else
+      settings.save!
+    end
+  end
+  
+  def run
+    $session = Tendersync::Session.new @dochome, @username, @password
+    $dry_run = @dry_run
+    case @command || 'help'
+    when 'help'
+      raise Error, @parser.to_s
+    when *%w[pull post create irb ls index]
+      send @command
+    else
+      raise Error, "Unknown command: #{@command}\n\n#{@parser}"
+    end
+  end
+  
+  private
+  
+  def ls
+    $session.ls *sections
+  end
+  
+  def pull
+    if @args.size > 0
+      @args.each do |url|
+        section = url =~ /\/faqs\/([^\/]*)\// && $1
+        raise Error, "Invalid URI for document: #{url}" if section.nil?
+        doc = Document.from_form(section, $session.edit_page_for(url).form_with(:action => /edit/))
+        puts "   #{doc.permalink}"
+        doc.save unless $dry_run
+      end
+    else
+      sections.each do |section|
+        puts "pulling #{section} docs ..."
+        $session.pull_from_tender(section)
+      end
+    end
+  end
+
+  def post
+    documents = @args.collect { |doc_name|
+      matches =  if doc_name =~ %r{/}
+        Dir.glob(doc_name)
+      else
+        Dir.glob("#{@root}/{#{sections.join(',')}}/#{doc_name}*")
+      end
+      if matches.empty?
+        puts "No documents match #{doc_name}"
+      else
+        matches.collect { |match| Tendersync::Document.read_from_file(match) }
+      end
+    }.flatten.compact
+    documents.each { |document|
+      if @dry_run
+        puts "would post #{document.section}/#{document.permalink} to tender."
+      else
+        $session.post(document)
+      end
+    }
+  end
+  alias push post
+  
+  def create
+    raise Error, "You must specify exactly one section to put the document in." if sections.length != 1 
+    raise Error, "You must specify exactly one document permalink."             if @args.length != 1 
+    section,permalink = sections.first,@args.first
+    filename = "#{@root}/#{section}/#{permalink}"
+    text = File.read(filename) rescue ""
+    text = "Put Text Here" if text.strip.empty?
+    if $dry_run
+      puts "would create document #{permalink}\nin #{section} as #{filename}"
+      puts "\ntext:\n------------\n#{text}"
+    else
+      document = $session.create_document(section,permalink,text)
+      document.save
+    end
+  end
+  
+  def irb
+    puts <<-EOF
+
+      Use $session to access the Tendersync::Session instance.
+      Use Tendersync::Document to manipulate documents local and remote.
+
+      Examples of crazy stuff you could try:
+
+          puts $session.all_sections.inspect
+
+          $session.pull_from_tender('troubleshooting')  
+
+          $session.post(Tendersync::Document.index('docs').save)
+
+          Tendersync::Document.each { |d| puts d.body.split(/\W/).join("\\n") }
+
+          doc = Tendersync::Document.read_from_file("./docs/agent-api")
+          doc.body.gsub! /api/,"API"
+          doc.save
+
+    EOF
+    ARGV.clear
+    require 'irb'
+    require 'irb/completion'
+    $sections = sections
+    IRB.start
+  end
+  def index
+    groups = settings['groups'].map do |title,regex|
+      regex = eval(regex) if regex =~ %r{^/.*/[a-z]*$}
+      Tendersync::Document::Group.new title, Regexp.new(regex)
+    end
+    section_details = $session.all_sections
+    sections.each do |section|
+      doc = Tendersync::Document.index_for section, section_details[section]
+      puts "indexing #{section}: #{doc.section}/#{doc.permalink}"
+      doc.refresh_index groups, settings['depth'] || 2
+      doc.save
+    end
+  end
+  def sections
+    @sections = $session.all_sections.keys if @sections.empty? 
+    @sections
+  end
+  def settings
+    case
+    when @settings
+      return @settings 
+    when File.exists?(".tendersync")
+      File.open(".tendersync", "r") { |f| @settings = YAML.load(f) }
+    else
+      @settings = {}
+    end
+    def @settings.save!
+      File.open(".tendersync","w") do |f|
+        f.write(self.to_yaml)
+      end
+    end
+    @settings
+  end
+end