wordlist 0.1.1 → 1.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3648b45622323ff7c7c9a39d3cdb458d40388f1f0840fb80b917239df7ae995a
+  data.tar.gz: 8302f069d5a7dfd68103cfa106df8ade1b1d00f7f6755be7158dd7acd2a4bf0e
+SHA512:
+  metadata.gz: 01662d4d752294cca6da0809f7dfc0a953f7702259c448d73b4cfc796d214ce90797e4eedc80ebd48d005ec67039124ab5e974add35fadc8552f5cc9e6683de1
+  data.tar.gz: d29b36e5319a75bd9d88617451e6a7fda6c0cf4d0fe4cb43fd41b19da1e960135a57982b09f3ce93ff106668443ba03f26479a664c73737cc7721d8e2d0f26e5

data/.github/workflows/ruby.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on: [ push, pull_request ]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby:
+          - 2.6
+          - 2.7
+          - 3.0
+          - 3.1
+          - jruby
+          - truffleruby
+    name: Ruby ${{ matrix.ruby }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+      - name: Install dependencies
+        run: bundle install --jobs 4 --retry 3
+      - name: Run tests
+        run: bundle exec rake test

data/.gitignore

@@ -1,6 +1,9 @@
-doc
-pkg
-tmp/*
+/Gemfile.lock
+/benchmarks
+/coverage
+/doc
+/pkg
+/tmp
 scripts/*_wordlist.txt
 .DS_Store
 .bundle

data/ChangeLog.md

@@ -1,6 +1,60 @@
+### 1.0.1 / 2023-07-17
+
+* Require Ruby >= 2.0.0.
+* Added `frozen_string_literal: true` magic comments.
+* Flush the output after each line is written in {Wordlist::Builder}.
+  This allows for watching the wordlist file grow using `tail -f`.
+* Fixed a bug in {Wordlist::Lexer#parse} where Unicode whitespace or other
+  Unicode characters caused an infinite loop.
+* Various documentation fixes.
+
+### 1.0.0 / 2021-11-01
+
+* Redesigned the API.
+* Added {Wordlist::AbstractWordlist}.
+* Added {Wordlist::ListMethods}.
+* Added {Wordlist::Operators}.
+* Added {Wordlist::Operators::Operator}.
+* Added {Wordlist::Operators::UnaryOperator}.
+* Added {Wordlist::Operators::BinaryOperator}.
+* Added {Wordlist::Operators::Concat}.
+* Added {Wordlist::Operators::Subtract}.
+* Added {Wordlist::Operators::Product}.
+* Added {Wordlist::Operators::Power}.
+* Added {Wordlist::Operators::Union}.
+* Added {Wordlist::Operators::Intersect}.
+* Added {Wordlist::Operators::Unique}.
+* Added {Wordlist::Modifiers}.
+* Added {Wordlist::Modifiers::Modifier}.
+* Added {Wordlist::Modifiers::Capitalize}.
+* Added {Wordlist::Modifiers::Downcase}.
+* Added {Wordlist::Modifiers::Upcase}.
+* Added {Wordlist::Modifiers::Tr}.
+* Added {Wordlist::Modifiers::Sub}.
+* Added {Wordlist::Modifiers::Gsub}.
+* Added {Wordlist::Modifiers::Mutate}.
+* Added {Wordlist::Modifiers::MutateCase}.
+* Added {Wordlist::Words}.
+* Added {Wordlist::Format}.
+* Added {Wordlist::Compression}.
+* Added {Wordlist::Compression::Reader}.
+* Added {Wordlist::Compression::Writer}.
+* Added {Wordlist::File}.
+* Added {Wordlist::Lexer}.
+* Added {Wordlist::Lexer::StopWords}.
+* Added {Wordlist::Builder}.
+* Added {Wordlist::CLI}.
+* Refactored {Wordlist::UniqueFilter} to only store Object hashes.
+* Removed `Wordlist::List`.
+* Removed `Wordlist::FlatFile`.
+* Removed `Wordlist::Mutator` in favor of {Wordlist::Modifiers::Mutate}.
+* Removed `Wordlist::Parsers` in favor of {Wordlist::Lexer}.
+* Removed `Wordlist::Builders`.
+* Removed `Wordlist::Runners`.
+
 ### 0.1.1 / 2012-06-11
 
-* Default {Wordlist::Builders::Website#proxy} to `Spidr.proxy`.
+* Default `Wordlist::Builders::Website#proxy` to `Spidr.proxy`.
 
 ### 0.1.0 / 2009-08-31
 

data/Gemfile

@@ -0,0 +1,15 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development do
+  gem 'rake'
+  gem 'rubygems-tasks', '~> 0.2'
+  gem 'rspec',          '~> 3.0'
+  gem 'simplecov',      '~> 0.20'
+
+  gem 'kramdown'
+  gem 'redcarpet',       platform: :mri
+  gem 'yard',           '~> 0.9'
+  gem 'yard-spellcheck', require: false
+end

data/LICENSE.txt

@@ -1,6 +1,4 @@
-Wordlist - A Ruby library for generating and working with word-lists.
-
-Copyright (c) 2009-2012 Hal Brodigan
+Copyright (c) 2009-2021 Hal Brodigan
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,96 +1,337 @@
 # Wordlist
 
-* [Source](https://github.com/sophsec/wordlist#readme)
-* [Issues](https://github.com/sophsec/wordlist/issues)
-* [Email](mailto:postmodern.mod3 at gmail.com)
+[![CI](https://github.com/postmodern/wordlist.rb/actions/workflows/ruby.yml/badge.svg)](https://github.com/postmodern/wordlist.rb/actions/workflows/ruby.yml)
+[![Code Climate](https://codeclimate.com/github/postmodern/wordlist.rb.svg)](https://codeclimate.com/github/postmodern/wordlist.rb)
+[![Gem Version](https://badge.fury.io/rb/wordlist.svg)](https://badge.fury.io/rb/wordlist)
+
+* [Source](https://github.com/postmodern/wordlist.rb#readme)
+* [Issues](https://github.com/postmodern/wordlist.rb/issues)
+* [Documentation](https://rubydoc.info/gems/wordlist/frames)
 
 ## Description
 
-A Ruby library for generating and working with word-lists. Wordlist allows
-one to efficiently generate unique word-lists from arbitrary text or
-other sources, such as website content. Wordlist can also quickly enumerate
-through words within an existing word-list, applying multiple mutation
-rules to each word in the list.
+Wordlist is a Ruby library and CLI for reading, combining, mutating, and
+building wordlists, efficiently.
 
 ## Features
 
-* Uses a bucket system of CRC32 hashes for efficient filtering of duplicate
-  words.
-* Can build wordlists containing multi-word phrases.
-* Can build wordlists containing phrases containing a minimum and maximum
-  number of words.
-* Supports adding mutation rules to a word-list, which are applied to
-  words as the list is enumerated.
-* Supports building word-lists from arbitrary text.
-* Supports custom word-list builders:
-  * Wordlist::Builders::Website: Build word-lists from website content.
-* Supports custom word-list formats:
-  * Wordlist::FlatFile: Enumerates through the words in a flat-file
-    word-list.
+* Supports reading `.txt` wordlists, and `.txt.gz`, `.txt.bz2`, and `.txt.xz`
+  compressed wordlists.
+* Supports building wordlists from arbitrary text. Also supports `.gz`, `.bz2,`
+  and `.xz` compression.
+* Provides an advanced lexer for parsing text into words.
+  * Can parse/skip digits, special characters, whole numbers, acronyms.
+  * Can normalize case, apostrophes, and acronyms.
+* Supports wordlist operations for combining multiple wordlists together.
+* Supports wordlist modify or mutating the words in the wordlist on the fly.
+* Also provides a `wordlist` [command](#synopsis).
+* [Fast-ish](#benchmarks)
 
 ## Examples
 
-Build a word-list from arbitrary text:
+### Reading
+
+Open a wordlist for reading:
+
+```ruby
+wordlist = Wordlist.open("passwords.txt")
+```
+
+Open a compressed wordlist for reading:
+
+```ruby
+wordlist = Wordlist.open("rockyou.txt.gz")
+```
+
+Enumerate through a wordlist:
+
+```ruby
+wordlist.each do |word|
+  puts word
+end
+```
+
+Create an in-memory list of literal words:
+
+```ruby
+words = Wordlist::Words["foo", "bar", "baz"]
+```
+
+### List Operations
+
+Concat two wordlists together:
+
+```ruby
+(wordlist1 + wordlist2).each do |word|
+  puts word
+end
+```
+
+Union two wordlists together:
+
+```ruby
+(wordlist1 | wordlist2).each do |word|
+  puts word
+end
+```
+
+Subtract one wordlist from the other:
+
+```ruby
+(wordlist1 - wordlist2).each do |word|
+  puts word
+end
+```
+
+Combine every word from `wordlist1` with the words from `wordlist2`:
+
+```ruby
+(wordlist1 * wordlist2).each do |word|
+  puts word
+end
+```
+
+Combine the wordlist with itself multiple times:
+
+```ruby
+(wordslist ** 3).each do |word|
+  puts word
+end
+```
+
+Filter out duplicates from multiple wordlists:
+
+```ruby
+(wordlist1 + wordlist2 + wordlist3).uniq.each do |word|
+  puts word
+end
+```
+
+### String Manipulation
+
+Convert every word in a wordlist to lowercase:
+
+```ruby
+wordlist.downcase.each do |word|
+  puts word
+end
+```
+
+Convert every word in a wordlist to UPPERCASE:
+
+```ruby
+wordlist.upcase.each do |word|
+  puts word
+end
+```
+
+Capitalize every word in a wordlist:
+
+```ruby
+wordlist.capitalize.each do |word|
+  puts word
+end
+```
 
-    Wordlist::Builder.build('list.txt') do |builder|
-      builder.parse(some_text)
-    end
+Run `String#tr` on every word in a wordlist:
 
-Build a word-list from another file:
+```ruby
+wordlist.tr('_','-').each do |word|
+  puts word
+end
+```
 
-    Wordlist::Builder.build('list.txt') do |builder|
-      builder.parse_file('some/file.txt')
-    end
+Run `String#sub` on every word in a wordlist:
 
-Build a word-list of phrases containing at most three words, from the
-arbitrary text:
+```ruby
+wordlist.sub("fish","phish").each do |word|
+  puts word
+end
+```
 
-    Wordlist::Builder.build('list.txt', :max_words => 3) do |builder|
-      builder.parse(some_text)
-    end
+Run `String#gsub` on every word in a wordlist:
 
-Build a word-list from content off a website:
+```ruby
+wordlist.gsub(/\d+/,"").each do |word|
+  puts word
+end
+```
 
-    require 'wordlist/builders/website'
+Performs every possible mutation of each word in a wordlist:
 
-    Wordlist::Builders::Website.build(
-      'list.txt',
-      :host => 'www.example.com'
-    )
+```ruby
+wordlist.mutate(/[oae]/, {'o' => '0', 'a' => '@', 'e' => '3'}).each do |word|
+  puts word
+end
+# dog
+# d0g
+# firefox
+# fir3fox
+# firef0x
+# fir3f0x
+# ...
+```
 
-Enumerate through each word in a flat-file word-list:
+Enumerates over every possible case variation of every word in a wordlist:
 
-    list = Wordlist::FlatFile.new('list.txt')
-    list.each_word do |word|
-      puts word
-    end
+```ruby
+wordlist.mutate_case.each do |word|
+  puts word
+end
+# cat
+# Cat
+# cAt
+# caT
+# CAt
+# CaT
+# cAT
+# CAT
+# ...
+```
 
-Enumerate through each unique word in a flat-file word-list:
+### Building a Wordlist
 
-    list.each_unique do |word|
-      puts word
-    end
+```ruby
+Wordlist::Builder.open('path/to/file.txt.gz') do |builder|
+  # ...
+end
+```
 
-Define mutation rules, and enumerate through each unique mutation of each
-unique word in the word-list:
+Add individual words:
 
-    list.mutate 'o', '0'
-    list.mutate '@', 0x41
-    list.mutate(/[hax]/i) { |match| match.swapcase }
+```ruby
+builder.add(word)
+```
 
-    list.each_mutation do |word|
-      puts word
-    end
+Adding an Array of words:
+
+```ruby
+builder.append(words)
+```
+
+Parsing text:
+
+```ruby
+builder.parse(text)
+```
+
+Parsing a file's content:
+
+```ruby
+builder.parse_file(path)
+```
 
 ## Requirements
 
-* [spidr](http://spidr.rubyforge.org) >= 0.1.9
+* [ruby] >= 2.0.0
+
+[ruby]: https://www.ruby-lang.org/
 
 ## Install
 
-    $ gem install wordlist
+```shell
+$ gem install wordlist
+```
+
+### gemspec
+
+```ruby
+gem.add_dependency 'wordlist', '~> 1.0'
+```
+
+### Gemfile
+
+```ruby
+gem 'wordlist', '~> 1.0'
+```
+
+### Synopsis
+
+Reading a wordlist:
+
+```shell
+$ wordlist rockyou.txt.gz
+```
+
+Reading multiple wordlists:
+
+```shell
+$ wordlist sport_teams.txt beers.txt
+```
+
+Combining every word from one wordlist with another:
+
+```shell
+$ wordlist sport_teams.txt -p beers.txt -p all_four_digits.txt
+coors0000
+coors0001
+coors0002
+coors0003
+...
+```
+
+Combining every word from one wordlist with itself, N times:
+
+```shell
+$ wordlist words.txt -P 3
+```
+
+Mutating every word in a wordlist:
+
+```shell
+$ wordlist passwords.txt -m o:0 -m e:3 -m a:@
+dog
+d0g
+firefox
+fir3fox
+firef0x
+fir3f0x
+...
+```
+
+Executing a command on each word in the wordlist:
+
+```shell
+$ wordlist directories.txt --exec "curl -X POST -F 'user=joe&password={}' -o /dev/null -w '%{http_code} {}' https://$TARGET/login"
+```
+
+Building a wordlist from a directory of `.txt` files:
+
+```shell
+$ wordlist --build wordlist.txt dir/*.txt
+```
+
+Building a wordlist from STDIN:
+
+```shell
+$ cat *.txt | wordlist --build wordlist.txt
+```
+
+## Benchmarks
+
+```
+                                               user     system      total        real
+Wordlist::Builder#parse_text (size=5.4M)   1.943605   0.003809   1.947414 (  1.955960)
+Wordlist::File#each (N=1000)               0.000544   0.000000   0.000544 (  0.000559)
+Wordlist::File#concat (N=1000)             0.001143   0.000000   0.001143 (  0.001153)
+Wordlist::File#subtract (N=1000)           0.001360   0.000000   0.001360 (  0.001375)
+Wordlist::File#product (N=1000)            0.536518   0.005959   0.542477 (  0.545536)
+Wordlist::File#power (N=1000)              0.000015   0.000001   0.000016 (  0.000014)
+Wordlist::File#intersect (N=1000)          0.001389   0.000000   0.001389 (  0.001407)
+Wordlist::File#union (N=1000)              0.001310   0.000000   0.001310 (  0.001317)
+Wordlist::File#uniq (N=1000)               0.000941   0.000000   0.000941 (  0.000948)
+Wordlist::File#tr (N=1000)                 0.000725   0.000000   0.000725 (  0.000736)
+Wordlist::File#sub (N=1000)                0.000863   0.000000   0.000863 (  0.000870)
+Wordlist::File#gsub (N=1000)               0.001240   0.000000   0.001240 (  0.001249)
+Wordlist::File#capittalize (N=1000)        0.000821   0.000000   0.000821 (  0.000828)
+Wordlist::File#upcase (N=1000)             0.000760   0.000000   0.000760 (  0.000769)
+Wordlist::File#downcase (N=1000)           0.000544   0.000001   0.000545 (  0.000545)
+Wordlist::File#mutate (N=1000)             0.004656   0.000000   0.004656 (  0.004692)
+Wordlist::File#mutate_case (N=1000)       24.178521   0.000000  24.178521 ( 24.294962)
+```
 
 ## License
 
-See {file:LICENSE.txt} for license information.
+Copyright (c) 2009-2021 Hal Brodigan
 
+See {file:LICENSE.txt} for details.

data/Rakefile

@@ -1,35 +1,10 @@
-require 'rubygems'
-require 'rake'
+require 'rubygems/tasks'
+Gem::Tasks.new
 
-begin
-  gem 'rubygems-tasks', '~> 0.1'
-  require 'rubygems/tasks'
-
-  Gem::Tasks.new
-rescue LoadError => e
-  warn e.message
-  warn "Run `gem install rubygems-tasks` to install 'rubygems/tasks'."
-end
-
-begin
-  gem 'rspec', '~> 2.4'
-  require 'rspec/core/rake_task'
-
-  RSpec::Core::RakeTask.new
-rescue LoadError => e
-  task :spec do
-    abort "Please run `gem install rspec` to install RSpec."
-  end
-end
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new
+task :test    => :spec
 task :default => :spec
 
-begin
-  gem 'yard', '~> 0.8'
-  require 'yard'
-
-  YARD::Rake::YardocTask.new  
-rescue LoadError => e
-  task :yard do
-    abort "Please run `gem install yard` to install YARD."
-  end
-end
+require 'yard'
+YARD::Rake::YardocTask.new

data/benchmarks.rb

@@ -0,0 +1,115 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift(File.expand_path(File.join(__dir__,'lib')))
+require 'wordlist'
+require 'fileutils'
+require 'benchmark'
+
+DIR       = File.join(__dir__,'benchmarks')
+TEXT_FILE = File.join(DIR,'shaks12.txt')
+WORDLIST1 = File.join(DIR,'wordlist1.txt')
+WORDLIST2 = File.join(DIR,'wordlist2.txt')
+N = 1_000
+
+FileUtils.mkdir_p(DIR)
+
+unless File.file?(TEXT_FILE)
+  require 'net/https'
+
+  uri = URI('https://www.gutenberg.org/files/100/old/shaks12.txt')
+
+  Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
+    request = Net::HTTP::Get.new(uri)
+
+    http.request(request) do |response|
+      File.open(TEXT_FILE,'w') do |file|
+        response.read_body do |chunk|
+          file.write(chunk)
+        end
+      end
+    end
+  end
+end
+
+unless File.file?(WORDLIST1)
+  system("head -n #{N} /usr/share/dict/words > #{WORDLIST1}")
+end
+
+unless File.file?(WORDLIST2)
+  system("tail -n #{N} /usr/share/dict/words > #{WORDLIST2}")
+end
+
+Benchmark.bm(40) do |b|
+  b.report("Wordlist::Builder#parse_text (size=5.4M)") do
+    Wordlist::Builder.open(File.join(DIR,'shakespeare.txt')) do |builder|
+      builder.parse_file(TEXT_FILE)
+    end
+  end
+
+  b.report("Wordlist::File#each (N=#{N})") do
+    Wordlist::File.open(WORDLIST1).each { |word| }
+  end
+
+  wordlist1 = Wordlist::File.open(WORDLIST1)
+  wordlist2 = Wordlist::File.open(WORDLIST2)
+
+  b.report("Wordlist::File#concat (N=#{N})") do
+    wordlist1.concat(wordlist2).each { |word| }
+  end
+
+  b.report("Wordlist::File#subtract (N=#{N})") do
+    wordlist1.subtract(wordlist2).each { |word| }
+  end
+
+  b.report("Wordlist::File#product (N=#{N})") do
+    wordlist1.product(wordlist2).each { |word| }
+  end
+
+  b.report("Wordlist::File#power (N=#{N})") do
+    wordlist1.power(2)
+  end
+
+  b.report("Wordlist::File#intersect (N=#{N})") do
+    wordlist1.intersect(wordlist2).each { |word| }
+  end
+
+  b.report("Wordlist::File#union (N=#{N})") do
+    wordlist1.union(wordlist2).each { |word| }
+  end
+
+  b.report("Wordlist::File#uniq (N=#{N})") do
+    wordlist1.uniq.each { |word| }
+  end
+
+  b.report("Wordlist::File#tr (N=#{N})") do
+    wordlist1.tr("e","3").each { |word| }
+  end
+
+  b.report("Wordlist::File#sub (N=#{N})") do
+    wordlist1.sub("e","3").each { |word| }
+  end
+
+  b.report("Wordlist::File#gsub (N=#{N})") do
+    wordlist1.gsub("e","3").each { |word| }
+  end
+
+  b.report("Wordlist::File#capittalize (N=#{N})") do
+    wordlist1.capitalize.each { |word| }
+  end
+
+  b.report("Wordlist::File#upcase (N=#{N})") do
+    wordlist1.upcase.each { |word| }
+  end
+
+  b.report("Wordlist::File#downcase (N=#{N})") do
+    wordlist1.downcase.each { |word| }
+  end
+
+  b.report("Wordlist::File#mutate (N=#{N})") do
+    wordlist1.mutate("e","3").each { |word| }
+  end
+
+  b.report("Wordlist::File#mutate_case (N=#{N})") do
+    wordlist1.mutate_case.each { |word| }
+  end
+end