fingerprint 1.4.0 → 3.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 2c6867e9566430e12dc2d913a013c469e3b2bc47
-  data.tar.gz: 58d9a1812dcd041743816328ac3e71fc79496014
+SHA256:
+  metadata.gz: d80181117a0710868e55f5218a6dd6b9241d151878826bfeffc81a4d79b3d2bf
+  data.tar.gz: 2ec8a17ce51398454fcaa082b0f08d87a37ef3753e28a3bb9c1a7252bc92db95
 SHA512:
-  metadata.gz: ac0bfcca4b51a31797849aa039c795626f78d7a5d0b5278aadbb9872b635d7ca3cd369b3cd08de77c48545ca58af4aad83ede79e99749b56e896a03b8e238e5e
-  data.tar.gz: 76f51cf43136114a35cafb99958da8e7099c98ee68a931efa7a1b853e76dc9025a2bedd6df697ebf69b7c7f47004f3616865514beea9c980d8a9e4798f7eec58
+  metadata.gz: a075aad140130acd196e13ee9c265d3b722d5879e681ff4852b81ea9d3885d34e649daf821a0da33a4e9c9b370f67905bee573dfe380d88db6e0195e95c41ce1
+  data.tar.gz: 594c2a23c98b18cd76be232c184b0176f4f6ac1f1fd240183e35969633440dc7deceeef442a43a80fb1d0d705e9b829e5ce5e9b4167fa022f0ad5facc843bb16

data/.gitignore

@@ -0,0 +1,15 @@
+.tags
+
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+.rspec_status
+.covered.db
+index.fingerprint

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--warnings
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,23 @@
+language: ruby
+dist: xenial
+cache: bundler
+
+matrix:
+  include:
+    - rvm: 2.3
+    - rvm: 2.4
+    - rvm: 2.5
+    - rvm: 2.6
+    - rvm: 2.6
+      env: COVERAGE=Summary,Coveralls
+    - rvm: truffleruby
+    - rvm: jruby-head
+      env: JRUBY_OPTS="--debug -X+O"
+    - rvm: ruby-head
+    - rvm: 2.6
+      os: osx
+  allow_failures:
+    - rvm: truffleruby
+    - rvm: ruby-head
+    - rvm: jruby-head
+    - rvm: truffleruby

data/GUIDE.md

@@ -0,0 +1,233 @@
+# Fingerprint Guide
+
+This guide gives an overview of the various ways in which fingerprint can be used.
+
+## Fingerprint Digests
+
+The following are the options for which digests can be generated:
+
+* `MD5`
+* `SHA1`
+* `SHA2.256`
+* `SHA2.384`
+* `SHA2.512`
+
+`SHA2.256` is the default.
+
+The use of `MD5` and `SHA1` are no longer recommended due to the
+risk of hash collisions.
+
+## Generating fingerprints
+
+Fingerprint is designed to index directories. The basic verb `scan` will index all paths specified in order (or the current path if none is given).
+
+``` text
+$ fingerprint scan /tmp/test
+C /tmp/test
+	fingerprint.version 2.1.5
+	options.checksums SHA2.256
+	options.extended false
+	summary.time.start 2016-12-09 16:42:41 -0800
+D
+F file-1.txt
+	file.size 0
+	key.SHA2.256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+F file-2.txt
+	file.size 0
+	key.SHA2.256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+F file-3.sh
+	file.size 0
+	key.SHA2.256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+S 3 files processed.
+	summary.directories 1
+	summary.excluded 0
+	summary.files 3
+	summary.size 0
+	summary.time.end 2016-12-09 16:42:41 -0800
+```
+
+## Verifing Fingerprints
+
+Files can be checked against a given fingerprint. In the case of `verify`, only changes and deletions will be reported. Additions are not reported.
+
+To generate a fingerprint for a given path:
+
+	fingerprint scan /tmp/test > /tmp/test.fingerprint
+
+This fingerprint can then be used to verify that no files have changed:
+
+	$ fingerprint verify -n /tmp/test.fingerprint /tmp/test
+	S 0 error(s) detected.
+		error.count 0
+
+## Comparing Fingerprints
+
+The `verify` operation checks a given fingerprint against a current filesystem, and can work efficiently according to the files in the source fingerprint. In the case that you want to compare two fingerprints, you can find all differences, including additions. This can be useful when comparing two backups to see what files changed (e.g. a tripwire).
+
+```
+/tmp$ fingerprint scan /tmp/test > /tmp/test1.fingerprint
+/tmp$ vi test/file-1.txt  (change this file)
+/tmp$ fingerprint scan /tmp/test > /tmp/test2.fingerprint
+/tmp$ fingerprint compare /tmp/test1.fingerprint /tmp/test2.fingerprint
+W file-1.txt
+	changes.file.size.new 4
+	changes.file.size.old 0
+	changes.key.SHA2.256.new b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
+	changes.key.SHA2.256.old e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+	error.code keys_different
+	error.message Key SHA2.256 does not match
+S 1 error(s) detected.
+	error.count 1
+```
+
+Here we can see files which have been changed and added to `/tmp/test` after a modifying a file.
+
+## Transmission and Archival Usage
+
+Fingerprint provides two high-level operations `analyze` which is roughly equivalent to `scan` but outputs to the file `index.fingerprint` by default, and `verify` which reads by default from `index.fingerprint`.
+
+Typical use cases would include analyzing a directory before copying it, and verifying it after the copy, or analyzing a backup and verifying it before restore.
+
+	-- Fingerprint the local data:
+	$ fingerprint --root /srv/http analyze
+	
+	-- Copy it to the remote system:
+	$ rsync -a /srv/http production:/srv/http
+	
+	-- Run fingerprint on the remote system to verify the data copied correctly:
+	$ ssh production fingerprint --root /srv/http verify
+	S 0 error(s) detected.
+		error.count 0
+
+This is equally useful for traditional backup mediums, e.g. write-only storage, offline backups, etc.
+
+## Data Preservation
+
+Data preservation means that data is available in a useful form for as long as it is required (which could be indefinitely). Faulty hardware, software, user error or malicious attack are the primary concerns affecting data preservation. Fingerprint can provide insight into data integrity problems, which can then be resolved appropriately.
+
+In almost all cases of data corruption, the sooner the corruption is detected, the less damage that occurs overall. It is in this sense that regular fingerprinting of critical files should be considered an important step in comprehensive data preservation policy.
+
+Data preservation almost always involves data replication and verification. If data becomes corrupt, it it essential that backups are available to recover the original data. However, it is also important that backup data can be verified otherwise there is no guarantee that the recovered data is correct.
+
+End to end testing on a regular basis is the only sane policy if your data is important.
+
+### Non-malicious Bit-rot
+
+Using standard analyze/verify procedure, fingerprint can detect file corruption. Make sure you record extended information using `-x`. The primary indication of file corruption is a change in checksum data but not in modified time. Standard operating system functions update the modified time when the file is changed. But, if the file changes without this, it may indicate hardware or software fault, and should be investigated further.
+
+Once data has been analyzed, you can store it on archival media (e.g. optical or tape storage). At a later date, you can verify this to ensure the data has not become damaged over time.
+
+### Malicious Changes
+
+Malicious modification of files can be detected using fingerprint. This setup is typically referred to as a tripwire, because when an attacker modifies some critical system files, the system administrator will be notified. To maintain the security of such a setup, the fingerprint should be stored on a separate server:
+
+	$ mv latest.fingerprint previous.fingerprint
+	$ ssh data.example.com fingerprint scan /etc > latest.fingerprint
+	$ fingerprint compare previous.fingerprint latest.fingerprint
+	S 0 error(s) detected.
+		error.count 0
+
+This can be scripted and run in an hourly cron job:
+
+	#!/usr/bin/env ruby
+
+	require 'fileutils'
+
+	REMOTE = "server.example.com"
+	DIRECTORY = "/etc"
+	PREVIOUS = "previous.fingerprint"
+	LATEST = "latest.fingerprint"
+
+	if File.exist? LATEST
+		FileUtils.mv LATEST, PREVIOUS
+	end
+
+	$stderr.puts "Generating fingerprint of #{REMOTE}:#{DIRECTORY}..."
+	system("ssh #{REMOTE} fingerprint #{DIRECTORY} > #{LATEST}")
+
+	if File.exist? PREVIOUS
+		$stderr.puts "Comparing fingerprints..."
+		system('fingerprint', 'compare', '-a', PREVIOUS, LATEST)
+	end
+
+## Backup integrity
+
+Data backup is typically an important part of any storage system. However, without end-to-end verification of backup data, it may not be possible to ensure that a backup system is working correctly. In the event that a failure occurs, data recovery may not be possible despite the existence of a backup, if that data has not been backed up reliably or correctly.
+
+If you are backing up online data, the backup tool you are using may backup files at non-deterministic times. This means that if software (such as a database) is writing to a file at the time the backup occurs, the data may be transferred incorrectly. Fingerprint can help to detect this, by running fingerprint before the backup on the local storage, and then verifying the backup data after it has been copied. Ideally, you'd expect to see minimal changes to critical files.
+
+However, the real world is often not so simple. Some software doesn't provide facilities for direct synchronization; other software provides facilities for dumping data (which may not be an option of the dataset is large). In these cases, fingerprint can give you some visibility about the potential issues you may face during a restore. You may want to consider [Synco](https://github.com/ioquatix/synco) which can coordinate more complex backup tasks.
+
+### Ensuring Data Validity
+
+To ensure that data has been backed up correctly, use fingerprint to analyze the data before it is backed up.
+
+	-- Perform the data analysis
+	$ sudo fingerprint analyze -f /etc
+
+	-- Backup the data to a remote system
+	$ sudo rsync --archive /etc backups.example.com:/mnt/backups/server.example.com/etc
+
+After the data has been copied to the remote backup device, restore the data to a temporary location and use fingerprint to verify the data. The exact procedure will depend on your backup system, e.g. if you use a tape you may need to restore from the tape to local storage first.
+
+	-- On the backup server
+	$ cd /mnt/backups/server.example.com/etc
+	$ fingerprint verify
+	S 0 error(s) detected.
+		error.count 0
+
+### Preserving Backups
+
+If your primary concern is ensuring that backup data is consistent over time (e.g. files are not modified or damaged), fingerprint can be used directly on the backup data to check for corruption. After the data has been backed up successfully, simply analyze the data as above, but on the backup server. Once this is done, at any point in the future you can verify the correctness of the backup set.
+
+## Cryptographic Sealing
+
+Fingerprint can be used to ensure that a set of files has been delivered without manipulation, by creating a fingerprint and signing this with a private key. The fingerprint and associated files can later be verified using the public key.
+
+### Generating Keys
+
+To sign fingerprints, the first step is to create a private and public key pair. This is easily achieved using OpenSSL:
+
+	-- Create a private key, which you must keep secure.
+	$ openssl genrsa -out private.pem 2048
+	Generating RSA private key, 2048 bit long modulus
+	.............+++
+	........+++
+	e is 65537 (0x10001)
+
+	-- Create a public key, which can be used to verify sealed fingerprints.
+	$ openssl rsa -in private.pem -pubout -out public.pem
+	writing RSA key
+
+### Signing Fingerprints
+
+After you have generated a fingerprint, you can sign it easily using the private key:
+
+	-- We assume here that you are using fingerprint analyze to generate fingerprints.
+	$ openssl dgst -sha1 -sign private.pem -out index.signature index.fingerprint
+
+### Verifying Fingerprint Signature
+
+You can easily verify the security of the fingerprint data:
+
+	$ openssl dgst -sha1 -verify public.pem -signature index.signature index.fingerprint
+	Verified OK
+	-- Fingerprint data has been cryptographically verified
+
+	$ fingerprint verify
+	S 0 error(s) detected.
+		error.count 0
+	Data verified, 0 errors found.
+	-- File list has been checked and no errors.
+
+As long as private key is kept secure, we can be sure that these files have not been tampered with.
+
+### Notarizing
+
+In many cases it is good to ensure that documents existed at a particular time. With modern document storage systems, it may be impossible to verify this by simply relying on databases and filesystems alone, especially because technology can be manipulated.
+
+Fingerprint can be used to produce printed documents which can be used to verify the existence of files at a given time. With appropriate physical signatures, these could be used to verify the existence of a given set of files at a specific time and place.
+
+Simply follow the procedure to produce a cryptographic hash of a directory, print these documents out, and get them signed.
+
+N.B. Please consult a lawyer for the correct procedure and legal standing of such techniques. This document is not legal advice and we accept no responsibility for any use or misuse of this tool.

data/Gemfile

@@ -1,4 +1,4 @@
 source 'https://rubygems.org'
 
-# Specify your gem's dependencies in fingerprint.gemspec
+# Specify your gem's dependencies in ..gemspec
 gemspec

data/README.md

@@ -1,43 +1,138 @@
 # Fingerprint
 
-Fingerprint is primarily a command line tool to compare directory structures on 
-disk. It also provides a programmatic interface for this procedure.
+> Matter and energy degrade to more probable, less informative states. The larger the amounts of information processed or diffused, the more likely it is that information will degrade toward meaningless variety, like noise or information overload, or sterile uniformity — Orrin Klapp
 
-Because Fingerprint produces output to `IO` like structures, it is easy to transmit
-this data across a network, or store it for later use. As an example, it could be
-used to check the integrity of a remote backup.
+Fingerprint is a general purpose data integrity tool that uses cryptographic hashes to detect changes in files. Fingerprint scans a directory tree and generates a small transcript file containing the names and hashes of the files. This snapshot file can then be used to generate a list of files that have been created, deleted, or modified. If so much as a single bit in a single file in the directory tree has changed, fingerprint will detect it.
 
-For examples and documentation please see the main [project page][1].
+Traditionally, the only way to preserve data was to take regular backups and hope that any unwanted changes that occurred would be major, obvious ones (such as loss of the disk). This approach means trusting all the software to which the data is exposed: operating systems, backup software, communications software, compression software, encryption software, and archiving software. Unfortunately, each of these systems is highly complex and can inflict all kinds of damage on data, much of the damage undetectable to humans. Fingerprint allows data to be monitored, detecting even the change of a single bit.
 
-[1]: http://www.codeotaku.com/projects/fingerprint/index
+Fingerprint can be used for:
+
+- Preservation: Detect corruption of important data, e.g. web server integrity, write-once storage verification.
+- Security: Detect changes made by intruders, e.g. firewall integrity, network configuration, software auditing.
+- Transfers:  Verify file copies and transfers between different systems, e.g. file transfer integrity.
+- Sealing: Cryptographically seal critical files, e.g. document verification.
+- Notarizing: Prove that documents existed at a particular time.
+- Backups: Verify restored backups to ensure that backups are sound, e.g. backup verification and integrity.
+
+A companion app is available in the [Mac App Store](https://itunes.apple.com/nz/app/fingerprint/id470866821). Purchasing this app helps fund the open source software development.
+
+[![Build Status](https://travis-ci.org/ioquatix/fingerprint.svg?branch=master)](https://travis-ci.org/ioquatix/fingerprint)
+[![Code Climate](https://codeclimate.com/github/ioquatix/fingerprint.svg)](https://codeclimate.com/github/ioquatix/fingerprint)
+[![Coverage Status](https://coveralls.io/repos/ioquatix/fingerprint/badge.svg)](https://coveralls.io/r/ioquatix/fingerprint)
+
+## Motivation
+
+As the world becomes further entrenched in digital data and storage, the accuracy and correctness of said data is going to become a bigger problem. As humans create information, we are ultimately decreasing the amount of entropy in the universe. By the second law of thermodynamics, when a closed system moves from "the least to the most probable, from differentiation to sameness, from ordered individuality to a kind of chaos," (Thomas Pynchon) the only logical conclusion is that what we consider to be important data is destined to become meaningless noise in the sands of time.
+
+When I first suffered data-loss, it wasn't catastrophic - it was the slow deterioration of a drive which silently corrupted many files. After this event, I wanted a tool which would allow me to minimize the chance of this happening in the future. When I take a backup now, I also make a fingerprint. If I ever need to restore from backup, I can be confident the data is as it was when it was backed up.
+
+As fingerprint provides a fast way to compare the files, I've also extended it to find duplicates within one or more fingerprints. This is useful for de-duplicating your home directory and I've also used it when marking assignments to find blatant copying.
+
+In cases where I've been concerned about the migration of data (e.g. copying my entire home directory from one system to another), I've used fingerprint to generate a transcript on the source machine, and then run it on the destination machine, to reassure me that the data was copied correctly and completely.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'fingerprint'
+	gem 'fingerprint'
 
 And then execute:
 
-    $ bundle
+	$ bundle
 
 Or install it yourself as:
 
-    $ gem install fingerprint
+	$ gem install fingerprint
 
 ## Usage
 
-Please refer to the [online documentation][http://www.codeotaku.com/projects/fingerprint/documentation/introduction].
+Please consult the [GUIDE](GUIDE.md) for an overview of how fingerprint command can be used.
+
+### RSpec
+
+The simplest usage of fingerprint is checking if two directories are equivalent:
+
+	Fingerprint.identical?(source_path, destination_path) do |record|
+		puts "#{record.path} is different"
+	end
+
+This would catch additions, removals, and changes. You can use this in RSpec:
+
+	expect(Fingerprint).to be_identical(source_path, destination_path)
+
+### Command Line
+
+The `fingerprint` command has a high-level and low-level interface.
+
+#### High-level Interface
+
+This usage is centered around analysing a given directory using `fingerprint analyze` and then, at a later date, checking that the directory is not missing any files and that all files are the same as they were originally, using `fingerprint verify`.
+
+```
+$ fingerprint analyze
+$ fingerprint verify
+S 0 error(s) detected.
+	error.count 0
+```
+
+If we modify a file (`file-1.txt` in this example), it will be reported:
+
+```
+$ fingerprint verify
+W file-1.txt
+	changes.file.size.new 8
+	changes.file.size.old 4
+	changes.key.SHA2.256.new 1f2ec52b774368781bed1d1fb140a92e0eb6348090619c9291f9a5a3c8e8d151
+	changes.key.SHA2.256.old b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
+	error.code keys_different
+	error.message Key SHA2.256 does not match
+S 1 error(s) detected.
+	error.count 1
+```
+
+This command does not report files which have been added.
+
+#### Low-level interface
+
+It is possible to generate a fingerprint using the scan command, which takes a list of paths and writes out the transcript.
+
+	% fingerprint scan spec 
+	C /home/samuel/Documents/Programming/ioquatix/fingerprint/spec
+		fingerprint.version 2.0.0
+		options.checksums MD5, SHA2.256
+		options.extended false
+		summary.time.start 2016-06-25 11:46:12 +1200
+	D 
+	D fingerprint
+	F fingerprint/check_paths_spec.rb
+		file.size 1487
+		key.MD5 ef77034977daa683bbaaed47c553f6f5
+		key.SHA2.256 970ec4663ffc257ec1d4f49f54711c38434108d580afc0c92ea7bf864e08a1e0
+	S 1 files processed.
+		summary.directories 2
+		summary.excluded 0
+		summary.files 1
+		summary.size 1487
+		summary.time.end 2016-06-25 11:46:12 +1200
+
+#### Duplicates
+
+Fingerprint can efficiently find duplicates in one or more fingerprints.
+
+	$ fingerprint duplicates index.fingerprint
+	F .git/refs/heads/master
+		file.size 41
+		fingerprint index.fingerprint
+		key.MD5 aaadaeee72126dedcd4044d687a74068
+		key.SHA2.256 6750f057b38c2ea93e3725545333b8167301b6d8daa0626b0a2a613a6a4f4f04
+		original.fingerprint index.fingerprint
+		original.path .git/refs/remotes/origin/master
 
 ## Todo
 
-* Command line option to show files that have been created (e.g. don't exist in master fingerprint).
-* Command line option to show files that have changed but have the same modified time (hardware corrutpion).
-* Command line option to check fingerprint files based on checksums, e.g. duplicate files, unique files, over a set of directories.
-* Command line tool for extracting duplicate and unique files over a set of directories?
+* Command line option to show files that have changed but have the same modified time (hardware corruption).
 * Supporting tools for signing fingerprints easily.
-* Support indexing specific files as well as whole directories (maybe?).
-* Support general filenames for `--archive`, e.g. along with `-n`, maybe support a file called `index.fingerprint` by default: improved visibility for end user.
 * Because fingerprint is currently IO bound in terms of performance, single-threaded checksumming is fine, but for SSD and other fast storage, it might be possible to improve speed somewhat by using a map-reduce style approach.
 
 ## Contributing
@@ -52,7 +147,7 @@ Please refer to the [online documentation][http://www.codeotaku.com/projects/fin
 
 Released under the MIT license.
 
-Copyright, 2012, by [Samuel G. D. Williams](http://www.codeotaku.com/samuel-williams).
+Copyright, 2019, by [Samuel G. D. Williams](http://www.codeotaku.com/samuel-williams).
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal