longleaf 0.1.0 → 1.1.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.circleci/config.yml +94 -0
- data/.editorconfig +13 -0
- data/.gitignore +4 -1
- data/.rubocop.yml +44 -0
- data/.rubocop_todo.yml +834 -0
- data/.yardopts +1 -0
- data/Gemfile +16 -1
- data/README.md +98 -12
- data/Rakefile +6 -0
- data/bin/setup +16 -1
- data/docs/aboutlongleaf.md +28 -0
- data/docs/extra.css +32 -0
- data/docs/img/change-file.png +0 -0
- data/docs/img/ll-example-preserved.png +0 -0
- data/docs/index.md +19 -0
- data/docs/install.md +66 -0
- data/docs/ll-example/config-example-relative.yml +33 -0
- data/docs/ll-example/files-dir/LLexample-PDF.pdf +0 -0
- data/docs/ll-example/files-dir/LLexample-TOCHANGE.txt +15 -0
- data/docs/ll-example/files-dir/LLexample-tokeep.txt +10 -0
- data/docs/ll-example/metadata-dir/.gitkeep +0 -0
- data/docs/ll-example/replica-files/.gitkeep +0 -0
- data/docs/ll-example/replica-metadata/.gitkeep +0 -0
- data/docs/quickstart.md +270 -0
- data/docs/rdocs/Longleaf.html +135 -0
- data/docs/rdocs/Longleaf/AppFields.html +178 -0
- data/docs/rdocs/Longleaf/ApplicationConfigDeserializer.html +631 -0
- data/docs/rdocs/Longleaf/ApplicationConfigManager.html +610 -0
- data/docs/rdocs/Longleaf/ApplicationConfigValidator.html +238 -0
- data/docs/rdocs/Longleaf/CLI.html +909 -0
- data/docs/rdocs/Longleaf/ChecksumMismatchError.html +151 -0
- data/docs/rdocs/Longleaf/ConfigBuilder.html +1339 -0
- data/docs/rdocs/Longleaf/ConfigurationError.html +143 -0
- data/docs/rdocs/Longleaf/ConfigurationValidator.html +227 -0
- data/docs/rdocs/Longleaf/DeregisterCommand.html +420 -0
- data/docs/rdocs/Longleaf/DeregisterEvent.html +453 -0
- data/docs/rdocs/Longleaf/DeregistrationError.html +151 -0
- data/docs/rdocs/Longleaf/DigestHelper.html +419 -0
- data/docs/rdocs/Longleaf/EventError.html +147 -0
- data/docs/rdocs/Longleaf/EventNames.html +163 -0
- data/docs/rdocs/Longleaf/EventStatusTracking.html +656 -0
- data/docs/rdocs/Longleaf/FileCheckService.html +540 -0
- data/docs/rdocs/Longleaf/FileHelpers.html +520 -0
- data/docs/rdocs/Longleaf/FileRecord.html +716 -0
- data/docs/rdocs/Longleaf/FileSelector.html +901 -0
- data/docs/rdocs/Longleaf/FixityCheckService.html +691 -0
- data/docs/rdocs/Longleaf/IndexManager.html +1155 -0
- data/docs/rdocs/Longleaf/InvalidDigestAlgorithmError.html +143 -0
- data/docs/rdocs/Longleaf/InvalidStoragePathError.html +143 -0
- data/docs/rdocs/Longleaf/Logging.html +405 -0
- data/docs/rdocs/Longleaf/Logging/RedirectingLogger.html +1213 -0
- data/docs/rdocs/Longleaf/LongleafError.html +139 -0
- data/docs/rdocs/Longleaf/MDFields.html +193 -0
- data/docs/rdocs/Longleaf/MetadataBuilder.html +787 -0
- data/docs/rdocs/Longleaf/MetadataDeserializer.html +537 -0
- data/docs/rdocs/Longleaf/MetadataError.html +143 -0
- data/docs/rdocs/Longleaf/MetadataPersistenceManager.html +539 -0
- data/docs/rdocs/Longleaf/MetadataRecord.html +1411 -0
- data/docs/rdocs/Longleaf/MetadataSerializer.html +786 -0
- data/docs/rdocs/Longleaf/PreservationServiceError.html +147 -0
- data/docs/rdocs/Longleaf/PreserveCommand.html +410 -0
- data/docs/rdocs/Longleaf/PreserveEvent.html +491 -0
- data/docs/rdocs/Longleaf/RegisterCommand.html +428 -0
- data/docs/rdocs/Longleaf/RegisterEvent.html +628 -0
- data/docs/rdocs/Longleaf/RegisteredFileSelector.html +446 -0
- data/docs/rdocs/Longleaf/RegistrationError.html +151 -0
- data/docs/rdocs/Longleaf/ReindexCommand.html +576 -0
- data/docs/rdocs/Longleaf/RsyncReplicationService.html +1180 -0
- data/docs/rdocs/Longleaf/SequelIndexDriver.html +1978 -0
- data/docs/rdocs/Longleaf/ServiceCandidateFilesystemIterator.html +572 -0
- data/docs/rdocs/Longleaf/ServiceCandidateIndexIterator.html +532 -0
- data/docs/rdocs/Longleaf/ServiceCandidateLocator.html +333 -0
- data/docs/rdocs/Longleaf/ServiceClassCache.html +725 -0
- data/docs/rdocs/Longleaf/ServiceDateHelper.html +425 -0
- data/docs/rdocs/Longleaf/ServiceDefinition.html +683 -0
- data/docs/rdocs/Longleaf/ServiceDefinitionManager.html +371 -0
- data/docs/rdocs/Longleaf/ServiceDefinitionValidator.html +269 -0
- data/docs/rdocs/Longleaf/ServiceFields.html +173 -0
- data/docs/rdocs/Longleaf/ServiceManager.html +1229 -0
- data/docs/rdocs/Longleaf/ServiceMappingManager.html +410 -0
- data/docs/rdocs/Longleaf/ServiceMappingValidator.html +347 -0
- data/docs/rdocs/Longleaf/ServiceRecord.html +821 -0
- data/docs/rdocs/Longleaf/StorageLocation.html +985 -0
- data/docs/rdocs/Longleaf/StorageLocationManager.html +729 -0
- data/docs/rdocs/Longleaf/StorageLocationUnavailableError.html +143 -0
- data/docs/rdocs/Longleaf/StorageLocationValidator.html +373 -0
- data/docs/rdocs/Longleaf/StoragePathValidator.html +253 -0
- data/docs/rdocs/Longleaf/SystemConfigBuilder.html +441 -0
- data/docs/rdocs/Longleaf/SystemConfigFields.html +163 -0
- data/docs/rdocs/Longleaf/ValidateConfigCommand.html +451 -0
- data/docs/rdocs/Longleaf/ValidateMetadataCommand.html +408 -0
- data/docs/rdocs/_index.html +660 -0
- data/docs/rdocs/class_list.html +51 -0
- data/docs/rdocs/css/common.css +1 -0
- data/docs/rdocs/css/full_list.css +58 -0
- data/docs/rdocs/css/style.css +496 -0
- data/docs/rdocs/file.README.html +165 -0
- data/docs/rdocs/file_list.html +56 -0
- data/docs/rdocs/frames.html +17 -0
- data/docs/rdocs/index.html +165 -0
- data/docs/rdocs/js/app.js +303 -0
- data/docs/rdocs/js/full_list.js +216 -0
- data/docs/rdocs/js/jquery.js +4 -0
- data/docs/rdocs/method_list.html +2051 -0
- data/docs/rdocs/top-level-namespace.html +110 -0
- data/lib/longleaf/candidates/file_selector.rb +150 -0
- data/lib/longleaf/candidates/manifest_digest_provider.rb +17 -0
- data/lib/longleaf/candidates/physical_path_provider.rb +17 -0
- data/lib/longleaf/candidates/registered_file_selector.rb +67 -0
- data/lib/longleaf/candidates/service_candidate_filesystem_iterator.rb +93 -0
- data/lib/longleaf/candidates/service_candidate_index_iterator.rb +84 -0
- data/lib/longleaf/candidates/service_candidate_locator.rb +23 -0
- data/lib/longleaf/candidates/single_digest_provider.rb +13 -0
- data/lib/longleaf/cli.rb +252 -46
- data/lib/longleaf/commands/deregister_command.rb +51 -0
- data/lib/longleaf/commands/preserve_command.rb +50 -0
- data/lib/longleaf/commands/register_command.rb +34 -43
- data/lib/longleaf/commands/reindex_command.rb +92 -0
- data/lib/longleaf/commands/validate_config_command.rb +33 -8
- data/lib/longleaf/commands/validate_metadata_command.rb +51 -0
- data/lib/longleaf/errors.rb +26 -7
- data/lib/longleaf/events/deregister_event.rb +53 -0
- data/lib/longleaf/events/event_names.rb +9 -0
- data/lib/longleaf/events/event_status_tracking.rb +59 -0
- data/lib/longleaf/events/preserve_event.rb +82 -0
- data/lib/longleaf/events/register_event.rb +59 -51
- data/lib/longleaf/helpers/case_insensitive_hash.rb +38 -0
- data/lib/longleaf/helpers/digest_helper.rb +56 -0
- data/lib/longleaf/helpers/s3_uri_helper.rb +86 -0
- data/lib/longleaf/helpers/selection_options_parser.rb +215 -0
- data/lib/longleaf/helpers/service_date_helper.rb +78 -0
- data/lib/longleaf/indexing/index_manager.rb +101 -0
- data/lib/longleaf/indexing/sequel_index_driver.rb +306 -0
- data/lib/longleaf/logging.rb +5 -4
- data/lib/longleaf/logging/redirecting_logger.rb +30 -25
- data/lib/longleaf/models/app_fields.rb +7 -2
- data/lib/longleaf/models/file_record.rb +31 -8
- data/lib/longleaf/models/filesystem_metadata_location.rb +56 -0
- data/lib/longleaf/models/filesystem_storage_location.rb +52 -0
- data/lib/longleaf/models/md_fields.rb +3 -1
- data/lib/longleaf/models/metadata_location.rb +47 -0
- data/lib/longleaf/models/metadata_record.rb +43 -16
- data/lib/longleaf/models/s3_storage_location.rb +138 -0
- data/lib/longleaf/models/service_definition.rb +7 -6
- data/lib/longleaf/models/service_fields.rb +7 -1
- data/lib/longleaf/models/service_record.rb +10 -6
- data/lib/longleaf/models/storage_location.rb +24 -19
- data/lib/longleaf/models/storage_types.rb +9 -0
- data/lib/longleaf/models/system_config_fields.rb +9 -0
- data/lib/longleaf/preservation_services/file_check_service.rb +59 -0
- data/lib/longleaf/preservation_services/fixity_check_service.rb +124 -0
- data/lib/longleaf/preservation_services/rsync_replication_service.rb +198 -0
- data/lib/longleaf/preservation_services/s3_replication_service.rb +131 -0
- data/lib/longleaf/services/application_config_deserializer.rb +81 -24
- data/lib/longleaf/services/application_config_manager.rb +20 -6
- data/lib/longleaf/services/application_config_validator.rb +19 -9
- data/lib/longleaf/services/configuration_validator.rb +67 -4
- data/lib/longleaf/services/filesystem_location_validator.rb +16 -0
- data/lib/longleaf/services/metadata_deserializer.rb +115 -42
- data/lib/longleaf/services/metadata_persistence_manager.rb +47 -0
- data/lib/longleaf/services/metadata_serializer.rb +156 -23
- data/lib/longleaf/services/metadata_validator.rb +76 -0
- data/lib/longleaf/services/s3_location_validator.rb +19 -0
- data/lib/longleaf/services/service_class_cache.rb +112 -0
- data/lib/longleaf/services/service_definition_manager.rb +10 -7
- data/lib/longleaf/services/service_definition_validator.rb +25 -18
- data/lib/longleaf/services/service_manager.rb +86 -11
- data/lib/longleaf/services/service_mapping_manager.rb +13 -12
- data/lib/longleaf/services/service_mapping_validator.rb +36 -26
- data/lib/longleaf/services/storage_location_manager.rb +76 -15
- data/lib/longleaf/services/storage_location_validator.rb +49 -35
- data/lib/longleaf/specs/config_builder.rb +47 -23
- data/lib/longleaf/specs/config_validator_helpers.rb +16 -0
- data/lib/longleaf/specs/custom_matchers.rb +9 -0
- data/lib/longleaf/specs/file_helpers.rb +61 -0
- data/lib/longleaf/specs/metadata_builder.rb +98 -0
- data/lib/longleaf/specs/system_config_builder.rb +27 -0
- data/lib/longleaf/version.rb +1 -1
- data/longleaf.gemspec +20 -7
- data/mkdocs.yml +21 -0
- metadata +308 -24
- data/.travis.yml +0 -4
- data/lib/longleaf/commands/abstract_command.rb +0 -37
- data/lib/longleaf/services/storage_path_validator.rb +0 -16
data/.yardopts
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
--output-dir docs/rdocs
|
data/Gemfile
CHANGED
@@ -1,4 +1,19 @@
|
|
1
1
|
source 'https://rubygems.org'
|
2
2
|
|
3
|
-
# Specify your gem's dependencies in longleaf.gemspec
|
4
3
|
gemspec
|
4
|
+
|
5
|
+
group :postgres, optional: true do
|
6
|
+
gem 'pg', '>= 1.0.0'
|
7
|
+
end
|
8
|
+
|
9
|
+
group :sqlite, optional: true do
|
10
|
+
gem 'sqlite3'
|
11
|
+
end
|
12
|
+
|
13
|
+
group :mysql2, optional: true do
|
14
|
+
gem 'mysql2', ">= 0.5.0"
|
15
|
+
end
|
16
|
+
|
17
|
+
group :mysql, optional: true do
|
18
|
+
gem 'mysql'
|
19
|
+
end
|
data/README.md
CHANGED
@@ -1,12 +1,39 @@
|
|
1
1
|
# Longleaf
|
2
|
+
Code: [![CircleCI](https://circleci.com/gh/UNC-Libraries/longleaf-preservation.svg?style=svg)](https://circleci.com/gh/UNC-Libraries/longleaf-preservation)
|
2
3
|
|
3
|
-
|
4
|
+
Longleaf is a command-line tool which allows users to configure a set of storage locations and define custom sets of preservation services to run on their contents. These services are executed in response to applicable preservation events issued by clients. Its primary goal is to provide tools to create a simple and customizable preservation environment. Longleaf:
|
4
5
|
|
5
|
-
|
6
|
+
* Offers a predictable command-line interface and integrates with standard command-line tools.
|
7
|
+
* Offers configurable and customizable criteria based preservation workflows.
|
8
|
+
* Provides a base set of tools and a framework for building extensions.
|
9
|
+
* Provides activity logging and notifications.
|
10
|
+
* Performs preservation services only when required.
|
6
11
|
|
7
12
|
## Installation
|
8
13
|
|
9
|
-
|
14
|
+
There are two primary ways to install Longleaf, depending on how you intend to use it:
|
15
|
+
|
16
|
+
#### Standalone gem
|
17
|
+
|
18
|
+
To use Longleaf as a command-line application, the gem can be installed using:
|
19
|
+
|
20
|
+
```
|
21
|
+
$ gem install longleaf
|
22
|
+
```
|
23
|
+
|
24
|
+
Or it may be built from source:
|
25
|
+
|
26
|
+
```
|
27
|
+
$ git clone git@github.com:UNC-Libraries/longleaf-preservation.git
|
28
|
+
$ cd longleaf-preservation
|
29
|
+
$ bin/setup
|
30
|
+
$ bundle exec rake install # builds the gem
|
31
|
+
$ gem install --local pkg/longleaf* # installs gem
|
32
|
+
```
|
33
|
+
|
34
|
+
#### Applicaton dependency
|
35
|
+
|
36
|
+
To make use of longleaf as a dependency of your application, add this line to your application's Gemfile:
|
10
37
|
|
11
38
|
```ruby
|
12
39
|
gem 'longleaf'
|
@@ -14,30 +41,89 @@ gem 'longleaf'
|
|
14
41
|
|
15
42
|
And then execute:
|
16
43
|
|
17
|
-
|
44
|
+
```
|
45
|
+
$ bundle
|
46
|
+
```
|
18
47
|
|
19
|
-
|
48
|
+
## Usage
|
20
49
|
|
21
|
-
|
50
|
+
#### Register a file
|
51
|
+
In order to register a new file with Longleaf, use the register command:
|
22
52
|
|
23
|
-
|
53
|
+
```
|
54
|
+
longleaf register -c <config.yml> -f <path to file>
|
55
|
+
```
|
56
|
+
|
57
|
+
In the case that a file's content is replaced, the file can be re-registered by providing the `--force` flag.
|
24
58
|
|
25
59
|
#### Validate configuration files
|
26
60
|
Application configuration files can be validated prior to usage with the following command:
|
27
|
-
|
61
|
+
|
62
|
+
```
|
63
|
+
longleaf validate_config -c <config.yml>
|
64
|
+
```
|
65
|
+
|
66
|
+
#### Output and logging
|
67
|
+
|
68
|
+
The primary output from Longleaf is directed to STDOUT, and contains both success and failure messages. If you would like to only return failure messages, you may provide the `--failure_only` flag.
|
69
|
+
|
70
|
+
Additional logging is sent to STDERR. To control the level of logging, you may provide the `--log-level` parameter, which expects the standard [Ruby Logger levels](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html). The default log level is 'WARN'.
|
71
|
+
|
72
|
+
Messages sent to STDOUT are duplicated to STDERR at 'INFO' level, so they are excluded by default. In order to store an ongoing log of activity and errors, you would perform the following:
|
73
|
+
|
74
|
+
```
|
75
|
+
longleaf <command> --log-level 'INFO' 2> /logs/longleaf.log
|
76
|
+
```
|
28
77
|
|
29
78
|
## Development
|
30
79
|
|
31
|
-
After checking out the repo, run `bin/setup` to install dependencies.
|
80
|
+
After checking out the repo, run `bin/setup` to install dependencies.
|
81
|
+
|
82
|
+
To perform the tests, run:
|
83
|
+
```
|
84
|
+
bundle exec rspec
|
85
|
+
```
|
86
|
+
|
87
|
+
To run Longleaf with local changes without needing to do a local install, you may run:
|
88
|
+
```
|
89
|
+
bundle exec exe/longleaf <command>
|
90
|
+
```
|
91
|
+
|
92
|
+
To install this gem onto your local machine, run:
|
93
|
+
```
|
94
|
+
bundle exec rake install
|
95
|
+
```
|
96
|
+
|
97
|
+
This places a newly built gem into the `pkg/` directory. This gem may then be installed in order to run commands in the `longleaf <command>` form.
|
98
|
+
_Note:_ Only files committed to git will be included in the installed gem.
|
99
|
+
|
100
|
+
To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
101
|
+
|
102
|
+
## Indexing
|
103
|
+
To use an index to improve performance, you will need to install the database drivers separately or bundle longleaf with the driver you wish to use:
|
104
|
+
|
105
|
+
```
|
106
|
+
bundle install --with postgres
|
107
|
+
```
|
108
|
+
|
109
|
+
Options include: postgres, mysql2, mysql, sqlite, amalgalite
|
110
|
+
|
111
|
+
To setup an index, you will need to add a `system > index` section to your configuration with the details of the database to use for the index. Then to setup the database, run:
|
32
112
|
|
33
|
-
|
113
|
+
```
|
114
|
+
longleaf setup_index -c <config_file>
|
115
|
+
```
|
116
|
+
And for a one-time indexing:
|
117
|
+
```
|
118
|
+
longleaf reindex -c <config_file>
|
119
|
+
```
|
34
120
|
|
35
121
|
## Contributing
|
36
122
|
|
37
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
123
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/UNC-Libraries/longleaf-preservation.
|
38
124
|
|
39
125
|
|
40
126
|
## License
|
41
127
|
|
42
|
-
The gem is available as open source under the terms of the [
|
128
|
+
The gem is available as open source under the terms of the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).
|
43
129
|
|
data/Rakefile
CHANGED
@@ -1,6 +1,12 @@
|
|
1
1
|
require "bundler/gem_tasks"
|
2
2
|
require "rspec/core/rake_task"
|
3
|
+
require "yard"
|
3
4
|
|
4
5
|
RSpec::Core::RakeTask.new(:spec)
|
5
6
|
|
6
7
|
task :default => :spec
|
8
|
+
|
9
|
+
YARD::Rake::YardocTask.new do |t|
|
10
|
+
t.options = ['--private', '--protected']
|
11
|
+
t.stats_options = ['--list-undoc']
|
12
|
+
end
|
data/bin/setup
CHANGED
@@ -1,8 +1,23 @@
|
|
1
1
|
#!/usr/bin/env bash
|
2
2
|
set -euo pipefail
|
3
3
|
IFS=$'\n\t'
|
4
|
+
|
5
|
+
INSTALL_DEST=(--path '.bundle/')
|
6
|
+
|
7
|
+
while [[ $# -gt 0 ]]
|
8
|
+
do
|
9
|
+
key="$1"
|
10
|
+
|
11
|
+
case $key in
|
12
|
+
-s|--system)
|
13
|
+
INSTALL_DEST=(--system)
|
14
|
+
shift # past argument
|
15
|
+
;;
|
16
|
+
esac
|
17
|
+
done
|
18
|
+
|
4
19
|
set -vx
|
5
20
|
|
6
|
-
bundle install
|
21
|
+
bundle install ${INSTALL_DEST[@]} --without ""
|
7
22
|
|
8
23
|
# Do any other automated setup that you need to do here
|
@@ -0,0 +1,28 @@
|
|
1
|
+
## About Longleaf
|
2
|
+
|
3
|
+
Longleaf, developed at the University of North Carolina at Chapel Hill University Libraries, is an open-source, portable digital preservation tool, designed to enable the creation and application of highly configurable preservation plans for large and varied collections of digital content across multiple systems.
|
4
|
+
|
5
|
+
Longleaf addresses challenges we have encountered over the past 20 years of growing and managing our digital collections infrastructure, which we feel are shared by other institutions:
|
6
|
+
|
7
|
+
1. Preservation activities being applied to files based on system affiliation (i.e. repository platform or lack thereof) rather than the needs of the content.
|
8
|
+
|
9
|
+
1. Difficulty maintaining an ideal schedule of fixity checks as the sizes of our collections grow.
|
10
|
+
|
11
|
+
1. Physical and computational costs to servers and storage devices imposed by ongoing cryptographic checksum algorithms (Altman et al., 2013).
|
12
|
+
|
13
|
+
1. Difficulty gradually introducing cloud storage services into our replication strategy for vulnerable files.
|
14
|
+
|
15
|
+
The goal for Longleaf is to provide organizations with a flexible tool for creating and implementing tailored preservation practices across the scope of their content, based on appropriate levels of digital preservation need (Phillips, Bailey, Goethals, & Owens, 2013). To this end, we have designed Longleaf according to the principles of high “software availability” (Davidson & Casden, 2016) that prioritize ease of use by a broad set of users in a variety of environments.
|
16
|
+
|
17
|
+
The Longleaf application is a command-line utility that will run on any modern Linux operating system with only a ruby interpreter. Longleaf can be applied to any content storage system with a file system, and requires no repository, no external database, and no storage system other than the file system. It can be run completely from the command line or triggered by arbitrary external systems (e.g. initiated on file ingest). Preservation processes are targeted at the file and storage level rather than through a repository system intermediary, allowing files managed in temporary storage or non-preservation asset management systems to benefit from the same replication and verification processes as those ingested into preservation repositories. For example, in our own digital collections context, we will be applying Longleaf to content across a wide variety of systems, including files managed entirely on shared drives, files managed by Fedora-based repositories, as well as digitization masters managed by CONTENTdm.
|
18
|
+
|
19
|
+
Longleaf’s modular architecture and flexible configuration system reduce the interference of repository system constraints, to enable needs-based digital preservation planning processes such as evaluating and configuring specific preservation activities across subsets of larger collections. For our collections, we are using Longleaf to begin addressing specific challenges such as managing the physical and computational costs of large-scale fixity verification, and integrating storage endpoints with different access costs. For example, we are increasing coverage of ongoing and transactional fixity checks (Gallinger, Bailey, Cariani, Owens, & Altman, 2017) by providing typical cryptographic checksums alongside more scaleable non-cryptographic checks and filesystem checks. We can also determine more appropriate replication and verification schedules and techniques overall, based on the characteristics of both the source and destination storage locations for content.
|
20
|
+
|
21
|
+
## References
|
22
|
+
Altman, M., Bailey, J., Cariani, K., Gallinger, M., Mandelbaum, J., & Owens, T. (2013). NDSA Storage Report: Reflections on National Digital Stewardship Alliance Member Approaches to Preservation Storage Technologies. D-Lib Magazine, 19(5/6). https://doi.org/10.1045/may2013-altman
|
23
|
+
|
24
|
+
Davidson, B., & Casden, J. (2016). Beyond open source. Code4Lib Journal, Issue 31. Retrieved from http://journal.code4lib.org/articles/11148
|
25
|
+
|
26
|
+
Gallinger, M., Bailey, J., Cariani, K., Owens, T., & Altman, M. (2017). Trends in Digital Preservation Capacity and Practice: Results from the 2nd Bi-annual National Digital Stewardship Alliance Storage Survey. D-Lib Magazine, 23(7/8). https://doi.org/10.1045/july2017-gallinger
|
27
|
+
|
28
|
+
Phillips, M., Bailey, J., Goethals, A., & Owens, T. (2013). The NDSA levels of digital preservation: Explanation and uses. In Archiving Conference (Vol. 2013, pp. 216–222). Society for Imaging Science and Technology.
|
data/docs/extra.css
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
/* added link colors to text that's not in side nav menu */
|
2
|
+
|
3
|
+
.rst-content a:link { color:#0000FF; text-decoration:none; font-weight:normal; }
|
4
|
+
|
5
|
+
.rst-content a:visited { color:#800080; text-decoration:none; font-weight:normal; }
|
6
|
+
|
7
|
+
.rst-content a:hover { color:aqua; text-decoration:none; font-weight:normal; }
|
8
|
+
|
9
|
+
.rst-content a:active { color:seagreen; text-decoration:none; font-weight:normal; }
|
10
|
+
|
11
|
+
|
12
|
+
/* changed background colors in side nav menu */
|
13
|
+
|
14
|
+
.wy-side-nav-search {
|
15
|
+
background-color: mediumseagreen;
|
16
|
+
}
|
17
|
+
|
18
|
+
.wy-menu-vertical span {
|
19
|
+
/* line-height: 18px;*/
|
20
|
+
line-height: 20px;
|
21
|
+
|
22
|
+
padding: 0.4045em 1.618em;
|
23
|
+
display: block;
|
24
|
+
position: relative;
|
25
|
+
font-size: 90%;
|
26
|
+
color: #838383;
|
27
|
+
}
|
28
|
+
|
29
|
+
.wy-nav-side {
|
30
|
+
background: ##404040;
|
31
|
+
}
|
32
|
+
/*
|
Binary file
|
Binary file
|
data/docs/index.md
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
## Longleaf Project Overview
|
2
|
+
|
3
|
+
Welcome to the Longleaf Project Documentation site.
|
4
|
+
|
5
|
+
This site contains supporting documentation for the Longleaf software project, currently in development by The University of North Carolina at Chapel Hill's University Libraries.
|
6
|
+
|
7
|
+
Longleaf is free, open-source software for creating highly configurable digital preservation environments. Longleaf provides a repository-agnostic command-line utility that enables users to configure and apply preservation processes such as monitoring and replication at the file level. Users can define preservation services using Longleaf's base set of commands, and then apply these preservation scripts to a set of user-defined storage locations that are relevant to the user's specific context.
|
8
|
+
|
9
|
+
For a step-by-step tutorial of Longleaf's basic functionality, the ["Basic Usage" page](quickstart.md) in this site walks through Longleaf's core operations using a pre-configured example data directory.
|
10
|
+
|
11
|
+
## Project URLs
|
12
|
+
|
13
|
+
Code repository [Longleaf-preservation ](https://github.com/UNC-Libraries/longleaf-preservation)
|
14
|
+
|
15
|
+
## Project Participants
|
16
|
+
|
17
|
+
* Ben Pennell, Technical lead and software developer
|
18
|
+
* Jason Casden, Product lead
|
19
|
+
* Morgan McKeehan, Documentation lead
|
data/docs/install.md
ADDED
@@ -0,0 +1,66 @@
|
|
1
|
+
# Installing Longleaf
|
2
|
+
|
3
|
+
### Ruby and other Prerequisites
|
4
|
+
Longleaf requires Ruby 2.3 or higher.
|
5
|
+
|
6
|
+
There also are optional gem dependencies if the user wishes to use an index to improve performance.
|
7
|
+
|
8
|
+
Additionally, Longleaf scripts rely on common Unix programs. In Mac OS X and Linux operating systems, these programs will likely already be installed, but some of these tools, such as `rsync` may be missing from a Windows system unless you have installed them.
|
9
|
+
|
10
|
+
### Download Longleaf
|
11
|
+
Download Longleaf from UNC Chapel Hill's University Libraries [Longleaf github repository](https://github.com/UNC-Libraries/longleaf-preservation).
|
12
|
+
|
13
|
+
### Install Longleaf
|
14
|
+
|
15
|
+
There are two ways to install Longleaf, depending on how you intend to use it:
|
16
|
+
|
17
|
+
**1. Standalone gem**
|
18
|
+
|
19
|
+
To use Longleaf as a command-line application, the gem can be installed using:
|
20
|
+
|
21
|
+
$ gem install longleaf
|
22
|
+
|
23
|
+
Or it may be built from source:
|
24
|
+
|
25
|
+
$ git clone git@github.com:UNC-Libraries/longleaf-preservation.git
|
26
|
+
$ cd longleaf-preservation
|
27
|
+
$ bin/setup --system
|
28
|
+
$ bundle exec rake install # builds the gem
|
29
|
+
$ gem install --local pkg/longleaf* # installs gem
|
30
|
+
|
31
|
+
**2. Application dependency**
|
32
|
+
|
33
|
+
To include longleaf as a dependency in an application, add this line to your application's Gemfile:
|
34
|
+
|
35
|
+
```ruby
|
36
|
+
gem 'longleaf'
|
37
|
+
```
|
38
|
+
|
39
|
+
And then execute:
|
40
|
+
|
41
|
+
$ bundle
|
42
|
+
|
43
|
+
### Confirm Longleaf Installation
|
44
|
+
If you have installed Longleaf using the "Standalone gem" approach, you can check to make sure that the installation succeeded by typing the following into your terminal:
|
45
|
+
|
46
|
+
```
|
47
|
+
longleaf
|
48
|
+
```
|
49
|
+
|
50
|
+
You should see the Longleaf help page:
|
51
|
+
```
|
52
|
+
Commands:
|
53
|
+
longleaf --version # Prints the Longleaf version number.
|
54
|
+
longleaf deregister # Deregister files with Longleaf
|
55
|
+
longleaf help [COMMAND] # Describe available commands or one specific command
|
56
|
+
longleaf preserve # Perform preservation services on files with Longleaf
|
57
|
+
longleaf register # Register files with Longleaf
|
58
|
+
longleaf reindex # Perform a full reindex of file metadata stored within the configured storage locations.
|
59
|
+
longleaf setup_index # Sets up the structure of the metadata index, if one is configured using the system configuration file provide...
|
60
|
+
longleaf validate_config # Validate an application configuration file, provided using --config.
|
61
|
+
longleaf validate_metadata # Validate metadata files.
|
62
|
+
|
63
|
+
|
64
|
+
```
|
65
|
+
### Installation Success!
|
66
|
+
If the Longleaf Help page printed successfully, you are ready to proceed to the [Basic Usage tutorial](quickstart.md) to try out basic Longleaf functionality.
|
@@ -0,0 +1,33 @@
|
|
1
|
+
# Storage locations below use relative paths. When validating and using
|
2
|
+
# the configuration file, paths for the storage locations will be evaluated
|
3
|
+
# relative to the location of the configuration file. In this example tutorial,
|
4
|
+
# the storage locations are directories ('files-dir' and 'replica-files')
|
5
|
+
# located at the same level as the config file, within the ll-example
|
6
|
+
# directory.
|
7
|
+
|
8
|
+
locations:
|
9
|
+
data-directory:
|
10
|
+
path: files-dir
|
11
|
+
metadata_path: metadata-dir
|
12
|
+
backup-directory:
|
13
|
+
path: replica-files
|
14
|
+
metadata_path: replica-metadata
|
15
|
+
services:
|
16
|
+
example_replication:
|
17
|
+
work_script: 'rsync_replication_service'
|
18
|
+
to:
|
19
|
+
- backup-directory
|
20
|
+
example_fixity:
|
21
|
+
work_script: 'fixity_check_service'
|
22
|
+
frequency: 30 seconds
|
23
|
+
absent_digest: generate
|
24
|
+
algorithms:
|
25
|
+
- sha1
|
26
|
+
service_mappings:
|
27
|
+
- locations: data-directory
|
28
|
+
services:
|
29
|
+
- example_replication
|
30
|
+
- example_fixity
|
31
|
+
- locations: backup-directory
|
32
|
+
services:
|
33
|
+
- example_fixity
|
Binary file
|
@@ -0,0 +1,15 @@
|
|
1
|
+
## MAKE CHANGES TO THIS FILE
|
2
|
+
|
3
|
+
* make any changes here!
|
4
|
+
* ( insert changed text )
|
5
|
+
|
6
|
+
## Longleaf
|
7
|
+
Longleaf is a command-line tool which allows users to configure a set of storage locations and define custom sets of preservation services to run on their contents. These services are executed in response to applicable preservation events issued by clients. Its primary goal is to provide tools to create a simple and customizable preservation environment.
|
8
|
+
|
9
|
+
Longleaf:
|
10
|
+
|
11
|
+
* Offers a predictable command-line interface and integrates with standard command-line tools.
|
12
|
+
* Offers configurable and customizable criteria based preservation workflows.
|
13
|
+
* Provides a base set of tools and a framework for building extensions.
|
14
|
+
* Provides activity logging and notifications.
|
15
|
+
* Performs preservation services only when required.
|
@@ -0,0 +1,10 @@
|
|
1
|
+
## Longleaf
|
2
|
+
Longleaf is a command-line tool which allows users to configure a set of storage locations and define custom sets of preservation services to run on their contents. These services are executed in response to applicable preservation events issued by clients. Its primary goal is to provide tools to create a simple and customizable preservation environment.
|
3
|
+
|
4
|
+
Longleaf:
|
5
|
+
|
6
|
+
* Offers a predictable command-line interface and integrates with standard command-line tools.
|
7
|
+
* Offers configurable and customizable criteria based preservation workflows.
|
8
|
+
* Provides a base set of tools and a framework for building extensions.
|
9
|
+
* Provides activity logging and notifications.
|
10
|
+
* Performs preservation services only when required.
|
File without changes
|
File without changes
|
File without changes
|
data/docs/quickstart.md
ADDED
@@ -0,0 +1,270 @@
|
|
1
|
+
## Longleaf basic usage tutorial
|
2
|
+
|
3
|
+
This tutorial provides an introduction to the basic functionality of Longleaf by demonstrating how to configure and execute a small set of preservation tasks, using an example data directory as a local sandbox on your own computer.
|
4
|
+
|
5
|
+
**Longleaf basic usage tasks covered in this tutorial:**
|
6
|
+
|
7
|
+
* Validate the mandatory Longleaf configuration file that contains the storage locations and preservation activities you will use
|
8
|
+
* Register example data files
|
9
|
+
* Validate metadata for registered files
|
10
|
+
* Run a Preservation action on example data files
|
11
|
+
* Alter an example data file to cause an error; catch the error by re-running the Preservation action
|
12
|
+
* Re-register the altered file, and preserve the file in its new, altered state
|
13
|
+
|
14
|
+
### System Requirements for this tutorial
|
15
|
+
Longleaf scripts rely on common UNIX programs. In Mac OS X and Linux operating systems, these programs will likely already be installed, but some of these tools, such as `rsync` may be missing from a Windows system unless you have installed it.
|
16
|
+
|
17
|
+
### Example data directory for this tutorial
|
18
|
+
The example data directory for this tutorial is named 'll-example'; it is located in the 'docs' folder of the longleaf-preservation repository.
|
19
|
+
|
20
|
+
The 'll-example' directory contains all the materials required for completing this tutorial:
|
21
|
+
|
22
|
+
* an example configuration file, 'config-example-relative.yml', that is pre-configured for tutorial tasks
|
23
|
+
* a folder of example data files to be preserved
|
24
|
+
* empty scaffolding folders that will be used for storing: 1) metadata files about the original data files, 2) the replicated data files created from the originals, and 3) metadata files about the replicated data files.
|
25
|
+
|
26
|
+
**folder structure of 'll-example' directory: materials for tutorial tasks**
|
27
|
+
```tree
|
28
|
+
└───ll-example
|
29
|
+
│ config-example-relative.yml
|
30
|
+
│
|
31
|
+
├───files-dir
|
32
|
+
│ LLexample-PDF.pdf
|
33
|
+
│ LLexample-TOCHANGE.txt
|
34
|
+
│ LLexample-tokeep.txt
|
35
|
+
│
|
36
|
+
├───metadata-dir
|
37
|
+
│
|
38
|
+
│
|
39
|
+
├───replica-files
|
40
|
+
│
|
41
|
+
│
|
42
|
+
└───replica-metadata
|
43
|
+
```
|
44
|
+
|
45
|
+
Subdirectories in 'll-example' are used as follows:
|
46
|
+
|
47
|
+
**'files-dir'**
|
48
|
+
Contains three example data files that will be preserved and replicated.
|
49
|
+
|
50
|
+
**'metadata-dir'**
|
51
|
+
Empty at start of tutorial; metadata files about the data files in 'files-dir' will be stored and automatically updated in this directory during Register and Preserve actions.
|
52
|
+
|
53
|
+
**'replica-files'**
|
54
|
+
Empty at start of tutorial; replicated data files (copies) created during Preserve actions run on the data files in 'files-dir' will be stored in this directory.
|
55
|
+
|
56
|
+
**'replica-metadata'**
|
57
|
+
Empty at start of tutorial; metadata files about the replicated data files in 'replica-files' will be stored in this directory.
|
58
|
+
|
59
|
+
### Set up your local sandbox: copy the example data directory to your Desktop
|
60
|
+
Copy the 'll-example' directory to a location where it will be easy to access and work with. For example,it might be convenient to copy 'll-example' to your 'Desktop' location.
|
61
|
+
|
62
|
+
Note: All paths in the commands in this tutorial are relative to the 'll-example' folder itself, so if you execute commands from within 'll-example,' you can use the paths as shown with no alterations.
|
63
|
+
|
64
|
+
### Review the example Longleaf configuration file
|
65
|
+
|
66
|
+
**Locations**
|
67
|
+
The example configuration file references storage locations based on the subdirectory names in 'll-example':
|
68
|
+
|
69
|
+
* The 'data-directory' location points to 'files-dir' as the location of the original data files to be preserved, and 'metadata-dir' for storing metadata files created about these original data files.
|
70
|
+
* The 'backup-directory' location refers to the 'replica-files' folder as the storage location where the copies of the original files will be stored, and 'replica-metadata' for metadata files.
|
71
|
+
|
72
|
+
**Services**
|
73
|
+
The 'services' area in the configuration file defines characteristics of work scripts, such as the target storage location for the replication script, and the checksum algorithm for the fixity script.
|
74
|
+
|
75
|
+
**Service Mappings**
|
76
|
+
The 'service-mappings' area specifies which services will run at a particular storage location. In the example configuration file, for example, the services 'example_replication' and 'example_fixity' are assigned to run at location 'data-directory'. At location 'backup_directory', however, only 'example_fixity' is assigned to run, so files there will have fixity checks, but will not get replicated.
|
77
|
+
|
78
|
+
```yml
|
79
|
+
locations: data-directory
|
80
|
+
services:
|
81
|
+
- example_replication
|
82
|
+
- example_fixity
|
83
|
+
- locations: backup-directory
|
84
|
+
services:
|
85
|
+
- example_fixity
|
86
|
+
```
|
87
|
+
|
88
|
+
**Relative Paths in the Configuration File**
|
89
|
+
Storage locations in this configuration file specify paths relative to the location of the configuration file itself, and will also be evaluated relative to the location of the configuration file itself when Longleaf commands are run. In this tutorial, the configuration file is located inside 'll-example', at the same level as the data and metadata folders.
|
90
|
+
|
91
|
+
### Check that you can run Longleaf, by viewing the help text page
|
92
|
+
Once you have copied the 'll-example' directory to an accessible working location, and have familiarized yourself with the configuration yaml file, you are ready to start using Longleaf!
|
93
|
+
|
94
|
+
First confirm that you can run Longleaf, as you did in the ["Installation Instructions" page](install.md) of this site. Open a terminal window and `cd` into the 'll-example' directory, then enter the command 'longleaf', with no arguments:
|
95
|
+
```
|
96
|
+
longleaf
|
97
|
+
```
|
98
|
+
This command will output the help page text, showing all available commands.
|
99
|
+
|
100
|
+
### Validate the example configuration file
|
101
|
+
Next, you will validate the Longleaf configuration file, to ensure that storage locations and paths are correctly specified and accessible.
|
102
|
+
|
103
|
+
As noted above, all paths in this tutorial are relative to the 'll-example' directory, so make sure that you execute commands from within that directory, or amend paths accordingly.
|
104
|
+
|
105
|
+
Command for validating configuration file:
|
106
|
+
|
107
|
+
```
|
108
|
+
longleaf validate_config -c config-example-relative.yml
|
109
|
+
```
|
110
|
+
|
111
|
+
Successful validation of the configuration file outputs the following message in terminal:
|
112
|
+
```
|
113
|
+
SUCCESS: Application configuration passed validation: config-example-relative.yml
|
114
|
+
```
|
115
|
+
|
116
|
+
### Register example data files
|
117
|
+
Once the configuration file has been validated, you can proceed to registering the example data files, so that they are ready for further preservation actions with Longleaf.
|
118
|
+
|
119
|
+
Note that Longleaf commands can be run on directories, or individual files. For example, when using the Register command, you can either:
|
120
|
+
|
121
|
+
1. register a whole directory at once, which will individually register every file in that directory
|
122
|
+
**or**
|
123
|
+
2. register individual files.
|
124
|
+
|
125
|
+
In this example of using the Register command, we will register an entire directory of files by specifying the directory path.
|
126
|
+
|
127
|
+
Use command:
|
128
|
+
```
|
129
|
+
longleaf register -c config-example-relative.yml -f files-dir
|
130
|
+
```
|
131
|
+
If files successfully registered, Longleaf will list the registration outcome for each file within the directory:
|
132
|
+
```
|
133
|
+
SUCCESS register /data/ll-example/files-dir/LLexample-PDF.pdf
|
134
|
+
SUCCESS register /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
135
|
+
SUCCESS register /data/ll-example/files-dir/LLexample-tokeep.txt
|
136
|
+
|
137
|
+
```
|
138
|
+
|
139
|
+
### Validate the new metadata files
|
140
|
+
Validate the metadata files that were created for the files you just registered.
|
141
|
+
Use command:
|
142
|
+
```
|
143
|
+
longleaf validate_metadata -c config-example-relative.yml -f files-dir
|
144
|
+
```
|
145
|
+
Confirmation of validation for the metadata files:
|
146
|
+
```
|
147
|
+
SUCCESS: Metadata for file passed validation: /data/ll-example/files-dir/LLexample-PDF.pdf
|
148
|
+
SUCCESS: Metadata for file passed validation: /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
149
|
+
SUCCESS: Metadata for file passed validation: /data/ll-example/files-dir/LLexample-tokeep.txt
|
150
|
+
|
151
|
+
```
|
152
|
+
|
153
|
+
|
154
|
+
### Run the Preserve command, to replicate and check fixity on the registered files.
|
155
|
+
Use command:
|
156
|
+
```
|
157
|
+
longleaf preserve -c config-example-relative.yml -f files-dir
|
158
|
+
```
|
159
|
+
Confirmation of successful Preservation action completed on all files in the files-dir directory:
|
160
|
+
```
|
161
|
+
SUCCESS register /data/ll-example/replica-files/LLexample-PDF.pdf
|
162
|
+
SUCCESS preserve[example_replication] /data/ll-example/files-dir/LLexample-PDF.pdf
|
163
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-PDF.pdf
|
164
|
+
SUCCESS register /data/ll-example/replica-files/LLexample-TOCHANGE.txt
|
165
|
+
SUCCESS preserve[example_replication] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
166
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
167
|
+
SUCCESS register /data/ll-example/replica-files/LLexample-tokeep.txt
|
168
|
+
SUCCESS preserve[example_replication] /data/ll-example/files-dir/LLexample-tokeep.txt
|
169
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-tokeep.txt
|
170
|
+
|
171
|
+
```
|
172
|
+
|
173
|
+
### Look at the output of the Preserve command
|
174
|
+
Take a look at the SUCCESS output messages from the `preserve` command above.
|
175
|
+
|
176
|
+
Notice that for each file included in the Preservation action, Longleaf indicates "SUCCESS" for the "register" and "preserve" components of the Preservation action, as well as the individual scripts within the "preserve" component.
|
177
|
+
|
178
|
+
Also take a look inside the subfolders in your 'll-example' sandbox folder, using Finder or command-line tools.
|
179
|
+
|
180
|
+
You'll notice that there are newly-created data files and metadata files in the 'replica-' directories. The original data files from 'files-dir' have been copied to the 'replica-files' directory, and metadata files for those copies have been created in the 'replica-metadata' directory.
|
181
|
+
|
182
|
+
![preserve files folders output](img/ll-example-preserved.png)
|
183
|
+
|
184
|
+
### Re-run the Preserve command, to check the integrity of the files that are being preserved.
|
185
|
+
|
186
|
+
Since the 'example_fixity' script in our example configuration file is set to frequency: 30 seconds, we can check fixity on the preserved files once 30 seconds have elapsed since the last preservation action.
|
187
|
+
|
188
|
+
Use command:
|
189
|
+
```
|
190
|
+
longleaf preserve -c config-example-relative.yml -f files-dir
|
191
|
+
```
|
192
|
+
Success output message:
|
193
|
+
```
|
194
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-PDF.pdf
|
195
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
196
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-tokeep.txt
|
197
|
+
|
198
|
+
```
|
199
|
+
Note that the output in terminal window is much shorter than the first `preserve` command. This output only shows "SUCCESS" and preserve[example_fixity] for each file. This time register and replication are not included because those actions already ran, and do not need to run this time. Only fixity checks ran in this Preserve action.
|
200
|
+
|
201
|
+
In the finder window, notice that the Date Modified is updated on the metadata files in the metadata-dir. The metadata files for the original data files are updated to reflect these Preservation actions running.
|
202
|
+
|
203
|
+
### Cause a change to a file, and catch the change with the Preservation command.
|
204
|
+
Now make a change to one of the files that is being preserved, which you will then catch by re-running `preserve` on the storage directory for that file.
|
205
|
+
|
206
|
+
Open up the file: 'll-example/files-dir/LLexample-TOCHANGE.txt'. On line 4 of the text file, you'll see "( insert changed text ) ". Insert your cursor near this line, and type something to make a change to the file. Then save the file and close it. In your 'll-example/files-dir' directory, notice that the Date Modified has changed for this file.
|
207
|
+
|
208
|
+
Run the `preserve` command again. This time it will fail, because one of the files under preservation has been altered since the file's last checksums were recorded.
|
209
|
+
|
210
|
+
Use command:
|
211
|
+
```
|
212
|
+
longleaf preserve -c config-example-relative.yml -f files-dir
|
213
|
+
```
|
214
|
+
Output showing failed fixity check on the file 'll-example/files-dir/LLexample-TOCHANGE.txt':
|
215
|
+
```
|
216
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-PDF.pdf
|
217
|
+
FAILURE preserve[example_fixity] /data/ll-example/files-dir/LLexample-TOCHANGE.txt: Fixity check using algorithm 'sha1' failed for file /data/ll-example/files-dir/LLexample-TOCHANGE.txt: expected 'effa1bfc1b93f1260a36044bcd668240cf14738a', calculated 'cd5e53f6945ceda2d90b003a39af509d7acb068f.'
|
218
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-tokeep.txt
|
219
|
+
|
220
|
+
```
|
221
|
+
|
222
|
+
### Re-register the altered file.
|
223
|
+
Now you will re-register the altered file, using the `--force` option for the `register` command. Run this registration command at the file level so that only this file is re-registered, not the entire directory.
|
224
|
+
|
225
|
+
Re-registration will update the file's preservation metadata to the new state of 'LLexample-TOCHANGE.txt'.
|
226
|
+
|
227
|
+
Note, however, that the re-registered altered file (ll-example/files-dir/LLexample-TOCHANGE.txt) will still not match the copy of LLexample-TOCHANGE.txt that is stored in "replica-files" (ll-example/replica-files/LLexample-TOCHANGE.txt), because the replica copy was created earlier, from the unaltered original file.
|
228
|
+
|
229
|
+
Use command:
|
230
|
+
```
|
231
|
+
longleaf register -c config-example-relative.yml -f files-dir/LLexample-TOCHANGE.txt --force
|
232
|
+
```
|
233
|
+
Success output message:
|
234
|
+
```
|
235
|
+
SUCCESS register /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
236
|
+
```
|
237
|
+
|
238
|
+
### Run Preserve on the re-registered file.
|
239
|
+
Running `preserve` on the re-registered altered file will replicate the new state of this file to the copied data location. As with `register --force`, run `preserve` at the file level, not the directory.
|
240
|
+
|
241
|
+
Use command:
|
242
|
+
```
|
243
|
+
longleaf preserve -c config-example-relative.yml -f files-dir/LLexample-TOCHANGE.txt
|
244
|
+
```
|
245
|
+
|
246
|
+
The terminal output shows the new registration of the file, and the replication and fixity scripts running on this file.
|
247
|
+
In finder, you can see that the Date Modified of the 'll-example-TOCHANGE.txt' file in the 'replica-files' directory now matches the original file. Open up 'replica-files/ll-example-TOCHANGE.txt' and you'll see the added line of text in this file.
|
248
|
+
|
249
|
+
Successful preservation of re-registered changed file:
|
250
|
+
```
|
251
|
+
SUCCESS register /data/ll-example/replica-files/LLexample-TOCHANGE.txt
|
252
|
+
SUCCESS preserve[example_replication] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
253
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
254
|
+
|
255
|
+
```
|
256
|
+
|
257
|
+
### Re-run Preserve at the directory level to check all three files.
|
258
|
+
After the 30-sec fixity check interval has elapsed, re-run the `preserve` command at the directory level as before, and now all three files pass again.
|
259
|
+
|
260
|
+
Use command:
|
261
|
+
```
|
262
|
+
longleaf preserve -c config-example-relative.yml -f files-dir
|
263
|
+
```
|
264
|
+
Success output message:
|
265
|
+
```
|
266
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-PDF.pdf
|
267
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-TOCHANGE.txt
|
268
|
+
SUCCESS preserve[example_fixity] /data/ll-example/files-dir/LLexample-tokeep.txt
|
269
|
+
|
270
|
+
```
|