store-digest 0.1.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c6d86e00c5ff94af896658561fdc0076c17650e704dc611f4520e5580043c45a
+  data.tar.gz: 68524ba5e9d3e253b1d3b7d589d777d617d3d360d9d7c9ed5f8cd019ff3ea2eb
+SHA512:
+  metadata.gz: 713edaa0ced8d81587b276f09a9740af9f0b25f39689336cf96377a3a38c5fcf081f9b3532e1b85e8051cfaed730cca1885a68a477e5d3f5a2a3664f4d4cf3d2
+  data.tar.gz: 7113db7fd653759cd6431e2c0c5f5ea06063daabb43ff767e93f47c942967915e54315c40d1ead17f51244e5c24f92dfb508df689977a2aad61ee527c61f87b1

data/.gitignore

@@ -0,0 +1,15 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+# rspec failure tracking
+.rspec_status
+.\#*
+\#*\#
+Gemfile.lock
+*.gem

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,7 @@
+---
+sudo: false
+language: ruby
+cache: bundler
+rvm:
+  - 2.5.5
+before_install: gem install bundler -v 1.17.3

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in store-digest.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,231 @@
+# Store::Digest - An RFC6920-compliant content-addressable store
+
+There are a number of content-addressable stores out there. [Git is
+one](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects),
+[Perkeep](https://perkeep.org/) (née CamliStore) is another. Why
+another one? Well:
+
+1. **[RFC6920](https://tools.ietf.org/html/rfc6920) URI interface:** the
+   primary way you talk to this thing is through `ni:///…` URIs.
+2. **Multiple digest algorithms:** every object in the store is
+  identified by more than one cryptographic digest, as cryptographic
+  digest algorithms tend to come and go.
+3. **Network-optional:** don't run another network service when an
+   embedded solution will suffice.
+4. **Rudimentary metadata:** Aside from internal bookeeping
+   information, store _only_ the handful of facts that would enable a
+   blob to be properly rendered in a Web browser without integrating
+   another metadata source, _and nothing else_.
+5. **Organizational memory:** Retain a record, not only of every
+   object _currently_ in the store, but also every object that has
+   _ever been_ in the store.
+
+## How to use Store::Digest
+
+```ruby
+require 'store/digest'
+require 'pathname'
+require 'uri/ni'
+
+store = Store::Digest.new driver: :LMDB, dir: '/var/lib/store-digest'
+
+objs = Pathname('~/Desktop').expand_path.glob(?*).map do |f|
+  store.add f if f.file?
+end
+# congratulations, you have just copied all the stuff on your desktop.
+
+# let's make an identifer
+
+uri = URI::NI.compute 'some data'
+# => #<URI::NI ni:///sha-256;EweZDmulyhRes16ZGCqb7EZTG8VN32VqYCx4D6AkDe4>
+
+store.get uri
+# => nil
+
+# of course not because we didn't put the content in there, so let's do that:
+
+store.add 'some data'
+# => #<Store::Digest::Object:0x00007fa00d5ee3e0
+#  @content=
+#   #<Proc:0x00007fa00d5ee430:1 (lambda)>,
+#  @ctime=2020-01-18 14:05:56 -0800,
+#  @digests=
+#   {:md5=>#<URI::NI ni:///md5;HlAhCgICSX-3m8OLat5sNA>,
+#    :"sha-1"=>#<URI::NI ni:///sha-1;uvNFUf7LSKzD2oaOuF4bbayd41Y>,
+#    :"sha-256"=>
+#     #<URI::NI ni:///sha-256;EweZDmulyhRes16ZGCqb7EZTG8VN32VqYCx4D6AkDe4>,
+#    :"sha-384"=>
+#     #<URI::NI ni:///sha-384;qcYaFi9LVypj5rDitFrvRztzAn1ZBVWWakwJGFg3_3KhAZHBNuw_RhTXkU0dqCPw>,
+#    :"sha-512"=>
+#     #<URI::NI ni:///sha-512;4WRedJLwMvtixnTbdVAL57Jgv8DaqWWCHds_ikm10zeI7j8EZ0TiuVr7XD2PJQDFScqJ15_GiQiF0o4FUAdCTw>},
+#  @dtime=nil,
+#  @flags=0,
+#  @mtime=2020-01-18 14:05:56 -0800,
+#  @ptime=2020-01-18 14:05:56 -0800,
+#  @size=9,
+#  @type="text/plain">
+
+store.get uri
+# ...same thing...
+```
+
+The main operations are, of course, `add`, `get`, `remove`, and
+`forget`. I am currently working on a `search` which will match
+partial digests and metadata values.
+
+For each object, immutable bookkeeping metadata include:
+
+* Size (bytes)
+* Added to store (timestamp)
+* Metadata modified (timestamp)
+* Blob deleted (timestamp if present)
+
+User-manipulable metadata consists of:
+
+* Modification time (timestamp)
+* Content-type (MIME identifier, e.g. `text/html`)
+* Language (optional [RFC5646](https://tools.ietf.org/html/rfc5646)
+  token, e.g. `en-ca`)
+* Character set (optional token, e.g. `utf-8`, `iso-8859-1`, `windows-1252`)
+* Content-encoding (optional token, e.g. `gzip`, `deflate`)
+* Flags (8-bit unsigned integer)
+
+There are four flags, each with two bits of information:
+
+* Content-type
+* Character set
+* Content-encoding
+* Syntax
+
+For each of these flags, the values 0 to 3 signify:
+
+0. Unverified
+1. Invalid
+2. Recheck validation
+3. Verified valid
+
+These metadata fields are "user-manipulable" for an _extremely_ loose
+definition of "user". The idea, in particular for the fields that have
+associated flags, is that any initial value is a _claim_ that may be
+subsequently verified, by, for example, a separate maintenance daemon
+that scans newly-inserted objects and supplants their metadata with
+whatever it finds. For example, one could insert a compressed file of
+type `application/gzip`, and some other maintenance process could
+come along and realize that in fact the file is `image/svg+xml` with
+an _encoding_ of `gzip`, but there is also a syntax error, so it
+should not be served without first attempting to repair it. Because of
+the enormous combination of types, encodings, and syntaxes, such a
+maintenance daemon is way out of scope for this project, despite being
+a desirable and likely future addition to it.
+
+## API Documentation
+
+Generated and deposited [in the usual
+place](https://www.rubydoc.info/github/doriantaylor/rb-store-digest/master).
+
+## Design
+
+This package is intended to be the simplest possible substrate for
+managing opaque data segments in terms of their contents. The central
+ambition is to solidify a consistent interface for all the desired
+behaviour, and then subsequently expand that interface to different
+languages and platforms, including, when possible, adapters to
+existing content-addressable storage platforms.
+
+This version is written in Ruby, and is the maturation of [an earlier
+version written in Perl](https://metacpan.org/pod/Store::Digest).
+Whereas the Perl implementation uses [Berkeley
+DB](https://www.oracle.com/database/berkeley-db/) to store its
+metadata, this version uses [LMDB](https://symas.com/lmdb/). Indeed, a
+subsidiary goal of this project is to define a _pattern_ for
+lower-level database storage, and key-value stores in particular, such
+that complying digest store interfaces running on different
+programming languages can attach to the _same_ storage repository.
+
+The current implementation uses the file system store its blobs. It
+does so by taking the primary digest algorithm (`sha-256` by default)
+of the given blob, encoding it as base-32 (to be case-insensitive),
+and transforming the first few bits into a hashed directory structure
+(similar to Git, though it uses hexadecimal encoding). Internally, the
+blob and metadata subsystems are decoupled, such that the two can be
+mixed and matched.
+
+### Basic Architecture
+
+`Store::Digest` proper is a unified interface over a `Driver` which
+provides the concrete implementation. A driver may be further
+decoupled (as is the case with `LMDB`) into `Blob` and `Meta`
+subcomponents, which themselves may share `Trait`s (like storing its
+state in a `RootDir`). We can imagine this bifurcation not being
+universal, e.g. a prospective PostgreSQL driver could handle both
+blobs _and_ metadata within its own confines.
+
+> I have yet to decide on the final layout of this system, so don't
+> get too used to it.
+
+### A note on storage efficiency
+
+Unlike Git, which uses `pigz` (or rather, `deflate`) to compress its
+contents (along with a small amount of embedded metadata), objects in
+this system are stored as-is. This is deliberate: if you want to
+compress your objects, you should compress them _before_ adding them
+to the store, and signal the fact that they are compressed in the
+metadata.
+
+I have also elected to have the system designate a "primary" digest
+algorithm, which defaults to `sha-256`. Its binary representation is
+32 bytes long. By all accounts, this is way too big for an internal
+identifier. This is how the original Perl version worked, and likely
+how I plan to keep it until I can work out the additional complexity
+of a shorter identifier (like an unsigned integer) which is guaranteed
+to be unique (namely, recycling discarded identifiers). For the time
+being however I am not worried, as storage is large and computers are
+fast.
+
+### Issues with hash collisions
+
+MD5 and SHA-1 are both considered unsafe for the purposes of
+cryptographic integrity, since collision attacks have been
+demonstrated against both algorithms. (This is why neither is
+recommended as a primary key.)
+
+Currently, this system does not support duplicate mappings from one
+digest algorithm to another. If two different blobs with the same hash
+are entered into the store, the second one will probably be ignored. I
+will probably change this eventually so smaller hashes can accommodate
+duplicate entries, as dumping crafted data into this system would
+likely be a convenient way to identify collision targets.
+
+### Future directions
+
+The first order of business is to create a search function for the
+various metadata elements. Then, probably a Rack app to put this whole
+business online (much like the analogous one I wrote in Perl).
+Following that, probably start poking around at that maintenance
+daemon I mentioned in the other section, and whatever else I think is
+a good idea and not too much effort.
+
+Afterward, I will probably take the show on the road: write a version
+in Python and/or JavaScript, for example. Maybe look at other
+back-ends. We'll see.
+
+## Installation
+
+When it is ready, you know how to do this:
+
+    $ gem install store-digest
+
+You will know when it's far enough along when you can [download it off
+rubygems.org](https://rubygems.org/gems/store-digest).
+
+## Contributing
+
+Bug reports and pull requests are welcome at
+[the GitHub repository](https://github.com/doriantaylor/rb-store-digest/issues).
+
+## Copyright & License
+
+©2019 [Dorian Taylor](https://doriantaylor.com/)
+
+This software is provided under
+the [Apache License, 2.0](https://www.apache.org/licenses/LICENSE-2.0).