store-digest 0.3.1 → 0.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 53000b2277712556bc9fdc4e3a8aa04bc1f2d6c4a432cd672ca1363901f1772d
-  data.tar.gz: ae7ff0deb6b21b7eb24031a44a74aed287866e792891331164fcf79df6894ab5
+  metadata.gz: '09eefcbc9e86e0b771a61edd3a16f135f559c228f9ea0ebd453bd198f3d70ae3'
+  data.tar.gz: 9bd41bdcd3709b07081daeaf49e2f7f8e7f5513f3c2469b3636d5ffcfc9f9948
 SHA512:
-  metadata.gz: fc4f0a08994b1382bb75cd34015e8984f65121439597bae891cfa5bdd55f7ab046b70f86995a4e3840eac699304540d0d92d408450366a11d324543f97ab8cb6
-  data.tar.gz: 19097b9434eb605b7ebfbc5c1af3d59caefae3d06d241437b7e66c24f9a95b9172fa5a40f754804dfd9192f7833dd4fdc0a51e38273345e0fe43fbac9f28ee4e
+  metadata.gz: d4f6199427973e30fda2380738bb8d7697c8e12f6c1f641e78f901c9827ab8a87742a62cd07fa422d89617e504181a41c9ba227427d64955b4416fdd0ebec6d6
+  data.tar.gz: b67f8805a959d787914b5adbc0854d969e4d0e5d5a2a5bb541c9aa259728f789f0a01e0e50bf7a4014bfbc9466d526f33ff24e4f809fbd0a8f644ae5cd4004fe

data/TODO.org

@@ -1,11 +1,122 @@
 #+STARTUP: showall indent hidestars
+* TODO 2026-06-07 remaining tasks for scanning overhaul
+- [ ] ~Store::Digest~
+  - [ ] create ~#has?~ predicate
+    - [ ] test it doyy
+  - [ ] overhaul ~#add~
+    - [ ] make it pass through to ~Entry~ constructor
+    - [ ] create ~#add_raw~
+      - [ ] canonicalize media type
+      - [ ] update TTL on cache entries on match
+      - [ ] return a hash
+  - [ ] overhaul ~#remove~
+    - [ ] add ~tombstone:~ parameter to return already-deleted entities
+    - [ ] create ~#remove_raw~
+      - [ ] do we /really/ need to create a ~#remove_raw~?
+      - [ ] can we add a ~remove:~ flag to ~#get_raw~?
+  - [ ] overhaul ~#get~
+    - [ ] add ~tombstone:~ parameter to return already-deleted entities
+    - [ ] make it pass through to ~Entry~ constructor
+    - [ ] create ~#get_raw~
+      - [ ] silently remove expired cache items
+        - could probably chuck some kinda ~#purge~ operation in a thread; run it every few seconds
+      - [ ] return the same hash as ~#add_raw~
+- [ ] ~Store::Digest::Entry~
+  - [ ] finish modifications to ~#initialize~
+    - [ ] prune out parameters that you shouldn't be able to mess with
+    - [ ] add ~#store.get_raw~ scenario to constructor
+      - [ ] if ~content~ is ~nil~ and there is a ~store:~ supplied
+      - [ ] (~content~ /must not/ be ~nil~ /except/ if there is a ~store:~)
+      - [ ] maybe implement as a block?
+  - [ ] implement ~#add~
+    - [ ] may be a noop if already associated with the store
+    - [ ] should produce an error if entry is a tombstone/has no content
+    - [ ] otherwise the entry will have to be rescanned
+  - [ ] implement ~#remove~
+  - [ ] implement ~#scan!~
+    - [ ] use ~store.add_raw~ if associated with a store
+      - (this returns a file handle or precursor thereto)
+    - [ ] otherwise use ~self.class.scan_raw~
+      - (this needs its own filehandle, or we can reuse the input if rewindable)
+* TODO 2026-05-26 beef up scanning
+- *GOAL*: We want to be able to drop one of these objects into a ~Rack~ message body — either request or response — and it should JFW.
+  - We should be able to take what's already in there and initialize the object with that.
+  - The result must be rewindable.
+- [-] ~Store::Digest::Object~ changes
+  - [X] rename to ~Store::Digest::Entry~
+    - [X] add a ~Store::Digest::Object~ constant for the time being
+  - [X] absorb ~Store::Digest::Object::IOWrapper~
+    - [X] expose a minimal ~#each~ ~#read~ ~#gets~ ~#rewind~ ~#close~ interface
+    - [X] ~#content~ becomes a no-op
+  - [-] overhaul ~#initialize~
+    - [X] [[https://github.com/rack/rack/blob/main/SPEC.rdoc#the-body][ingest all kinds of objects]] (via ~Store::Digest::ReadWrapper~)
+      - [X] strings
+      - [X] arrays of strings / enumerators that yield strings / ~#each~
+      - [X] [[https://github.com/rack/rack/blob/main/SPEC.rdoc#the-input-stream][things that quack like]] ~#gets~ ~#read~ ~#close~
+      - [X] things that respond to ~#call~
+        - [X] that takes no arguments and returns one of the above
+        - [X] that takes a write handle as its first (only?) argument
+    - [ ] provide a ~store:~ named parameter for associating with a ~Store::Digest~ instance
+    - [ ] make it so ~digests:~ can take an ~Array~ of ~Symbol~ objects that represent the desired algorithms when the content is eventually scanned
+    - [ ] add a ~cache:~ named parameter which can be falsy, truthy, an integer representing a TTL, or a ~Time~ object indicating the desired expiry date
+    - [ ] add scan ~scan:~ named parameter (effectively the same as invoking ~.new(…).scan~)
+  - [ ] do not scan input until an attempt to access its contents
+    - [ ] first call to ~#each~, ~#gets~, ~#read~, or ~#content~
+    - [ ] first call to the ~#digests~ hash or any of the dependent metadata, such as ~#size~
+    - [ ] may not scan at all if passed in an object which is already scanned
+  - [ ] overhaul ~#scan~ instance method
+    - [ ] make it idempotent
+    - [ ] create a ~#scan!~ method that forces the issue
+    - [ ] deprecate the arguments/parameters and have it just operate over itself
+    - [ ] scanning the input adds it to the store if a backreference is present and replaces the input with the settled blob in the store
+    - [ ] swap in a different read handle if there is no backreference to the store
+      - [ ] the passed-in handle itself if it responds to ~#binmode~ and ~#rewind~
+      - [ ] ~StringIO~ for small blobs (1k? 10k? 64k?)
+        - how would I do that? start writing to a ~StringIO~ and if it ticks over the limit, dump that into a temp file and switch?
+          - (probably)
+          - (the benefit to doing that, aside from being faster, would be to mitigate the number of concurrent open file handles)
+          - (at the cost of bigger stuff being one blit slower)
+          - (honestly it'd probably get buffered in ram anyway)
+      - [ ] otherwise an ordinary temporary file
+  - [ ] Overhaul ~.scan~ class method to align with new constructor
+    - [ ] ~store:~ named parameter
+    - [ ] ~cache:~ named parameter
+    - [ ] initialize with ~scan: true~, obviously
+  - [X] create a ~.scan_raw~ class method so ~Store::Digest~ can use it independently
+    - [X] make it return a minimal set: digest hashes, byte count, and optional (scanned) content type
+  - [ ] if hashes are already supplied in the constructor then it is assumed to be scanned already
+    - [ ] a call to ~#scan~ is a no-op (because ~#scan~ should be idempotent)
+    - [ ] a call to ~#scan!~ is assumed to mean you want to verify the content and should blow up if *any* of the supplied hashes don't match
+  - [ ] implement ~IO~ and/or ~Enumerable~ emulation methods
+    - [ ] Any call to any of these should start a scan that ultimately replaces the underlying content with a rewindable file handle
+      - [ ] (unless the original object implements ~#rewind~ and there is no association with a store)
+        - (although note the caveat that the original object can be written to in situ, like it's more efficient to reuse it but less secure)
+          - [ ] maybe make an option in the constructor to not do this?
+    - [ ] ~#each~
+    - [ ] ~#gets~
+    - [ ] ~#read~
+    - [ ] ~#rewind~
+    - [ ] ~#open~ (no-op)
+    - [ ] ~#close~ (no-op)
+  - [ ] make ~#content~ a no-op
+  - [ ] going to have to overhaul ~#to_h~
+    - [ ] make a version that doesn't scan?
+  - [ ] add ~#stored?~
+- [-] ~Store::Digest~ changes
+  - [ ] add something like a ~#add_raw~ method that will take the entry's readable object and return a ~Hash~ of metadata plus a lambda that returns a rewindable file handle
+  - [ ] overhaul ~#add~
+  - [ ] overhaul ~#get~
+  - [X] nuke ~#lazy_add~; it is no longer necessary
+- [X] ~MimeMagic~ monkeypatch
+  - [X] lobby maintainers to merge my patches (again; not a peep in years)
+  - [X] failing that, ship a fork
 * TODO upgrade metadata format
-- [ ] get rid of the idea of a 'primary' digest algorithm
-  - [ ] change the main record to being keyed by integers and contain all hashes
+- [X] get rid of the idea of a 'primary' digest algorithm
+  - [X] change the main record to being keyed by integers and contain all hashes
     - old: ≥209 bytes + 316 mapping
     - new: ≥218 + 220 mapping
-  - [ ] make digest -> integer mapping
-- [ ] add version to metadata
+  - [X] make digest -> integer mapping
+- [X] add version to metadata
 - [ ] add function to upgrade metadata store
 * TODO add cache capability
 - [ ] add fields to control:

data/lib/store/digest/blob/filesystem.rb

@@ -57,20 +57,24 @@ module Store::Digest::Blob::FileSystem
   # Return an open tempfile in the designated temp directory
   # @return [Tempfile]
   def temp_blob
-    Tempfile.new 'blob', tmp
+    Tempfile.new 'blob', tmp, encoding: Encoding::BINARY
   end
 
   # Settle a blob from its temporary location to its permanent location.
+  #
   # @param bin [String] The binary representation of the keying digest
   # @param fh  [File] An open filehandle, presumably a temp file
   # @param mtime [nil, Time, DateTime, Integer] the modification time
   #  (defaults to now)
   # @param overwrite [false, true] whether to overwrite the target
-  # @return [true] a throwaway return value
+  #
   # @raise [SystemCallError] as we are mucking with the file system
-  def settle_blob bin, fh, mtime: nil, overwrite: false
+  #
+  # @return [Proc] a lambda that returns a read handle
+  #
+  def settle_blob bin, fh, mtime: nil, overwrite: false, direct: false
     # get the mtimes
-    mtime ||= Time.now
+    mtime ||= Time.now(in: ?Z)
     mtime = case mtime
             when Time    then mtime.to_i
             when Integer then mtime
@@ -100,7 +104,8 @@ module Store::Digest::Blob::FileSystem
       target.utime mtime, mtime
     end
 
-    true
+    # return a proc that returns the open file handle
+    direct ? target.open('rb') : -> { target.open 'rb' }
   end
 
   # Return a blob filehandle (or closure that will return said blob).

data/lib/store/digest/driver.rb

@@ -30,4 +30,8 @@ module Store::Digest::Driver
   def setup **options
     raise NotImplementedError, 'gotta roll your own, holmes'
   end
+
+  def close_internal
+    raise NotImplementedError, 'close_internal not implemented'
+  end
 end