webvac 0.1.5 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dd9d4c498fdf6c8497e3f3ac3799621488304111
-  data.tar.gz: 3f41f908d153a5aa880a098389f0bad4e6e95640
+  metadata.gz: c0be13ddc03fb863eaf5ada9ed4f554d3afc3d64
+  data.tar.gz: a4dc5592881d294c45bd2fbfe77ff4a68881b19a
 SHA512:
-  metadata.gz: 1359633e7236896e71eecd91bd0fdc04d0941b3868292979d42a6dfd5dbf58d29aa7ca827bb0c9ef56b8d84f7b6087a4e53f4f70826f6fdff445925c65d89350
-  data.tar.gz: bf08f77ec60e7b61eea7855de97139b3e180d2447e4bb863ef768ad93d12b00f1f71409bd1d3a6760c617768b2f4115ba3931b8ce5bd8e6b865e381be132a109
+  metadata.gz: 297b5ac456d6a199bc216a9079610f2e0b8c8a82745de232973443542de47e4d7ce3df7ef18fac2b0e8bb7667a7ec19547eb98990cea5ef58d777358317515f8
+  data.tar.gz: 568910a94852ee97daa01977aef1623ad989ddc07ef0f45236ae4929e9bd0b81bc70d33c1ba2c8f5d0ddecafd45f64777846f4ab74a94b1905c6a87860107a45

data/Changelog

@@ -0,0 +1,3 @@
+Sat Jun 20 22:59:30 PDT 2020
+* webvac now streams.  This cuts RAM usage and latency.  I tested it in production, because I am a gangsta (and because rollback is easy).
+* First Changelog entry!

data/README

@@ -1,29 +1,29 @@
 This is a somewhat specialized chunk of code!
 
-THIS IS ALSO BETA SOFTWARE.  CAVEAT EMPTOR.
-
 Using a lookup table in Redis and venti as the backing storage, webvac
-serves static content.  It comes with utilities "webvac-sweep" and
-"webvac-unsweep" that push things into venti and remove them from the
-filesystem and place the things back into the filesystem, respectively.
-The 'webvac' program itself starts up a webserver.  Give these programs
-'-h' or '--help' to see helpful(?) information, and see below for
-configuration.
-
-Currently I am only using it to serve uploads for UGC in Pleroma.  I
-have been using venti to do incremental backups of the data, and since
+serves static content.  It comes with utility `webvac-sweep`, that
+pushes content into venti, optionally removing it from the filesystem.
+The `webvac-server` program itself starts up a webserver.  Give these
+programs '-h' or '--help' to see helpful(?) information, and see below
+for configuration.
+
+Currently I am only using it to serve uploads for UGC in some small- to
+moderately high-traffic Pleroma instances.  I have been using venti to
+do incremental backups of the data (replication is speedy), and since
 the uploads are WORM ("write once, read many") data, I thought it'd be
-cool to serve the data directly out of venti.
+cool to serve the data directly out of venti.  It worked better than
+expected!
 
 I expect to use this more often and thus expect it to become a bit more
-general as a result.
+general as a result, but for right now, it makes a couple of assumptions
+about where it serves files.
 
 = Quick Start
 
   [Check that your venti and Redis servers are operational.]
   $ sudo ed /etc/webvac.json
   a
-  '{"server_path_prepend":"/where/uploads/get/put/in/the/filesystem"}'
+  {"server_path_prepend":"/where/uploads/get/put/in/the/filesystem"}
   .
   wq
   $ sudo $EDITOR /etc/nginx/whatever
@@ -47,12 +47,12 @@ to configure probably one thing.
 
 = Overhead
 
-For about 100GB of files, venti takes 86GB of disk (no surprise, since
-it's mostly JPGs and MP4s, so it's already compressed; the savings are
-probably from dedup), and Redis takes about 60MB of RAM for this.  All
-of the files were put into venti as part of the backup solution, but the
-originals weren't removed if they were bigger than 4MB (see doc/TODO).
-The workers take some RAM to run.  CPU overhead is negligible.
+Just empirically, for about 100GB of files, venti takes 86GB of disk (no
+surprise, since it's mostly JPGs and MP4s, so it's already compressed;
+the savings are probably from dedup), and Redis takes about 60MB of RAM
+for this.  All of the files were put into venti as part of the backup
+solution.  CPU overhead is negligible, the server takes about 200MB of
+RAM for both workers.
 
 = Installation
 
@@ -110,7 +110,12 @@ server_path_prepend.
 
 = Usage
 
-Afer configuring, you can run the server with `webvac-server`.  This will actually serve the content from venti, as long as it is present in the path→score index in Redis (so you can remove content as needed by just removing items from the index).  In order to add items, you run `webvac-sweep`.  
+Afer configuring, you can run the server with `webvac-server`.  This
+will actually serve the content from venti, as long as it is present in
+the path→score index in Redis (so you can remove content as needed by
+just removing items from the index).  In order to add items, you run
+`webvac-sweep`.  You can also delete the file (which will only happen if
+the sweep is successful) with `-d`.
 
 = TODO
 
@@ -124,4 +129,4 @@ See doc/TODO
 
 = I feel dirty
 
-You can throw BTC at 1BZz3ndJUoWhEvm1BfW3FzceAjFqKTwqWV .  Proceeds will go to funding the instance hosting thing.
+You can throw Bitcoin (BTC) at this address:  1BZz3ndJUoWhEvm1BfW3FzceAjFqKTwqWV .  Proceeds will go to funding the instance hosting thing.

data/Rakefile

@@ -31,3 +31,14 @@ desc "Runs IRB, automatically require()ing #{spec.name}."
 task(:irb) {
 	exec "irb -Ilib -r#{spec.name}"
 }
+
+desc "Runs IRB, automatically require()ing #{spec.name}, with "\
+	"acme-suitable options"
+task(:airb) {
+	exec "irb -Ilib -r#{spec.name} --prompt default --noreadline"
+}
+
+desc "Runs nginx test server."
+task(:nginx) {
+	exec "nginx", "-c", "#{__dir__}/doc/nginx.example.conf"
+}

data/config.ru

@@ -32,6 +32,9 @@ module WebVac
 				'Content-Disposition' => "filename=\"#{name}\"",
 				'Fortune' => Fortunes.sample,
 			}.merge!(t.metadata(score) || {}).tap { |h|
+				# Now that we stream, this condition never happens.
+				# Maybe it's a good idea to wrap the IO object and fill in
+				# the extra data as needed.
 				if contents
 					h['Content-Type'] ||= t.guess_mime(contents) rescue nil
 					h['Content-Length'] ||= contents.bytesize.to_s
@@ -66,14 +69,12 @@ module WebVac
 			s = tab.path2score p
 			return [404, {}, ["404 Not found\nNo such path: #{p}\n"]] unless s
 			ct = Time.parse(env['HTTP_IF_MODIFIED_SINCE']) rescue nil
+			hs = headers(tab, p, s, nil)
 			if ct && ct.to_i > 0
-				hs = headers(tab, p, s, nil)
 				mt = Time.parse(hs['Last-Modified']) rescue nil
 				return [304, hs, []] if mt && mt > ct
 			end
-			contents = vac.load! s
-			hs = headers(tab, p, s, contents)
-			[200, headers(tab, p, s, contents), [contents]]
+			[200, hs, vac.load_io(s)]
 		}
 	end
 

data/doc/TODO

@@ -2,9 +2,6 @@ Unordered:
 
 · The closure abuse in Serv precludes using the URL to generate the routes.
   This needs a fix in order to generalize beyond Pleroma.
-· Should be easy to stream rather than loading everything into memory,
-  but until then, big-ish files (≈4MB) take a second to get out of venti.
-  Obviously, it'll be faster and more reliable to implement the venti protocol.
 · Stats and webvac-unsweep.  This will allow hot objects to be swapped out of
   venti.
 · Implement the venti protocol instead of calling $plan9bin/vac.

data/lib/webvac.rb

@@ -89,14 +89,18 @@ module WebVac
 			io.read.chomp.sub(/^vac:/, '')
 		end
 
-		def load! vac
+		def load_io vac
 			unless /^vac:[a-f0-9]{40}$/.match(vac)
 				raise ArgumentError, "#{vac.inspect} not a vac score?"
 			end
 			IO.popen(
 				{'venti' => config.venti_server},
 				["#{config.plan9bin}/unvac", '-c', vac]
-			).tap { |io| Thread.new { Process.wait(io.pid) } }.read
+			).tap { |io| Thread.new { Process.wait(io.pid) } }			
+		end
+
+		def load! vac
+			load_io(vac).read
 		end
 	end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: webvac
 version: !ruby/object:Gem::Version
-  version: 0.1.5
+  version: 0.1.6
 platform: ruby
 authors:
 - Pete
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-06-09 00:00:00.000000000 Z
+date: 2020-06-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redic
@@ -91,7 +91,9 @@ extra_rdoc_files:
 - doc/TODO
 - doc/nginx.example.conf
 - README
+- Changelog
 files:
+- Changelog
 - README
 - Rakefile
 - bin/webvac-server