xferase 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: eb45d4b708f6d607c40971f8081a4778d45887fdd3f5b6a16e4d3b7ada1b809b
+  data.tar.gz: ff1906f90f5b01b0b47c0f197444d546ac00fef6203d6225bf116fb2ff9e6187
+SHA512:
+  metadata.gz: e7c192878a8a3f13634d2b203edff06e2818cccb0dd79e1df9518ddf257e0397769a27c3f76ec6190da759cdde9b65da73c04c1c21c774056835f75eef3e4b88
+  data.tar.gz: 7c279579641f9e91866971167eb0aa75a1231dde5f983c3b474bd91d0c402fb164cd5c8f7f8053403b91764444cbdbfa45f3f617ccd00e0f3f8314aa17749767

data/README.md

@@ -0,0 +1,275 @@
+🔀ferase
+========
+
+All your photos under one roof.
+-------------------------------
+
+Xferase is an always-on background service
+that automatically imports pictures & videos
+into your personal photo library as they come in.
+
+When combined with other software,
+it can be used as a kind of self-hosted / DIY alternative
+to cloud photo services like Google Photos or iCloud.
+
+Why?
+----
+
+My photo library has always been a hot mess.
+I have photos from my cell phone, photos from my digital camera,
+and photos I’ve gotten from other people.
+Some live in the cloud, some live on my cell phone, some live on my laptop,
+some live on a backup drive in my shoe closet,
+and others will sit on my digital camera’s SD card for a couple weeks
+before I remember that I wanted to share them with someone.
+
+I got tired of taking lots of photos
+and then having to look in three places to find the ones I wanted.
+I just wanted my old photos to live in the same place as my new photos,
+plus a few other modest requirements:
+
+1. **automatic photo import** from any source[\*](#caveat)
+
+   > “Automatic” means no mouse/keyboard interaction:
+   >
+   > * 📱 **cell phone** pictures get imported as soon as they’re taken;
+   > * 💬 **chat app** downloads, as soon as they’re saved;
+   > * 📷 **digital camera** pictures, as soon as it’s plugged into USB.
+
+2. master copy of library **stored locally, on disk**
+
+   > My digital photo/video library belongs to _me._
+   > But if I don’t own the pipeline for storing and managing it,
+   > then does it really? 🤔
+   >
+   > Cloud services can raise their prices, go offline,
+   > or fall victim to ransomware attacks.
+   > My own Internet can fail, too.
+   > I’m happy to use the cloud for Netflix & Spotify,
+   > but for something as personal and unreplaceable as my photos,
+   > I want the master copy in my own hands.
+
+3. automatic syncing of library back to phone[\*](#caveat)
+
+   > Obviously, I don’t just want to _collect_ my personal photos;
+   > I want to _use_ them, and to have them with me wherever I go.
+   > But your complete photo collection is probably
+   > too large to fit on your cell phone,
+   > which is why Google and Apple host it in the cloud.
+   >
+   > Xferase gets around this
+   > by maintaining two parallel copies of your library:
+   > one master, and one optimized for web.
+   > Keep both on your computer and sync the latter to your phone;
+   > Xferase will make sure that when a photo is deleted from one,
+   > it’s automatically removed from the other, too.
+
+4. clean, consistent, **user-visible directory & filename scheme**
+
+   > Call me obsessive-compulsive, but which would you rather have—
+   >
+   > ```sh
+   > # this?                                 # ...or this?
+   >
+   > ~/Pictures                              ~/Pictures
+   > ├── 1619593208911.jpeg                  ├── 2020
+   > ├── DCIM                                │   ├── 2020-08-01_113129.heic
+   > │   └── 2021_03_26                      │   └── 2020-05-20_160209.png
+   > │       ├── R0014285.MOV                └── 2021
+   > │       ├── R0014286.DNG                    ├── 2021-02-12_081933a.jpg
+   > │       ├── R0014286.JPG                    ├── 2021-02-12_081933b.jpg
+   > │       ├── R0014287.DNG                    ├── 2021-02-12_081939.mp4
+   > │       └── R0014287.JPG                    ├── 2021-03-26_161245.mp4
+   > ├── IMG_20210212_081933_001.jpg             ├── 2021-03-26_161518.dng
+   > ├── IMG_20210212_081933_002.jpg             ├── 2021-03-26_161518.jpg
+   > ├── IMG_8953.HEIC                           ├── 2021-03-26_170304.dng
+   > ├── Screenshot_20200520_160209.png          ├── 2021-03-26_170304.jpg
+   > └── VID_20210212_081939.mp4                 └── 2021-04-28_000008.jpg
+   > ```
+   >
+   > I also want to know where my files are
+   > so I can find them in a “Browse...” dialog
+   > or mirror them to other devices with Dropbox, Syncthing, or even rsync.
+
+5. available on Linux
+
+   > Xferase has not been tested on macOS,
+   > but it should work when run in a Docker container.
+
+#### \*Caveat
+
+For points 1 and 3, Xferase needs the help of additional software
+([Syncthing][] & systemd).
+If you have no experience (or interest in) tinkering with Linux,
+Xferase is **not** for you.
+
+See the next section for more details.
+
+[Syncthing]: https://syncthing.net
+
+What _Exactly_ Does It Do?
+--------------------------
+
+Xferase watches a directory of your choosing (its “inbox”),
+and whenever any files are placed there,
+it automatically optimizes and imports them into your photo library,
+like so:
+
+```sh
+# Before                                # After
+
+~/Pictures                              ~/Pictures
+├── .inbox                              ├── .inbox
+│   ├── 1619593208911.jpeg              └── library
+│   ├── DCIM                                ├── 2020
+│   │   └── 2021_03_26                      │   ├── 2020-08-01_113129.heic
+│   │       ├── R0014285.MOV                │   └── 2020-05-20_160209.png
+│   │       ├── R0014286.DNG                └── 2021
+│   │       ├── R0014286.JPG                    ├── 2021-02-12_081933a.jpg
+│   │       ├── R0014287.DNG                    ├── 2021-02-12_081933b.jpg
+│   │       └── R0014287.JPG                    ├── 2021-02-12_081939.mp4
+│   ├── IMG_20210212_081933_001.jpg             ├── 2021-03-26_161245.mp4
+│   ├── IMG_20210212_081933_002.jpg             ├── 2021-03-26_161518.dng
+│   ├── IMG_8953.HEIC                           ├── 2021-03-26_161518.jpg
+│   ├── Screenshot_20200520_160209.png          ├── 2021-03-26_170304.dng
+│   └── VID_20210212_081939.mp4                 ├── 2021-03-26_170304.jpg
+└── library                                     └── 2021-04-28_000008.jpg
+```
+
+(You may have noticed that this is identical to the snippet from above,
+except for the `.inbox` and `library` parent directories.)
+
+How you get those files from your phone or camera into the inbox is up to you.
+
+> ### 🤷 Up to me?? I thought Xferase was supposed to “import from many sources”.
+>
+> Yes, but only with the help of other software.
+> Automatically getting files from other devices onto your computer
+> is not a trivial problem,
+> and there are existing utilities that already do it better than Xferase could.
+>
+> Consider that different users may have a different technical requirements:
+>
+> * Should digital camera photos be transferred over USB,
+>   or using a Wi-Fi SD card?
+> * Should cell phone photos be transferred over cellular data,
+>   or only over Wi-Fi?
+> * How many other devices should the library be mirrored/synced to?
+>
+> Limiting the scope of what Xferase does
+> gives you the flexibility to use the best available tools
+> for your needs, every step of the way.
+>
+> Here is a diagram outlining the recommended approach:
+>
+> <img src="/i/system-diagram.gif" width="480">
+>
+> For more on each step, see the [guides](#guides) below.
+
+Quick Start
+-----------
+
+> 💡 **The examples below assume you have [Docker][] installed.**
+>
+> Why use Docker? Xferase can be [installed natively as a Ruby gem][],
+> but there are lots of external dependencies.
+>
+> [Docker]: https://docs.docker.com/get-docker/
+> [installed natively as a Ruby gem]: guides/ingest.md#option-2-rubygems--systemd
+
+### Basic Usage
+
+```sh
+$ docker run -d \
+    --name xferase \
+    --user $(id -u) \
+    --env TZ=$(timedatectl show --property=Timezone --value) \
+    --volume $HOME/Pictures:/data \
+    --env INBOX=/data/.inbox \
+    --env STAGING=/data/.staging \
+    --env LIBRARY=/data/master \
+    rlue/xferase
+```
+
+Any photos or videos placed in the **inbox**
+will be automatically moved to the **staging folder** for processing.
+From there, files are moved to the **library**,
+with videos being compressed to save space on disk.
+
+> 🤔 **What’s with the staging folder?**
+>
+> Using a temp folder makes it easier to resume from a crash,
+> especially when the `LIBRARY_WEB` env var is set.
+
+#### Option: `LIBRARY_WEB`
+
+```sh
+$ docker run -d \
+    --name xferase \
+    --user $(id -u) \
+    --env TZ=$(timedatectl show --property=Timezone --value) \
+    --volume $HOME/Pictures:/data \
+    --env INBOX=/data/.inbox \
+    --env STAGING=/data/.staging \
+    --env LIBRARY=/data/master \
+    --env LIBRARY_WEB=/data/web \
+    rlue/xferase
+```
+
+Xferase will create a separate, lo-res copy of each imported file
+and save it to the **web-optimized library**.
+
+**Xferase keeps both libraries in sync**,
+meaning that when a photo is deleted from one,
+it will be automatically deleted from the other.
+
+> ⚠️ **Warning**
+>
+> This also applies to copies of the same image in different formats:
+> if you shoot RAW+JPEG, deleting a .jpg will cause Xferase to delete the
+> corresponding raw image file (and vice versa).
+
+#### Option: `GRACE_PERIOD`
+
+```sh
+$ docker run -d \
+    --name xferase \
+    --user $(id -u) \
+    --env TZ=$(timedatectl show --property=Timezone --value) \
+    --volume $HOME/Pictures:/data \
+    --env INBOX=/data/.inbox \
+    --env STAGING=/data/.staging \
+    --env LIBRARY=/data/master \
+    --env LIBRARY_WEB=/data/web \
+    --env GRACE_PERIOD=60 \
+    rlue/xferase
+```
+
+Xferase will wait 60 seconds before initiating the import process.
+
+Why would you want this?
+Because not every picture you take belongs in your collection—maybe
+you just wanted to show a friend a weird bug you found on the sidewalk.
+With the grace period set, you can film it, send it off, and delete it
+before Xferase wastes CPU time transcoding it (twice).
+
+Guides
+------
+
+### 1. Upload (with systemd/Syncthing)
+
+* [📷➡️🖥️ Get photos from your camera into Xferase’s inbox](guides/upload-camera.md)
+* [📱➡️🖥️ Get photos from your phone into Xferase’s inbox](guides/upload-phone.md)
+
+### 2. Ingest (with Xferase)
+
+* [🖼️➡️📂 Rename, optimize, and move new photos into your library](guides/ingest.md)
+
+### 3. Propagate (with Syncthing)
+
+* [🖥️🔄📱 Sync your library (back) to other devices](guides/propagate.md)
+
+License
+-------
+
+© 2021 Ryan Lue. This project is licensed under the terms of the MIT License.

data/bin/xferase

@@ -0,0 +1,137 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Config -----------------------------------------------------------------------
+require 'xferase'
+require 'photein'
+
+Xferase::Config.parse_opts!
+Xferase.logger.open
+Photein.logger = Xferase.logger
+
+ENV['MEDIAINFO_XML_PARSER'] ||= 'nokogiri'
+
+SRC_DEST_MAP = {
+  "#{Xferase::Config.staging}/desktop" => Xferase::Config.library,
+  "#{Xferase::Config.staging}/web" => Xferase::Config.library_web,
+}.compact
+
+FORMAT_GROUPS = [%w(.jpg .dng .heic), %w(.mov .mp4), %w(.png)]
+
+# Setup ------------------------------------------------------------------------
+require 'debouncer'
+require 'fileutils'
+require 'pathname'
+require 'rb-inotify'
+require 'tmpdir'
+
+%i[inbox staging library library_web]
+  .map { |dir| Xferase::Config.send(dir) }
+  .compact
+  .reject(&File.method(:directory?))
+  .each do |dir|
+    Xferase.logger.info("#{dir}: not found; creating now...")
+    FileUtils.mkdir_p(dir)
+  end
+
+Thread.abort_on_exception = true
+
+mutex = Mutex.new
+debouncer = Debouncer.new(Xferase::Config.grace_period.to_i) do |*files|
+  files = files.map(&Pathname.method(:new))
+               .select(&:exist?)
+               .reject do |file|
+                 file.dirname.join(".syncthing.#{file.basename}.tmp").exist?
+               end
+
+  SRC_DEST_MAP.keys.each { |dir| FileUtils.ln(files, dir) }
+  FileUtils.rm(files + Dir["#{Xferase::Config.staging}/web/*.DNG"]) # FIXME (ugly hack)
+
+  break if debouncer.inspect_params[:threads] > 2 # don't let threads pile up
+
+  mutex.synchronize do
+    SRC_DEST_MAP.each do |src, dest|
+      Photein::Config.set(
+        source: src,
+        dest: dest,
+        'optimize-for': Pathname(src).basename.to_sym
+      )
+
+      Photein.run
+    end
+  rescue => e
+    warn e.message
+  end
+end.reducer(:+)
+
+# Resume from interruption/failure ---------------------------------------------
+debouncer.call unless SRC_DEST_MAP.keys.all?(&Dir.method(:empty?))
+
+Dir["#{Xferase::Config.inbox}/**/*"]
+  .select(&File.method(:file?))
+  .each { |path| debouncer.call(path) }
+
+# Start ------------------------------------------------------------------------
+Thread.new do
+  call_debouncer = ->(event) do
+    next if event.name.start_with?('.')
+
+    Xferase.logger.debug("#{event.name}: new file detected in watch directory")
+    debouncer.call(event.absolute_name)
+  rescue => e
+    warn e.message
+  end
+
+  import_notifier = INotify::Notifier.new
+  import_notifier.watch(Xferase::Config.inbox, :close_write, &call_debouncer)
+
+  # NOTE: inotify is not recursive,
+  # so subdirectories must be watched separately!
+  # (Why do Syncthing folders get special treatment?
+  # Because ST works by creating hidden tempfiles and moving them upon completion)
+  stfolders, simple_subdirs = Dir["#{Xferase::Config.inbox}/**/*"]
+    .select(&File.method(:directory?))
+    .partition { |dir| Dir.children(dir).include?('.stfolder') }
+
+  simple_subdirs.each { |dir| import_notifier.watch(dir, :close_write, &call_debouncer) }
+  stfolders.each { |dir| import_notifier.watch(dir, :moved_to, &call_debouncer) }
+
+  import_notifier.run
+end
+
+Thread.new do
+  sync_deletions = ->(event) do
+    Xferase.logger.info("#{event.name} has disappeared!")
+
+    deleted_file   = Pathname(event.absolute_name).expand_path
+
+    sister_file    = if deleted_file.to_s.start_with?(File.expand_path(Xferase::Config.library))
+                      deleted_file.sub(Xferase::Config.library, Xferase::Config.library_web)
+                    else
+                      deleted_file.sub(Xferase::Config.library_web, Xferase::Config.library)
+                    end
+
+    sister_formats = FORMAT_GROUPS.find { |group| group.include?(deleted_file.extname) }
+
+    related_files  = [deleted_file, sister_file]
+      .product(sister_formats)
+      .map { |file, ext| file.sub_ext(ext) }
+      .select(&:file?)
+
+    # (Why mv to tmpdir first? Why not rm straight away?
+    # Because rm would recursively trigger this inotify callback.)
+    related_files.each { |f| Xferase.logger.info("deleting #{f.realpath}") }
+    FileUtils.mv(related_files, Dir.tmpdir)
+    FileUtils.rm(related_files.map { |f| File.join(Dir.tmpdir, f.basename) })
+  end
+
+  deletion_notifier = INotify::Notifier.new
+
+  # NOTE: inotify is not recursive,
+  # so subdirectories must be watched separately!
+  (Dir["#{Xferase::Config.library}/**/*"] + Dir["#{Xferase::Config.library_web}/**/*"])
+    .select(&File.method(:directory?))
+    .each { |dir| deletion_notifier.watch(dir, :delete, &sync_deletions) }
+
+  deletion_notifier.run
+end.join

data/lib/xferase/config.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require 'singleton'
+require 'optparse'
+
+module Xferase
+  class Config
+    include Singleton
+
+    OPTIONS = [
+      ['-v',              '--verbose',                 'print verbose output'],
+      ['-i INBOX',        '--inbox=INBOX',             'path to the inbox (required)'],
+      ['-s STAGING',      '--staging=STAGING',         'path to the staging directory (required)'],
+      ['-l LIBRARY',      '--library=LIBRARY',         'path to the master library (required)'],
+      ['-w LIBRARY_WEB',  '--library-web=LIBRARY_WEB', 'path to the web-optimized library'],
+      ['-g INTERVAL',     '--grace-period=INTERVAL',   'wait n seconds for additional files before import'],
+    ].freeze
+
+    OPTION_NAMES = OPTIONS
+      .flatten
+      .grep(/^--/)
+      .map { |option| option[/\w[a-z\-]+/] }
+      .map(&:to_sym)
+
+    class << self
+      def parse_opts!
+        @params = {}
+
+        parser = OptionParser.new do |opts|
+          opts.version = Xferase::VERSION
+          opts.banner  = <<~BANNER
+            Usage: xferase [--version] [-h | --help] [<args>]
+          BANNER
+
+          OPTIONS.each { |opt| opts.on(*opt) }
+        end.tap { |p| p.parse!(into: @params) }
+
+        @params.freeze
+
+        raise "no inbox directory given" if !@params.key?(:inbox)
+        raise "no staging directory given" if !@params.key?(:staging)
+        raise "no master library given" if !@params.key?(:library)
+      rescue => e
+        warn("#{parser.program_name}: #{e.message}")
+        warn(parser.help) if e.is_a?(OptionParser::ParseError)
+        exit 1
+      end
+
+      def [](key)
+        @params[key]
+      end
+
+      def method_missing(m, *args, &blk)
+        m.to_s.tr('_', '-').to_sym
+          .then { |key| OPTION_NAMES.include?(key) ? self[key] : super }
+      end
+
+      def respond_to_missing?(m, *args)
+        @params.key?(m.to_s.tr('_', '-').to_sym) || super
+      end
+    end
+  end
+end

data/lib/xferase/logger.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require 'logger'
+require 'singleton'
+
+module Xferase
+  class << self
+    attr_writer :logger
+
+    def logger
+      @logger ||= Xferase::Logger
+    end
+  end
+
+  class Logger
+    include Singleton
+
+    class << self
+      attr_reader :stdout, :stderr
+
+      def open
+        @stdout = ::Logger.new($stdout)
+        @stderr = ::Logger.new($stderr)
+
+        Xferase::Config.verbose ? stdout.debug! : stdout.info!
+        Xferase::Config.verbose ? stderr.warn!  : stderr.fatal!
+      end
+
+      %i[unknown fatal error warn].each do |m|
+        define_method(m) { |*args| stderr.send(m, *args) }
+      end
+
+      %i[info debug].each do |m|
+        define_method(m) { |*args| stdout.send(m, *args) }
+      end
+    end
+  end
+end

data/lib/xferase/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Xferase
+  VERSION = '0.0.1'
+end

data/lib/xferase.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require 'xferase/version'
+require 'xferase/config'
+require 'xferase/logger'
+
+module Xferase
+end

metadata

@@ -0,0 +1,126 @@
+--- !ruby/object:Gem::Specification
+name: xferase
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Ryan Lue
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2021-12-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: debouncer
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: photein
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.9
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.9
+- !ruby/object:Gem::Dependency
+  name: rb-inotify
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.10'
+description: ''
+email: hello@ryanlue.com
+executables:
+- xferase
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- bin/xferase
+- lib/xferase.rb
+- lib/xferase/config.rb
+- lib/xferase/logger.rb
+- lib/xferase/version.rb
+homepage: https://github.com/rlue/xferase
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/rlue/xferase
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.6.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.2.3
+signing_key: 
+specification_version: 4
+summary: Import/rename photos & videos from one directory to another.
+test_files: []