dcat 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 21aa70d4fd2961f0000538bb28d339c50379aee433f35ff4663a26217a8c58e1
+  data.tar.gz: 9f66cab093be3f67b938d4bc6ea3102449c50a390ef6672a8553866baaaf8e51
+SHA512:
+  metadata.gz: c8af4dd1942c96e0945cca7630e9f0d2e9193a8ceb9e241ec6fa0ae282d8a27b67651567983b2949580391d1d2e838b0f257fba3c929151bf9d7e9dc4c46dc6e
+  data.tar.gz: cb88673b172b06d4ee42b4c59774e5a358ef7a8b3ec4423e4f4d86c60974125c4970005067bd1437adf7c384e4e2b883161aaa91772cbced32ce463e220eab30

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 src
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.ja.md

@@ -0,0 +1,132 @@
+# Dcat
+
+[English](README.md) | **日本語**
+
+ディスクカタログを作成・検索するためのRuby gemです。ファイルリストとメタデータ（サイズ、更新日時）を含む検索可能なカタログを作成し、効率的な保存のためにgzipで圧縮できます。
+
+## 機能
+
+- **カタログ作成** - ディレクトリ構造からファイルメタデータ付きのカタログを作成
+- **カタログ検索** - ripgrepを使用した効率的な検索
+- **カタログ圧縮** - gzip圧縮による省スペース保存
+- **クロスプラットフォーム対応** - Windows互換性
+- **人間が読みやすいファイルサイズ** - 出力での可読性
+
+## インストール
+
+gemをインストールするには：
+
+```bash
+$ gem install dcat
+```
+
+または、Gemfileに追加：
+
+```ruby
+gem 'dcat'
+```
+
+## 使用方法
+
+### 設定
+
+`~/.catalogrc`に設定ファイルを作成：
+
+```yaml
+general:
+    catalogdir: ./data
+```
+
+### コマンド
+
+#### カタログの作成
+
+```bash
+$ dcatc [ディレクトリパス]
+```
+
+指定されたディレクトリ（未指定の場合は現在のディレクトリ）からカタログファイルを作成します。
+
+#### カタログの検索
+
+```bash
+$ dcats [検索パターン]
+```
+
+パターンに一致するファイルをカタログファイルから検索します。
+
+#### 圧縮カタログの読み込み
+
+```bash
+$ dcatz [カタログファイル.gz]
+```
+
+圧縮されたカタログファイルを読み込むための`zcat`の代替コマンド。特にWindowsで有用です。
+
+### カタログファイル形式
+
+カタログファイルはタブ区切り値として保存されます：
+- 列1: フルファイルパス
+- 列2: 更新日時（YYYY-MM-DD形式）
+- 列3: ファイルサイズ（バイト）
+- 列4: KB単位に切り上げたファイルサイズ（例: `1 KB`、`120,564 KB`）
+
+ファイルは効率的な保存のためにgzip（`.gz`拡張子）で圧縮できます。
+
+### 使用例
+
+```bash
+# 現在のディレクトリのカタログを作成
+$ dcatc
+
+# 特定のディレクトリのカタログを作成
+$ dcatc /path/to/directory
+
+# "document"を含むファイルを検索
+$ dcats document
+
+# 特定の拡張子のファイルを検索
+$ dcats "*.pdf"
+
+# 圧縮されたカタログを読み込み
+$ dcatz catalog.gz
+```
+
+## 開発
+
+リポジトリをチェックアウト後、`bin/setup`を実行して依存関係をインストールします。その後、`rake test`でテストを実行できます。
+
+```bash
+# 依存関係のインストール
+$ bundle install
+
+# テストの実行
+$ rake test
+
+# gemのビルド
+$ bundle exec rake build
+
+# ローカルインストール
+$ bundle exec rake install
+
+# 対話型コンソール
+$ bin/console
+```
+
+## 要件
+
+- Ruby >= 2.6.0
+- 検索機能用の`ripgrep`（rg）コマンド
+
+## 依存関係
+
+- `sys-filesystem` - ディスク容量情報の取得
+- `zlib` - gzip圧縮/展開
+
+## 貢献
+
+バグレポートとプルリクエストは、GitHub https://github.com/src256/dcat で歓迎します。
+
+## ライセンス
+
+このgemは[MIT License](https://opensource.org/licenses/MIT)の条件の下でオープンソースとして利用できます。

data/README.md

@@ -0,0 +1,132 @@
+# Dcat
+
+**English** | [日本語](README.ja.md)
+
+A Ruby gem for creating and searching disk catalogs. Generate searchable catalogs containing file lists and metadata (size, modification date) with efficient gzip compression for storage.
+
+## Features
+
+- **Create catalogs** from directory structures with file metadata
+- **Search catalogs** efficiently using ripgrep
+- **Compress catalogs** with gzip for space-efficient storage
+- **Cross-platform support** with Windows compatibility
+- **Human-readable file sizes** in output
+
+## Installation
+
+Install the gem by executing:
+
+```bash
+$ gem install dcat
+```
+
+Or add to your Gemfile:
+
+```ruby
+gem 'dcat'
+```
+
+## Usage
+
+### Configuration
+
+Create a configuration file at `~/.catalogrc`:
+
+```yaml
+general:
+    catalogdir: ./data
+```
+
+### Commands
+
+#### Create a catalog
+
+```bash
+$ dcatc [directory_path]
+```
+
+Creates a catalog file from the specified directory (or current directory if not specified).
+
+#### Search catalogs
+
+```bash
+$ dcats [search_pattern]
+```
+
+Searches through catalog files for files matching the pattern.
+
+#### Read compressed catalogs
+
+```bash
+$ dcatz [catalog_file.gz]
+```
+
+Alternative to `zcat` for reading compressed catalog files, especially useful on Windows.
+
+### Catalog File Format
+
+Catalog files are stored as tab-separated values:
+- Column 1: Full file path
+- Column 2: Modification date (YYYY-MM-DD format)
+- Column 3: File size (bytes)
+- Column 4: File size rounded up to KB (e.g. `1 KB`, `120,564 KB`)
+
+Files can be compressed with gzip (`.gz` extension) for efficient storage.
+
+### Examples
+
+```bash
+# Create a catalog of the current directory
+$ dcatc
+
+# Create a catalog of a specific directory
+$ dcatc /path/to/directory
+
+# Search for files containing "document"
+$ dcats document
+
+# Search for files with specific extension
+$ dcats "*.pdf"
+
+# Read a compressed catalog
+$ dcatz catalog.gz
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests.
+
+```bash
+# Install dependencies
+$ bundle install
+
+# Run tests
+$ rake test
+
+# Build the gem
+$ bundle exec rake build
+
+# Install locally
+$ bundle exec rake install
+
+# Interactive console
+$ bin/console
+```
+
+## Requirements
+
+- Ruby >= 2.6.0
+- `ripgrep` (rg) command for searching functionality
+
+## Dependencies
+
+- `sys-filesystem` - For disk space information
+- `zlib` - For gzip compression/decompression
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/src256/dcat.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/exe/dcatc

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "dcat"
+
+Dcat::CreateCommand.run(ARGV)

data/exe/dcats

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "dcat"
+
+Dcat::SearchCommand.run(ARGV)

data/exe/dcatz

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+
+require 'zlib'
+
+
+def zcat(filename)
+  return unless filename
+  return unless FileTest.file?(filename)
+  if filename =~ /\.gz$/
+    Zlib::GzipReader.open(filename) do |gz|
+      print gz.read
+    end
+    return
+  end
+  File.open(filename, 'rb') do |f|
+    while (line = f.gets)
+      print line
+    end
+  end
+end
+
+filename = ARGV[0]
+zcat(filename)
+
+

data/lib/dcat/my_find.rb

@@ -0,0 +1,61 @@
+module MyFind
+
+  #
+  # Calls the associated block with the name of every file and directory listed
+  # as arguments, then recursively on their subdirectories, and so on.
+  #
+  # Returns an enumerator if no block is given.
+  #
+  # See the +Find+ module documentation for an example.
+  #
+  def find(*paths, ignore_error: true) # :yield: path
+	block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error)
+    
+	fs_encoding = Encoding.find("filesystem")
+    
+	paths.collect!{|d| raise Errno::ENOENT, d unless File.exist?(d); d.dup}.each do |path|
+	  path = path.to_path if path.respond_to? :to_path
+	  enc = path.encoding == Encoding::US_ASCII ? fs_encoding : path.encoding
+	  ps = [path]
+	  while file = ps.shift
+		catch(:prune) do
+		  yield file.dup
+		  begin
+			s = File.lstat(file)
+		  rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP, Errno::ENAMETOOLONG, Errno::EPERM
+			raise unless ignore_error
+			next
+		  end
+		  if s.directory? then
+			begin
+			  fs = Dir.children(file, encoding: enc)
+			rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP, Errno::ENAMETOOLONG, Errno::EPERM
+			  raise unless ignore_error
+			  next
+			end
+			fs.sort!
+			fs.reverse_each {|f|
+			  f = File.join(file, f)
+			  ps.unshift f
+			}
+		  end
+		end
+	  end
+	end
+	nil
+  end
+  
+  #
+  # Skips the current file or directory, restarting the loop with the next
+  # entry. If the current file is a directory, that directory will not be
+  # recursively entered. Meaningful only within the block associated with
+  # Find::find.
+  #
+  # See the +Find+ module documentation for an example.
+  #
+  def prune
+	throw :prune
+  end
+  
+  module_function :find, :prune
+end

data/lib/dcat/utils.rb

@@ -0,0 +1,32 @@
+module Dcat
+  class Utils
+    def self.number_to_human_size(size)
+      kb = 1024
+      mb = 1024**2
+      gb = 1024**3
+      tb = 1024**4
+      case
+      when size < kb
+        '%d Bytes' % size
+      when size < mb
+        '%.1f KB' % (size.to_f / kb)
+      when size < gb
+        '%.1f MB' % (size.to_f / mb)
+      when size < tb
+        '%.1f GB' % (size.to_f / gb)
+      else
+        '%.1f TB' % (size.to_f / tb)
+      end.sub('.0', '')
+    end
+
+    def self.windows?
+      RUBY_PLATFORM =~ /mswin|mingw|bccwin|cygwin/
+    end
+
+    def self.number_to_kb(bytes)
+      kb = (bytes.to_f / 1024).ceil
+      kb.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\\1,').reverse + " KB"
+    end
+  end
+end
+

data/lib/dcat/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Dcat
+  VERSION = "1.0.0"
+end

data/lib/dcat.rb

@@ -0,0 +1,308 @@
+# frozen_string_literal: true
+
+require "dcat/version"
+require "dcat/my_find"
+require "dcat/utils"
+
+
+require 'optparse'
+require 'fileutils'
+require 'yaml'
+require 'sys/filesystem'
+require 'zlib'
+
+module Dcat
+  DEF_CONFIG = File.expand_path("~/.catalogrc")
+  SEARCH_CMD = "rg"
+
+  class Error < StandardError; end
+ 
+  class Config
+    CATALOGDIR = File.expand_path("~/.catalog")
+
+    def self.create_from_yaml(yaml)
+      config = Config.new
+      catalogdir = config_value(yaml, "general", "catalogdir", true)
+      if catalogdir
+        catalogdir = File.expand_path(catalogdir)
+      end
+      config.catalogdir = catalogdir
+
+  ##    puts config.catalogdir
+      config.init_default
+      if config.catalogdir.nil? || !FileTest.directory?(config.catalogdir)
+        raise RuntimeError, "catalogdir is not exist: #{config.catalogdir}"
+      end      
+      config
+    end
+
+    def self.config_value(yaml, section, key, require)
+      return nil unless yaml
+      return nil unless yaml[section]
+      value = yaml[section][key]
+      if require && (value.nil? || value.empty?)
+        raise RuntimeError, "#{section}:#{key}: is empty"
+      end
+      value
+    end
+    
+    def init_default
+      @catalogdir ||= CATALOGDIR
+    end
+
+    attr_accessor :catalogdir
+  end
+
+  class Option
+    def initialize(opts, argv, config)
+      @opts = opts
+      @argv = argv
+      @config = config
+      @debug = @opts[:d]
+      @list = false
+      if @opts[:l]
+        @list = true
+      end
+    end
+    attr_reader :debug, :list
+    def catalogdir
+      @config.catalogdir
+    end
+  end
+
+  class CreateOption < Option
+    def initialize(opts, argv, config)
+      super
+      @dir = nil
+      if argv.size > 0 && FileTest.directory?(argv[0])
+       @dir = argv[0]
+      end
+
+      @debug = @opts[:d]
+      @compress = @opts[:z]
+      @output = @opts[:o]
+      if @output
+        @output_files = File.join(catalogdir, @output + '.files')
+        @output_info = File.join(catalogdir, @output + '.info')
+      end
+    end
+    attr_reader :dir, :search, :debug, :output, :output_files, :output_info, :compress, :debug
+
+    # def output_files
+    #   if compress
+    #     @output_files + '.gz'
+    #   else
+    #     @output_files
+    #   end
+    # end
+
+    def compress_output_files
+      @output_files + '.gz'
+    end
+
+    def real_output_files
+      if compress
+        compress_output_files
+      else
+        @output_files
+      end
+    end
+  end
+
+  class SearchOption < Option
+    def initialize(opts, argv, config)
+      super
+      @search = nil
+      if argv.size > 0
+        @search = argv[0]
+      end      
+      raise "SEARCH is not specified" unless @search
+    end
+    attr_reader :search
+  end
+
+  class Command
+    def self.load_yaml(config_file)
+      config_file ||=  DEF_CONFIG
+      config_file = File.expand_path(config_file)
+      puts "Config: #{config_file}"
+      yaml = nil
+      if FileTest.file?(config_file)
+        yaml = YAML.load_file(config_file)
+      end
+      Config.create_from_yaml(yaml)
+    end
+
+    def list
+      puts "List catalog in #{@option.catalogdir}"
+      Dir.glob(File.join(@option.catalogdir, "*.info")).each do |path|
+        puts File.basename(path)
+      end
+    end
+  end
+
+  class CreateCommand < Command
+    def self.run(argv)
+      STDOUT.sync = true
+      opts = {}
+      opt = OptionParser.new(argv)
+      opt.version = VERSION
+      opt.banner = "Usage: #{opt.program_name} [options] <directory>"
+      opt.separator('Options')
+      opt.on_head('-h', '--help', 'show this message') do |v|
+	      puts opt.help
+	      exit
+      end      
+      opt.on('-v', '--verbose', 'verbose message') {|v| opts[:v] = v}
+      opt.on('-o FILENAME', '--output=FILENAME') {|v| opts[:o] = v}
+      opt.on('--config=CONFIGFILE', 'Config file') {|v| opts[:c] = v }
+      opt.on('-d', '--debug', 'debug message') {|v| opts[:d] = v}
+      opt.on('-z', '--gzip', 'compress with gzip') {|v| opts[:z] = v}
+      opt.on('-l', '--list') {|v| opts[:l] = v}
+      opt.parse!(argv)
+      # if opts[:o].nil?
+	    #   puts opt.help
+	    #   exit
+      # end
+      config = load_yaml(opts[:c])
+      option = CreateOption.new(opts, argv, config)
+      begin
+	      command = CreateCommand.new(option)
+	      command.run
+      rescue => e
+        puts e.message
+	      if option.debug
+          puts e.backtrace
+	      end
+	      puts opt.help
+      end
+    end
+
+    def initialize(option)
+      @option = option
+    end
+
+    def run
+      if @option.list
+        list
+      else
+        unless @option.dir
+          raise "directory is not specified"
+        end
+        make_files
+        make_info
+      end
+    end
+
+    private
+    def make_files
+      puts "Create catalog file for #{@option.dir}: #{@option.real_output_files}"
+      unless FileTest.directory?(@option.dir)
+	       raise RuntimeError
+      end
+
+      File.unlink(@option.output_files) if FileTest.file?(@option.output_files)
+      File.unlink(@option.compress_output_files) if FileTest.file?(@option.compress_output_files)
+
+      if @option.compress
+        out = File.open(@option.compress_output_files, "w:utf-8")
+        out = Zlib::GzipWriter.open(out)
+      else
+        out = File.open(@option.output_files, "w:utf-8")
+      end
+
+      begin
+        MyFind.find(@option.dir, ignore_error: true) do |f|
+          if FileTest.file?(f)
+            size = File.size(f)
+            mtime = File.mtime(f).strftime("%Y-%m-%d")
+            human_readable_size = Utils.number_to_kb(size)
+            out.puts "#{f}\t#{mtime}\t#{size}\t#{human_readable_size}"
+          elsif FileTest.directory?(f)
+            STDERR.puts f
+          end
+        end  
+      rescue => e
+        puts e.message
+        if option.debug
+          p e.backtrace
+          p e.class
+        end
+      end
+      out.close if out
+    end    
+
+    def make_info
+      puts "Create info for #{@option.dir}: #{@option.output_info}"
+      unless FileTest.directory?(@option.dir)
+        raise RuntimeError
+      end
+
+      stat = Sys::Filesystem.stat(@option.dir)
+      free_bytes = Utils::number_to_human_size(stat.blocks_available * stat.block_size)
+      #      puts number_to_human_size(free_bytes)
+      puts "total=#{stat.blocks * stat.block_size}"
+      total_bytes = Utils::number_to_human_size(stat.blocks * stat.block_size)
+      puts "total_bytes=#{total_bytes}"
+      #puts number_to_human_size(total_bytes)
+
+      out = File.open(@option.output_info, "w:utf-8")
+      created = Time.now.strftime("%Y-%m-%dT%H:%M:%S")
+      out.puts "Created: #{created}"
+      out.puts "Volume: #{@option.dir}"
+      out.puts "Total: #{total_bytes}"
+      out.puts "Free: #{free_bytes}"
+      out.close if out
+    end
+  end
+
+  class SearchCommand < Command
+    def self.run(argv)
+      STDOUT.sync = true
+      opts = {}
+      opt = OptionParser.new(argv)
+      opt.version = VERSION
+      opt.banner = "Usage: #{opt.program_name} [options] SEARCH <directory>"
+      opt.separator('Options')
+      opt.on_head('-h', '--help', 'show this message') do |v|
+        puts opt.help
+        exit
+      end
+      opt.on('-v', '--verbose', 'verbose message') {|v| opts[:v] = v}
+      opt.on('--config=CONFIGFILE', 'Config file') {|v| opts[:c] = v }
+      opt.on('-d', '--debug', 'debug message') {|v| opts[:d] = v}
+      opt.parse!(argv)
+      config = load_yaml(opts[:c])
+      begin
+        option = SearchOption.new(opts, argv, config)
+        command = SearchCommand.new(option)
+        command.run
+      rescue => e
+        puts e.message
+        if opts[:d]
+          puts e.backtrace
+        end
+        puts opt.help
+      end
+    end
+
+    def initialize(option)
+      @option = option
+    end
+
+    def run
+      if @option.list
+        list
+      else
+        search
+      end
+    end
+
+    def search
+      # rg の -z オプションが gzip 圧縮されたカタログを展開しながら検索する
+      cmd = "#{SEARCH_CMD} -z #{@option.search} #{@option.catalogdir}"
+      puts cmd if @option.debug
+      system(cmd)
+    end
+  end
+end

data/sig/dcat.rbs

@@ -0,0 +1,4 @@
+module Dcat
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,92 @@
+--- !ruby/object:Gem::Specification
+name: dcat
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- src
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2026-09-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sys-filesystem
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: zlib
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: |
+  Dcat builds searchable catalogs of directory trees, recording each file's
+  full path, modification date and size as tab-separated text. Catalogs can be
+  gzip-compressed for compact storage and searched with ripgrep. It ships three
+  commands: dcatc to create a catalog, dcats to search catalogs, and dcatz to
+  read compressed catalogs as an alternative to zcat.
+email:
+- src@srcw.net
+executables:
+- dcatc
+- dcats
+- dcatz
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.ja.md
+- README.md
+- exe/dcatc
+- exe/dcats
+- exe/dcatz
+- lib/dcat.rb
+- lib/dcat/my_find.rb
+- lib/dcat/utils.rb
+- lib/dcat/version.rb
+- sig/dcat.rbs
+homepage: https://github.com/src256/dcat
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/src256/dcat
+  source_code_uri: https://github.com/src256/dcat
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.6.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Create and search disk catalogs of files and their metadata.
+test_files: []