schwifty 0.0.0 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +4 -9
- data/.rspec +1 -2
- data/.rubocop.yml +17 -0
- data/.ruby-version +1 -0
- data/.travis.yml +1 -2
- data/Gemfile +11 -2
- data/README.md +55 -21
- data/Rakefile +10 -4
- data/bin/console +6 -10
- data/bin/schwifty +7 -0
- data/lib/schwifty.rb +2 -5
- data/lib/schwifty/cli.rb +51 -0
- data/lib/schwifty/client.rb +31 -0
- data/lib/schwifty/commands/add.rb +47 -0
- data/lib/schwifty/commands/bootstrap.rb +39 -0
- data/lib/schwifty/commands/gc.rb +39 -0
- data/lib/schwifty/commands/get.rb +21 -0
- data/lib/schwifty/content/defaults.rb +12 -0
- data/lib/schwifty/content/hash_file.rb +50 -0
- data/lib/schwifty/content/objects_file.rb +25 -0
- data/lib/schwifty/content/parsers.rb +20 -0
- data/lib/schwifty/content/pinned_file.rb +31 -0
- data/lib/schwifty/info.rb +7 -0
- data/schwifty.gemspec +16 -14
- metadata +43 -28
- data/LICENSE.txt +0 -21
- data/bin/setup +0 -8
- data/lib/schwifty/version.rb +0 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2f96cb81f4a5fb3d0ca084725d860f86961c183
|
4
|
+
data.tar.gz: 601aac7b592f8bebaa8a3d58ffbf50676af3cecc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 640b9aa83d011135e7729a35743e6b2b2fb2e86a25e7380c4635353e8f70447305be171a7a8977bc65cd4d32181d648666a9435c82ec5f3ba52b971b789f6e24
|
7
|
+
data.tar.gz: 77c301fb13c23543672a471b39c5e1981cd61f309058ae91c93018418b66473923b1a219d641b67b2ef87e1e99ea288781f16a393b9ff260e1f2900e6cfe3e51
|
data/.gitignore
CHANGED
data/.rspec
CHANGED
@@ -1,2 +1 @@
|
|
1
|
-
--format documentation
|
2
|
-
--color
|
1
|
+
--color --format documentation
|
data/.rubocop.yml
ADDED
data/.ruby-version
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
2.3.1
|
data/.travis.yml
CHANGED
data/Gemfile
CHANGED
@@ -1,4 +1,13 @@
|
|
1
|
-
source 'https://rubygems.org'
|
1
|
+
source 'https://rubygems.org' do
|
2
|
+
group :development do
|
3
|
+
gem 'pry', '~>0.10.3'
|
4
|
+
gem 'rubocop', '~>0.40.0'
|
5
|
+
end
|
6
|
+
|
7
|
+
group :test do
|
8
|
+
gem 'rake', '~>11.1.2'
|
9
|
+
gem 'rspec', '~>3.4.0'
|
10
|
+
end
|
11
|
+
end
|
2
12
|
|
3
|
-
# Specify your gem's dependencies in schwifty.gemspec
|
4
13
|
gemspec
|
data/README.md
CHANGED
@@ -1,41 +1,75 @@
|
|
1
|
-
#
|
1
|
+
# schwifty
|
2
2
|
|
3
|
-
|
3
|
+
[ipfs][1] is a distributed file system that is content-addressed and uses keys (hashes) to files
|
4
|
+
(objects). This repo uses the hash/object terminology consistent with the rest of ipfs docs.
|
4
5
|
|
5
|
-
|
6
|
+
## Usage
|
6
7
|
|
7
|
-
|
8
|
+
```bash
|
9
|
+
schwifty add <files>...
|
10
|
+
schwifty bootstrap (--clear | <nodes>... | --file=<bootstrap_list_yaml>)
|
11
|
+
schwifty get <files_or_hashes>...
|
12
|
+
schwifty gc
|
13
|
+
schwifty -h | --help
|
14
|
+
schwifty --version
|
15
|
+
```
|
8
16
|
|
9
|
-
|
17
|
+
## Hash Files
|
10
18
|
|
11
|
-
|
12
|
-
|
13
|
-
|
19
|
+
Because ipfs does not have a typical Unix file system structure, we keep track of files added using
|
20
|
+
the `ipfs add` operation. This task is entrusted to two files, one of which any client code needs to
|
21
|
+
interact with and the other for garbage collection purposes. Both of these files are in `.yaml` form.
|
14
22
|
|
15
|
-
|
23
|
+
For the examples below, we'll use the convention in `bin/console` where `client = IPFS::Client`.
|
16
24
|
|
17
|
-
|
25
|
+
### Objects File: `ipfs_objects.yaml`
|
18
26
|
|
19
|
-
|
27
|
+
It's your job read and manipulate this file and pass it between steps of your build/test pipeline.
|
28
|
+
The objects file contains a mapping of filename : hash
|
20
29
|
|
21
|
-
|
30
|
+
`client.add('spec/samples/files/file1')`
|
22
31
|
|
23
|
-
|
32
|
+
```yaml
|
33
|
+
---
|
34
|
+
spec/samples/files/file1: QmXMSyJvaz912Wi6533MegwUn4mJ4kQikaBZpAdeFwoWkj
|
35
|
+
```
|
24
36
|
|
25
|
-
|
37
|
+
`client.get('spec/samples/files/file1')` will download the file to the current directory.
|
38
|
+
Both `add` and `get` can take multiple files.
|
26
39
|
|
27
|
-
|
40
|
+
### Pinned Objects File: `~/.ipfs/ipfs_pinned_objects.yaml`
|
28
41
|
|
29
|
-
|
42
|
+
A hash : timestamp is written in this file every time `ipfs add` is run. `ipfs add` pins objects by
|
43
|
+
default. You aren't expected to interact with this file, but it may help in debugging purposes.
|
30
44
|
|
31
|
-
|
45
|
+
```yaml
|
46
|
+
---
|
47
|
+
Qmb1CkJmfpwmyLvuPMMRKKpYPuiwUMW9V1WdbFLVZgAdpL: 2016-06-01 01:04:49 UTC
|
48
|
+
QmZXYwLpuGd8kR51FU5f8U5M8M1LHPSUxMeFBtrNWwJGdE: 2016-05-31 12:41:19 UTC
|
49
|
+
```
|
32
50
|
|
33
|
-
|
51
|
+
Garbage collection (GC) follows these rules:
|
52
|
+
1. Check for disk space. If disk space is below the threshold (0.85), do not GC.
|
53
|
+
2. Run `ipfs repo gc` to remove all unpinned objects.
|
54
|
+
3. Check for disk space. If disk space is above the threshold, go to step 4.
|
55
|
+
4. Unpin one third of objects from the Pinned Objects File.
|
56
|
+
5. Run `ipfs repo gc` to remove the unpinned objects.
|
57
|
+
6. Back to step 3.
|
34
58
|
|
35
|
-
|
59
|
+
There's no backoff in the GC unpin rate in step 4, and the gem doesn't know how much disk space will
|
60
|
+
be freed every call. The 1/3 of hashes removed each time does get smaller though.
|
36
61
|
|
62
|
+
### Bootstrapping
|
37
63
|
|
38
|
-
|
64
|
+
We recommend checking in an approved bootstrap list that, in tandem with the CLI, bootstraps nodes
|
65
|
+
via a configuration managed provisioning tool (like Ansible). The CLI supports bootstrapping via
|
66
|
+
a `.yaml` file with hostname : multiaddr/peerID formats. See `ipfs bootstrap` and `ipfs id` docs
|
67
|
+
for more info.
|
39
68
|
|
40
|
-
|
69
|
+
```yaml
|
70
|
+
---
|
71
|
+
hostname1: /ip4/127.0.0.1/tcp/4001/ipfs/QaowiA9HvLCinkCLwRFauHkAZUP3DQDogku98r9BctEhgc
|
72
|
+
hostname2: /ip4/192.112.0.101/tcp/4001/ipfs/QmcpER9AOSUinkCLwRFi8zkAZU73DQDogkF98r9BctEhgc
|
73
|
+
```
|
41
74
|
|
75
|
+
[1]: https://ipfs.io/
|
data/Rakefile
CHANGED
@@ -1,6 +1,12 @@
|
|
1
|
-
|
2
|
-
require "rspec/core/rake_task"
|
1
|
+
# encoding: utf-8
|
3
2
|
|
4
|
-
|
3
|
+
require 'bundler/setup'
|
4
|
+
require 'bundler/gem_tasks'
|
5
5
|
|
6
|
-
task :
|
6
|
+
task default: %w(spec rubocop)
|
7
|
+
|
8
|
+
require 'rspec/core/rake_task'
|
9
|
+
RSpec::Core::RakeTask.new
|
10
|
+
|
11
|
+
require 'rubocop/rake_task'
|
12
|
+
RuboCop::RakeTask.new
|
data/bin/console
CHANGED
@@ -1,14 +1,10 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
require
|
4
|
-
require
|
3
|
+
require 'bundler/setup'
|
4
|
+
require 'schwifty'
|
5
|
+
require 'pry'
|
5
6
|
|
6
|
-
#
|
7
|
-
|
7
|
+
# Debugging shortcuts
|
8
|
+
client = Schwifty::Client
|
8
9
|
|
9
|
-
|
10
|
-
# require "pry"
|
11
|
-
# Pry.start
|
12
|
-
|
13
|
-
require "irb"
|
14
|
-
IRB.start
|
10
|
+
binding.pry
|
data/bin/schwifty
ADDED
data/lib/schwifty.rb
CHANGED
data/lib/schwifty/cli.rb
ADDED
@@ -0,0 +1,51 @@
|
|
1
|
+
require 'docopt'
|
2
|
+
|
3
|
+
require 'schwifty/client'
|
4
|
+
require 'schwifty/info'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
# This class acts as the command line interface for the IPFS client. Args are parsed using the
|
8
|
+
# docopt template below.
|
9
|
+
class CLI
|
10
|
+
def run
|
11
|
+
opts = parse_args
|
12
|
+
|
13
|
+
if opts['add']
|
14
|
+
Schwifty::Client.add(*opts['<files>'])
|
15
|
+
elsif opts['bootstrap']
|
16
|
+
Schwifty::Client.bootstrap(*opts['<nodes>'], clear: opts['--clear'], filename: opts['--file'])
|
17
|
+
elsif opts['get']
|
18
|
+
Schwifty::Client.add(*opts['<files_or_hashes>'])
|
19
|
+
elsif opts['gc']
|
20
|
+
Schwifty::Client.gc
|
21
|
+
elsif opts['--version']
|
22
|
+
puts "schwifty version #{Schwifty::VERSION}"
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
# Modify this to change how the CLI works -- see http://docopt.org/
|
27
|
+
def parse_args
|
28
|
+
doc = <<DOCOPT
|
29
|
+
schwifty saves and downloads objects from ipfs, keeping track of their hashes in a garbage collection file in ~/.ipfs/ipfs_pinned_objects.yaml and an objects file in ./ipfs_objects.yaml
|
30
|
+
|
31
|
+
Usage:
|
32
|
+
schwifty add <files>...
|
33
|
+
schwifty bootstrap (--clear | <nodes>... | --file=<bootstrap_list_yaml>)
|
34
|
+
schwifty get <files_or_hashes>...
|
35
|
+
schwifty gc
|
36
|
+
schwifty -h | --help
|
37
|
+
schwifty --version
|
38
|
+
|
39
|
+
Options:
|
40
|
+
-h --help Show this screen.
|
41
|
+
--version Show version.
|
42
|
+
DOCOPT
|
43
|
+
begin
|
44
|
+
Docopt.docopt(doc)
|
45
|
+
rescue Docopt::Exit => e
|
46
|
+
puts e.message
|
47
|
+
exit
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
|
3
|
+
require 'schwifty/commands/add'
|
4
|
+
require 'schwifty/commands/bootstrap'
|
5
|
+
require 'schwifty/commands/gc'
|
6
|
+
require 'schwifty/commands/get'
|
7
|
+
|
8
|
+
module Schwifty
|
9
|
+
# This class calls all the ipfs commands for the CLI or other use.
|
10
|
+
class Client
|
11
|
+
def self.add(*files)
|
12
|
+
Commands::Add.run(*files)
|
13
|
+
nil
|
14
|
+
end
|
15
|
+
|
16
|
+
def self.bootstrap(*nodes, clear: false, filename: nil)
|
17
|
+
Commands::Bootstrap.run(*nodes, clear: clear, filename: filename)
|
18
|
+
nil
|
19
|
+
end
|
20
|
+
|
21
|
+
def self.gc
|
22
|
+
Commands::GC.run
|
23
|
+
nil
|
24
|
+
end
|
25
|
+
|
26
|
+
def self.get(*filenames_or_hashes, dst: '.')
|
27
|
+
Commands::Get.run(*filenames_or_hashes, dst)
|
28
|
+
nil
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'time'
|
3
|
+
|
4
|
+
require 'schwifty/content/objects_file'
|
5
|
+
require 'schwifty/content/parsers'
|
6
|
+
require 'schwifty/content/pinned_file'
|
7
|
+
|
8
|
+
module Schwifty
|
9
|
+
module Commands
|
10
|
+
# This class adds objects to ipfs and tracks their hashes using the Objects File and Pinned
|
11
|
+
# Objects File.
|
12
|
+
class Add
|
13
|
+
def self.run(*filenames)
|
14
|
+
result_filenames = []
|
15
|
+
result_hashes = []
|
16
|
+
|
17
|
+
# Parse add command outputs
|
18
|
+
filenames.each do |name|
|
19
|
+
parsed_filenames, parsed_hashes = Dir.exist?(name) ? add_dir(name) : add_file(name)
|
20
|
+
result_filenames += parsed_filenames
|
21
|
+
result_hashes += parsed_hashes
|
22
|
+
end
|
23
|
+
|
24
|
+
update_hash_files(result_filenames, result_hashes)
|
25
|
+
end
|
26
|
+
|
27
|
+
private_class_method
|
28
|
+
|
29
|
+
def self.add_dir(name)
|
30
|
+
output = fail_on_error('ipfs', 'add', '-r', name)
|
31
|
+
parsed_filenames, parsed_hashes = Schwifty::Content::AddParser.run(output)
|
32
|
+
parsed_filenames.map! { |tail| File.join(File.dirname(name), tail) }
|
33
|
+
[parsed_filenames, parsed_hashes]
|
34
|
+
end
|
35
|
+
|
36
|
+
def self.add_file(name)
|
37
|
+
[[name], [fail_on_error('ipfs', 'add', '-q', name)]]
|
38
|
+
end
|
39
|
+
|
40
|
+
def self.update_hash_files(filenames, hashes)
|
41
|
+
Schwifty::Content::ObjectsFile.new.add_entries(filenames, hashes)
|
42
|
+
Schwifty::Content::PinnedFile.new.
|
43
|
+
add_entries(hashes, [Time.now.getutc.to_s] * hashes.size, allow_collisions: true)
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'yaml'
|
3
|
+
|
4
|
+
module Schwifty
|
5
|
+
module Commands
|
6
|
+
# Bootstrapping individual worker nodes. '<multiaddr>/<peerID>' format required.
|
7
|
+
class Bootstrap
|
8
|
+
def self.run(*nodes, clear: false, filename: nil)
|
9
|
+
if clear
|
10
|
+
bootstrap_clear
|
11
|
+
elsif filename
|
12
|
+
bootstrap_file(filename)
|
13
|
+
else
|
14
|
+
bootstrap_nodes(*nodes)
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
private_class_method
|
19
|
+
|
20
|
+
def self.bootstrap_clear
|
21
|
+
existing_nodes = fail_on_error('ipfs', 'bootstrap', 'list').split("\n")
|
22
|
+
if existing_nodes.empty?
|
23
|
+
puts "#{self.class.name}: No nodes in bootstrap list!"
|
24
|
+
else
|
25
|
+
fail_on_error('ipfs', 'bootstrap', 'rm', *existing_nodes)
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
# Bootstrap file is a mapping of hostname : '<multiaddr>/<peerID>' formats.
|
30
|
+
def self.bootstrap_file(filename)
|
31
|
+
bootstrap_nodes(*YAML.load_file(filename).values)
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.bootstrap_nodes(*nodes)
|
35
|
+
fail_on_error('ipfs', 'bootstrap', 'add', *nodes)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'sys/filesystem'
|
3
|
+
|
4
|
+
require 'schwifty/content/pinned_file'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
module Commands
|
8
|
+
# Garbage collection to be called via CLI. If there's not enough disk space after a blind call
|
9
|
+
# to GC unpinned objects, we GC pinned objects as well. Pinned GC works by removing hashes
|
10
|
+
# repeatedly until we're below the disk space threshold. We get hashes to remove from the
|
11
|
+
# Pinned Objects File.
|
12
|
+
class GC
|
13
|
+
def self.run
|
14
|
+
return unless disk_space_exceeds_threshold?
|
15
|
+
gc
|
16
|
+
|
17
|
+
while disk_space_exceeds_threshold?
|
18
|
+
unpin(Schwifty::Content::PinnedFile.new.cull)
|
19
|
+
gc
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
private_class_method
|
24
|
+
|
25
|
+
def self.disk_space_exceeds_threshold?
|
26
|
+
stat = Sys::Filesystem.stat('/')
|
27
|
+
(stat.blocks - stat.blocks_free) / stat.blocks.to_f > Schwifty::Content::Defaults::GC_DISK_THRESHOLD
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.gc
|
31
|
+
fail_on_error('ipfs', 'repo', 'gc')
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.unpin(hashes)
|
35
|
+
fail_on_error('ipfs', 'pin', 'rm', *hashes)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
|
3
|
+
require 'schwifty/content/objects_file'
|
4
|
+
|
5
|
+
module Schwifty
|
6
|
+
module Commands
|
7
|
+
# Get allows us to download valid files in the Objects File or try a hash directly.
|
8
|
+
class Get
|
9
|
+
def self.run(*filenames_or_hashes, dst: '.')
|
10
|
+
filenames_or_hashes.each do |key|
|
11
|
+
key = Schwifty::Content::ObjectsFile.new.get(key) unless hash?(key)
|
12
|
+
fail_on_error('ipfs', 'get', key, "--output='#{dst}'")
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
private_class_method def self.hash?(str)
|
17
|
+
str.length == 46 && str[/[a-zA-Z0-9]+/] == str
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,12 @@
|
|
1
|
+
module Schwifty
|
2
|
+
module Content
|
3
|
+
module Defaults
|
4
|
+
OBJECTS_FILE_NAME = 'ipfs_objects.yaml'.freeze
|
5
|
+
PINNED_FILE_NAME = File.join(ENV['HOME'], '.ipfs', 'ipfs_pinned_objects.yaml').freeze
|
6
|
+
|
7
|
+
# If we're above this threshold after an unpinned GC, we GC pinned objects as well.
|
8
|
+
GC_DISK_THRESHOLD = 0.85 # TODO: Tune this value
|
9
|
+
GC_CULL_THRESHOLD = 0.67 # TODO: Tune this value
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
@@ -0,0 +1,50 @@
|
|
1
|
+
require 'yaml'
|
2
|
+
|
3
|
+
module Schwifty
|
4
|
+
module Content
|
5
|
+
# IPFS file base class
|
6
|
+
class HashFile
|
7
|
+
def initialize(filename)
|
8
|
+
@filename = filename
|
9
|
+
@entries = load_yaml
|
10
|
+
end
|
11
|
+
|
12
|
+
def add_entries(keys, vals, allow_collisions: false)
|
13
|
+
keys.zip(vals).each do |key, val|
|
14
|
+
if @entries.key?(key) && !allow_collisions
|
15
|
+
raise "#{self.class.name} collision when inserting #{key}, #{val}"
|
16
|
+
end
|
17
|
+
|
18
|
+
@entries[key] = val
|
19
|
+
end
|
20
|
+
|
21
|
+
save_yaml
|
22
|
+
end
|
23
|
+
|
24
|
+
def get(key)
|
25
|
+
val = @entries[key.to_s]
|
26
|
+
raise "#{self.class.name} value does not exist for: #{key}" unless val
|
27
|
+
val
|
28
|
+
end
|
29
|
+
|
30
|
+
def remove_entry(key)
|
31
|
+
@entries.delete(key)
|
32
|
+
end
|
33
|
+
|
34
|
+
def save_yaml
|
35
|
+
File.open(@filename, 'w') { |f| f.write(@entries.to_yaml) }
|
36
|
+
end
|
37
|
+
|
38
|
+
private
|
39
|
+
|
40
|
+
def load_yaml
|
41
|
+
FileUtils.touch(@filename) unless File.exist?(@filename)
|
42
|
+
YAML.load_file(@filename) || {}
|
43
|
+
end
|
44
|
+
|
45
|
+
def num_entries
|
46
|
+
@entries.length
|
47
|
+
end
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
require 'yaml'
|
2
|
+
|
3
|
+
require 'schwifty/content/defaults'
|
4
|
+
require 'schwifty/content/hash_file'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
module Content
|
8
|
+
# The Objects File keeps track of filename : hash mappings. Directories added recursively will
|
9
|
+
# have all of their files' individual hashes added as well.
|
10
|
+
#
|
11
|
+
# ipfs add -r spec/samples
|
12
|
+
#
|
13
|
+
# ---
|
14
|
+
# spec/samples/files/dir1/file2: Qmenn1xp6bq5JJhE1hjsishPLeohEsWTq2GHWc2CspixZH
|
15
|
+
# spec/samples/files/file1: QmXMSyJvaz912Wi6533MegwUn4mJ4kQikaBZpAdeFwoWkj
|
16
|
+
# spec/samples/files/dir1: QmSsF4xjCAcpY3634TpKdBjHNnncWcQCrjyJqePUpdVjoX
|
17
|
+
# spec/samples/files: QmXYciy4T1dZAWp2JW2caPoK9ktJ54cuJoG8aAmkXRkJWq
|
18
|
+
#
|
19
|
+
class ObjectsFile < Schwifty::Content::HashFile
|
20
|
+
def initialize(filename = Schwifty::Content::Defaults::OBJECTS_FILE_NAME)
|
21
|
+
super(filename)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module Schwifty
|
2
|
+
module Content
|
3
|
+
# Parses ipfs add -r output to return a list of filenames and their hashes
|
4
|
+
class AddParser
|
5
|
+
def self.run(output)
|
6
|
+
filenames = []
|
7
|
+
hashes = []
|
8
|
+
|
9
|
+
output.split("\n").each do |line|
|
10
|
+
words = line.split(' ')
|
11
|
+
index = words.index('added')
|
12
|
+
hashes << words[index + 1]
|
13
|
+
filenames << words[index + 2]
|
14
|
+
end
|
15
|
+
|
16
|
+
[filenames, hashes]
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require 'time'
|
2
|
+
require 'yaml'
|
3
|
+
|
4
|
+
require 'schwifty/content/defaults'
|
5
|
+
require 'schwifty/content/hash_file'
|
6
|
+
|
7
|
+
module Schwifty
|
8
|
+
module Content
|
9
|
+
# The Pinned Objects File keeps track of hash : timestamp mappings. Remember: a file must
|
10
|
+
# remain on a single host to exist in the cluster, but we need to keep track of space usage.
|
11
|
+
#
|
12
|
+
# ---
|
13
|
+
# Qmb1CkJmfpwmyLvuPMMRKKpYPuiwUMW9V1WdbFLVZgAdpL: 2016-06-01 01:04:49 UTC
|
14
|
+
#
|
15
|
+
class PinnedFile < Schwifty::Content::HashFile
|
16
|
+
def initialize(filename = Schwifty::Content::Defaults::PINNED_FILE_NAME)
|
17
|
+
super(filename)
|
18
|
+
end
|
19
|
+
|
20
|
+
# Remove hashes from the Pinned Objects File using a threshold. Does not do any garbage
|
21
|
+
# collection on its own. Return a list of hashes to be GC'ed.
|
22
|
+
def cull(threshold = Schwifty::Content::Defaults::GC_CULL_THRESHOLD)
|
23
|
+
num_to_remove = num_entries - (threshold * num_entries).to_int
|
24
|
+
entries_to_remove = @entries.sort_by { |_hash, date| date }[0..num_to_remove].map(&:first)
|
25
|
+
entries_to_remove.each { |key| remove_entry(key) }
|
26
|
+
save_yaml
|
27
|
+
entries_to_remove
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,7 @@
|
|
1
|
+
module Schwifty
|
2
|
+
VERSION = '0.1.0'.freeze
|
3
|
+
SUMMARY = 'Ruby bindings for ipfs.'.freeze
|
4
|
+
DESCRIPTION = "Designed for distributed artifact storage, this gem keeps track of objects
|
5
|
+
added using ipfs with an objects file and also provides a garbage collection solution when a
|
6
|
+
particular node's space runs low.".freeze
|
7
|
+
end
|
data/schwifty.gemspec
CHANGED
@@ -1,24 +1,26 @@
|
|
1
|
-
#
|
1
|
+
# encoding: utf-8
|
2
2
|
lib = File.expand_path('../lib', __FILE__)
|
3
3
|
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
-
require 'schwifty/
|
4
|
+
require 'schwifty/info'
|
5
5
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name =
|
7
|
+
spec.name = 'schwifty'
|
8
8
|
spec.version = Schwifty::VERSION
|
9
|
-
spec.authors = [
|
10
|
-
spec.email = [
|
9
|
+
spec.authors = ['James Chang']
|
10
|
+
spec.email = ['longboardcat13@sgmail.com']
|
11
11
|
|
12
|
-
spec.summary =
|
13
|
-
spec.
|
14
|
-
spec.
|
12
|
+
spec.summary = Schwifty::SUMMARY
|
13
|
+
spec.description = Schwifty::DESCRIPTION
|
14
|
+
spec.homepage = 'https://github.com/longboardcat/schwifty'
|
15
15
|
|
16
16
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
17
|
-
spec.bindir =
|
18
|
-
spec.executables =
|
19
|
-
spec.require_paths = [
|
17
|
+
spec.bindir = 'bin'
|
18
|
+
spec.executables = ['schwifty']
|
19
|
+
spec.require_paths = ['lib']
|
20
20
|
|
21
|
-
spec.
|
22
|
-
spec.
|
23
|
-
spec.
|
21
|
+
spec.add_runtime_dependency 'docopt', '~>0.5.0'
|
22
|
+
spec.add_runtime_dependency 'sys-filesystem', '~>1.1.6'
|
23
|
+
spec.add_runtime_dependency 'build_execution'
|
24
|
+
|
25
|
+
spec.required_ruby_version = '>= 2.0.0'
|
24
26
|
end
|
metadata
CHANGED
@@ -1,79 +1,94 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: schwifty
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- James Chang
|
8
8
|
autorequire:
|
9
|
-
bindir:
|
9
|
+
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2017-05-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
|
-
name:
|
14
|
+
name: docopt
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
17
|
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version:
|
20
|
-
type: :
|
19
|
+
version: 0.5.0
|
20
|
+
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version:
|
26
|
+
version: 0.5.0
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
|
-
name:
|
28
|
+
name: sys-filesystem
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version:
|
34
|
-
type: :
|
33
|
+
version: 1.1.6
|
34
|
+
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version:
|
40
|
+
version: 1.1.6
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
|
-
name:
|
42
|
+
name: build_execution
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
|
-
- - "
|
45
|
+
- - ">="
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '
|
48
|
-
type: :
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
|
-
- - "
|
52
|
+
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '
|
55
|
-
description:
|
54
|
+
version: '0'
|
55
|
+
description: |-
|
56
|
+
Designed for distributed artifact storage, this gem keeps track of objects
|
57
|
+
added using ipfs with an objects file and also provides a garbage collection solution when a
|
58
|
+
particular node's space runs low.
|
56
59
|
email:
|
57
|
-
-
|
58
|
-
executables:
|
60
|
+
- longboardcat13@sgmail.com
|
61
|
+
executables:
|
62
|
+
- schwifty
|
59
63
|
extensions: []
|
60
64
|
extra_rdoc_files: []
|
61
65
|
files:
|
62
66
|
- ".gitignore"
|
63
67
|
- ".rspec"
|
68
|
+
- ".rubocop.yml"
|
69
|
+
- ".ruby-version"
|
64
70
|
- ".travis.yml"
|
65
71
|
- Gemfile
|
66
|
-
- LICENSE.txt
|
67
72
|
- README.md
|
68
73
|
- Rakefile
|
69
74
|
- bin/console
|
70
|
-
- bin/
|
75
|
+
- bin/schwifty
|
71
76
|
- lib/schwifty.rb
|
72
|
-
- lib/schwifty/
|
77
|
+
- lib/schwifty/cli.rb
|
78
|
+
- lib/schwifty/client.rb
|
79
|
+
- lib/schwifty/commands/add.rb
|
80
|
+
- lib/schwifty/commands/bootstrap.rb
|
81
|
+
- lib/schwifty/commands/gc.rb
|
82
|
+
- lib/schwifty/commands/get.rb
|
83
|
+
- lib/schwifty/content/defaults.rb
|
84
|
+
- lib/schwifty/content/hash_file.rb
|
85
|
+
- lib/schwifty/content/objects_file.rb
|
86
|
+
- lib/schwifty/content/parsers.rb
|
87
|
+
- lib/schwifty/content/pinned_file.rb
|
88
|
+
- lib/schwifty/info.rb
|
73
89
|
- schwifty.gemspec
|
74
|
-
homepage: https://github.com/
|
75
|
-
licenses:
|
76
|
-
- MIT
|
90
|
+
homepage: https://github.com/longboardcat/schwifty
|
91
|
+
licenses: []
|
77
92
|
metadata: {}
|
78
93
|
post_install_message:
|
79
94
|
rdoc_options: []
|
@@ -83,7 +98,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
83
98
|
requirements:
|
84
99
|
- - ">="
|
85
100
|
- !ruby/object:Gem::Version
|
86
|
-
version:
|
101
|
+
version: 2.0.0
|
87
102
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
88
103
|
requirements:
|
89
104
|
- - ">="
|
@@ -91,7 +106,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
91
106
|
version: '0'
|
92
107
|
requirements: []
|
93
108
|
rubyforge_project:
|
94
|
-
rubygems_version: 2.
|
109
|
+
rubygems_version: 2.5.1
|
95
110
|
signing_key:
|
96
111
|
specification_version: 4
|
97
112
|
summary: Ruby bindings for ipfs.
|
data/LICENSE.txt
DELETED
@@ -1,21 +0,0 @@
|
|
1
|
-
The MIT License (MIT)
|
2
|
-
|
3
|
-
Copyright (c) 2016 James Chang
|
4
|
-
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
7
|
-
in the Software without restriction, including without limitation the rights
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
10
|
-
furnished to do so, subject to the following conditions:
|
11
|
-
|
12
|
-
The above copyright notice and this permission notice shall be included in
|
13
|
-
all copies or substantial portions of the Software.
|
14
|
-
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
-
THE SOFTWARE.
|
data/bin/setup
DELETED
data/lib/schwifty/version.rb
DELETED