schwifty 0.0.0 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +4 -9
- data/.rspec +1 -2
- data/.rubocop.yml +17 -0
- data/.ruby-version +1 -0
- data/.travis.yml +1 -2
- data/Gemfile +11 -2
- data/README.md +55 -21
- data/Rakefile +10 -4
- data/bin/console +6 -10
- data/bin/schwifty +7 -0
- data/lib/schwifty.rb +2 -5
- data/lib/schwifty/cli.rb +51 -0
- data/lib/schwifty/client.rb +31 -0
- data/lib/schwifty/commands/add.rb +47 -0
- data/lib/schwifty/commands/bootstrap.rb +39 -0
- data/lib/schwifty/commands/gc.rb +39 -0
- data/lib/schwifty/commands/get.rb +21 -0
- data/lib/schwifty/content/defaults.rb +12 -0
- data/lib/schwifty/content/hash_file.rb +50 -0
- data/lib/schwifty/content/objects_file.rb +25 -0
- data/lib/schwifty/content/parsers.rb +20 -0
- data/lib/schwifty/content/pinned_file.rb +31 -0
- data/lib/schwifty/info.rb +7 -0
- data/schwifty.gemspec +16 -14
- metadata +43 -28
- data/LICENSE.txt +0 -21
- data/bin/setup +0 -8
- data/lib/schwifty/version.rb +0 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2f96cb81f4a5fb3d0ca084725d860f86961c183
|
4
|
+
data.tar.gz: 601aac7b592f8bebaa8a3d58ffbf50676af3cecc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 640b9aa83d011135e7729a35743e6b2b2fb2e86a25e7380c4635353e8f70447305be171a7a8977bc65cd4d32181d648666a9435c82ec5f3ba52b971b789f6e24
|
7
|
+
data.tar.gz: 77c301fb13c23543672a471b39c5e1981cd61f309058ae91c93018418b66473923b1a219d641b67b2ef87e1e99ea288781f16a393b9ff260e1f2900e6cfe3e51
|
data/.gitignore
CHANGED
data/.rspec
CHANGED
@@ -1,2 +1 @@
|
|
1
|
-
--format documentation
|
2
|
-
--color
|
1
|
+
--color --format documentation
|
data/.rubocop.yml
ADDED
data/.ruby-version
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
2.3.1
|
data/.travis.yml
CHANGED
data/Gemfile
CHANGED
@@ -1,4 +1,13 @@
|
|
1
|
-
source 'https://rubygems.org'
|
1
|
+
source 'https://rubygems.org' do
|
2
|
+
group :development do
|
3
|
+
gem 'pry', '~>0.10.3'
|
4
|
+
gem 'rubocop', '~>0.40.0'
|
5
|
+
end
|
6
|
+
|
7
|
+
group :test do
|
8
|
+
gem 'rake', '~>11.1.2'
|
9
|
+
gem 'rspec', '~>3.4.0'
|
10
|
+
end
|
11
|
+
end
|
2
12
|
|
3
|
-
# Specify your gem's dependencies in schwifty.gemspec
|
4
13
|
gemspec
|
data/README.md
CHANGED
@@ -1,41 +1,75 @@
|
|
1
|
-
#
|
1
|
+
# schwifty
|
2
2
|
|
3
|
-
|
3
|
+
[ipfs][1] is a distributed file system that is content-addressed and uses keys (hashes) to files
|
4
|
+
(objects). This repo uses the hash/object terminology consistent with the rest of ipfs docs.
|
4
5
|
|
5
|
-
|
6
|
+
## Usage
|
6
7
|
|
7
|
-
|
8
|
+
```bash
|
9
|
+
schwifty add <files>...
|
10
|
+
schwifty bootstrap (--clear | <nodes>... | --file=<bootstrap_list_yaml>)
|
11
|
+
schwifty get <files_or_hashes>...
|
12
|
+
schwifty gc
|
13
|
+
schwifty -h | --help
|
14
|
+
schwifty --version
|
15
|
+
```
|
8
16
|
|
9
|
-
|
17
|
+
## Hash Files
|
10
18
|
|
11
|
-
|
12
|
-
|
13
|
-
|
19
|
+
Because ipfs does not have a typical Unix file system structure, we keep track of files added using
|
20
|
+
the `ipfs add` operation. This task is entrusted to two files, one of which any client code needs to
|
21
|
+
interact with and the other for garbage collection purposes. Both of these files are in `.yaml` form.
|
14
22
|
|
15
|
-
|
23
|
+
For the examples below, we'll use the convention in `bin/console` where `client = IPFS::Client`.
|
16
24
|
|
17
|
-
|
25
|
+
### Objects File: `ipfs_objects.yaml`
|
18
26
|
|
19
|
-
|
27
|
+
It's your job read and manipulate this file and pass it between steps of your build/test pipeline.
|
28
|
+
The objects file contains a mapping of filename : hash
|
20
29
|
|
21
|
-
|
30
|
+
`client.add('spec/samples/files/file1')`
|
22
31
|
|
23
|
-
|
32
|
+
```yaml
|
33
|
+
---
|
34
|
+
spec/samples/files/file1: QmXMSyJvaz912Wi6533MegwUn4mJ4kQikaBZpAdeFwoWkj
|
35
|
+
```
|
24
36
|
|
25
|
-
|
37
|
+
`client.get('spec/samples/files/file1')` will download the file to the current directory.
|
38
|
+
Both `add` and `get` can take multiple files.
|
26
39
|
|
27
|
-
|
40
|
+
### Pinned Objects File: `~/.ipfs/ipfs_pinned_objects.yaml`
|
28
41
|
|
29
|
-
|
42
|
+
A hash : timestamp is written in this file every time `ipfs add` is run. `ipfs add` pins objects by
|
43
|
+
default. You aren't expected to interact with this file, but it may help in debugging purposes.
|
30
44
|
|
31
|
-
|
45
|
+
```yaml
|
46
|
+
---
|
47
|
+
Qmb1CkJmfpwmyLvuPMMRKKpYPuiwUMW9V1WdbFLVZgAdpL: 2016-06-01 01:04:49 UTC
|
48
|
+
QmZXYwLpuGd8kR51FU5f8U5M8M1LHPSUxMeFBtrNWwJGdE: 2016-05-31 12:41:19 UTC
|
49
|
+
```
|
32
50
|
|
33
|
-
|
51
|
+
Garbage collection (GC) follows these rules:
|
52
|
+
1. Check for disk space. If disk space is below the threshold (0.85), do not GC.
|
53
|
+
2. Run `ipfs repo gc` to remove all unpinned objects.
|
54
|
+
3. Check for disk space. If disk space is above the threshold, go to step 4.
|
55
|
+
4. Unpin one third of objects from the Pinned Objects File.
|
56
|
+
5. Run `ipfs repo gc` to remove the unpinned objects.
|
57
|
+
6. Back to step 3.
|
34
58
|
|
35
|
-
|
59
|
+
There's no backoff in the GC unpin rate in step 4, and the gem doesn't know how much disk space will
|
60
|
+
be freed every call. The 1/3 of hashes removed each time does get smaller though.
|
36
61
|
|
62
|
+
### Bootstrapping
|
37
63
|
|
38
|
-
|
64
|
+
We recommend checking in an approved bootstrap list that, in tandem with the CLI, bootstraps nodes
|
65
|
+
via a configuration managed provisioning tool (like Ansible). The CLI supports bootstrapping via
|
66
|
+
a `.yaml` file with hostname : multiaddr/peerID formats. See `ipfs bootstrap` and `ipfs id` docs
|
67
|
+
for more info.
|
39
68
|
|
40
|
-
|
69
|
+
```yaml
|
70
|
+
---
|
71
|
+
hostname1: /ip4/127.0.0.1/tcp/4001/ipfs/QaowiA9HvLCinkCLwRFauHkAZUP3DQDogku98r9BctEhgc
|
72
|
+
hostname2: /ip4/192.112.0.101/tcp/4001/ipfs/QmcpER9AOSUinkCLwRFi8zkAZU73DQDogkF98r9BctEhgc
|
73
|
+
```
|
41
74
|
|
75
|
+
[1]: https://ipfs.io/
|
data/Rakefile
CHANGED
@@ -1,6 +1,12 @@
|
|
1
|
-
|
2
|
-
require "rspec/core/rake_task"
|
1
|
+
# encoding: utf-8
|
3
2
|
|
4
|
-
|
3
|
+
require 'bundler/setup'
|
4
|
+
require 'bundler/gem_tasks'
|
5
5
|
|
6
|
-
task :
|
6
|
+
task default: %w(spec rubocop)
|
7
|
+
|
8
|
+
require 'rspec/core/rake_task'
|
9
|
+
RSpec::Core::RakeTask.new
|
10
|
+
|
11
|
+
require 'rubocop/rake_task'
|
12
|
+
RuboCop::RakeTask.new
|
data/bin/console
CHANGED
@@ -1,14 +1,10 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
require
|
4
|
-
require
|
3
|
+
require 'bundler/setup'
|
4
|
+
require 'schwifty'
|
5
|
+
require 'pry'
|
5
6
|
|
6
|
-
#
|
7
|
-
|
7
|
+
# Debugging shortcuts
|
8
|
+
client = Schwifty::Client
|
8
9
|
|
9
|
-
|
10
|
-
# require "pry"
|
11
|
-
# Pry.start
|
12
|
-
|
13
|
-
require "irb"
|
14
|
-
IRB.start
|
10
|
+
binding.pry
|
data/bin/schwifty
ADDED
data/lib/schwifty.rb
CHANGED
data/lib/schwifty/cli.rb
ADDED
@@ -0,0 +1,51 @@
|
|
1
|
+
require 'docopt'
|
2
|
+
|
3
|
+
require 'schwifty/client'
|
4
|
+
require 'schwifty/info'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
# This class acts as the command line interface for the IPFS client. Args are parsed using the
|
8
|
+
# docopt template below.
|
9
|
+
class CLI
|
10
|
+
def run
|
11
|
+
opts = parse_args
|
12
|
+
|
13
|
+
if opts['add']
|
14
|
+
Schwifty::Client.add(*opts['<files>'])
|
15
|
+
elsif opts['bootstrap']
|
16
|
+
Schwifty::Client.bootstrap(*opts['<nodes>'], clear: opts['--clear'], filename: opts['--file'])
|
17
|
+
elsif opts['get']
|
18
|
+
Schwifty::Client.add(*opts['<files_or_hashes>'])
|
19
|
+
elsif opts['gc']
|
20
|
+
Schwifty::Client.gc
|
21
|
+
elsif opts['--version']
|
22
|
+
puts "schwifty version #{Schwifty::VERSION}"
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
# Modify this to change how the CLI works -- see http://docopt.org/
|
27
|
+
def parse_args
|
28
|
+
doc = <<DOCOPT
|
29
|
+
schwifty saves and downloads objects from ipfs, keeping track of their hashes in a garbage collection file in ~/.ipfs/ipfs_pinned_objects.yaml and an objects file in ./ipfs_objects.yaml
|
30
|
+
|
31
|
+
Usage:
|
32
|
+
schwifty add <files>...
|
33
|
+
schwifty bootstrap (--clear | <nodes>... | --file=<bootstrap_list_yaml>)
|
34
|
+
schwifty get <files_or_hashes>...
|
35
|
+
schwifty gc
|
36
|
+
schwifty -h | --help
|
37
|
+
schwifty --version
|
38
|
+
|
39
|
+
Options:
|
40
|
+
-h --help Show this screen.
|
41
|
+
--version Show version.
|
42
|
+
DOCOPT
|
43
|
+
begin
|
44
|
+
Docopt.docopt(doc)
|
45
|
+
rescue Docopt::Exit => e
|
46
|
+
puts e.message
|
47
|
+
exit
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
|
3
|
+
require 'schwifty/commands/add'
|
4
|
+
require 'schwifty/commands/bootstrap'
|
5
|
+
require 'schwifty/commands/gc'
|
6
|
+
require 'schwifty/commands/get'
|
7
|
+
|
8
|
+
module Schwifty
|
9
|
+
# This class calls all the ipfs commands for the CLI or other use.
|
10
|
+
class Client
|
11
|
+
def self.add(*files)
|
12
|
+
Commands::Add.run(*files)
|
13
|
+
nil
|
14
|
+
end
|
15
|
+
|
16
|
+
def self.bootstrap(*nodes, clear: false, filename: nil)
|
17
|
+
Commands::Bootstrap.run(*nodes, clear: clear, filename: filename)
|
18
|
+
nil
|
19
|
+
end
|
20
|
+
|
21
|
+
def self.gc
|
22
|
+
Commands::GC.run
|
23
|
+
nil
|
24
|
+
end
|
25
|
+
|
26
|
+
def self.get(*filenames_or_hashes, dst: '.')
|
27
|
+
Commands::Get.run(*filenames_or_hashes, dst)
|
28
|
+
nil
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'time'
|
3
|
+
|
4
|
+
require 'schwifty/content/objects_file'
|
5
|
+
require 'schwifty/content/parsers'
|
6
|
+
require 'schwifty/content/pinned_file'
|
7
|
+
|
8
|
+
module Schwifty
|
9
|
+
module Commands
|
10
|
+
# This class adds objects to ipfs and tracks their hashes using the Objects File and Pinned
|
11
|
+
# Objects File.
|
12
|
+
class Add
|
13
|
+
def self.run(*filenames)
|
14
|
+
result_filenames = []
|
15
|
+
result_hashes = []
|
16
|
+
|
17
|
+
# Parse add command outputs
|
18
|
+
filenames.each do |name|
|
19
|
+
parsed_filenames, parsed_hashes = Dir.exist?(name) ? add_dir(name) : add_file(name)
|
20
|
+
result_filenames += parsed_filenames
|
21
|
+
result_hashes += parsed_hashes
|
22
|
+
end
|
23
|
+
|
24
|
+
update_hash_files(result_filenames, result_hashes)
|
25
|
+
end
|
26
|
+
|
27
|
+
private_class_method
|
28
|
+
|
29
|
+
def self.add_dir(name)
|
30
|
+
output = fail_on_error('ipfs', 'add', '-r', name)
|
31
|
+
parsed_filenames, parsed_hashes = Schwifty::Content::AddParser.run(output)
|
32
|
+
parsed_filenames.map! { |tail| File.join(File.dirname(name), tail) }
|
33
|
+
[parsed_filenames, parsed_hashes]
|
34
|
+
end
|
35
|
+
|
36
|
+
def self.add_file(name)
|
37
|
+
[[name], [fail_on_error('ipfs', 'add', '-q', name)]]
|
38
|
+
end
|
39
|
+
|
40
|
+
def self.update_hash_files(filenames, hashes)
|
41
|
+
Schwifty::Content::ObjectsFile.new.add_entries(filenames, hashes)
|
42
|
+
Schwifty::Content::PinnedFile.new.
|
43
|
+
add_entries(hashes, [Time.now.getutc.to_s] * hashes.size, allow_collisions: true)
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'yaml'
|
3
|
+
|
4
|
+
module Schwifty
|
5
|
+
module Commands
|
6
|
+
# Bootstrapping individual worker nodes. '<multiaddr>/<peerID>' format required.
|
7
|
+
class Bootstrap
|
8
|
+
def self.run(*nodes, clear: false, filename: nil)
|
9
|
+
if clear
|
10
|
+
bootstrap_clear
|
11
|
+
elsif filename
|
12
|
+
bootstrap_file(filename)
|
13
|
+
else
|
14
|
+
bootstrap_nodes(*nodes)
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
private_class_method
|
19
|
+
|
20
|
+
def self.bootstrap_clear
|
21
|
+
existing_nodes = fail_on_error('ipfs', 'bootstrap', 'list').split("\n")
|
22
|
+
if existing_nodes.empty?
|
23
|
+
puts "#{self.class.name}: No nodes in bootstrap list!"
|
24
|
+
else
|
25
|
+
fail_on_error('ipfs', 'bootstrap', 'rm', *existing_nodes)
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
# Bootstrap file is a mapping of hostname : '<multiaddr>/<peerID>' formats.
|
30
|
+
def self.bootstrap_file(filename)
|
31
|
+
bootstrap_nodes(*YAML.load_file(filename).values)
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.bootstrap_nodes(*nodes)
|
35
|
+
fail_on_error('ipfs', 'bootstrap', 'add', *nodes)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
require 'sys/filesystem'
|
3
|
+
|
4
|
+
require 'schwifty/content/pinned_file'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
module Commands
|
8
|
+
# Garbage collection to be called via CLI. If there's not enough disk space after a blind call
|
9
|
+
# to GC unpinned objects, we GC pinned objects as well. Pinned GC works by removing hashes
|
10
|
+
# repeatedly until we're below the disk space threshold. We get hashes to remove from the
|
11
|
+
# Pinned Objects File.
|
12
|
+
class GC
|
13
|
+
def self.run
|
14
|
+
return unless disk_space_exceeds_threshold?
|
15
|
+
gc
|
16
|
+
|
17
|
+
while disk_space_exceeds_threshold?
|
18
|
+
unpin(Schwifty::Content::PinnedFile.new.cull)
|
19
|
+
gc
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
private_class_method
|
24
|
+
|
25
|
+
def self.disk_space_exceeds_threshold?
|
26
|
+
stat = Sys::Filesystem.stat('/')
|
27
|
+
(stat.blocks - stat.blocks_free) / stat.blocks.to_f > Schwifty::Content::Defaults::GC_DISK_THRESHOLD
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.gc
|
31
|
+
fail_on_error('ipfs', 'repo', 'gc')
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.unpin(hashes)
|
35
|
+
fail_on_error('ipfs', 'pin', 'rm', *hashes)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'build_execution'
|
2
|
+
|
3
|
+
require 'schwifty/content/objects_file'
|
4
|
+
|
5
|
+
module Schwifty
|
6
|
+
module Commands
|
7
|
+
# Get allows us to download valid files in the Objects File or try a hash directly.
|
8
|
+
class Get
|
9
|
+
def self.run(*filenames_or_hashes, dst: '.')
|
10
|
+
filenames_or_hashes.each do |key|
|
11
|
+
key = Schwifty::Content::ObjectsFile.new.get(key) unless hash?(key)
|
12
|
+
fail_on_error('ipfs', 'get', key, "--output='#{dst}'")
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
private_class_method def self.hash?(str)
|
17
|
+
str.length == 46 && str[/[a-zA-Z0-9]+/] == str
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,12 @@
|
|
1
|
+
module Schwifty
|
2
|
+
module Content
|
3
|
+
module Defaults
|
4
|
+
OBJECTS_FILE_NAME = 'ipfs_objects.yaml'.freeze
|
5
|
+
PINNED_FILE_NAME = File.join(ENV['HOME'], '.ipfs', 'ipfs_pinned_objects.yaml').freeze
|
6
|
+
|
7
|
+
# If we're above this threshold after an unpinned GC, we GC pinned objects as well.
|
8
|
+
GC_DISK_THRESHOLD = 0.85 # TODO: Tune this value
|
9
|
+
GC_CULL_THRESHOLD = 0.67 # TODO: Tune this value
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
@@ -0,0 +1,50 @@
|
|
1
|
+
require 'yaml'
|
2
|
+
|
3
|
+
module Schwifty
|
4
|
+
module Content
|
5
|
+
# IPFS file base class
|
6
|
+
class HashFile
|
7
|
+
def initialize(filename)
|
8
|
+
@filename = filename
|
9
|
+
@entries = load_yaml
|
10
|
+
end
|
11
|
+
|
12
|
+
def add_entries(keys, vals, allow_collisions: false)
|
13
|
+
keys.zip(vals).each do |key, val|
|
14
|
+
if @entries.key?(key) && !allow_collisions
|
15
|
+
raise "#{self.class.name} collision when inserting #{key}, #{val}"
|
16
|
+
end
|
17
|
+
|
18
|
+
@entries[key] = val
|
19
|
+
end
|
20
|
+
|
21
|
+
save_yaml
|
22
|
+
end
|
23
|
+
|
24
|
+
def get(key)
|
25
|
+
val = @entries[key.to_s]
|
26
|
+
raise "#{self.class.name} value does not exist for: #{key}" unless val
|
27
|
+
val
|
28
|
+
end
|
29
|
+
|
30
|
+
def remove_entry(key)
|
31
|
+
@entries.delete(key)
|
32
|
+
end
|
33
|
+
|
34
|
+
def save_yaml
|
35
|
+
File.open(@filename, 'w') { |f| f.write(@entries.to_yaml) }
|
36
|
+
end
|
37
|
+
|
38
|
+
private
|
39
|
+
|
40
|
+
def load_yaml
|
41
|
+
FileUtils.touch(@filename) unless File.exist?(@filename)
|
42
|
+
YAML.load_file(@filename) || {}
|
43
|
+
end
|
44
|
+
|
45
|
+
def num_entries
|
46
|
+
@entries.length
|
47
|
+
end
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
require 'yaml'
|
2
|
+
|
3
|
+
require 'schwifty/content/defaults'
|
4
|
+
require 'schwifty/content/hash_file'
|
5
|
+
|
6
|
+
module Schwifty
|
7
|
+
module Content
|
8
|
+
# The Objects File keeps track of filename : hash mappings. Directories added recursively will
|
9
|
+
# have all of their files' individual hashes added as well.
|
10
|
+
#
|
11
|
+
# ipfs add -r spec/samples
|
12
|
+
#
|
13
|
+
# ---
|
14
|
+
# spec/samples/files/dir1/file2: Qmenn1xp6bq5JJhE1hjsishPLeohEsWTq2GHWc2CspixZH
|
15
|
+
# spec/samples/files/file1: QmXMSyJvaz912Wi6533MegwUn4mJ4kQikaBZpAdeFwoWkj
|
16
|
+
# spec/samples/files/dir1: QmSsF4xjCAcpY3634TpKdBjHNnncWcQCrjyJqePUpdVjoX
|
17
|
+
# spec/samples/files: QmXYciy4T1dZAWp2JW2caPoK9ktJ54cuJoG8aAmkXRkJWq
|
18
|
+
#
|
19
|
+
class ObjectsFile < Schwifty::Content::HashFile
|
20
|
+
def initialize(filename = Schwifty::Content::Defaults::OBJECTS_FILE_NAME)
|
21
|
+
super(filename)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module Schwifty
|
2
|
+
module Content
|
3
|
+
# Parses ipfs add -r output to return a list of filenames and their hashes
|
4
|
+
class AddParser
|
5
|
+
def self.run(output)
|
6
|
+
filenames = []
|
7
|
+
hashes = []
|
8
|
+
|
9
|
+
output.split("\n").each do |line|
|
10
|
+
words = line.split(' ')
|
11
|
+
index = words.index('added')
|
12
|
+
hashes << words[index + 1]
|
13
|
+
filenames << words[index + 2]
|
14
|
+
end
|
15
|
+
|
16
|
+
[filenames, hashes]
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require 'time'
|
2
|
+
require 'yaml'
|
3
|
+
|
4
|
+
require 'schwifty/content/defaults'
|
5
|
+
require 'schwifty/content/hash_file'
|
6
|
+
|
7
|
+
module Schwifty
|
8
|
+
module Content
|
9
|
+
# The Pinned Objects File keeps track of hash : timestamp mappings. Remember: a file must
|
10
|
+
# remain on a single host to exist in the cluster, but we need to keep track of space usage.
|
11
|
+
#
|
12
|
+
# ---
|
13
|
+
# Qmb1CkJmfpwmyLvuPMMRKKpYPuiwUMW9V1WdbFLVZgAdpL: 2016-06-01 01:04:49 UTC
|
14
|
+
#
|
15
|
+
class PinnedFile < Schwifty::Content::HashFile
|
16
|
+
def initialize(filename = Schwifty::Content::Defaults::PINNED_FILE_NAME)
|
17
|
+
super(filename)
|
18
|
+
end
|
19
|
+
|
20
|
+
# Remove hashes from the Pinned Objects File using a threshold. Does not do any garbage
|
21
|
+
# collection on its own. Return a list of hashes to be GC'ed.
|
22
|
+
def cull(threshold = Schwifty::Content::Defaults::GC_CULL_THRESHOLD)
|
23
|
+
num_to_remove = num_entries - (threshold * num_entries).to_int
|
24
|
+
entries_to_remove = @entries.sort_by { |_hash, date| date }[0..num_to_remove].map(&:first)
|
25
|
+
entries_to_remove.each { |key| remove_entry(key) }
|
26
|
+
save_yaml
|
27
|
+
entries_to_remove
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,7 @@
|
|
1
|
+
module Schwifty
|
2
|
+
VERSION = '0.1.0'.freeze
|
3
|
+
SUMMARY = 'Ruby bindings for ipfs.'.freeze
|
4
|
+
DESCRIPTION = "Designed for distributed artifact storage, this gem keeps track of objects
|
5
|
+
added using ipfs with an objects file and also provides a garbage collection solution when a
|
6
|
+
particular node's space runs low.".freeze
|
7
|
+
end
|
data/schwifty.gemspec
CHANGED
@@ -1,24 +1,26 @@
|
|
1
|
-
#
|
1
|
+
# encoding: utf-8
|
2
2
|
lib = File.expand_path('../lib', __FILE__)
|
3
3
|
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
-
require 'schwifty/
|
4
|
+
require 'schwifty/info'
|
5
5
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
|
-
spec.name =
|
7
|
+
spec.name = 'schwifty'
|
8
8
|
spec.version = Schwifty::VERSION
|
9
|
-
spec.authors = [
|
10
|
-
spec.email = [
|
9
|
+
spec.authors = ['James Chang']
|
10
|
+
spec.email = ['longboardcat13@sgmail.com']
|
11
11
|
|
12
|
-
spec.summary =
|
13
|
-
spec.
|
14
|
-
spec.
|
12
|
+
spec.summary = Schwifty::SUMMARY
|
13
|
+
spec.description = Schwifty::DESCRIPTION
|
14
|
+
spec.homepage = 'https://github.com/longboardcat/schwifty'
|
15
15
|
|
16
16
|
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
17
|
-
spec.bindir =
|
18
|
-
spec.executables =
|
19
|
-
spec.require_paths = [
|
17
|
+
spec.bindir = 'bin'
|
18
|
+
spec.executables = ['schwifty']
|
19
|
+
spec.require_paths = ['lib']
|
20
20
|
|
21
|
-
spec.
|
22
|
-
spec.
|
23
|
-
spec.
|
21
|
+
spec.add_runtime_dependency 'docopt', '~>0.5.0'
|
22
|
+
spec.add_runtime_dependency 'sys-filesystem', '~>1.1.6'
|
23
|
+
spec.add_runtime_dependency 'build_execution'
|
24
|
+
|
25
|
+
spec.required_ruby_version = '>= 2.0.0'
|
24
26
|
end
|
metadata
CHANGED
@@ -1,79 +1,94 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: schwifty
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- James Chang
|
8
8
|
autorequire:
|
9
|
-
bindir:
|
9
|
+
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2017-05-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
|
-
name:
|
14
|
+
name: docopt
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
17
|
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version:
|
20
|
-
type: :
|
19
|
+
version: 0.5.0
|
20
|
+
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version:
|
26
|
+
version: 0.5.0
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
|
-
name:
|
28
|
+
name: sys-filesystem
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version:
|
34
|
-
type: :
|
33
|
+
version: 1.1.6
|
34
|
+
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version:
|
40
|
+
version: 1.1.6
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
|
-
name:
|
42
|
+
name: build_execution
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
|
-
- - "
|
45
|
+
- - ">="
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '
|
48
|
-
type: :
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
|
-
- - "
|
52
|
+
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '
|
55
|
-
description:
|
54
|
+
version: '0'
|
55
|
+
description: |-
|
56
|
+
Designed for distributed artifact storage, this gem keeps track of objects
|
57
|
+
added using ipfs with an objects file and also provides a garbage collection solution when a
|
58
|
+
particular node's space runs low.
|
56
59
|
email:
|
57
|
-
-
|
58
|
-
executables:
|
60
|
+
- longboardcat13@sgmail.com
|
61
|
+
executables:
|
62
|
+
- schwifty
|
59
63
|
extensions: []
|
60
64
|
extra_rdoc_files: []
|
61
65
|
files:
|
62
66
|
- ".gitignore"
|
63
67
|
- ".rspec"
|
68
|
+
- ".rubocop.yml"
|
69
|
+
- ".ruby-version"
|
64
70
|
- ".travis.yml"
|
65
71
|
- Gemfile
|
66
|
-
- LICENSE.txt
|
67
72
|
- README.md
|
68
73
|
- Rakefile
|
69
74
|
- bin/console
|
70
|
-
- bin/
|
75
|
+
- bin/schwifty
|
71
76
|
- lib/schwifty.rb
|
72
|
-
- lib/schwifty/
|
77
|
+
- lib/schwifty/cli.rb
|
78
|
+
- lib/schwifty/client.rb
|
79
|
+
- lib/schwifty/commands/add.rb
|
80
|
+
- lib/schwifty/commands/bootstrap.rb
|
81
|
+
- lib/schwifty/commands/gc.rb
|
82
|
+
- lib/schwifty/commands/get.rb
|
83
|
+
- lib/schwifty/content/defaults.rb
|
84
|
+
- lib/schwifty/content/hash_file.rb
|
85
|
+
- lib/schwifty/content/objects_file.rb
|
86
|
+
- lib/schwifty/content/parsers.rb
|
87
|
+
- lib/schwifty/content/pinned_file.rb
|
88
|
+
- lib/schwifty/info.rb
|
73
89
|
- schwifty.gemspec
|
74
|
-
homepage: https://github.com/
|
75
|
-
licenses:
|
76
|
-
- MIT
|
90
|
+
homepage: https://github.com/longboardcat/schwifty
|
91
|
+
licenses: []
|
77
92
|
metadata: {}
|
78
93
|
post_install_message:
|
79
94
|
rdoc_options: []
|
@@ -83,7 +98,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
83
98
|
requirements:
|
84
99
|
- - ">="
|
85
100
|
- !ruby/object:Gem::Version
|
86
|
-
version:
|
101
|
+
version: 2.0.0
|
87
102
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
88
103
|
requirements:
|
89
104
|
- - ">="
|
@@ -91,7 +106,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
91
106
|
version: '0'
|
92
107
|
requirements: []
|
93
108
|
rubyforge_project:
|
94
|
-
rubygems_version: 2.
|
109
|
+
rubygems_version: 2.5.1
|
95
110
|
signing_key:
|
96
111
|
specification_version: 4
|
97
112
|
summary: Ruby bindings for ipfs.
|
data/LICENSE.txt
DELETED
@@ -1,21 +0,0 @@
|
|
1
|
-
The MIT License (MIT)
|
2
|
-
|
3
|
-
Copyright (c) 2016 James Chang
|
4
|
-
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
7
|
-
in the Software without restriction, including without limitation the rights
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
10
|
-
furnished to do so, subject to the following conditions:
|
11
|
-
|
12
|
-
The above copyright notice and this permission notice shall be included in
|
13
|
-
all copies or substantial portions of the Software.
|
14
|
-
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
-
THE SOFTWARE.
|
data/bin/setup
DELETED
data/lib/schwifty/version.rb
DELETED