coinsync 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 59e420867e256a9ed7fef1d87ae08279e7e4dfcc
+  data.tar.gz: 69fbe095ab411e0a0a65223063bb30267af81682
+SHA512:
+  metadata.gz: fa574e1fcecefe950ab83f576754e24ae4a867bf913ff2ea34cdf0673149e6787edc75663d6500500dec5e8e668fb5a6e34163e0ee106395296d6beef721cc11
+  data.tar.gz: 8f60e49910ffc061406cf20f7186f31bd0246e2c663b69753f343daf87c5b155b2a82d0039095c36ebf1ccea0bf559cba4b9c3b2eaacf8cbcbbd273206bc670b

data/MIT-LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Kuba Suder
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,255 @@
+# CoinSync
+
+CoinSync is a command-line tool for crypto traders written in Ruby that helps you import data like your transaction histories from multiple exchanges, convert it into a single unified format and process it in various ways.
+
+
+## IMPORTANT ⚠️
+
+These tools are provided without any warranty (just like the license says), and I cannot guarantee that they work correctly. The project is currently in an alpha stage, so a lot of parts might be incomplete, might not do error handling properly, not take various edge cases into account, and generally might fail in unexpected ways. And don't even look at the test suite…
+
+Basically, please verify and double-check any data you get from these scripts before you use it for anything remotely serious. You take full responsibility for any problems you might run into if you use them e.g. for your crypto tax calculations.
+
+
+## Assumptions
+
+The tool makes several assumptions about how it calculates things - if they are different than what you expect, you might have to work around them by writing some code yourself based on the provided classes. For example:
+
+- fiat amounts are rounded to 2 decimal digits (4 digits for numbers smaller than 10), and crypto amounts are rounded to 8 digits
+- exchange fees are simply added/subtracted from the amounts, i.e. they're treated as if you simply bought less units of an asset or paid more for it
+- withdrawal fees are ignored
+- for crypto-to-crypto transactions (also called "swaps" here), the base currency is determined automatically if possible, e.g. for a BTC-XMR pair XMR will always be the asset you buy/sell and BTC the currency you buy/sell it for
+
+
+## Installation
+
+You're going to need some version of Ruby installed, but any fairly recent version should do.
+
+To just install the gem in the system, call:
+
+    $ gem install coinsync
+
+You might need to add `sudo` if you don't use any Ruby version manager like [RVM](https://rvm.io), though it's better if you get one.
+
+### Installing with a Gemfile
+
+Alternatively, you can add it to your application's Gemfile:
+
+```ruby
+gem 'coinsync'
+```
+
+And then call `bundle` to install it, and `bundle exec coinsync` to run it.
+
+You can also create a project directory with a Gemfile specifically for CoinSync:
+
+```
+gem install bundle
+mkdir mycrypto
+cd mycrypto
+bundle init
+echo "gem 'coinsync'" >> Gemfile
+bundle
+bundle exec coinsync
+```
+
+This also lets you install the gem locally in that directory, by adding a path argument to `bundle`, e.g. `bundle --path ./bundle`.
+
+### Installing development version
+
+To use the latest development version, check out the repository to a local directory:
+
+```
+git clone git@github.com:mackuba/coinsync.git
+```
+
+And then run `bin/coinsync` from inside that directory.
+
+
+## Configuration
+
+To use CoinSync, you will need to create a config file first. By default it looks for the config file at `config.yml` in the current directory, though you can specify a different path with `-c path/to/config`.
+
+The config uses the [YAML format](http://yaml.org), and the expected structure is:
+
+```
+sources:
+  bittrex:
+    type: bittrex
+    file: data/bittrex.csv
+  kraken:
+    type: kraken
+    file: data/kraken-ledgers.csv
+  bitbay_api:
+    file: data/bitbay.json
+    api_public_key: 12345-678-90abc
+    api_private_key: 45678-90a-bcdef
+settings:
+  timezone: Europe/Warsaw
+  time_format: "%Y-%m-%d %H:%M"
+  column_separator: ";"
+  decimal_separator: ","
+  convert_to: PLN
+include:
+  - extras.rb
+```
+
+### Sources
+
+The `sources` list is a map of { key => definition } entries which lists the exchanges from which you want to import the transaction history and other data. There are generally two types of sources:
+
+- those that import a CSV file that you've downloaded yourself from your profile on the exchange, and don't connect to the exchange API at all
+- those that can connect to an exchange's API and download its transaction history and other data like current balances
+
+The key is any valid identifier you want to use to refer to this source, and the definition is a list of parameters:
+
+- `type` specifies which importer module to use
+- `file` specifies from where it should load a file you've downloaded, or where it should save the transaction history it imports itself
+
+Depending on the exchange, some sources might allow or require additional parameters like API keys or addresses. You can have more than one source of the same type, if you want to import multiple transaction histories from one exchange.
+
+If the type of the importer is the same as the source key, the `type` parameter might be skipped. If no other parameters are needed, you can also use a shorthand format, passing just a filename instead of a full hash:
+
+    kraken: kraken-ledgers.csv
+
+See the separate ["Importers"](doc/importers.md) doc file for a full list of supported exchanges.
+
+### Settings
+
+`settings` is an optional section that lets you change the behavior of the tool in various ways. Here's the list of currently supported keys:
+
+- `base_cryptocurrencies`: an array listing which cryptocurrencies might be considered the base currency for a trading pair; if both sides of the pair are included in the list, the one earlier in the list takes priority (default: `['USDT', 'BTC', 'ETH', 'BNB', 'KCS', 'LTC', 'BCH', 'NEO']`)
+- `column_separator`: what character is used to separate columns in saved CSV files (default: `","`)
+- `convert_to`: what fiat currency should fiat amounts be converted to (default: none)
+- `convert_with`: what currency converter module should be used to do the currency conversions (default: `fixer`)
+- `decimal_separator`: what character is used to separate decimal digits in numbers (default: `"."`)
+- `time_format`: the [time format string](http://ruby-doc.org/core-2.5.0/Time.html#method-i-strftime) to use when printing dates (default: `"%Y-%m-%d %H:%M:%S"`)
+- `timezone`: an explicit timezone to use for printing dates and currency conversion (default: system timezone)
+
+### Includes
+
+If you want to extend the tool with support for additional importers, build tasks, currency converters etc., you can add an `include` key to the config and list there any local Ruby files you want to be loaded when CoinSync runs.
+
+
+### Currency conversion
+
+If you make transactions in multiple fiat currencies (e.g. USD on Bitfinex, EUR on Kraken) and you want to have all values converted to one currency (for example, to calculate profits for tax purposes), use the `convert_to` and `convert_with` options in the settings. Currency conversion is done using pluggable modules that load currency rates from specific sources. Currently, two are available:
+
+- `fixer` loads exchange rates from [fixer.io](http://fixer.io) API (note: they've now decided to deprecate this API in June and the new one requires an API key, let me know if you know any better option)
+- `nbp` loads rates from [Polish National Bank](http://www.nbp.pl/home.aspx?f=/statystyka/kursy.html) (this will be moved to a separate gem)
+
+You can always write another module that connects to your preferred source and plug it in using `include`.
+
+
+## Using the tool
+
+Once you have a config file, you can run one of the commands described below to import or process your data:
+
+
+### Balance
+
+```
+coinsync balance [sources...]
+```
+
+This connects to the exchange APIs using any importers that support it (and where you've provided the keys), downloads the wallet balances, and prints them in a list like this:
+
+```
+Coin   |      binance          kucoin    |     TOTAL
+-------------------------------------------------------
+BTC    |          0.1 (+)         0.1    |          0.2
+ETH    |             10.5                |         10.5
+LTC    |                           20    |           20
+NANO   |               15          25    |           40
+XMR    |               33         1.8    |         34.8
+```
+
+The "(+)" means that some amount of funds is locked in an open order.
+
+You can filter the list of sources by listing one or more source names as arguments, or by listing source names with a '^' symbol (e.g. `^kucoin`) to *exclude* them.
+
+
+### Import
+
+```
+coinsync import [sources...]
+```
+
+This connects to the exchange APIs using any importers that support it (and where you've provided the keys) and downloads the transaction histories to specified files, which are later used in the build tasks below. Again, you can choose or exclude specific sources as above.
+
+
+### Build
+
+```
+coinsync build <task>
+```
+
+The build tasks process all downloaded transaction history files, combine them into a single chronologically sorted list in memory, and then output it in a selected format or run some additional calculations on it:
+
+
+#### Build List
+
+```
+coinsync build list
+```
+
+This will just print all your transactions to a single unified CSV file (in `build/list.csv`).
+
+
+#### Build Raw
+
+```
+coinsync build raw
+```
+
+This prints a list of transactions in a way similar to `build list`, but in the same format as CoinSync stores transactions internally in memory: each transaction, no matter the type or source, is stored in exactly the same way - the amount and code of the fiat or crypto currency received (bought) and the amount & code of the fiat/crypto currency paid (sold). This means that when e.g. buying BTC for USD, the bought currency will be BTC and the sold one will be USD, and when selling BTC for USD, it will be the other way around (bought: USD, sold: BTC).
+
+This output is mostly meant if you intend to use it as input for some other tools written in other languages and process the data further there.
+
+The columns in the CSV are:
+
+- exchange: name of the exchange or source
+- date: transaction date
+- bought amount: amount of the bought (received) asset/currency
+- bought currency: code of the bought (received) asset/currency
+- sold amount: amount of the asset/currency you sold (paid in)
+- sold currency: code of the asset/currency you sold (paid in)
+
+
+#### Build Summary
+
+```
+coinsync build summary
+```
+
+This will calculate how much of each traded asset you're supposed to have right now (this will differ more or less from the actual amounts because of transaction and withdrawal fees paid, payments/donations or mined coins you haven't counted, etc.):
+
+```
+Coin       Amount
+---------------------
+BTC               0.5
+BCH                 0
+ETH              12.5
+LTC                40
+NEO                15
+IOTA             1800
+```
+
+#### Custom build tasks
+
+You can output a compiled transactions list in any other format you specifically need, by creating a custom output class inheriting from `CoinSync::Outputs::Base` based on the [provided output classes](lib/coinsync/outputs) and loading it with the `include` option in the config.
+
+
+### Run command
+
+```
+coinsync run <source> <command> [args...]
+```
+
+Some importers (currently only Binance) may have custom commands implemented that only make sense for a given importer. This allows you to run these commands from the command line. See the ["Importers"](doc/importers.md) doc for more info.
+
+
+## Credits & contributing
+
+Copyright © 2018 [Kuba Suder](https://mackuba.eu). Licensed under [MIT License](http://opensource.org/licenses/MIT).
+
+If anything doesn't work as expected, please [file a bug report](https://github.com/mackuba/coinsync/issues), and any contributions in the form of pull requests (especially adding support for new exchanges) are [very welcome](https://github.com/mackuba/coinsync/pulls).

data/bin/coinsync

@@ -0,0 +1,78 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'optparse'
+
+require 'coinsync/config'
+require 'coinsync/balance_task'
+require 'coinsync/build_task'
+require 'coinsync/import_task'
+require 'coinsync/run_command_task'
+require 'coinsync/source_filter'
+
+def load_config(options)
+  CoinSync::Config.load_from_file(options[:config_file])
+end
+
+options = {}
+
+OptionParser.new do |opts|
+  opts.on('-cCONFIG', '--config CONFIG') { |c| options[:config_file] = c }
+
+  opts.parse!
+end
+
+command = ARGV.shift
+
+case command
+when nil
+  puts "Usage:"
+  puts "  coinsync balance [source1 source2 ^excluded_source...]"
+  puts "    - imports and prints wallet balances from all or selected sources"
+  puts
+  puts "  coinsync import [source1 source2 ^excluded_source...]"
+  puts "    - imports transaction histories from all or selected sources to files listed in the config"
+  puts
+  puts "  coinsync build list"
+  puts "    - merges all transaction histories into a single list and saves it to build/list.csv"
+  puts
+  puts "  coinsync build fifo"
+  puts "    - merges all transaction histories into a single list, calculates transaction"
+  puts "      profits using FIFO and saves the result to build/fifo.csv"
+  puts
+  puts "  coinsync build summary"
+  puts "    - merges all transaction histories into a single list and calculates how many"
+  puts "      units of each token you should have in total now"
+  puts
+  puts "  coinsync run <source> <command> [args]"
+  puts "    - executes a custom action from one of the configured importers (see docs for more info)"
+  puts
+  puts "  * add -c file.yml / --config file.yml to use a custom config path instead of config.yml"
+  puts
+
+when 'balance'
+  selected, except = CoinSync::SourceFilter.new.parse_command_line_args(ARGV)
+  task = CoinSync::BalanceTask.new(load_config(options))
+  task.run(selected, except)
+
+when 'import'
+  selected, except = CoinSync::SourceFilter.new.parse_command_line_args(ARGV)
+  task = CoinSync::ImportTask.new(load_config(options))
+  task.run(selected, except)
+
+when 'build'
+  output_name = ARGV.shift
+  task = CoinSync::BuildTask.new(load_config(options))
+  task.run(output_name, ARGV)
+
+when 'run'
+  source = ARGV.shift or (puts "Usage: coinsync run <source> <command> [args]"; exit 1)
+  command = ARGV.shift or (puts "Usage: coinsync run <source> <command> [args]"; exit 1)
+
+  task = CoinSync::RunCommandTask.new(load_config(options))
+  task.run(source, command, ARGV)
+
+else
+  raise "Unknown command #{command}"
+
+end

data/doc/importers.md

@@ -0,0 +1,201 @@
+## Supported exchanges / importers
+
+### Ark Voting Rewards (`ark_voting`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- address: your Ark address
+
+The importer loads a list of transactions sent to a given address from the Ark explorer, picks those that are voting rewards sent from the delegates, and saves them as purchases made with zero cost.
+
+### Binance API (`binance_api`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- api_key: API key
+- secret_key: secret API key
+- traded_pairs: a list of all pairs you trade there
+
+Create the API keys on your Binance profile page. Only the "Read Info" permission is required.
+
+Unfortunately, the Binance API currently doesn't allow loading transaction history for all pairs in one go, and checking all possible pairs would take too much time, so you need to explicitly specify the list of pairs to be downloaded, in such format:
+
+```
+traded_pairs:
+  - XRPBTC
+  - ETHBTC
+  - NANOETH
+  - BTCUSDT
+  - LTCUSDT
+```
+
+The Binance importer has a custom command that you can use to generate this list. This task scans all available trading pairs and finds those with some trades present on your account. It may take about 5-10 minutes to complete, that's why this isn't done automatically during the import.
+
+```
+coinsync run binance_api find_all_pairs
+```
+
+
+### BitBay API (`bitbay_api`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- api_public_key: API public key
+- api_private_key: API secret key
+
+Create the API keys on your BitBay profile page. You will need to select the "History" permission for transaction history, and "Crypto deposit" + "Updating a wallets list" for downloading balances.
+
+### BitBay 2.0 (`bitbay20`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+This is meant for importing transaction history from the older version of BitBay (before September 2017), available at <https://old.bitbay.net>.
+
+### Bitcurex (`bitcurex`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+This is used to import transaction history from the late Polish Bitcurex exchange (RIP in peace), in case you happen to have downloaded one before the owners disappeared with all the money.
+
+### Bittrex API (`bittrex_api`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- api_key: API public key
+- api_secret: API secret key
+
+Note: this importer can **_only_** import wallet balances. Unfortunately, the "get order history" endpoint in the Bittrex API has the same problem as the "Orders" page on your profile page: it only lists orders from last month or so, and any older orders are cleaned up. This makes it pretty much useless for our purposes. The only way to download a complete transaction history seems to be by manually pressing the (captcha-protected) "Load All" button on the site, which returns a CSV file that can be parsed with the `bittrex_csv` importer.
+
+The API key can be created on your Bittrex profile page - you only need the "Read Info" permission.
+
+### Bittrex CSV (`bittrex_csv`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+This parses the full transaction history CSV, downloaded using the "Load All" button on the History page.
+
+### Changelly (`changelly`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+Download the history from the My Account / History page, using the "Export .csv" button.
+
+### Circle (`circle`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+You can get the transaction history from the Settings / Advanced page - it will be sent to you via email.
+
+### Default (`default`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+This is a generic CSV format that you can use to import a list of random transactions from any other sources.
+
+The expected columns in the CSV are:
+
+- number: an incrementing integer number (ignored, it's just to make the file more readable)
+- exchange: name of the exchange or source
+- type: "Purchase" or "Sale"
+- date: date in a recognizable format
+- amount: amount of the asset bought or sold (uses decimal separator defined in the settings)
+- value: amount received or paid for the asset (as above)
+- currency: currency in which you've paid for the asset
+
+Any other columns after that are ignored.
+
+To list crypto-to-crypto transactions in the CSV (e.g. BTC-NANO), put the base currency in the "currency" column with a "$" prefix (e.g. `$BTC`) and the other one as asset.
+
+Airdrops and mined coins can be listed as a purchase with value 0 and an empty "currency" field.
+
+### EtherDelta / ForkDelta (`etherdelta`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+To download a transaction history, use the [DeltaBalances tool](https://deltabalances.github.io), look up your transactions on the History page specifying the time range you need, and then download the "Default" CSV in the top-right section.
+
+### Kraken API (`kraken_api`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- api_key: API public key
+- private_key: API secret key
+
+Create the API key on your Kraken profile page. Select the "Query Funds" permission to import balances and "Query Ledger Entries" to import transactions.
+
+### Kraken CSV (`kraken_csv`)
+
+Connects to the API: **NO**
+
+Parameters:
+
+- file: path from where the transaction history will be loaded
+
+The Kraken importer expects a "Ledgers" CSV file downloaded from the Kraken's History page.
+
+### Kucoin API (`kucoin`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- api_key: API public key
+- api_secret: API secret key
+
+Create the API keys on your Kucoin profile page.
+
+**Warning**: there's currently no way to create an API key on Kucoin with partial permissions; any key you create will have full access to your account, including placing orders and making withdrawals. Be careful with it.
+
+### Lisk Voting Rewards (`lisk_voting`)
+
+Connects to the API: **YES**
+
+Parameters:
+
+- file: path where the transaction history will be saved
+- address: your Lisk address
+
+The importer loads a list of transactions sent to a given address from the Lisk explorer, picks those that are voting rewards sent from the delegates, and saves them as purchases made with zero cost.