blz 0.2.0.20180305 → 0.2.0.20180604

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 23d2e9f80c8d7f92347738f5df1997372ea32617
-  data.tar.gz: 829f4a0f0ace3c8929a94f845c14975a2b7e7ae2
+  metadata.gz: f56ab0ea1e3baa5ffbd5cc9fac34376f96949a00
+  data.tar.gz: 2d761008414baa0d5ba0cd324a11652c648f144a
 SHA512:
-  metadata.gz: 20302a1fd1d0eb6b1944b116798cbc41b1c38dc173b07faf313b7be934aab3f9e3bb276aef2daca68ba08d223ef7bfd88ac366de715c438f021e1c65cf7c22a7
-  data.tar.gz: 285e92bb3165ad5b48ecedb4f665c84693cfc541f2dd6797dadefb079beaf40fb6cad27431fbcb51e268dd61d713757efe0a49a2f66a39eb33484d09c83896ae
+  metadata.gz: e38a7c35894455fc7439692292bdc64b570d52cdc42336f8a8c4963cdf80e28bc51abc7d4d96a2ef6fa97fd983656fdaa4a268d477a3708cbac4e0d37f4844ed
+  data.tar.gz: 55f222f01e38faa765c81da67de9fb24c7a6af4d404ed2398d1be257a1eaefdb7941709af9915e396793d81bf817d8b4243689df93ebae133321332f1b25cfdd

data/CHANGELOG.md

@@ -1,8 +1,11 @@
+# 0.2.0.20180604, released 2018-05-04
+
+* BLZ data file for cycle 2018-06-04 to 2018-08-02 added
+
 # 0.2.0.20180305, released 2018-03-16
 
 * BLZ data file for cycle 2018-03-05 to 2018-06-03 added
 
-
 # 0.2.0.20171204, released 2017-05-08
 
 * BLZ data file for cycle 2017-12-04 to 2018-03-04 added

data/Gemfile

@@ -6,3 +6,7 @@ group :test do
   gem 'test-unit'
 end
 
+group :development do
+  gem "mechanize"
+  gem "nokogiri"
+end

data/README.md

@@ -3,7 +3,7 @@
 ## Description
 
 BLZ or Bankleitzahlen is a bank identifier code system used by German
-and Austrian banks (see [Wikipedia](http://en.wikipedia.org/wiki/Bankleitzahl)).
+and Austrian banks (see [Wikipedia](https://en.wikipedia.org/wiki/Bankleitzahl)).
 
 This gem provides for searching and returning the information
 represented by the BLZ, such as the name of the bank etc.
@@ -16,45 +16,50 @@ represented by the BLZ, such as the name of the bank etc.
 
 I provided a command line tool to search for BLZ:
 
-    blz exact 70150000
-
-    70150000, Stadtsparkasse München, 80791 München, SSKMDEMMXXX
+```sh
+$ blz exact 70150000
+70150000, Stadtsparkasse München, 80791 München, SSKMDEMMXXX
+```
 
 Or search for BIC matches:
 
-    blz bic SSKMDE
-
-    70150000, Stadtsparkasse München, 80791 München, SSKMDEMMXXX
+```sh
+$ blz bic SSKMDE
+70150000, Stadtsparkasse München, 80791 München, SSKMDEMMXXX
+```
 
 Or use substring matches:
 
-    blz match 4945
-
-    49450120, Sparkasse Herford, 32045 Herford, WLAHDE44XXX
-    49450120, Sparkasse Herford, 32285 Rödinghausen, Westf
-    49450120, Sparkasse Herford, 32132 Spenge
-    49450120, Sparkasse Herford, 32269 Kirchlengern
-    49450120, Sparkasse Herford, 32122 Enger, Westf
-    49450120, Sparkasse Herford, 32591 Vlotho
-    49450120, Sparkasse Herford, 32120 Hiddenhausen
-    49450120, Sparkasse Herford, 32221 Bünde
-    49450120, Sparkasse Herford, 32556 Löhne
-    49451210, Sparkasse Bad Salzuflen -alt-, 32102 Bad Salzuflen, WELADED1BSU
+```sh
+$ blz match 4945
+49450120, Sparkasse Herford, 32045 Herford, WLAHDE44XXX
+49450120, Sparkasse Herford, 32285 Rödinghausen, Westf
+49450120, Sparkasse Herford, 32132 Spenge
+49450120, Sparkasse Herford, 32269 Kirchlengern
+49450120, Sparkasse Herford, 32122 Enger, Westf
+49450120, Sparkasse Herford, 32591 Vlotho
+49450120, Sparkasse Herford, 32120 Hiddenhausen
+49450120, Sparkasse Herford, 32221 Bünde
+49450120, Sparkasse Herford, 32556 Löhne
+49451210, Sparkasse Bad Salzuflen -alt-, 32102 Bad Salzuflen, WELADED1BSU
+```
 
 Or find by city (rather uncommon, but useful):
 
-    blz city Münstereifel
-
-    37069627, Raiffeisenbank Rheinbach Voreifel, 53902 Bad Münstereifel
-    37070024, Deutsche Bank Privat und Geschäftskunden, 53902 Bad Münstereifel, DEUTDEDB379
-    37070060, Deutsche Bank, 53902 Bad Münstereifel, DEUTDEDK379
-    38250110, Kreissparkasse Euskirchen, 53896 Bad Münstereifel
-    38260082, Volksbank Euskirchen, 53895 Bad Münstereifel
+```sh
+$ blz city Münstereifel
+37069627, Raiffeisenbank Rheinbach Voreifel, 53902 Bad Münstereifel
+37070024, Deutsche Bank Privat und Geschäftskunden, 53902 Bad Münstereifel, DEUTDEDB379
+37070060, Deutsche Bank, 53902 Bad Münstereifel, DEUTDEDK379
+38250110, Kreissparkasse Euskirchen, 53896 Bad Münstereifel
+38260082, Volksbank Euskirchen, 53895 Bad Münstereifel
+```
 
 ## Downloads
 
 You can download the current list of BLZ (free of charge)
-[from the German Bundesbank](http://www.bundesbank.de/Redaktion/DE/Standardartikel/Kerngeschaeftsfelder/Unbarer_Zahlungsverkehr/bankleitzahlen_download.html).
+[from the German Bundesbank](http://www.bundesbank.de/Redaktion/DE/Standardartikel/Kerngeschaeftsfelder/Unbarer_Zahlungsverkehr/bankleitzahlen_download.html),
+however, you need to complete their [registration process](https://extranet.bundesbank.de/bsvpub/).
 
 The Bundesbank releases a new list every three months. The last part of the
 version number of this gem mirrors the last date the provided data is valid.
@@ -63,25 +68,19 @@ Now go and build your own BLZ gem ;-)
 
 ### Note on converting
 
-One of the easiest ways to contribute is by updating the `data/*.tsv.gz`
-files. Currently, this is a 3-monthly manual process of:
-
-1. Downloading the file linked above (it helps to be subscribed to the
-  Bundesbank ExtraNet INTBA mailing list to get notifications, but the
-  [registration process](https://extranet.bundesbank.de/bsvpub/)
-  is a bit cumbersome...).
-
-2. Converting the `BLZ_yyyymmdd.xlsx` file into a tab seperated format
-  (plain old CSV format with `\t` as field seperator; omit the header).
-  Save it as `data/yyyy_mm_dd.tsv`
+If you have an Extranet account, you may use the `scripts/fetch.rb` helper,
+which will perform login, download, conversion (and logout). You need to
+provide it with your credentials, like so:
 
-3. Gzip'ing the `.tsv` file (i.e. `gzip -9 yyyy_mm_dd.tsv`, resulting
-  in `data/yyyy_mm_dd.tsv.gz`).
+```sh
+$ EXTRANET_USERNAME="..." EXTRANET_PASSWORD="..." scripts/fetch.rb
+```
 
-4. Run the tests (`rake`).
+It will create a new `data/*.tsv.gz` file or do nothing if the download
+provided already exists.
 
-Obviously, this process tends to be error prone. A pull request with a
-Rake task completing these steps would be nice to have.
+Note, that you'll need to clone this repository and execute `bundle` in
+the repository directory to fetch development dependencies.
 
 ## Contributors
 

data/blz.gemspec

@@ -4,7 +4,7 @@ extra_rdoc_files = ['CHANGELOG.md', 'LICENSE', 'README.md']
 
 Gem::Specification.new do |s|
   s.name            = 'blz'
-  s.version         = '0.2.0.20180305'
+  s.version         = '0.2.0.20180604'
   s.platform        = Gem::Platform::RUBY
   s.summary         = "BLZ (Bankleitzahlen) for Ruby"
 

data/scripts/Makefile

@@ -0,0 +1,20 @@
+# see https://github.com/olivere/blz/issues/14
+TEST_SOURCE_URL = https://github.com/olivere/blz/files/1820218/BLZ_20180305.txt
+TEST_TARGET_URL = https://github.com/olivere/blz/files/1820224/2018_03_05.tsv.gz
+
+.PHONY: test
+test: target.tsv compiled.tsv
+	diff -a target.tsv compiled.tsv && echo "SUCCESS: files identical" || echo "ERROR: file not identical"
+
+.PHONY: clean
+clean:
+	rm -f source.txt target.tsv compiled.tsv
+
+source.txt:
+	curl -sSL $(TEST_SOURCE_URL) >$@
+
+target.tsv:
+	curl -sSL $(TEST_TARGET_URL) | gunzip --stdout | sort >$@
+
+compiled.tsv: source.txt
+	./convert.rb $< | sort >$@

data/scripts/convert.rb

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+#
+# Helper to convert an Extranet BLZ file download in a more sane format.
+#
+
+if ARGV[0].nil?
+  $stderr.puts "Usage: #{$0} FILE|-"
+  exit 1
+end
+
+# length (and name as comment) of each column
+METRIC = [
+  8,  # BLZ
+  1,  # Merkmal
+  58, # Bezeichnung
+  5,  # PLZ
+  35, # Ort
+  27, # Kurzbezeichnung
+  5,  # PAN
+  11, # BIC
+  2,  # Prüfzifferberechnungsmethode
+  6,  # Datensatznummer
+  1,  # Änderungskennzeichen
+  1,  # BLZ-Löschung
+  8,  # Nachfolge-BLZ
+]
+
+METRIC_LEN = METRIC.inject(:+)
+
+# read from stdin if arg is "-", and setup encoding transformation
+SOURCE = (ARGV[0] == "-" ? $stdin : File.open(ARGV[0], "rb"))
+  .set_encoding(Encoding::ISO_8859_15, Encoding::UTF_8)
+
+while line = SOURCE.gets
+  line.chomp!
+  if line.length != METRIC_LEN
+    $stderr.puts "expected line length #{METRIC_LEN}, got #{line.length} in '#{line}'"
+    next
+  end
+
+  i = 0
+  puts METRIC.inject([]) {|s, m|
+    # extract columns, and strip leading/trailing whitespace
+    s << line[i ... (i+=m)].strip
+  }.join("\t")
+end

data/scripts/fetch.rb

@@ -0,0 +1,89 @@
+#!/usr/bin/env ruby
+
+require "mechanize"
+require "logger"
+require "zlib"
+
+if (f = File.expand_path("~/.extranet")) && File.exist?(f)
+  username, password = File.read(f).strip.split(":", 2)
+else
+  username = ENV.fetch("EXTRANET_USERNAME")
+  password = ENV.fetch("EXTRANET_PASSWORD")
+end
+
+# from ./convert.rb
+METRIC = [8, 1, 58, 5, 35, 27, 5, 11, 2, 6, 1, 1, 8]
+METRIC_LEN = METRIC.inject(:+)
+
+NAVIGATION = {
+  login:  "https://extranet.bundesbank.de/FT/",
+  table:  "https://extranet.bundesbank.de/FT/dirHTML.do?dirKind=DOWN&ftclient=browser&tableView=kompakt",
+  logout: "https://extranet.bundesbank.de/FT/logout.do",
+}
+
+# don't double download already existing files
+last = Dir[ File.expand_path('../data/*.tsv.gz', __dir__) ].sort.last
+last_match = last.match(/(?<y>\d{4})_(?<m>\d\d)_(?<d>\d\d)\.tsv\.gz$/)
+
+logger = Logger.new($stderr).tap{|l| l.level = Logger::INFO }
+agent = Mechanize.new do |m|
+  # m.log = logger
+  m.user_agent_alias = "Windows IE 11"
+  m.pluggable_parser["application/octet-stream"] = Mechanize::Download
+end
+
+begin
+  logger.info("login")
+  page = agent.get NAVIGATION[:login]
+  form = page.form_with action: "/pkmslogin.form"
+  form["username"] = username
+  form["password"] = password
+  agent.submit(form)
+
+  logger.info("get list of downloads")
+  sleep 1
+  page = agent.get NAVIGATION[:table]
+
+  target_link = page.links.find do |link|
+    !!(link.text.strip =~ /BLZ_\d{8}.txt/)
+  end
+
+  if !target_link
+    logger.info("no new download found")
+    exit 0
+  end
+
+  name = target_link.text.strip
+  if name == format("BLZ_%4d%02d%02d.txt", last_match[:y], last_match[:m], last_match[:d])
+    logger.info("no matching link found")
+    exit 0
+  end
+
+  logger.info("downloading new file #{name}")
+  sleep 1
+  blz = agent.get(target_link.href)
+
+  name_match = name.match(/BLZ_(?<y>\d{4})(?<m>\d\d)(?<d>\d\d)\.txt$/)
+  target_name = format("../data/%4d_%02d_%02d.tsv.gz", name_match[:y], name_match[:m], name_match[:d])
+  target_file = File.expand_path(target_name, __dir__)
+
+  logger.info("reformatting, saving as #{target_file}")
+  Zlib::GzipWriter.open(target_file, Zlib::BEST_COMPRESSION) do |gz|
+    while line = blz.body_io.gets
+      line = line.encode(Encoding::UTF_8, Encoding::ISO_8859_15).chomp!
+      if line.length != METRIC_LEN
+        logger.error("expected line length #{METRIC_LEN}, got #{line.length} in '#{line}'")
+        next
+      end
+
+      i = 0
+      gz.puts METRIC.inject([]) {|s, m| s << line[i ... (i+=m)].strip }.join("\t")
+    end
+  end
+
+  logger.info("suggestion new version: 0.2.0.#{name_match[:y]}#{name_match[:m]}#{name_match[:d]}")
+ensure
+  logger.info("logout")
+  sleep 1
+  agent.get NAVIGATION[:logout]
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: blz
 version: !ruby/object:Gem::Version
-  version: 0.2.0.20180305
+  version: 0.2.0.20180604
 platform: ruby
 authors:
 - Oliver Eilhard
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-03-16 00:00:00.000000000 Z
+date: 2018-05-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -84,8 +84,12 @@ files:
 - data/2017_06_05.tsv.gz
 - data/2017_12_04.tsv.gz
 - data/2018_03_05.tsv.gz
+- data/2018_06_04.tsv.gz
 - lib/blz.rb
 - lib/blz/bank.rb
+- scripts/Makefile
+- scripts/convert.rb
+- scripts/fetch.rb
 - test/test_bank.rb
 homepage: http://github.com/olivere/blz
 licenses: