ydim 1.1.5 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +3 -1
- data/CLAUDE.md +147 -0
- data/History.txt +12 -0
- data/bin/ydim-camt +120 -0
- data/bin/ydim-edit +1 -1
- data/lib/ydim/camt.rb +219 -0
- data/lib/ydim/config.rb +5 -0
- data/lib/ydim/reconciler.rb +223 -0
- data/lib/ydim/root_session.rb +46 -0
- data/lib/ydim/server_config.rb +6 -1
- data/lib/ydim/version.rb +1 -1
- data/lib/ydim/ydim-camt +120 -0
- data/lib/ydim/ydim-edit +1 -1
- data/readme.md +88 -3
- data/test/data/camt053.xml +215 -0
- data/test/test_camt.rb +113 -0
- data/test/test_reconciler.rb +186 -0
- data/test/test_root_session.rb +109 -0
- data/ydim.gemspec +13 -2
- metadata +40 -14
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cf1f40eabcc24aeef2f6c0e5eb3a1c6d207297dc5e3009aadbf073a9ce07233d
|
|
4
|
+
data.tar.gz: 01ebd91434ee1121c038e1dfd2fc7601f354d0175d08c0c8028055cbcd04c1bf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d8c31ab23b9df6679605589ba8a02e742415a0dad1fedc35b1ab307b68694024c622200da569fbaf55f74c24380fd2b19fad9bfdb5440581b4a244a380de00b7
|
|
7
|
+
data.tar.gz: 443601812147a9c983bf692deb8a4fc67c66379a2dccc21bc5eba77afb6789ccd8d9f79d65790d9486bc087223842c3844d28265ae1462707b0a5dc1c8f8fbd3
|
data/.github/workflows/ruby.yml
CHANGED
|
@@ -20,7 +20,7 @@ jobs:
|
|
|
20
20
|
fail-fast: false
|
|
21
21
|
matrix:
|
|
22
22
|
os: [ubuntu-latest]
|
|
23
|
-
ruby: [
|
|
23
|
+
ruby: ['3.0', '3.1', '3.2']
|
|
24
24
|
runs-on: ${{ matrix.os }}
|
|
25
25
|
steps:
|
|
26
26
|
- uses: actions/checkout@v3
|
|
@@ -29,3 +29,5 @@ jobs:
|
|
|
29
29
|
ruby-version: ${{ matrix.ruby }}
|
|
30
30
|
bundler-cache: true # runs 'bundle install' and caches installed gems automatically
|
|
31
31
|
- run: bundle exec rake test
|
|
32
|
+
env:
|
|
33
|
+
MT_COMPAT: '1'
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
+
|
|
5
|
+
## Project
|
|
6
|
+
|
|
7
|
+
`ydim` — ywesee distributed invoice manager. A Ruby gem (GPLv2) providing a DRb daemon that
|
|
8
|
+
stores debitors/invoices in PostgreSQL via ODBA, renders PDF invoices, and mails them.
|
|
9
|
+
Version lives in `lib/ydim/version.rb`; changes are recorded in `History.txt`.
|
|
10
|
+
|
|
11
|
+
## Commands
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
bundle install # gems come from ydim.gemspec via Gemfile
|
|
15
|
+
MT_COMPAT=1 bundle exec rake test # full suite (test/suite.rb); this is what CI runs
|
|
16
|
+
MT_COMPAT=1 bundle exec ruby -Ilib -Itest test/test_invoice.rb # a single test file
|
|
17
|
+
MT_COMPAT=1 bundle exec ruby -Ilib -Itest test/test_invoice.rb -n test_add_item # one test
|
|
18
|
+
bundle exec rake # default: clobber + test + build gem into pkg/
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
`MT_COMPAT=1` is required — the tests use `flexmock/test_unit`, which needs minitest's
|
|
22
|
+
`Minitest::Unit::TestCase` compatibility shim. Without it (or with a too-new
|
|
23
|
+
flexmock/minitest) loading a test file dies with `undefined method 'teardown'`.
|
|
24
|
+
CI (`.github/workflows/ruby.yml`) runs Ruby 3.0/3.1/3.2 on ubuntu; `.travis.yml` is dead.
|
|
25
|
+
`Gemfile.lock` is gitignored — a stale one in the working tree will make `bundle exec` fail
|
|
26
|
+
with `Bundler::GemNotFound`; delete it and re-run `bundle install`.
|
|
27
|
+
|
|
28
|
+
`rake spec` exists (RSpec task) but there are no spec files.
|
|
29
|
+
|
|
30
|
+
## Architecture
|
|
31
|
+
|
|
32
|
+
Client/server over **DRb**, authenticated by DSA key challenge (`rrba`), persisted with
|
|
33
|
+
**ODBA** (object database over `ydbi`/`ydbd-pg` → PostgreSQL).
|
|
34
|
+
|
|
35
|
+
**Server side** (`lib/ydim/ydimd` is the daemon entry point, not `bin/`):
|
|
36
|
+
`ydimd` wires `ODBA.storage.dbi` to a connection pool, installs `ODBA::DRbIdConv`, then
|
|
37
|
+
serves a `YDIM::Server` over `druby://`. `Server#initialize` builds a **Needle registry**
|
|
38
|
+
(`@serv`) — the service locator threaded through nearly every class. Registered services:
|
|
39
|
+
`:auth_server`, `:clients`, `:config`, `:currency_converter`, `:factory`, `:id_server`,
|
|
40
|
+
`:logger`. Anything that takes a `serv` argument (`Factory`, `AutoInvoicer`,
|
|
41
|
+
`CurrencyUpdater`) reads its collaborators from this registry, so tests mock `serv` rather
|
|
42
|
+
than the individual dependencies.
|
|
43
|
+
|
|
44
|
+
`Server` also spawns daily background threads via `repeat_at(hour)`: AutoInvoicer,
|
|
45
|
+
CurrencyUpdater, and a StatusUpdater that re-saves every invoice so derived `status` stays
|
|
46
|
+
current in the DB.
|
|
47
|
+
|
|
48
|
+
**Session/API surface**: `Server#login` → `RootUser#new_session` → `RootSession` wrapped in
|
|
49
|
+
`ODBA::DRbWrapper`. **`RootSession` is the entire remote API** — every method a client can
|
|
50
|
+
call (`create_invoice`, `add_items`, `send_invoice`, `debitors`, `collect_garbage`,
|
|
51
|
+
`mark_paid`, `reconcile_camt`, …) is a public method there. Adding a client-callable
|
|
52
|
+
operation means adding it to `RootSession`. `Client#method_missing` forwards everything to
|
|
53
|
+
the session, so clients need no stubs.
|
|
54
|
+
|
|
55
|
+
**Domain**: `Debitor` (1→n `Invoice` and `AutoInvoice`) → `Item`. `Invoice#status` is
|
|
56
|
+
computed, not stored (`is_trash` / `is_paid` / `is_due` / `is_open`). `AutoInvoice` is a
|
|
57
|
+
recurring template: `AutoInvoicer#run` walks all debitors daily and, when `auto.date` is
|
|
58
|
+
today, calls `Factory#generate_invoice` to materialise a real `Invoice` from it (copying
|
|
59
|
+
items, stamping `expiry_time`, applying the current `vat_rate`) and mails it; a month ahead
|
|
60
|
+
it sends a reminder instead. `Factory` is the only place invoice IDs are allocated
|
|
61
|
+
(`id_server.next_id(:invoice, config.invoice_number_start)`).
|
|
62
|
+
|
|
63
|
+
**Persistence is declared in one file**: `lib/ydim/odba.rb` reopens the domain classes to
|
|
64
|
+
include `ODBA::Persistable`, list `ODBA_SERIALIZABLE` ivars, and declare `odba_index`
|
|
65
|
+
(e.g. `Debitor` by email/name/unique_id, `Invoice` by status/unique_id). Those indexes are
|
|
66
|
+
what power `find_by_unique_id` / `search_by_status` / `search_by_exact_email` used in
|
|
67
|
+
`RootSession`. New persisted classes or lookups must be registered here — and each
|
|
68
|
+
`odba_index` needs a matching `ydim_<class>_<attr>` table, see `set_initial_ydim_db.sql`.
|
|
69
|
+
|
|
70
|
+
**VAT**: `config.vat_rate` (currently 8.1, in `server_config.rb`) is applied when items are
|
|
71
|
+
added (`RootSession#add_items`), when autoinvoices are materialised (`Factory`), and reset
|
|
72
|
+
by `Invoice#suppress_vat=`. `Debitor#foreign?` (country != `config.home_country`) makes
|
|
73
|
+
`Factory` suppress VAT at creation time. Changing the rate touches all of these plus the
|
|
74
|
+
`texts.tax` string in pdfinvoice config.
|
|
75
|
+
|
|
76
|
+
**Payment reconciliation**: `lib/ydim/camt.rb` parses ISO-20022 camt.052/053/054 statements
|
|
77
|
+
(rexml only, no DB — the namespace is read off the document root, so any minor version
|
|
78
|
+
works), and `lib/ydim/reconciler.rb` matches booked credits against invoices. The CLI
|
|
79
|
+
(`ydim-camt`) parses client-side and sends `Camt::Entry` objects to
|
|
80
|
+
`RootSession#reconcile_camt`, so the daemon never touches the files. Results carry
|
|
81
|
+
`Reconciler::InvoiceRef`, not `Invoice::Info`, so a client can unmarshal them with
|
|
82
|
+
`ydim/reconciler` alone instead of loading the whole server.
|
|
83
|
+
|
|
84
|
+
Three rules that the real UBS data forces and that are easy to break:
|
|
85
|
+
- **Filter by IBAN.** An e-banking download holds every account the login sees, private ones
|
|
86
|
+
included; `config.camt_accounts` says which are ydim's, and `Reconciler#reconcile` raises
|
|
87
|
+
rather than run without it.
|
|
88
|
+
- **Dedup by `AcctSvcrRef`.** UBS re-sends the same day under a new `MsgId`; in the sample
|
|
89
|
+
set 60 of 142 entries were redeliveries.
|
|
90
|
+
- **Whole digit runs only** (`Camt::Entry::TOKEN_PATTERN`) — never a substring, and never a
|
|
91
|
+
run touching a letter. TWINT credits carry the payer's phone number, and bank
|
|
92
|
+
`EndToEndId`s are hex that happens to contain 5-digit sequences.
|
|
93
|
+
|
|
94
|
+
Only `:exact` and `:split` matches (invoice named *and* amount equal to the cent) are ever
|
|
95
|
+
applied; everything else is reported for review.
|
|
96
|
+
|
|
97
|
+
**PDF**: `lib/pdfinvoice/` is a vendored sub-library (PDF::Writer based) with its own config;
|
|
98
|
+
`Invoice#pdf_invoice` maps ydim items onto it and overrides `formats`/`texts.tax` per invoice.
|
|
99
|
+
|
|
100
|
+
**Mail**: `lib/ydim/mail.rb` configures `::Mail.defaults` with SMTP settings at load time.
|
|
101
|
+
|
|
102
|
+
## Configuration
|
|
103
|
+
|
|
104
|
+
Three independent `rclconf` config objects, all merging ARGV over defaults:
|
|
105
|
+
|
|
106
|
+
| Object | Defaults defined in | YAML read from |
|
|
107
|
+
| --- | --- | --- |
|
|
108
|
+
| `YDIM::Server.config` (daemon) | `lib/ydim/server_config.rb` | `/etc/ydim/ydimd.yml` |
|
|
109
|
+
| `YDIM::Client::CONFIG` | `lib/ydim/config.rb` | `/etc/ydim/ydim.yml` |
|
|
110
|
+
| `PdfInvoice.config` | `lib/pdfinvoice/config.rb` | `~/.pdfinvoice/config.yml`, `/etc/pdfinvoice/config.yml` |
|
|
111
|
+
|
|
112
|
+
Gotcha: `server_config.rb` only builds `CONFIG` with defaults — the YAML file is actually
|
|
113
|
+
loaded by the `config.load(config.config)` call at the top of `lib/ydim/mail.rb`, which is
|
|
114
|
+
why `mail` is required early in the daemon's require chain. Don't reorder those requires.
|
|
115
|
+
Since 1.1.4/1.1.5 all server paths are rooted at `/etc/ydim` by design; keep them there.
|
|
116
|
+
|
|
117
|
+
## Executables
|
|
118
|
+
|
|
119
|
+
`bin/ydim-edit` and `bin/ydim-inject` are **byte-identical copies** of `lib/ydim/ydim-edit`
|
|
120
|
+
and `lib/ydim/ydim-inject` (the gemspec takes executables from `bin/`, but the working
|
|
121
|
+
copies live in `lib/ydim/` — see History.txt 1.0.3). Edit both, or the gem and the checkout
|
|
122
|
+
diverge. Note `ydim-edit` carries its own duplicated defaults hash, separate from
|
|
123
|
+
`server_config.rb`.
|
|
124
|
+
|
|
125
|
+
- `ydim-edit` — IRB console with a live `$server` / `$needle` (Needle registry) against the DB.
|
|
126
|
+
- `ydim-inject` — reads a YAML invoice on stdin, creates and mails it through a `Client`.
|
|
127
|
+
- `ydim-camt` — reconciles a camt.053 zip/directory/file against the open invoices; reports
|
|
128
|
+
by default, books only with `--apply`. Config overrides are RCLConf's `key=value` form,
|
|
129
|
+
not `--key value`, so the OptionParser call filters those out of the file arguments.
|
|
130
|
+
- `ydim_migrate_to_utf_8` — one-off LATIN1→UTF-8 DB migration (repo root, not `bin/`).
|
|
131
|
+
- `get_db_ydim` — shell script pulling a nightly Postgres dump and restoring it locally.
|
|
132
|
+
|
|
133
|
+
`install.rb` is a generated setup.rb-style installer; `Manifest.txt` is stale (lists
|
|
134
|
+
`bin/ydimd`, `lib/ydim/smtp_tls.rb`, `README.txt` — none exist).
|
|
135
|
+
|
|
136
|
+
## Testing conventions
|
|
137
|
+
|
|
138
|
+
Minitest + FlexMock, no DB. `test/stub/odba.rb` replaces `ODBA.transaction` with a plain
|
|
139
|
+
yield and `odba_store` with a counter, so tests that touch persistence require it
|
|
140
|
+
(`require 'stub/odba'`) *before* the class under test. Tests mock the Needle registry with a
|
|
141
|
+
bare `FlexMock` and stub `serv.config` / `serv.logger` individually. `test/suite.rb` just
|
|
142
|
+
globs `test_*.rb`; SimpleCov is present but disabled (`if false`).
|
|
143
|
+
|
|
144
|
+
## Style
|
|
145
|
+
|
|
146
|
+
Existing code is hard-tabbed in the older files and two-space indented in the newer ones,
|
|
147
|
+
often mixed within a single file. Match the surrounding block rather than reformatting.
|
data/History.txt
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
=== 1.1.6 / 09.08.2026
|
|
2
|
+
|
|
3
|
+
* Reconcile payments from ISO-20022 camt.053 bank statements (UBS Z53)
|
|
4
|
+
* ydim-camt reads a statement zip, directory or single xml and reports
|
|
5
|
+
which open invoices the credits settle; --apply books them
|
|
6
|
+
* RootSession#reconcile_camt and #mark_paid make this available over DRb
|
|
7
|
+
* new config camt_accounts: the IBANs ydim invoices are paid into. The
|
|
8
|
+
e-banking download also contains the private accounts, and reconciling
|
|
9
|
+
without this set is refused rather than guessed
|
|
10
|
+
* Pin minitest below 6, which dropped the Minitest::Unit::TestCase shim that
|
|
11
|
+
flexmock/test_unit needs
|
|
12
|
+
|
|
1
13
|
=== 1.1.5/ 21.01.2023
|
|
2
14
|
|
|
3
15
|
* Read ALL configuration values only from /etc/ydim
|
data/bin/ydim-camt
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# encoding: utf-8
|
|
3
|
+
# ydim-camt -- ydim -- 09.08.2026 -- zdavatz@ywesee.com
|
|
4
|
+
#
|
|
5
|
+
# Reconciles bank statements (ISO-20022 camt.053, as downloaded from UBS
|
|
6
|
+
# e-banking) against the open invoices. Reports by default; only books a
|
|
7
|
+
# payment when told to with --apply, and even then only when the payer named
|
|
8
|
+
# the invoice and the amount matches to the cent.
|
|
9
|
+
#
|
|
10
|
+
# ydim-camt ~/Downloads/statements.zip
|
|
11
|
+
# ydim-camt --apply ~/Downloads/statements.zip
|
|
12
|
+
# ydim-camt --account CH87... /var/ydim/camt/
|
|
13
|
+
|
|
14
|
+
require 'openssl'
|
|
15
|
+
require 'optparse'
|
|
16
|
+
require 'rrba/error'
|
|
17
|
+
require 'ydim/camt'
|
|
18
|
+
require 'ydim/client'
|
|
19
|
+
require 'ydim/config'
|
|
20
|
+
require 'ydim/reconciler'
|
|
21
|
+
|
|
22
|
+
options = { :apply => false, :verbose => false, :accounts => [] }
|
|
23
|
+
parser = OptionParser.new { |opt|
|
|
24
|
+
opt.banner = "Usage: ydim-camt [options] <statements.zip|directory|file.xml>"
|
|
25
|
+
opt.on('--apply', 'mark unambiguously matched invoices as paid') {
|
|
26
|
+
options[:apply] = true }
|
|
27
|
+
opt.on('--account IBAN', 'account to reconcile (repeatable); ',
|
|
28
|
+
'defaults to camt_accounts from the config') { |iban|
|
|
29
|
+
options[:accounts].push(iban) }
|
|
30
|
+
opt.on('-v', '--verbose', 'also list credits that matched nothing') {
|
|
31
|
+
options[:verbose] = true }
|
|
32
|
+
opt.on('-h', '--help') { puts opt; exit }
|
|
33
|
+
}
|
|
34
|
+
# Every ydim executable also takes its configuration on the command line, in
|
|
35
|
+
# RCLConf's "key=value" form -- those are not statements to read, they were
|
|
36
|
+
# already picked up from ARGV when ydim/config was required.
|
|
37
|
+
paths = parser.parse(ARGV.dup).reject { |arg| arg =~ /\A[\w.]+=/ }
|
|
38
|
+
if paths.empty?
|
|
39
|
+
warn parser
|
|
40
|
+
exit 1
|
|
41
|
+
end
|
|
42
|
+
missing = paths.reject { |path| File.exist?(path) }
|
|
43
|
+
unless missing.empty?
|
|
44
|
+
warn "no such file: #{missing.join(', ')}"
|
|
45
|
+
exit 1
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
entries = paths.flat_map { |path| YDIM::Camt.entries(path) }
|
|
49
|
+
if entries.empty?
|
|
50
|
+
warn "no camt entries found in #{paths.join(', ')}"
|
|
51
|
+
exit 1
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
config = YDIM::Client::CONFIG
|
|
55
|
+
accounts = options[:accounts]
|
|
56
|
+
accounts = [config.camt_accounts].flatten.compact if accounts.empty?
|
|
57
|
+
if accounts.empty?
|
|
58
|
+
warn "no account to reconcile: pass --account IBAN, or set camt_accounts " \
|
|
59
|
+
"in the config to the IBAN of the ywesee business account. The private " \
|
|
60
|
+
"accounts in the same download must not be matched against invoices."
|
|
61
|
+
exit 1
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def show(match, marker)
|
|
65
|
+
ids = match.invoice_ids
|
|
66
|
+
printf(" %-11s %10.2f %s %s %s\n", ids.first || '?',
|
|
67
|
+
match.entry.amount.to_f, match.entry.currency,
|
|
68
|
+
match.entry.booking_date, match.entry.counterparty.to_s[0, 30])
|
|
69
|
+
ids[1..-1].to_a.each { |id| printf(" %-11s\n", id) }
|
|
70
|
+
puts " #{' ' * 11} #{marker} #{match.reason}"
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
server = DRb::DRbObject.new(nil, config.server_url)
|
|
74
|
+
client = YDIM::Client.new(config)
|
|
75
|
+
key = OpenSSL::PKey::DSA.new(File.read(config.private_key))
|
|
76
|
+
|
|
77
|
+
DRb.start_service
|
|
78
|
+
client.login(server, key)
|
|
79
|
+
begin
|
|
80
|
+
result = client.reconcile_camt(entries,
|
|
81
|
+
:apply => options[:apply], :accounts => accounts)
|
|
82
|
+
|
|
83
|
+
applicable = result.applicable
|
|
84
|
+
puts(options[:apply] ? "BOOKED AS PAID" \
|
|
85
|
+
: "MATCHED (would be marked paid with --apply)")
|
|
86
|
+
if applicable.empty?
|
|
87
|
+
puts " none"
|
|
88
|
+
else
|
|
89
|
+
applicable.each { |match| show(match, match.applied? ? '+' : '~') }
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
review = result.review
|
|
93
|
+
unless review.empty?
|
|
94
|
+
puts
|
|
95
|
+
puts "REVIEW (never booked automatically)"
|
|
96
|
+
review.each { |match| show(match, '?') }
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
unmatched = result.unmatched
|
|
100
|
+
if options[:verbose] && !unmatched.empty?
|
|
101
|
+
puts
|
|
102
|
+
puts "UNMATCHED CREDITS"
|
|
103
|
+
unmatched.each { |match| show(match, '-') }
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
puts
|
|
107
|
+
summary = ["#{applicable.size} matched", "#{review.size} to review",
|
|
108
|
+
"#{unmatched.size} unmatched"]
|
|
109
|
+
summary.push("#{result.applied.size} booked") if options[:apply]
|
|
110
|
+
result.skipped.sort_by { |reason, count| reason.to_s }.each { |reason, count|
|
|
111
|
+
summary.push("#{count} #{reason.to_s.tr('_', ' ')}") if count > 0
|
|
112
|
+
}
|
|
113
|
+
puts summary.join(', ')
|
|
114
|
+
unless options[:apply] || applicable.empty?
|
|
115
|
+
puts "run again with --apply to mark the #{applicable.size} matched " \
|
|
116
|
+
"invoice(s) paid"
|
|
117
|
+
end
|
|
118
|
+
ensure
|
|
119
|
+
client.logout
|
|
120
|
+
end
|
data/bin/ydim-edit
CHANGED
data/lib/ydim/camt.rb
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# encoding: utf-8
|
|
3
|
+
# YDIM::Camt -- ydim -- 09.08.2026 -- zdavatz@ywesee.com
|
|
4
|
+
|
|
5
|
+
require 'date'
|
|
6
|
+
require 'fileutils'
|
|
7
|
+
require 'rexml/document'
|
|
8
|
+
require 'tmpdir'
|
|
9
|
+
|
|
10
|
+
module YDIM
|
|
11
|
+
# Reader for ISO-20022 bank-to-customer statements (camt.052/053/054), as
|
|
12
|
+
# delivered by UBS and the other Swiss banks. This is pure parsing: no DB and
|
|
13
|
+
# no ODBA, so it can run on the client side. The matching of entries against
|
|
14
|
+
# invoices lives in YDIM::Reconciler.
|
|
15
|
+
module Camt
|
|
16
|
+
class Error < StandardError; end
|
|
17
|
+
|
|
18
|
+
# A single booked movement on an account (a camt <Ntry>).
|
|
19
|
+
class Entry
|
|
20
|
+
attr_accessor :account_iban, :account_currency, :amount, :currency,
|
|
21
|
+
:credit, :status, :booking_date, :value_date, :reference,
|
|
22
|
+
:counterparty, :additional_info, :source
|
|
23
|
+
attr_reader :remittance, :end_to_end_ids, :creditor_references
|
|
24
|
+
def initialize
|
|
25
|
+
@remittance = []
|
|
26
|
+
@end_to_end_ids = []
|
|
27
|
+
@creditor_references = []
|
|
28
|
+
@credit = false
|
|
29
|
+
end
|
|
30
|
+
def credit?
|
|
31
|
+
@credit
|
|
32
|
+
end
|
|
33
|
+
# ydim invoices are only ever paid into the ywesee business account --
|
|
34
|
+
# the private accounts share the same e-banking download, so entries have
|
|
35
|
+
# to be filtered by IBAN before anything is matched. An empty filter
|
|
36
|
+
# accepts everything here; refusing to run without one is the
|
|
37
|
+
# Reconciler's job, since only it knows the entries are about to be
|
|
38
|
+
# matched against invoices.
|
|
39
|
+
def account?(ibans)
|
|
40
|
+
ibans = [ibans].flatten.compact
|
|
41
|
+
ibans.empty? || ibans.any? { |iban|
|
|
42
|
+
normalize_iban(iban) == normalize_iban(@account_iban)
|
|
43
|
+
}
|
|
44
|
+
end
|
|
45
|
+
# Only BOOK entries have actually hit the account; PDNG ones can still
|
|
46
|
+
# disappear, so they must never mark an invoice as paid.
|
|
47
|
+
def booked?
|
|
48
|
+
@status.nil? || @status == 'BOOK'
|
|
49
|
+
end
|
|
50
|
+
# Money as an integer so that amounts can be compared without float
|
|
51
|
+
# rounding surprises.
|
|
52
|
+
def amount_cents
|
|
53
|
+
(@amount.to_f * 100).round
|
|
54
|
+
end
|
|
55
|
+
# Every string a payer might have put an invoice number into.
|
|
56
|
+
def texts
|
|
57
|
+
(@remittance + @end_to_end_ids + @creditor_references \
|
|
58
|
+
+ [@additional_info]).compact
|
|
59
|
+
end
|
|
60
|
+
# Runs of digits found in the remittance information. Whole runs only --
|
|
61
|
+
# never substrings, or the phone number in a TWINT payment
|
|
62
|
+
# ("+41796723413") would match half the invoices in the database. A run
|
|
63
|
+
# touching a letter is machine noise rather than something a human typed:
|
|
64
|
+
# this drops the hex EndToEndIds that banks generate
|
|
65
|
+
# ("0ebf22116f364fc394da3e2776a05643" would otherwise offer up "22116"),
|
|
66
|
+
# while "RG 13363 VOM 26.6.2026" and "ISO 13368" still yield their
|
|
67
|
+
# invoice number.
|
|
68
|
+
TOKEN_PATTERN = /(?<![0-9A-Za-z])\d+(?![0-9A-Za-z])/
|
|
69
|
+
def numeric_tokens
|
|
70
|
+
texts.flat_map { |text| text.scan(TOKEN_PATTERN) }.uniq
|
|
71
|
+
end
|
|
72
|
+
# Two statements can deliver the same entry (UBS re-sends a day under a
|
|
73
|
+
# new MsgId), so entries are identified by the bank's own reference.
|
|
74
|
+
def dedup_key
|
|
75
|
+
[@account_iban, @reference || texts.join('|'), amount_cents, @credit,
|
|
76
|
+
@booking_date]
|
|
77
|
+
end
|
|
78
|
+
def to_s
|
|
79
|
+
sprintf("%s %s %s %s %s", @booking_date, @credit ? 'CRDT' : 'DBIT',
|
|
80
|
+
@currency, sprintf('%.2f', @amount.to_f), @counterparty)
|
|
81
|
+
end
|
|
82
|
+
private
|
|
83
|
+
def normalize_iban(iban)
|
|
84
|
+
iban.to_s.gsub(/\s+/, '').upcase
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# A camt <Stmt> -- one account over one period.
|
|
89
|
+
class Statement
|
|
90
|
+
attr_accessor :id, :account_iban, :currency, :owner, :created_at,
|
|
91
|
+
:opening_balance, :closing_balance, :source
|
|
92
|
+
attr_reader :entries
|
|
93
|
+
def initialize
|
|
94
|
+
@entries = []
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
class << self
|
|
99
|
+
# Reads whatever the bank handed you: a single xml file, a directory of
|
|
100
|
+
# them, or the zip you downloaded from e-banking. Returns Statements.
|
|
101
|
+
def read(path)
|
|
102
|
+
if File.directory?(path)
|
|
103
|
+
read_files(Dir[File.join(path, '**', '*.[xX][mM][lL]')].sort)
|
|
104
|
+
elsif path =~ /\.zip\z/i
|
|
105
|
+
read_zip(path)
|
|
106
|
+
else
|
|
107
|
+
read_files([path])
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
def read_files(paths)
|
|
111
|
+
paths.flat_map { |path|
|
|
112
|
+
parse(File.read(path), File.basename(path))
|
|
113
|
+
}
|
|
114
|
+
end
|
|
115
|
+
def read_zip(path)
|
|
116
|
+
Dir.mktmpdir('ydim-camt') { |dir|
|
|
117
|
+
unless system('unzip', '-q', '-o', path, '-d', dir)
|
|
118
|
+
raise Error,
|
|
119
|
+
"unable to unzip #{path} -- extract it and pass the directory"
|
|
120
|
+
end
|
|
121
|
+
read_files(Dir[File.join(dir, '**', '*.[xX][mM][lL]')].sort)
|
|
122
|
+
}
|
|
123
|
+
end
|
|
124
|
+
# Parses one camt document into its Statements.
|
|
125
|
+
def parse(xml, source = nil)
|
|
126
|
+
doc = REXML::Document.new(xml)
|
|
127
|
+
root = doc.root or raise Error, "#{source}: not an XML document"
|
|
128
|
+
# camt.053.001.02 through .08 differ only in the namespace URI for
|
|
129
|
+
# everything we read, so take whatever the document declares.
|
|
130
|
+
ns = { 'c' => root.namespace }
|
|
131
|
+
REXML::XPath.match(doc, '//c:Stmt | //c:Rpt | //c:Ntfctn', ns).collect { |node|
|
|
132
|
+
parse_statement(node, ns, source)
|
|
133
|
+
}
|
|
134
|
+
end
|
|
135
|
+
# All entries of all statements below path, duplicates included -- the
|
|
136
|
+
# Reconciler drops those, and it can only report how many deliveries were
|
|
137
|
+
# doubled up if it gets to see them.
|
|
138
|
+
def entries(path)
|
|
139
|
+
read(path).flat_map { |stmt| stmt.entries }
|
|
140
|
+
end
|
|
141
|
+
def dedup(entries)
|
|
142
|
+
seen = {}
|
|
143
|
+
entries.select { |entry| !seen.key?(entry.dedup_key) \
|
|
144
|
+
&& seen[entry.dedup_key] = true }
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
private
|
|
148
|
+
def parse_statement(node, ns, source)
|
|
149
|
+
stmt = Statement.new
|
|
150
|
+
stmt.source = source
|
|
151
|
+
stmt.id = text(node, 'c:Id', ns)
|
|
152
|
+
stmt.account_iban = text(node, 'c:Acct/c:Id/c:IBAN', ns)
|
|
153
|
+
stmt.currency = text(node, 'c:Acct/c:Ccy', ns)
|
|
154
|
+
stmt.owner = text(node, 'c:Acct/c:Ownr/c:Nm', ns)
|
|
155
|
+
stmt.created_at = date(text(node, 'c:CreDtTm', ns))
|
|
156
|
+
stmt.opening_balance = balance(node, ns, 'OPBD')
|
|
157
|
+
stmt.closing_balance = balance(node, ns, 'CLBD')
|
|
158
|
+
REXML::XPath.each(node, 'c:Ntry', ns) { |entry_node|
|
|
159
|
+
stmt.entries.push(parse_entry(entry_node, ns, stmt))
|
|
160
|
+
}
|
|
161
|
+
stmt
|
|
162
|
+
end
|
|
163
|
+
def parse_entry(node, ns, stmt)
|
|
164
|
+
entry = Entry.new
|
|
165
|
+
entry.source = stmt.source
|
|
166
|
+
entry.account_iban = stmt.account_iban
|
|
167
|
+
entry.account_currency = stmt.currency
|
|
168
|
+
amount = REXML::XPath.first(node, 'c:Amt', ns)
|
|
169
|
+
entry.amount = amount && amount.text.to_f
|
|
170
|
+
entry.currency = (amount && amount.attributes['Ccy']) || stmt.currency
|
|
171
|
+
entry.credit = text(node, 'c:CdtDbtInd', ns) == 'CRDT'
|
|
172
|
+
entry.status = text(node, 'c:Sts/c:Cd', ns) || text(node, 'c:Sts', ns)
|
|
173
|
+
entry.booking_date = date(text(node, 'c:BookgDt/c:Dt', ns) \
|
|
174
|
+
|| text(node, 'c:BookgDt/c:DtTm', ns))
|
|
175
|
+
entry.value_date = date(text(node, 'c:ValDt/c:Dt', ns) \
|
|
176
|
+
|| text(node, 'c:ValDt/c:DtTm', ns))
|
|
177
|
+
entry.reference = text(node, 'c:AcctSvcrRef', ns) \
|
|
178
|
+
|| text(node, 'c:NtryRef', ns)
|
|
179
|
+
entry.additional_info = text(node, 'c:AddtlNtryInf', ns)
|
|
180
|
+
# The party on the other side: for money coming in that is the debtor,
|
|
181
|
+
# for money going out the creditor. Either can be given as a name or as
|
|
182
|
+
# loose address lines.
|
|
183
|
+
side = entry.credit? ? 'Dbtr' : 'Cdtr'
|
|
184
|
+
entry.counterparty = collect(node,
|
|
185
|
+
".//c:#{side}/c:Pty/c:Nm | .//c:#{side}/c:Pty/c:PstlAdr/c:AdrLine",
|
|
186
|
+
ns).first
|
|
187
|
+
# An entry can bundle several transactions (a batch booking), so the
|
|
188
|
+
# remittance information of all of them counts.
|
|
189
|
+
entry.remittance.concat(collect(node, './/c:RmtInf/c:Ustrd', ns))
|
|
190
|
+
entry.remittance.concat(collect(node, './/c:Strd/c:AddtlRmtInf', ns))
|
|
191
|
+
entry.end_to_end_ids.concat(collect(node, './/c:EndToEndId', ns)\
|
|
192
|
+
.reject { |id| id == 'NOTPROVIDED' })
|
|
193
|
+
entry.creditor_references.concat(
|
|
194
|
+
collect(node, './/c:CdtrRefInf/c:Ref', ns))
|
|
195
|
+
entry
|
|
196
|
+
end
|
|
197
|
+
def balance(node, ns, code)
|
|
198
|
+
amount = REXML::XPath.first(node,
|
|
199
|
+
"c:Bal[c:Tp/c:CdOrPrtry/c:Cd='#{code}']/c:Amt", ns)
|
|
200
|
+
amount && amount.text.to_f
|
|
201
|
+
end
|
|
202
|
+
def collect(node, path, ns)
|
|
203
|
+
REXML::XPath.match(node, path, ns).collect { |el|
|
|
204
|
+
el.text.to_s.strip
|
|
205
|
+
}.reject { |str| str.empty? }.uniq
|
|
206
|
+
end
|
|
207
|
+
def text(node, path, ns)
|
|
208
|
+
el = REXML::XPath.first(node, path, ns)
|
|
209
|
+
str = el && el.text.to_s.strip
|
|
210
|
+
str unless str.nil? || str.empty?
|
|
211
|
+
end
|
|
212
|
+
def date(str)
|
|
213
|
+
Date.parse(str) if str
|
|
214
|
+
rescue ArgumentError
|
|
215
|
+
nil
|
|
216
|
+
end
|
|
217
|
+
end
|
|
218
|
+
end
|
|
219
|
+
end
|
data/lib/ydim/config.rb
CHANGED
|
@@ -13,6 +13,11 @@ module YDIM
|
|
|
13
13
|
File.join(ydim_default_dir, 'ydim.yml'),
|
|
14
14
|
]
|
|
15
15
|
defaults = {
|
|
16
|
+
# IBANs of the accounts ydim invoices are paid into -- see
|
|
17
|
+
# camt_accounts in server_config.rb. ydim-camt reconciles these and
|
|
18
|
+
# nothing else, so that the private accounts in the same e-banking
|
|
19
|
+
# download are left alone.
|
|
20
|
+
'camt_accounts' => [],
|
|
16
21
|
'client_url' => 'druby://localhost:0',
|
|
17
22
|
'config' => default_config_files,
|
|
18
23
|
'private_key' => File.join(home_dir, '.ssh', 'id_dsa'),
|