feidee_utils 0.0.4 → 0.0.4.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: ef73043dafb067cba94f418475c9b87acfd1b064
4
- data.tar.gz: 1740946281e699563aec46dce68bb6c9d97e837c
3
+ metadata.gz: 293b859c218934656fc1b096076284f44fe7e17b
4
+ data.tar.gz: bf5e50c65f90d910f75b39bc667f63d9ff61fbf3
5
5
  SHA512:
6
- metadata.gz: 33c241f66d3457a4879b1ef18226186b122cefac1e64e734e13f087af2ec6ff874e0f3dc8197f1512b51340b568d00c00a44f3e0a2300e86f6577df4ee95652b
7
- data.tar.gz: 6805623ecc37f3a3b70e1f81a84c5921c08b3d4c55f6c8b2e28338da0cbeefea050fc32ec21282f5c8e8ad66d9d5b9a4a8766a530fdd01a26b503ec243d5b9c6
6
+ metadata.gz: 30ee3bc1c7e4b1fa59ec6a2a5e29b5750261f08f2f335e0dbf3fc6e3fd462c5958803df7e07fc04c162750dcee5a8482dd16ce851bc5533a63363fda0613286f
7
+ data.tar.gz: 101e8d20b5db4048f9f9f994d35fe285f65d0f0b2881a7e236f5fc16e9fbc6fbb7fa2afbc5f58a80643c6794931e05b999624ca0f2e675c07f0df3dd9958787f
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
data/MIT-LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2015 Liqing Muyi (muyiliqing@gmail.com)
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,67 @@
1
+ Feidee Utils
2
+ ============
3
+ [![Build Status](https://travis-ci.org/muyiliqing/feidee_utils.svg?branch=master)](https://travis-ci.org/muyiliqing/feidee_utils)
4
+
5
+ Free users' data from [Feidee](http://www.feidee.com) private backups (.kbf).
6
+
7
+ Feidee And The KBF Format
8
+ -----------
9
+ [Feidee MyMoney](http://www.feidee.com/money/) is a set of popular book-keeping software in China. It includes Android, iOS, web apps and several desktop software. The Android and iOS apps produces backup in kbf format.
10
+
11
+ A kbf file is in fact a zip file contains a modified SQLite database and other attachments, such as photos.
12
+ The first 16 bits of the SQLite database is modified so that it could not be read directly by SQLite.
13
+ The database itself is NOT encrypted. Almost all useful information is in the SQLite database.
14
+
15
+ Install
16
+ ---------
17
+ ```bash
18
+ gem install feidee_utils
19
+ ```
20
+
21
+ Usage
22
+ ----------
23
+ A set of ActiveRecord-like classes are provided to access the information in the backup. See the quick example below.
24
+
25
+ ```ruby
26
+ require 'feidee_utils'
27
+
28
+ kbf = FeideeUtils::Kbf.open_file(path_to_kbf_file)
29
+ database = kbf.sqlite_db
30
+ all_accounts = database.namespaced::Account.all
31
+ all_transactions = database.namespaced::Transaction.all
32
+ ```
33
+
34
+ For more examples see ```examples/``` (To be added).
35
+
36
+ Supported Entities
37
+ -----------------
38
+
39
+ * Account
40
+ * Transaction
41
+ * AccountGroup
42
+ * Category
43
+
44
+ Chinese Characters
45
+ -----------------
46
+
47
+ The database contains many Chinese characters such as builtin category names. Some of the characters are also included in tests. The gem is developed under OSX so presumably the gem should work fine in Unix-like environments with Unicode/UTF8 or whatever the encoding is.
48
+
49
+ Why not ActiveRecord
50
+ ----------------
51
+ Sometimes we have to compare the content of two backups and must open them at the same time.
52
+ Only one database can be opened using ActiveRecord. It is not designed to be used in such a way.
53
+
54
+ Why Feidee Utils At All
55
+ -----------
56
+ Originally the Feidee Android and iOS app let users export their personal data recorded by the app.
57
+ Since some version last year, the functionality is removed from the app and user data is trapped inside Feidee's private system forever. The uses may pay to get pro version, or upload their data to Feidee's server in order to export.
58
+
59
+ As a user of Feidee MyMoney, I'm truelly grateful that such great apps are available free of charge. However I also believe that users' data belongs to users and should be controlled by its owner. Thus I decided to build the utils to help Feidee MyMoney users access their own data.
60
+
61
+ Disclaimer
62
+ ---------
63
+ Use at your own risk. Study purpose only. Please do NOT use for any illegal purpose. For details see MIT-LICENSE in the repo.
64
+
65
+ This software DOES NOT involve any kind of jail break, reverse engineering or crack of any Feidee's software or app.
66
+
67
+ The trademark Feidee, Feidee MyMoney, kbf file format and database design are intellecture properties of Feidee, or whoever the owners are.
data/Rakefile ADDED
@@ -0,0 +1,24 @@
1
+ require 'rake/testtask'
2
+ require 'fileutils'
3
+
4
+ Rake::TestTask.new do |t|
5
+ t.libs << 'test'
6
+ t.pattern = 'test/**/*_test.rb'
7
+ end
8
+
9
+ namespace :sync do
10
+ desc "Sync source file with the main project."
11
+ task :source do
12
+ puts "copying source code..."
13
+ FileUtils.cp_r Dir.glob("lib/*"), "/Users/muyiliqing/Git/cloud-ruby-dev/lib/"
14
+ end
15
+ task :scallion do
16
+ puts "copying source code..."
17
+ FileUtils.cp_r Dir.glob("lib/*"), "/Users/muyiliqing/Git/scallion/lib/"
18
+ end
19
+ end
20
+
21
+ desc "Run tests"
22
+ task :default => :test
23
+ task :deploy => :'sync:source'
24
+ task :scallion => :'sync:scallion'
@@ -0,0 +1,132 @@
1
+ require 'feidee_utils/record'
2
+ require 'feidee_utils/account_group'
3
+ require 'bigdecimal'
4
+
5
+ module FeideeUtils
6
+ class Account < Record
7
+ def validate_integrity
8
+ raise "Account type should always be 0, but it's #{field["type"]}.\n" + inspect unless field["type"] == 0
9
+ raise "Account usedCount should always be 0, but it's #{field["usedCount"]}.\n" + inspect unless field["usedCount"] == 0
10
+ raise "Account uuid should always be empty, but it's #{field["uuid"]}.\n" + inspect unless field["uuid"].to_s.empty?
11
+ raise "Account hierachy contains more than 2 levels.\n" + inspect unless flat_parent_hierachy?
12
+ raise "Account hidden should be either 0 or 1, but it's #{raw_hidden}.\n" + inspect unless (raw_hidden == 1 or raw_hidden == 0)
13
+ end
14
+
15
+ def self.validate_global_integrity
16
+ if self.find_by_id(-1) != nil
17
+ raise "-1 is used as the parent POID placeholder of a parent account. " +
18
+ "Account of POID -1 should not exist."
19
+ end
20
+ end
21
+
22
+ FieldMappings = {
23
+ name: "name",
24
+ raw_balance: "balance",
25
+ raw_credit: "amountOfCredit",
26
+ raw_debit: "amountOfLiability",
27
+ currency: "currencyType",
28
+ parent_poid: "parent",
29
+ memo: "memo",
30
+ ordered: "ordered",
31
+ # Examples: saving accounts, credit cards, cash, insurances and so on.
32
+ account_group_poid: "accountGroupPOID",
33
+ raw_hidden: "hidden",
34
+ }.freeze
35
+
36
+ IgnoredFields = [
37
+ "tradingEntityPOID",
38
+ "type", # Always 0
39
+ "usedCount", # Always 0
40
+ "uuid", # Always empty.
41
+ "code", # WTF
42
+ "clientID", # WTF
43
+ ].freeze
44
+
45
+ define_accessors(FieldMappings)
46
+
47
+ # NOTE: balance is not set for credit cards etc. Instead
48
+ # credit/debit are used.
49
+ # Guess: The special behavior is be controlled by
50
+ # account_group_poid. Again, the code can work in all cases,
51
+ # thus no check is done.
52
+ def balance
53
+ to_bigdecimal(raw_balance) + credit - debit
54
+ end
55
+
56
+ def credit
57
+ to_bigdecimal(raw_credit)
58
+ end
59
+
60
+ def debit
61
+ to_bigdecimal(raw_debit)
62
+ end
63
+
64
+ def hidden?
65
+ raw_hidden == 1
66
+ end
67
+
68
+ def account_group
69
+ self.class.environment::AccountGroup.find_by_id(account_group_poid)
70
+ end
71
+
72
+ # Parent related.
73
+ def parent
74
+ self.class.find_by_id(parent_poid)
75
+ end
76
+
77
+ def has_parent?
78
+ parent_poid != 0 && !flagged_as_parent?
79
+ end
80
+
81
+ def flagged_as_parent?
82
+ # Account with POID -1 doesn't exist. It's just a special
83
+ # POID used to indicate that this account itself is the parent
84
+ # of some other accounts.
85
+ parent_poid == -1
86
+ end
87
+
88
+ def flat_parent_hierachy?
89
+ !has_parent? or parent.flagged_as_parent?
90
+ end
91
+
92
+ def children
93
+ arr = []
94
+ self.class.database.query("SELECT * FROM #{self.class.table_name} WHERE parent = ?", poid) do |result|
95
+ result.each do |raw_row|
96
+ arr << self.class.new(result.columns, result.types, raw_row)
97
+ end
98
+ end
99
+ arr
100
+ end
101
+
102
+ class ModifiedAccount < Record::ModifiedRecord
103
+ define_custom_methods([
104
+ :balance,
105
+ :credit,
106
+ :debit,
107
+ :parent,
108
+ ])
109
+ define_default_methods(FieldMappings)
110
+ end
111
+
112
+ # Schema:
113
+ # accountPOID LONG NOT NULL,
114
+ # name varchar(100) NOT NULL,
115
+ # tradingEntityPOID integer NOT NULL,
116
+ # lastUpdateTime] LONG,
117
+ # usedCount integer DEFAULT 0,
118
+ # accountGroupPOID integer,
119
+ # balance decimal(12, 2),
120
+ # currencyType varchar(50) default 'CNY',
121
+ # memo varchar(200),
122
+ # type integer DEFAULT 0,
123
+ # amountOfLiability decimal(12, 2)DEFAULT 0,
124
+ # amountOfCredit decimal(12, 2)DEFAULT 0,
125
+ # ordered integer default 0,
126
+ # code VARCHAR(20),
127
+ # parent LONG default 0,
128
+ # hidden integer default 0,
129
+ # clientID LONG default 0,
130
+ # uuid varchar(200),
131
+ end
132
+ end
@@ -0,0 +1,54 @@
1
+ require 'feidee_utils/record'
2
+ require 'feidee_utils/mixins/parent_and_path'
3
+ require 'feidee_utils/mixins/type'
4
+
5
+ module FeideeUtils
6
+ class AccountGroup < Record
7
+ include FeideeUtils::Mixins::ParentAndPath
8
+ include FeideeUtils::Mixins::Type
9
+
10
+ def validate_integrity
11
+ validate_depth_integrity
12
+ validate_one_level_path_integrity
13
+
14
+ unless (poid == 1 and raw_type == -1) or (raw_type >= 0 and raw_type <= 2)
15
+ raise "Unexpected account group type #{raw_type}.\n" + inspect
16
+ end
17
+ end
18
+
19
+ FieldMappings = {
20
+ name: "name",
21
+ parent_poid: "parentAccountGroupPOID",
22
+ raw_path: "path",
23
+ depth: "depth",
24
+ raw_type: "type",
25
+ ordered: "ordered",
26
+ }.freeze
27
+
28
+ IgnoredFields = [
29
+ "userTradingEntityPOID", # WTF
30
+ "_tempIconName", # Icon name in the app
31
+ "clientID", # WTF
32
+ ].freeze
33
+
34
+ define_accessors(FieldMappings)
35
+
36
+ define_type_enum({
37
+ 0 => :asset,
38
+ 1 => :liability,
39
+ 2 => :claim,
40
+ })
41
+
42
+ # Schema
43
+ # accountGroupPOID long not null
44
+ # name varchar(100) not null
45
+ # parentAccountGroupPOID long not null
46
+ # path varchar(200)
47
+ # depth integer
48
+ # lastUpdateTime long
49
+ # userTradingEntityPOID long
50
+ # _tempIconName varchar(100) default defaultAccountGroupIcon
51
+ # type integer default 1
52
+ # ordered integer default 0
53
+ end
54
+ end
@@ -0,0 +1,76 @@
1
+ require 'feidee_utils/record'
2
+ require 'feidee_utils/mixins/parent_and_path'
3
+ require 'feidee_utils/mixins/type'
4
+
5
+ module FeideeUtils
6
+ class Category < Record
7
+ include FeideeUtils::Mixins::ParentAndPath
8
+ include FeideeUtils::Mixins::Type
9
+
10
+ def validate_integrity
11
+ validate_depth_integrity
12
+ validate_one_level_path_integrity
13
+ raise "Category usedCount should always be 0, but it's #{field["usedCount"]}.\n" + inspect unless field["usedCount"] == 0
14
+ end
15
+
16
+ def self.validate_global_integrity
17
+ project_root_code = 2
18
+ if TypeEnum[project_root_code] != :project_root
19
+ raise "The type code of project root has been changed, please update the code."
20
+ end
21
+
22
+ rows = self.database.execute <<-SQL
23
+ SELECT #{id_field_name}, #{FieldMappings[:name]} FROM #{table_name}
24
+ WHERE #{FieldMappings[:raw_type]}=#{project_root_code};
25
+ SQL
26
+
27
+ if rows.length > 1
28
+ poids = rows.map do |row| row[0] end
29
+ raise "More than one category have type project_root. IDs are #{poids.inspect}."
30
+ elsif rows.length == 1
31
+ category_name = rows[0][1]
32
+ if category_name != "projectRoot" and category_name != "root"
33
+ raise "Category #{category_name} has type project_root. ID: #{rows[0][0]}."
34
+ end
35
+ end
36
+ end
37
+
38
+ FieldMappings = {
39
+ name: "name",
40
+ parent_poid: "parentCategoryPOID",
41
+ raw_path: "path",
42
+ depth: "depth",
43
+ raw_type: "type",
44
+ ordered: "ordered",
45
+ }.freeze
46
+
47
+ IgnoredFields = [
48
+ "userTradingEntityPOID", # WTF
49
+ "_tempIconName", # Icon name in the app
50
+ "usedCount", # Always 0.
51
+ "clientID", # WTF
52
+ ].freeze
53
+
54
+ define_accessors(FieldMappings)
55
+
56
+ define_type_enum({
57
+ 0 => :expenditure,
58
+ 1 => :income,
59
+ 2 => :project_root, # unkown
60
+ })
61
+
62
+ # Schema
63
+ # categoryPOID LONG NOT NULL
64
+ # name varchar(100) NOT NULL
65
+ # parentCategoryPOID LONG NOT NULL
66
+ # path VARCHAR(200)
67
+ # depth INTEGER
68
+ # lastUpdateTime LONG
69
+ # userTradingEntityPOID LONG
70
+ # _tempIconName VARCHAR(100) DEFAULT defaultIcon,
71
+ # usedCount integer default 0,
72
+ # type integer default 0,
73
+ # ordered integer default 0,
74
+ # clientID LONG default 0,
75
+ end
76
+ end
@@ -0,0 +1,190 @@
1
+ require 'sqlite3'
2
+ require 'feidee_utils/record'
3
+
4
+ # A thin wrapper around SQLite3
5
+ module FeideeUtils
6
+ class Database < SQLite3::Database
7
+ Tables = {
8
+ accounts: "t_account",
9
+ account_groups: "t_account_group",
10
+ categories: "t_category",
11
+ transactions: "t_transaction",
12
+
13
+ metadata: "t_metadata",
14
+ profile: "t_profile",
15
+ }.freeze
16
+
17
+ PotentialUsefulTables = %w(
18
+ t_account_book
19
+ t_account_info
20
+ t_budget_item
21
+ t_fund_holding
22
+ t_fund_trans
23
+ t_module_stock_holding
24
+ t_module_stock_tran
25
+ ).freeze
26
+
27
+ attr_reader :sqlite_file
28
+ attr_reader :platform, :sqlite_name, :sqlite_timestamp
29
+ attr_reader :missing_tables
30
+ attr_reader :namespaced
31
+
32
+ def initialize(private_sqlite, strip = false)
33
+ @sqlite_file = Database.feidee_to_sqlite(private_sqlite)
34
+
35
+ super(@sqlite_file.path)
36
+
37
+ extract_metadata
38
+ drop_unused_tables if strip
39
+
40
+ @namespaced = Record.generate_namespaced_record_classes(self)
41
+ end
42
+
43
+ def sqlite_backup(dest_file_path)
44
+ self.execute("vacuum;")
45
+
46
+ backup_sqlite_db = SQLite3::Database.new(dest_file_path.to_s)
47
+ backup_obj = SQLite3::Backup.new(backup_sqlite_db, "main", self, "main")
48
+ backup_obj.step(-1)
49
+ backup_obj.finish
50
+ backup_sqlite_db.close
51
+ end
52
+
53
+ def validate_global_integrity
54
+ @namespaced.constants.each do |const|
55
+ @namespaced.const_get(const).validate_global_integrity if const != :Database
56
+ end
57
+ end
58
+
59
+ private
60
+ def all_tables
61
+ rows = self.execute <<-SQL
62
+ SELECT name FROM sqlite_master
63
+ WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
64
+ UNION ALL
65
+ SELECT name FROM sqlite_temp_master
66
+ WHERE type IN ('table','view')
67
+ ORDER BY 1
68
+ SQL
69
+ rows.map do |row| row[0] end.sort
70
+ end
71
+
72
+ def drop_unused_tables
73
+ useful_tables = Tables.values + PotentialUsefulTables
74
+ useful_tables = (useful_tables + useful_tables.map do |x| self.class.trash_table_name(x) end).sort
75
+
76
+ # TODO: Record all tables droped.
77
+ (all_tables - useful_tables).each do |table|
78
+ self.execute("DROP TABLE IF EXISTS #{table}");
79
+ end
80
+
81
+ @missing_tables = Tables.values - all_tables
82
+ if !@missing_tables.empty?
83
+ raise "Missing tables: #{@missing_tables} from kbf backup."
84
+ end
85
+ end
86
+
87
+ def extract_metadata
88
+ @platform = self.execute("SELECT platform from #{Tables[:metadata]}")[0][0];
89
+
90
+ @sqlite_name = self.get_first_row("SELECT accountBookName FROM #{Tables[:profile]};")[0];
91
+
92
+ # This is not recorded in the database, so the lastest lastUpdateTime of all
93
+ # transactions is chosen.
94
+ timestamp = self.get_first_row("SELECT max(lastUpdateTime) FROM #{Tables[:transactions]};")[0]
95
+ @sqlite_timestamp = timestamp == nil ? Time.at(0) : Time.at(timestamp / 1000)
96
+ end
97
+
98
+ class << self
99
+ def open_file(file_name)
100
+ Database.new(File.open(file_name))
101
+ end
102
+
103
+ Header = "SQLite format 3\0".force_encoding("binary")
104
+ FeideeHeader_iOS = "%$^#&!@_@- -!F\xff\0".force_encoding('binary')
105
+ FeideeHeader_Android = ("\0" * 13 + "F\xff\0").force_encoding("binary")
106
+
107
+ def feidee_to_sqlite(private_sqlite, sqlite_file = nil)
108
+ # Discard the first a few bytes content.
109
+ private_header = private_sqlite.read(Header.length)
110
+
111
+ unless [FeideeHeader_iOS, FeideeHeader_Android, Header].include? private_header
112
+ raise "Unexpected header #{private_header.inspect} in private sqlite file."
113
+ end
114
+
115
+ # Write the rest to a tempfile.
116
+ sqlite_file ||= Tempfile.new("kingdee_sqlite", binmode: true)
117
+ sqlite_file.binmode
118
+ sqlite_file.write(Header)
119
+ sqlite_file.write(private_sqlite.read)
120
+ sqlite_file.fsync
121
+ sqlite_file
122
+ end
123
+ end
124
+
125
+ class << self
126
+ NoDeleteSuffixTables = %w(account category tag tradingEntity transaction transaction_template)
127
+
128
+ def trash_table_name name
129
+ NoDeleteSuffixTables.each do |core_name|
130
+ if name == "t_" + core_name then
131
+ return "t_" + "deleted_" + core_name;
132
+ end
133
+ end
134
+
135
+ name + "_delete"
136
+ end
137
+ end
138
+
139
+ AllKnownTables = {
140
+ t_account: "As named",
141
+ t_account_book: "A group of accounts, travel accounts etc.",
142
+ t_account_extra: "Extra Feidee account configs, key/value pair.",
143
+ t_account_group: "A group of accounts, saving/chekcing etc.",
144
+ t_account_info: "Additional info of accounts: banks etc.",
145
+ t_accountgrant: "Feidee account VIP related stuff.",
146
+ t_budget_item: "Used to create budgets. An extension of category.",
147
+ t_binding: "Netbank / credit card / Taobao bindings.",
148
+ t_category: "Transaction categories.",
149
+ t_currency: "Currency types.",
150
+ t_exchange: "Currency exchange rates.",
151
+ t_fund: "List of money manage institute names. Abandoned.",
152
+ t_fund_holding: "Fund accounts.",
153
+ t_fund_trans: "Fund transactions.",
154
+ t_fund_price_history: "Fund price history",
155
+ t_id_seed: "ID seeds for all tables.",
156
+ t_import_history: "As named.",
157
+ t_import_source: "Import data from text messages etc.",
158
+
159
+ # JCT stands for Jia Cai Tong, a software for family book keeping.
160
+ # See http://www.feidee.com/jct/
161
+ # JCT is quite obsolete. All the related tables can be safely ignored.
162
+ t_jct_clientdeviceregist: "Client devices.",
163
+ t_jct_clientdevicestatus: "Client devices status.",
164
+ t_jct_syncbookfilelist: "Name of files synced from other devices.",
165
+ t_jct_usergrant: "Maybe if the user has purchased any service.",
166
+ t_jct_userlog: "As named.",
167
+
168
+ t_local_recent: "Local merchandise used recently",
169
+ t_message: "Kingdee ads.",
170
+ t_metadata: "Database version, client version etc.",
171
+ t_module_stock_holding: "Stock accounts.",
172
+ t_module_stock_info: "Stock rates.",
173
+ t_module_stock_trans: "Stock transactions.",
174
+ t_notification: "If and when a notification has been delivered.",
175
+ t_profile: "User profile, default stuff when configuring account books.",
176
+ t_property: "Data collected on user settings, key/value pair.",
177
+ t_syncResource: "???",
178
+ t_sync_logs: "As named.",
179
+ t_tag: "Other support like roles/merchandise.",
180
+ t_tradingEntity: "Merchandise. Used together with t_user in Debt Center.",
181
+ t_trans_debt: "Transactions in Debt Center.",
182
+ t_trans_debt_group: "Transaction groups in Debt Center.",
183
+ t_transaction: "As named.",
184
+ t_transaction_projectcategory_map: "Transaction project.",
185
+ t_transaction_template: "As named. UI.",
186
+ t_user: "Multi-user support.",
187
+ t_usage_count: "As named. Abandoned."
188
+ }
189
+ end
190
+ end
@@ -0,0 +1,36 @@
1
+ require 'zip'
2
+ require 'sqlite3'
3
+ require 'feidee_utils/database'
4
+ require 'feidee_utils/record'
5
+ require 'feidee_utils/transaction'
6
+ require 'feidee_utils/account'
7
+
8
+ module FeideeUtils
9
+ class Kbf
10
+ DatabaseName = 'mymoney.sqlite'
11
+
12
+ attr_reader :zipfile, :sqlite_db
13
+
14
+ def initialize(input_stream)
15
+ @zipfile = Zip::File.open_buffer(input_stream) do |zipfile|
16
+ zipfile.each do |entry|
17
+ if entry.name == DatabaseName
18
+ # Each call to get_input_stream will create a new stream
19
+ @original_sqlite_db_entry = entry
20
+ @sqlite_db = FeideeUtils::Database.new(entry.get_input_stream, true)
21
+ end
22
+ end
23
+ end
24
+ end
25
+
26
+ def extract_original_sqlite(dest_file_path = nil)
27
+ FeideeUtils::Database.feidee_to_sqlite(@original_sqlite_db_entry.get_input_stream, dest_file_path)
28
+ end
29
+
30
+ class << self
31
+ def open_file(file_name)
32
+ return Kbf.new(File.open(file_name))
33
+ end
34
+ end
35
+ end
36
+ end
@@ -0,0 +1,69 @@
1
+ module FeideeUtils
2
+ module Mixins
3
+ # Requires:
4
+ # instance methods: poid, parent_poid, raw_path
5
+ # class methods: find_by_id
6
+ module ParentAndPath
7
+ class InconsistentDepthException < Exception
8
+ end
9
+
10
+ class InconsistentPathException < Exception
11
+ end
12
+
13
+ def validate_depth_integrity
14
+ path_depth = path.length - 1
15
+ if path_depth != depth
16
+ raise InconsistentDepthException,
17
+ "Path is #{path}, but the given depth is #{depth}.\n" +
18
+ inspect
19
+ end
20
+ end
21
+
22
+ def validate_one_level_path_integrity
23
+ path_array = path.clone
24
+ last_poid = path_array.pop
25
+
26
+ if last_poid != poid
27
+ raise InconsistentPathException,
28
+ "The last node in path is #{last_poid}, but the current poid is #{poid}.\n" +
29
+ inspect
30
+ end
31
+
32
+ if has_parent? and path_array != parent.path
33
+ raise InconsistentPathException,
34
+ "Path is #{path}, but path of parent is #{parent.path}.\n" +
35
+ inspect
36
+ end
37
+ end
38
+
39
+ def validate_path_integrity_hard
40
+ cur = self
41
+ step = 0
42
+ while cur or step != path.length
43
+ step += 1
44
+ poid = path[-step]
45
+ if !cur or poid == nil or poid != cur.poid
46
+ raise InconsistentPathException,
47
+ "Reverse path and trace are different at step #{step}. " +
48
+ "Path shows #{poid}, but trace shows #{cur and cur.poid}.\n" +
49
+ inspect
50
+ end
51
+
52
+ cur = cur.has_parent? && cur.parent
53
+ end
54
+ end
55
+
56
+ def path
57
+ @path ||= (raw_path.split("/").map do |poid| poid.to_i end)[1..-1]
58
+ end
59
+
60
+ def parent
61
+ self.class.find_by_id(parent_poid)
62
+ end
63
+
64
+ def has_parent?
65
+ parent_poid != 0
66
+ end
67
+ end
68
+ end
69
+ end
@@ -0,0 +1,34 @@
1
+ module FeideeUtils
2
+ module Mixins
3
+ # Requires:
4
+ # instance methods: raw_type
5
+ module Type
6
+ module ClassMethods
7
+ def define_type_enum type_enum, reverse_lookup = true
8
+ const_set :TypeEnum, type_enum.freeze
9
+
10
+ if reverse_lookup
11
+ enum_values = type_enum.values
12
+ if enum_values.size != enum_values.uniq.size
13
+ raise "Duplicate values in enum #{type_enum}."
14
+ end
15
+
16
+ const_set :TypeCode, type_enum.invert.freeze
17
+ define_singleton_method :type_code do |type_enum_value|
18
+ self::TypeCode[type_enum_value]
19
+ end
20
+ end
21
+ end
22
+ end
23
+
24
+ def self.included klass
25
+ klass.extend ClassMethods
26
+ end
27
+
28
+ def type
29
+ self.class::TypeEnum[raw_type]
30
+ end
31
+ end
32
+ end
33
+ end
34
+
@@ -0,0 +1,22 @@
1
+ module FeideeUtils
2
+ class Record
3
+ module Accessors
4
+ def poid
5
+ @field[self.class.id_field_name]
6
+ end
7
+
8
+ def last_update_time
9
+ timestamp_to_time(@field["lastUpdateTime"])
10
+ end
11
+
12
+ module ClassMethods
13
+ def define_accessors field_mappings
14
+ field_mappings.each do |name, key|
15
+ raise "Accessor #{name} already exists in #{self.name}." if method_defined? name
16
+ define_method name do field[key] end
17
+ end
18
+ end
19
+ end
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,76 @@
1
+ module FeideeUtils
2
+ class Record
3
+ # TODO: Reconsider this class and ship full support to all entities.
4
+ class ModifiedRecord
5
+ attr_reader :poid
6
+ attr_reader :base, :head
7
+ attr_reader :modified_fields
8
+
9
+ def initialize(poid, base, head)
10
+ raise "Base row doesn't have the given poid." if base.poid != poid
11
+ raise "Head row doesn't have the given poid." if head.poid != poid
12
+ @poid = poid
13
+ @base = base
14
+ @head = head
15
+ @modified_fields = self.class.fields_diff(base.field, head.field)
16
+ end
17
+
18
+ class ValuePair
19
+ attr_reader :old_value, :new_value
20
+ def initialize(old_value, new_value)
21
+ @old = old_value
22
+ @new = new_value
23
+ end
24
+ end
25
+
26
+ def self.fields_diff base, head
27
+ (base.keys.sort | head.keys.sort).inject({}) do |hash, key|
28
+ if base[key] != head[key]
29
+ hash[key] = ValuePair.new(base[key], head[key])
30
+ end
31
+ hash
32
+ end
33
+ end
34
+
35
+ def touched?
36
+ !modified_fields.empty?
37
+ end
38
+
39
+ def changed?
40
+ methods.inject(false) do |acc, name|
41
+ if name.to_s.end_with? "_changed?"
42
+ acc ||= send name
43
+ end
44
+ acc
45
+ end
46
+ end
47
+
48
+ protected
49
+ def self.define_custom_methods fields
50
+ fields.each do |name|
51
+ if !respond_to? name
52
+ define_method name do
53
+ ValuePair.new((base.send name), (head.send name))
54
+ end
55
+ define_method (name.to_s + "_changed?").to_sym do
56
+ (base.send name) != (head.send name)
57
+ end
58
+ end
59
+ end
60
+ end
61
+
62
+ def self.define_default_methods field_mappings
63
+ field_mappings.each do |name, key|
64
+ if !respond_to? name
65
+ define_method name do
66
+ modified_fields[key]
67
+ end
68
+ define_method (name.to_s + "_changed?").to_sym do
69
+ modified_fields.has_key? key
70
+ end
71
+ end
72
+ end
73
+ end
74
+ end
75
+ end
76
+ end
@@ -0,0 +1,39 @@
1
+ require 'set'
2
+
3
+ module FeideeUtils
4
+ class Record
5
+ module Namespaced
6
+ module ClassMethods
7
+ attr_reader :child_classes
8
+
9
+ # Must be invoked by Record.inherited
10
+ def collect_subclass(child_class)
11
+ @child_classes ||= Set.new
12
+ @child_classes.add(child_class)
13
+ end
14
+
15
+ # To use Record with different databases, generate a set of classes for each db
16
+ def generate_namespaced_record_classes(db)
17
+ @child_classes ||= Set.new
18
+ this = self
19
+ Module.new do |mod|
20
+ const_set(:Database, Module.new {
21
+ define_method("database") { db }
22
+ define_method("environment") { mod }
23
+ })
24
+
25
+ this.child_classes.each do |child_class|
26
+ if child_class.name.start_with? FeideeUtils.name
27
+ class_name = child_class.name.sub(/#{FeideeUtils.name}::/, '')
28
+ # Generate a const for the child class
29
+ const_set(class_name, Class.new(child_class) {
30
+ extend mod::Database
31
+ })
32
+ end
33
+ end
34
+ end
35
+ end
36
+ end
37
+ end
38
+ end
39
+ end
@@ -0,0 +1,55 @@
1
+
2
+ module FeideeUtils
3
+ class Record
4
+ module Persistent
5
+ module ClassMethods
6
+ # Names
7
+ # Must be invoked by Record.inherited
8
+ def genereate_names subclass
9
+ entity_name =
10
+ if i = subclass.name.rindex("::")
11
+ subclass.name[(i+2)..-1]
12
+ else
13
+ subclass.name
14
+ end
15
+
16
+ id_field_name = entity_name.sub(/^[A-Z]/) { $&.downcase } + "POID"
17
+ table_name = "t_" + entity_name.gsub(/([a-z\d])([A-Z\d])/, '\1_\2').downcase
18
+ subclass.class_exec do
19
+ define_singleton_method :entity_name do entity_name end
20
+ define_singleton_method :id_field_name do id_field_name end
21
+ define_singleton_method :table_name do table_name end
22
+ end
23
+ end
24
+
25
+ # Persistent
26
+ def all
27
+ arr = []
28
+ database.query("SELECT * FROM #{self.table_name}") do |result|
29
+ result.each do |raw_row|
30
+ arr << self.new(result.columns, result.types, raw_row)
31
+ end
32
+ end
33
+ arr
34
+ end
35
+
36
+ def find_by_id(id)
37
+ raw_result = database.query("SELECT * FROM #{self.table_name} WHERE #{self.id_field_name} = ?", id)
38
+
39
+ raw_row = raw_result.next
40
+ return nil if raw_row == nil
41
+
42
+ if raw_result.next != nil
43
+ raise "Getting more than one result with the same ID #{id} in table #{self.table_name}."
44
+ end
45
+
46
+ self.new(raw_result.columns, raw_result.types, raw_row)
47
+ end
48
+
49
+ def find(id)
50
+ find_by_id(id) or raise "No #{self.name} of poid #{id} found"
51
+ end
52
+ end
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,15 @@
1
+ module FeideeUtils
2
+ class Record
3
+ module Utils
4
+ protected
5
+ def timestamp_to_time num
6
+ Time.at(num / 1000.0, num % 1000)
7
+ end
8
+
9
+ def to_bigdecimal number
10
+ # Be aware of the precision lost from String -> Float -> BigDecimal.
11
+ BigDecimal.new(number, 12).round(2)
12
+ end
13
+ end
14
+ end
15
+ end
@@ -0,0 +1,66 @@
1
+ require 'feidee_utils/record/accessors'
2
+ require 'feidee_utils/record/namespaced'
3
+ require 'feidee_utils/record/persistent'
4
+ require 'feidee_utils/record/utils'
5
+ require 'feidee_utils/record/modified_record'
6
+
7
+ module FeideeUtils
8
+ # The implementation here is wired.
9
+ # The goal is to create a class hierachy similar to ActiveRecord, where every table is represented by a
10
+ # subclass of ActiveRecord::Base. Class methods, attribute accessors and almost all other functionalities
11
+ # are provided by ActiveRecord::Base. For example, Base.all(), Base.find_by_id() are tied to a specific
12
+ # table in a specific database.
13
+ # The problem we are solving here is not the same as ActiveRecord. In ActiveRecord, the databases
14
+ # are static, i.e. they won't be changed at runtime. Meanwhile, in our case, new databases can be created
15
+ # at runtime, when a new KBF backup file is uploaded. Furthermore, multiple instances of different databases
16
+ # can co-exist at the same time. To provide the same syntax as ActiveRecord, a standalone "Base" class has
17
+ # to be created for each database.
18
+ # In our implementation, when a new database is created, a subclass of Record is created in a new namepsace.
19
+ # For each subclass of Record, a new subclass is copied to the new namespace, with it's database method
20
+ # overloaded.
21
+ class Record
22
+ attr_reader :field, :field_type
23
+
24
+ public
25
+ def initialize(columns, types, raw_row)
26
+ @field = Hash[ columns.zip(raw_row) ]
27
+ @field_type = Hash[ columns.zip(types) ]
28
+
29
+ validate_integrity
30
+ end
31
+
32
+ def validate_integrity
33
+ # Do nothing.
34
+ end
35
+
36
+ def self.validate_global_integrity
37
+ # Do nothing.
38
+ end
39
+
40
+ class << self
41
+ protected
42
+ def database
43
+ raise NotImplementedError.new("Subclasses must set database")
44
+ end
45
+
46
+ private
47
+ def inherited subclass
48
+ if subclass.name != nil and subclass.name.start_with? FeideeUtils.name
49
+ collect_subclass subclass
50
+ genereate_names subclass
51
+ end
52
+ end
53
+ end
54
+
55
+ # Basic accessors, poid, last update time, etc.
56
+ include Accessors
57
+ # Helper methods to define accessors
58
+ extend Accessors::ClassMethods
59
+ # Helper methods to define new classes in a given namespace.
60
+ extend Namespaced::ClassMethods
61
+ # Helper methods to look up records.
62
+ extend Persistent::ClassMethods
63
+ # Helper methods to convert data types.
64
+ include Utils
65
+ end
66
+ end
@@ -0,0 +1,232 @@
1
+ require 'feidee_utils/record'
2
+ require 'feidee_utils/mixins/type'
3
+
4
+ module FeideeUtils
5
+ class Transaction < Record
6
+ include FeideeUtils::Mixins::Type
7
+
8
+ class TransferLackBuyerOrSellerException < Exception
9
+ end
10
+
11
+ class TransferWithCategoryException < Exception
12
+ end
13
+
14
+ class InconsistentBuyerAndSellerSetException < Exception
15
+ end
16
+
17
+ class InconsistentCategoryException < Exception
18
+ end
19
+
20
+ class InconsistentAmountException < Exception
21
+ end
22
+
23
+ def validate_integrity
24
+ if is_transfer?
25
+ unless buyer_account_poid != 0 and seller_account_poid != 0
26
+ raise TransferLackBuyerOrSellerException,
27
+ "Both buyer and seller should be set in a transfer. " +
28
+ "Buyer account POID: #{buyer_account_poid}. Seller account POID: #{seller_account_poid}.\n" +
29
+ inspect
30
+ end
31
+ unless buyer_category_poid == 0 and seller_category_poid == 0
32
+ raise TransferWithCategoryException,
33
+ "Neither buyer or seller category should be set in a transfer. " +
34
+ "Buyer category POID: #{buyer_category_poid}. Seller category POID: #{seller_category_poid}.\n" +
35
+ inspect
36
+ end
37
+ else
38
+ unless (buyer_account_poid == 0) ^ (seller_account_poid == 0)
39
+ raise InconsistentBuyerAndSellerSetException,
40
+ "Exactly one of buyer and seller should be set in a non-transfer transaction. " +
41
+ "Buyer account POID: #{buyer_account_poid}. Seller account POID: #{seller_account_poid}.\n" +
42
+ inspect
43
+ end
44
+
45
+ # We could enforce that category is set to the matching party (buyer or seller) of account.
46
+ # However the implementation could handle all situations, as long as only one of them is set.
47
+ # Thus no extra check is done here.
48
+ unless buyer_category_poid == 0 or seller_category_poid == 0
49
+ raise InconsistentCategoryException,
50
+ "Only one of buyer and seller category should be set in a non-transfer transaction. "
51
+ "Buyer category POID: #{buyer_category_poid}. Seller category POID: #{seller_category_poid}.\n" +
52
+ inspect
53
+ end
54
+ end
55
+
56
+ unless buyer_deduction == seller_addition
57
+ raise InconsistentAmountException,
58
+ "Buyer and seller should have the same amount set. " +
59
+ "Buyer deduction: #{buyer_deduction}, seller_addition: #{seller_addition}.\n" +
60
+ inspect
61
+ end
62
+ end
63
+
64
+ class TransfersNotPaired < Exception
65
+ end
66
+
67
+ def self.validate_global_integrity
68
+ uuids_map = all.inject({}) do |uuids, transaction|
69
+ if transaction.is_transfer?
70
+ uuid = transaction.uuid
71
+ uuids[uuid] ||= [nil, nil]
72
+ uuids[uuid][transaction.raw_type - 2] = transaction
73
+ end
74
+ uuids
75
+ end
76
+
77
+ uuids_map.each do |uuid, transfers|
78
+ valid = true
79
+ valid &&= transfers[0] != nil
80
+ valid &&= transfers[1] != nil
81
+ valid &&= transfers[0].buyer_account_poid == transfers[1].buyer_account_poid
82
+ valid &&= transfers[0].seller_account_poid == transfers[1].seller_account_poid
83
+ raise TransfersNotPaired.new([uuid] + transfers) unless valid
84
+ end
85
+ end
86
+
87
+ FieldMappings = {
88
+ raw_created_at: "createdTime",
89
+ raw_modified_at: "modifiedTime",
90
+ raw_trade_at: "tradeTime",
91
+ raw_type: "type",
92
+ memo: "memo",
93
+ buyer_account_poid: "buyerAccountPOID",
94
+ buyer_category_poid: "buyerCategoryPOID",
95
+ seller_account_poid: "sellerAccountPOID",
96
+ seller_category_poid: "sellerCategoryPOID",
97
+ raw_buyer_deduction: "buyerMoney",
98
+ raw_seller_addition: "sellerMoney",
99
+ uuid: "relation",
100
+ }.freeze
101
+
102
+ IgnoredFields = [
103
+ "creatorTradingEntityPOID",
104
+ "modifierTradingEntityPOID",
105
+ "ffrom", # The signature of the App writting this transaction.
106
+ "photoName", # To be added
107
+ "photoNeedUpload", # To be added
108
+ "relationUnitPOID", # WTF
109
+ "clientID", # WTF
110
+ "FSourceKey", # WTF
111
+ ].freeze
112
+
113
+ define_accessors(FieldMappings)
114
+
115
+ define_type_enum({
116
+ 0 => :expenditure,
117
+ 1 => :income,
118
+ 2 => :transfer_buyer,
119
+ 3 => :transfer_seller,
120
+ 8 => :positive_initial_balance,
121
+ 9 => :negative_initial_balance,
122
+ })
123
+
124
+ def created_at
125
+ timestamp_to_time(raw_created_at)
126
+ end
127
+
128
+ def modified_at
129
+ timestamp_to_time(raw_modified_at)
130
+ end
131
+
132
+ def trade_at
133
+ timestamp_to_time(raw_trade_at)
134
+ end
135
+
136
+ def has_category?
137
+ category_poid != 0
138
+ end
139
+
140
+ def category_poid
141
+ # At least one of those two must be 0.
142
+ buyer_category_poid + seller_category_poid
143
+ end
144
+
145
+ # Amount accessors
146
+
147
+ def buyer_deduction
148
+ sign_by_type(raw_buyer_deduction)
149
+ end
150
+
151
+ def seller_addition
152
+ sign_by_type(raw_seller_addition)
153
+ end
154
+
155
+ def amount
156
+ # Buyer deduction is always equal to seller addition.
157
+ (buyer_deduction + seller_addition) / 2
158
+ end
159
+
160
+ def is_transfer?
161
+ type == :transfer_buyer or type == :transfer_seller
162
+ end
163
+
164
+ def is_initial_balance?
165
+ type == :positive_initial_balance or type == :negative_initial_balance
166
+ end
167
+
168
+ def revised_account_poid
169
+ if type == :transfer_buyer
170
+ buyer_account_poid
171
+ elsif type == :transfer_seller
172
+ seller_account_poid
173
+ else
174
+ buyer_account_poid + seller_account_poid
175
+ end
176
+ end
177
+
178
+ def revised_amount
179
+ account_poid = revised_account_poid
180
+ if account_poid == buyer_account_poid
181
+ -buyer_deduction
182
+ elsif account_poid == seller_account_poid
183
+ seller_addition
184
+ else
185
+ raise "Unexpected revised account poid #{account_poid}."
186
+ end
187
+ end
188
+
189
+ class ModifiedTransaction < ModifiedRecord
190
+ define_custom_methods([
191
+ :created_at,
192
+ :modified_at,
193
+ :trade_at,
194
+ :type,
195
+ :category_poid,
196
+ :buyer_deduction,
197
+ :seller_addition,
198
+ :amount,
199
+ ])
200
+ define_default_methods(FieldMappings)
201
+ end
202
+
203
+ private
204
+ def sign_by_type num
205
+ raw_type == 9 ? -num : num
206
+ end
207
+
208
+ # Schema:
209
+ # transactionPOID LONG NOT NULL,
210
+ # createdTime LONG NOT NULL,
211
+ # modifiedTime LONG NOT NULL,
212
+ # tradeTime LONG NOT NULL,
213
+ # memo varchar(100),
214
+ # type integer NOT NULL,
215
+ # creatorTradingEntityPOID LONG,
216
+ # modifierTradingEntityPOID LONG,
217
+ # buyerAccountPOID LONG,
218
+ # buyerCategoryPOID LONG default 0,
219
+ # buyerMoney decimal(12, 2),
220
+ # sellerAccountPOID LONG,
221
+ # sellerCategoryPOID LONG default 0,
222
+ # sellerMoney decimal(12, 2),
223
+ # lastUpdateTime LONG,
224
+ # photoName VARCHAR(100),
225
+ # photoNeedUpload integer default 0,
226
+ # relation varchar(200) default '',
227
+ # relationUnitPOID LONG,
228
+ # ffrom varchar(250) default '',
229
+ # clientID LONG default 0,
230
+ # FSourceKey varchar(100) DEFAULT NULL,
231
+ end
232
+ end
@@ -0,0 +1,3 @@
1
+ module FeideeUtils
2
+ VERSION = '0.0.4.1'
3
+ end
@@ -0,0 +1,9 @@
1
+ require 'feidee_utils/kbf'
2
+ require 'feidee_utils/database'
3
+ require 'feidee_utils/record'
4
+ require 'feidee_utils/account'
5
+ require 'feidee_utils/transaction'
6
+ require 'feidee_utils/category'
7
+
8
+ module FeideeUtils
9
+ end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: feidee_utils
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.4
4
+ version: 0.0.4.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Liqing Muyi
@@ -97,7 +97,27 @@ email: muyiliqing@gmail.com
97
97
  executables: []
98
98
  extensions: []
99
99
  extra_rdoc_files: []
100
- files: []
100
+ files:
101
+ - Gemfile
102
+ - MIT-LICENSE
103
+ - README.md
104
+ - Rakefile
105
+ - lib/feidee_utils.rb
106
+ - lib/feidee_utils/account.rb
107
+ - lib/feidee_utils/account_group.rb
108
+ - lib/feidee_utils/category.rb
109
+ - lib/feidee_utils/database.rb
110
+ - lib/feidee_utils/kbf.rb
111
+ - lib/feidee_utils/mixins/parent_and_path.rb
112
+ - lib/feidee_utils/mixins/type.rb
113
+ - lib/feidee_utils/record.rb
114
+ - lib/feidee_utils/record/accessors.rb
115
+ - lib/feidee_utils/record/modified_record.rb
116
+ - lib/feidee_utils/record/namespaced.rb
117
+ - lib/feidee_utils/record/persistent.rb
118
+ - lib/feidee_utils/record/utils.rb
119
+ - lib/feidee_utils/transaction.rb
120
+ - lib/feidee_utils/version.rb
101
121
  homepage: http://github.com/muyiliqing/feidee_utils
102
122
  licenses:
103
123
  - MIT