tsp_scraper 0.4.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (11) hide show
  1. checksums.yaml +5 -5
  2. data/.travis.yml +9 -0
  3. data/README.md +60 -19
  4. data/bin/tsp_scraper +2 -7
  5. data/lib/tsp_scraper.rb +1 -0
  6. data/lib/tsp_scraper/cli.rb +30 -0
  7. data/lib/tsp_scraper/client.rb +8 -9
  8. data/lib/tsp_scraper/converter.rb +4 -4
  9. data/lib/tsp_scraper/version.rb +1 -1
  10. data/tsp_scraper.gemspec +6 -6
  11. metadata +19 -18
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 4d558e5f4183a926944d88c114e5a5b74c086606
4
- data.tar.gz: 04f3e0301c9e82e0a966c52bb5050990e008271d
2
+ SHA256:
3
+ metadata.gz: 06f326a47003377a09a52f6a6943355119373cfd73740f645a7e91f7f4583a91
4
+ data.tar.gz: 0e8f2a09436a87940f7b7fc66b7243d8c4c0b28e88b32e72ebf984b283c28237
5
5
  SHA512:
6
- metadata.gz: f934b572611f8ffdd20237898a87ffaac9b63d2566360c4f970e716bea4b224f549c8d4893a094bb16e0a1f1ac29a508fe43f4557ff65e6aea00a1088dce1fcb
7
- data.tar.gz: bbedffaa821617810c40dba8fed793145d8dca020e5e4ff710a53bc9516dae3b311c29d9e1bc435232e85f3e984422b8a06156056d7dbff13b1420fe8147e25d
6
+ metadata.gz: 397fa873097f209732982fcdf156b2062e240aaf71ea53470861fca6e6a56715aa2b257099b825a1aa28438fa3206ab4fb775ad81b74eadbe136271dabc12540
7
+ data.tar.gz: 342ccc4844395e03c838d88fdcc79bf1bbe139406228cb7304738e720f9311cc6f7edd47e1915d16a8a4e32f3a78f806be6cd7a66881ec484858d99ade698805
data/.travis.yml ADDED
@@ -0,0 +1,9 @@
1
+ language: ruby
2
+ rvm:
3
+ - 2.6
4
+ - 2.5
5
+ - 2.4
6
+ before_install:
7
+ - gem update --system
8
+ - gem install bundler
9
+ cache: bundler
data/README.md CHANGED
@@ -1,6 +1,7 @@
1
1
  # tsp_scraper
2
2
 
3
3
  [![Gem Version](https://badge.fury.io/rb/tsp_scraper.svg)](https://badge.fury.io/rb/tsp_scraper)
4
+ [![Travis CI](https://travis-ci.org/jfredrickson/tsp_scraper.svg?branch=master)](https://travis-ci.org/jfredrickson/tsp_scraper)
4
5
 
5
6
  Scrapes fund price history from the TSP.gov website.
6
7
 
@@ -12,6 +13,10 @@ gem install tsp_scraper
12
13
 
13
14
  ## Usage
14
15
 
16
+ The `tsp_scraper` gem includes both an API and a command-line interface (CLI).
17
+
18
+ ### API Usage
19
+
15
20
  The `scrape` method accepts `Date` objects as parameters and returns TSP price data as an array of hashes:
16
21
 
17
22
  ```ruby
@@ -25,38 +30,74 @@ The returned array of hashes has the format:
25
30
  ```
26
31
  [
27
32
  {
28
- date: DATE1,
29
- funds: [
30
- { fund: "FUND NAME 1", price: "12.3456" },
31
- { fund: "FUND NAME 2", price: "12.3456" },
32
- { fund: "FUND NAME 3", price: "12.3456" }
33
- ]
33
+ :date => DATE1,
34
+ :funds => {
35
+ "FUND NAME 1" => #<BigDecimal>,
36
+ "FUND NAME 2" => #<BigDecimal>,
37
+ "FUND NAME 3" => #<BigDecimal>
38
+ }
34
39
  },
35
40
  {
36
- date: DATE2,
37
- funds: [
38
- { fund: "FUND NAME 1", price: "12.3456" },
39
- { fund: "FUND NAME 2", price: "12.3456" },
40
- { fund: "FUND NAME 3", price: "12.3456" }
41
- ]
41
+ :date => DATE2,
42
+ :funds => {
43
+ "FUND NAME 1" => #<BigDecimal>,
44
+ "FUND NAME 2" => #<BigDecimal>,
45
+ "FUND NAME 3" => #<BigDecimal>
46
+ }
42
47
  },
43
48
  {
44
- date: DATE3,
45
- funds: [
46
- { fund: "FUND NAME 1", price: "12.3456" },
47
- { fund: "FUND NAME 2", price: "12.3456" },
48
- { fund: "FUND NAME 3", price: "12.3456" }
49
- ]
49
+ :date => DATE3,
50
+ :funds => {
51
+ "FUND NAME 1" => #<BigDecimal>,
52
+ "FUND NAME 2" => #<BigDecimal>,
53
+ "FUND NAME 3" => #<BigDecimal>
54
+ }
50
55
  }
51
56
  ]
52
57
  ```
53
58
 
54
59
  You can get the raw CSV data as returned by the TSP website by using `TSPScraper::Client.scrape_raw()`, which accepts the same parameters as the `scrape` method.
55
60
 
56
- ### HTTParty Options
61
+ #### HTTParty Options
57
62
 
58
63
  The `scrape` and `scrape_raw` methods also accept an options hash for the HTTP request. The options hash should contain [HTTParty](https://github.com/jnunemaker/httparty) options.
59
64
 
60
65
  ```ruby
61
66
  TSPScraper::Client.scrape(start_date, end_date, verify: false) # Don't verify SSL certificate
62
67
  ```
68
+
69
+ ### CLI Usage
70
+
71
+ Installing the gem makes the `tsp_scraper` command available to you.
72
+
73
+ ```
74
+ tsp_scraper START_DATE END_DATE
75
+ ```
76
+
77
+ START_DATE and END_DATE should be in the format YYYY-MM-DD.
78
+
79
+ ```
80
+ $ tsp_scraper 2017-01-23 2017-01-24
81
+ 2017-01-24,L Income,18.5121
82
+ 2017-01-24,L 2010,0.0
83
+ 2017-01-24,L 2020,24.7183
84
+ 2017-01-24,L 2030,27.2166
85
+ 2017-01-24,L 2040,29.1416
86
+ 2017-01-24,L 2050,16.6275
87
+ 2017-01-24,G Fund,15.2106
88
+ 2017-01-24,F Fund,17.4619
89
+ 2017-01-24,C Fund,31.4713
90
+ 2017-01-24,S Fund,42.0024
91
+ 2017-01-24,I Fund,25.2087
92
+ 2017-01-23,L Income,18.4934
93
+ 2017-01-23,L 2010,0.0
94
+ 2017-01-23,L 2020,24.6629
95
+ 2017-01-23,L 2030,27.12
96
+ 2017-01-23,L 2040,29.0183
97
+ 2017-01-23,L 2050,16.5456
98
+ 2017-01-23,G Fund,15.2096
99
+ 2017-01-23,F Fund,17.5237
100
+ 2017-01-23,C Fund,31.2661
101
+ 2017-01-23,S Fund,41.434
102
+ 2017-01-23,I Fund,25.1846
103
+ ```
data/bin/tsp_scraper CHANGED
@@ -1,10 +1,5 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- require 'tsp_scraper'
4
- require 'date'
3
+ require "tsp_scraper"
5
4
 
6
- start_date = Date.parse(ARGV[0])
7
- end_date = Date.parse(ARGV[1])
8
-
9
- csv = TSPScraper::Client.scrape(start_date, end_date)
10
- puts csv
5
+ TSPScraper::CLI.start
data/lib/tsp_scraper.rb CHANGED
@@ -1,3 +1,4 @@
1
1
  require 'tsp_scraper/version'
2
2
  require 'tsp_scraper/converter'
3
3
  require 'tsp_scraper/client'
4
+ require 'tsp_scraper/cli'
data/lib/tsp_scraper/cli.rb ADDED
@@ -0,0 +1,30 @@
1
+ module TSPScraper
2
+ class CLI
3
+ def self.start
4
+ begin
5
+ end_date_raw = ARGV.pop
6
+ start_date_raw = ARGV.pop
7
+ start_date = Date.parse(start_date_raw)
8
+ end_date = Date.parse(end_date_raw)
9
+ rescue
10
+ print_help
11
+ exit 1
12
+ end
13
+ quotes = TSPScraper::Client.scrape(start_date, end_date)
14
+ print_quotes quotes
15
+ end
16
+
17
+ def self.print_help
18
+ puts "Usage: tsp_scraper START_DATE END_DATE"
19
+ puts "START_DATE and END_DATE should be in the format YYYY-MM-DD."
20
+ end
21
+
22
+ def self.print_quotes(quotes)
23
+ quotes.each do |quote|
24
+ quote[:funds].each do |name, price|
25
+ puts "#{quote[:date]},#{name},#{price.to_s('F')}"
26
+ end
27
+ end
28
+ end
29
+ end
30
+ end
data/lib/tsp_scraper/client.rb CHANGED
@@ -4,22 +4,21 @@ module TSPScraper
4
4
  class Client
5
5
  include HTTParty
6
6
  disable_rails_query_string_format
7
- base_uri "https://www.tsp.gov/InvestmentFunds/FundPerformance"
7
+ base_uri "https://secure.tsp.gov/components/CORS"
8
8
 
9
9
  def self.scrape_raw(start_date = Date.today.prev_month, end_date = Date.today, options = {})
10
- funds = ["Linc", "L2010", "L2020", "L2030", "L2040", "L2050", "G", "F", "C", "S", "I"]
11
10
  default_options = {
12
11
  query: {
13
- reloaded: 1,
14
- startdate: start_date.strftime("%m/%d/%Y"),
15
- enddate: end_date.strftime("%m/%d/%Y"),
16
- whichButton: "CSV",
17
- fundgroup: funds
12
+ download: 1,
13
+ startdate: start_date.strftime("%Y%m%d"),
14
+ enddate: end_date.strftime("%Y%m%d"),
15
+ format: "CSV",
16
+ Lfunds: 1,
17
+ InvFunds: 1
18
18
  }
19
19
  }
20
20
  options = default_options.merge(options)
21
-
22
- response = self.get("/index.html", options)
21
+ response = self.get("/getSharePrices.html", options)
23
22
  response.body.strip
24
23
  end
25
24
 
data/lib/tsp_scraper/converter.rb CHANGED
@@ -2,20 +2,20 @@ module TSPScraper
2
2
  class Converter
3
3
  # TSP CSV has a janky format:
4
4
  # - Each field in a row, except for the first field, is prepended by a space
5
- # - Each row has a final column containing just a space
6
5
  # For this reason, we manually clean up the CSV instead of using the Ruby CSV library, especially since we know TSP
7
6
  # does not use quotation marks in their CSV fields... at least for now.
8
7
  def self.raw_csv_to_hash(raw_csv)
9
- raw_data = raw_csv.lines.map { |row| row.strip.chomp(',').split(',').map { |field| field.strip } }
8
+ raw_data = raw_csv.lines.map { |row| row.strip.split(',').map { |field| field.strip } }
10
9
  headers = raw_data.shift
11
10
  data = []
12
11
  raw_data.each do |d|
13
12
  hash = {
14
13
  date: Date.parse(d.first),
15
- funds: []
14
+ funds: {}
16
15
  }
17
16
  headers.each_with_index do |header, index|
18
- hash[:funds].push({ fund: header, price: BigDecimal.new(d[index]) }) unless index == 0
17
+ next if index == 0
18
+ hash[:funds][header] = BigDecimal(d[index]) unless d[index].empty?
19
19
  end
20
20
  data.push(hash)
21
21
  end
data/lib/tsp_scraper/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module TSPScraper
2
- VERSION = "0.4.0"
2
+ VERSION = "0.9.0"
3
3
  end
data/tsp_scraper.gemspec CHANGED
@@ -13,12 +13,12 @@ Gem::Specification.new do |s|
13
13
  s.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
14
14
  s.require_paths = ['lib']
15
15
  s.license = 'MIT'
16
- s.required_ruby_version = '>= 1.9.3'
16
+ s.required_ruby_version = '>= 2.4.0'
17
17
  s.executables = ['tsp_scraper']
18
18
 
19
- s.add_runtime_dependency 'httparty', '~> 0.13'
20
- s.add_development_dependency 'bundler', '~> 1.13'
21
- s.add_development_dependency 'rake', '~> 10.0'
22
- s.add_development_dependency 'pry', '~> 0.10'
23
- s.add_development_dependency 'minitest', '~> 5.9'
19
+ s.add_runtime_dependency 'httparty', '~> 0.17.1'
20
+ s.add_development_dependency 'bundler', '~> 2.1.4'
21
+ s.add_development_dependency 'rake', '~> 13.0.0'
22
+ s.add_development_dependency 'pry', '~> 0.12.2'
23
+ s.add_development_dependency 'minitest', '~> 5.12.2'
24
24
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: tsp_scraper
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.0
4
+ version: 0.9.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jeff Fredrickson
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-09-16 00:00:00.000000000 Z
11
+ date: 2020-07-16 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: httparty
@@ -16,70 +16,70 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '0.13'
19
+ version: 0.17.1
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '0.13'
26
+ version: 0.17.1
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: bundler
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
31
  - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: '1.13'
33
+ version: 2.1.4
34
34
  type: :development
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '1.13'
40
+ version: 2.1.4
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: rake
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
45
  - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: '10.0'
47
+ version: 13.0.0
48
48
  type: :development
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: '10.0'
54
+ version: 13.0.0
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: pry
57
57
  requirement: !ruby/object:Gem::Requirement
58
58
  requirements:
59
59
  - - "~>"
60
60
  - !ruby/object:Gem::Version
61
- version: '0.10'
61
+ version: 0.12.2
62
62
  type: :development
63
63
  prerelease: false
64
64
  version_requirements: !ruby/object:Gem::Requirement
65
65
  requirements:
66
66
  - - "~>"
67
67
  - !ruby/object:Gem::Version
68
- version: '0.10'
68
+ version: 0.12.2
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: minitest
71
71
  requirement: !ruby/object:Gem::Requirement
72
72
  requirements:
73
73
  - - "~>"
74
74
  - !ruby/object:Gem::Version
75
- version: '5.9'
75
+ version: 5.12.2
76
76
  type: :development
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
80
  - - "~>"
81
81
  - !ruby/object:Gem::Version
82
- version: '5.9'
82
+ version: 5.12.2
83
83
  description: Scrapes fund price history from the TSP.gov website.
84
84
  email:
85
85
  - jeff.fredrickson@gmail.com
@@ -89,6 +89,7 @@ extensions: []
89
89
  extra_rdoc_files: []
90
90
  files:
91
91
  - ".gitignore"
92
+ - ".travis.yml"
92
93
  - Gemfile
93
94
  - LICENSE
94
95
  - README.md
@@ -97,6 +98,7 @@ files:
97
98
  - bin/setup
98
99
  - bin/tsp_scraper
99
100
  - lib/tsp_scraper.rb
101
+ - lib/tsp_scraper/cli.rb
100
102
  - lib/tsp_scraper/client.rb
101
103
  - lib/tsp_scraper/converter.rb
102
104
  - lib/tsp_scraper/version.rb
@@ -105,7 +107,7 @@ homepage: https://github.com/jfredrickson/tsp_scraper
105
107
  licenses:
106
108
  - MIT
107
109
  metadata: {}
108
- post_install_message:
110
+ post_install_message:
109
111
  rdoc_options: []
110
112
  require_paths:
111
113
  - lib
@@ -113,16 +115,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
113
115
  requirements:
114
116
  - - ">="
115
117
  - !ruby/object:Gem::Version
116
- version: 1.9.3
118
+ version: 2.4.0
117
119
  required_rubygems_version: !ruby/object:Gem::Requirement
118
120
  requirements:
119
121
  - - ">="
120
122
  - !ruby/object:Gem::Version
121
123
  version: '0'
122
124
  requirements: []
123
- rubyforge_project:
124
- rubygems_version: 2.5.0
125
- signing_key:
125
+ rubygems_version: 3.0.3
126
+ signing_key:
126
127
  specification_version: 4
127
128
  summary: Scrapes fund price history from the TSP.gov website
128
129
  test_files: []