ny_frb_as_dataframe 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 037b31b64679ee33a6a8632a80e5a9fcc9d09939b9e45c8499a561c624f98816
+  data.tar.gz: 96a90d12efa519f0633a2162cf2f263769395cfcd2b992af3dbb235e37a0577c
+SHA512:
+  metadata.gz: 30eea77268f8ebab0ca02efbcfb1f5531046123738b1bb57796663cc9677ada2fbcdf61d688ce27a92352bb9c3bc6526b8ec37b82900d838575a60965466a803
+  data.tar.gz: d34d36721197a590d1e16c4fec87657e68c5723047a4c674ded9d6233cf4280f61929420d253ef7df9bb7a5afd1b5479f3b4df6ba95123e602d2b9faa8cdaab2

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+Gemfile.lock

data/.travis.yml

@@ -0,0 +1,6 @@
+---
+language: ruby
+cache: bundler
+rvm:
+  - 3.0.5
+before_install: gem install bundler -v 2.2.33

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2024-08-12
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,84 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at bill.mckinnon@bmck.org. All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior,  harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0,
+available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations.

data/Gemfile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in ny_frb_as_dataframe.gemspec
+gemspec
+
+gem "rake", "~> 13.0"

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2024 Bill McKinnon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,43 @@
+# NyFrbAsDataframe
+
+Up to date remote economic data access for ruby, using Polars dataframes. 
+
+This package will fetch economic and financial information from the Federal Reserve Bank of Philadelphia, and return the results as a Polars dataframe. 
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'ny_frb_as_dataframe'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install ny_frb_as_dataframe
+
+## Usage
+
+TODO: Write usage instructions here
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/ny_frb_as_dataframe. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/ny_frb_as_dataframe/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the NyFrbAsDataframe project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/ny_frb_as_dataframe/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+task default: %i[]

data/bin/console

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "ny_frb_as_dataframe"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/ny_frb_as_dataframe/hhdc.rb

@@ -0,0 +1,176 @@
+require 'polars-df'
+require 'httparty'
+require 'simple_xlsx_reader'
+require 'csv'
+require 'active_support/core_ext/date/calculations'
+require 'active_support/core_ext/string/conversions'
+
+
+module NyFrbAsDataframe
+  class Hhdc
+
+    include ::HTTParty
+    base_uri 'https://www.newyorkfed.org/medialibrary/interactives/householdcredit/data/xls/'
+
+    attr_reader :tag
+
+    def initialize(series = nil)
+      @tag = series
+    end
+
+    def fetch(start = nil, fin = nil)
+      df = nil
+
+      SimpleXlsxReader.configuration.auto_slurp = true
+
+      Tempfile.open(['foo', '.xlsx'], '/tmp', binmode: true) do |f|
+        qtr = Date.today.year * 4 + Date.today.quarter
+        dta = nil
+        resp = nil
+
+        loop do
+          u = "https://www.newyorkfed.org/medialibrary/interactives/householdcredit/data/xls/hhd_c_report_#{qtr.div(4)}q#{qtr.modulo(4)+1}.xlsx"
+          dta = self.class.get(u)
+          resp = dta.parsed_response
+          break if (dta.response.code.to_i == 200 && dta.parsed_response.encoding.to_s == "ASCII-8BIT")
+          qtr -= 1
+        end
+        f.write(resp)
+        f.rewind
+
+        xlsx_file = SimpleXlsxReader.open(f.path)
+
+        col_nm_hsh = 
+          {
+            'Total Debt Balance and Its Composition' => {'Mortgage' => 'Mort Debt Balance ($T)', 'HE Revolving' => 'HELOC Debt Balance ($T)',  'Auto Loan' => 'Car Loan Debt Balance ($T)', 'Credit Card' => 'CC Debt Balance ($T)', 'Student Loan' => 'Student Debt Balance ($T)',  'Other' => 'Other Debt Balance ($T)', 'Total' => 'Total Debt Balance ($T)'},
+            'Number of Accounts by Loan Type' => {'Auto Loan' => 'Num Car Loan Accts (M)', 'Credit Card' => 'Num CC Accts (M)', 'Mortgage' => 'Num Mort Accts (M)',  'HE Revolving' => 'Num HELOC Accts (M)'},
+            'Total Number of New and Closed Accounts and Consumer Credit Inquiries' => {'inquiry within 6 mo' => 'Num Credit Inquiries and New Accts', 'closed within 12 mo' => 'Num New Credit Accts Closed in 12 mo', 'open within 12 mo' => 'Num New Credit Accts Open after 12 mo'},
+            'Mortgage Origination Volume by Riskscore' => {'<620' => 'New Mort w Riskscore <620 ($B)','620-659' => 'New Mort w Riskscore 620-659 ($B)','660-719' => 'New Mort w Riskscore 660-719 ($B)','720-759' => 'New Mort w Riskscore 720-759 ($B)','760+' => 'New Mort w Riskscore 760+ ($B)','TOTAL' => 'New Mort ($B)'},
+            'Distribution of Riskscore of Mortgage Originations' => {'Median' => 'Median Riskscore for New Mort', '25th percentile' => 'Bottom 25%ile Riskscore for New Mort', '10th percentile' => 'Bottom 10%ile Riskscore for New Mort'},
+            'Auto Loan Origination Volume by Riskscore' => {'<620' => 'New Car Loans w Riskscore <620 ($B)','620-659' => 'New Car Loans w Riskscore 620-659 ($B)','660-719' => 'New Car Loans w Riskscore 660-719 ($B)','720-759' => 'New Car Loans w Riskscore 720-759 ($B)','760+' => 'New Car Loans w Riskscore 760+ ($B)','TOTAL' => 'All New Car Loans ($B)'},
+            'Distribution of Riskscore of Auto Loan Originations' => {'Median' => 'Median Riskscore for New Car Loan', '25th percentile' => 'Bottom 25%ile Riskscore for New Car Loan', '10th percentile' => 'Bottom 10%ile Riskscore for New Car Loan'},
+            'Credit Limit and Balance for Credit Cards and HE Revolving' => {'Credit Card Balance' => 'CC Balance ($T)', 'Credit Card Available Credit' => 'CC Avail Credit ($T)',  'Credit Card Limit' => 'CC Limit ($T)', 'HE Revolving Balance' => 'HELOC Balance ($T)',  'HE Revolving Available Credit' => 'HELOC Avail Credit ($T)', 'HE Revolving Limit' => 'HELOC Limit ($T)'},
+            'Total Balance by Delinquency Status' => {'Current' => '% Balances Current', '30 days late' => '% Balances 30d late',  '60 days late' => '% Balances 60d late',  '90 days late' => '% Balances 90d late',  '120+ days late' => '% Balances 120+d late',  'Severely Derogatory' => '% Balances Severely Impair'},
+            'Percent of Balance 90+ Days Delinquent by Loan Type' => {'MORTGAGE' => '% Mort Bal 90d+ late',  'HELOC' => '% HELOC Bal 90d+ late', 'AUTO' => '% Car Loan Bal 90d+ late',  'CC' => '% CC Bal 90d+ late',  'STUDENT LOAN' => '% Student Loan Bal 90d+ late',  'OTHER' => '% Other Bal 90d+ late', 'ALL' => '% All Bal 90d+ late'},
+            'New Delinquent* Balances by Loan Type' => {'AUTO' => '% Car Loan Bal 0d->30d+ Late',  'CC' => '% CC Bal 0d->30d+ Late',  'MORTGAGE' => '% Mort Bal 0d->30d+ Late',  'HELOC' => '% HELOC Bal 0d->30d+ Late', 'STUDENT LOAN' => '% Student Loan Bal 0d->30d+ Late',  'OTHER' => '% Other Bal 0d->30d+ Late', 'Total' => '% All Bal 0d->30d+ Late'},
+            'New Seriously Delinquent* Balances by Loan Type' => {'AUTO' => '% Car Loan Bal ->90d+ Late',  'CC' => '% CC Bal ->90d+ Late',  'MORTGAGE' => '% Mort Bal ->90d+ Late',  'HELOC' => '% HELOC Bal ->90d+ Late', 'STUDENT LOAN' => '% Student Loan Bal ->90d+ Late',  'OTHER' => '% Other Bal ->90d+ Late', 'ALL' => '% All Bal ->90d+ Late'},
+            'Quarterly Transition Rates for Current Mortgage Accounts' => {'To 30-60 days late' => 'Qtrly % Mort 0->30-60d late', 'To 90+ days Late' => 'Qtrly % Mort ->90d+ Late'},
+            'Quarterly Transition Rates for 30-60 Day Late Mortgage Accounts' => {'To current'  => 'Qtrly % Mort % 30-60d->0d Late', 'To 90+ days Late' => 'Qtrly % Mort 30-60d->90d+ Late'},
+            'Number of Consumers with New Foreclosures and Bankruptcies' => {'foreclosure' => 'Num Consumers w New Foreclosure',  'bankruptcy' => 'Num Consumers w New Bankruptcy'},
+            'Third Party Collections' => {'proportion of consumers with collection' => '% Consumers w Collection', 'average collection amount per person with item' => 'Avg Collection Amt'},
+            'Total Debt Balance by Age' => {'18-29' => 'Avg Total Debt Bal for 18-29 yo ($T)', '30-39' => 'Avg Total Debt Bal for 30-39 yo ($T)', '40-49' => 'Avg Total Debt Bal for 40-49 yo ($T)', '50-59' => 'Avg Total Debt Bal for 50-59 yo ($T)', '60-69' => 'Avg Total Debt Bal for 60-69 yo ($T)', '70+' => 'Avg Total Debt Bal for 70+ yo ($T)'},
+            'Auto Loan Originations By Age' => {'18-29' => 'New Car Loans for 18-29 yo ($B)', '30-39' => 'New Car Loans for 30-39 yo ($B)', '40-49' => 'New Car Loans for 40-49 yo ($B)', '50-59' => 'New Car Loans for 50-59 yo ($B)', '60-69' => 'New Car Loans for 60-69 yo ($B)', '70+' => 'New Car Loans for 70+ yo ($B)'},
+            'Total Mortgage Orginations by Age' => {'18-29' => 'New Mort for 18-29 yo ($B)', '30-39' => 'New Mort for 30-39 yo ($B)', '40-49' => 'New Mort for 40-49 yo ($B)', '50-59' => 'New Mort for 50-59 yo ($B)', '60-69' => 'New Mort for 60-69 yo ($B)', '70+' => 'New Mort for 70+ yo ($B)'},
+            'Transition into Serious Delinquency (90+) by Age' => {'18-29' => 'Qtrly % ->90d+ late for 18-29 yo', '30-39' => 'Qtrly % ->90d+ late for 30-39 yo', '40-49' => 'Qtrly % ->90d+ late for 40-49 yo', '50-59' => 'Qtrly % ->90d+ late for 50-59 yo', '60-69' => 'Qtrly % ->90d+ late for 60-69 yo', '70+' => 'Qtrly % ->90d+ late for 70+ yo', 'all' => 'Qtrly % ->90d+ late'},
+            'Transition into Serious Delinquency (90+) for Mortgages by Age' => {'18-29' => 'Qtrly % Mort ->90d+ late for 18-29 yo', '30-39' => 'Qtrly % Mort ->90d+ late for 30-39 yo', '40-49' => 'Qtrly % Mort ->90d+ late for 40-49 yo', '50-59' => 'Qtrly % Mort ->90d+ late for 50-59 yo', '60-69' => 'Qtrly % Mort ->90d+ late for 60-69 yo', '70+' => 'Qtrly % Mort ->90d+ late for 70+ yo', 'all' => 'Qtrly % Mort ->90d+ late'},
+            'Transition into Serious Delinquency (90+) for Auto Loans by Age' => {'18-29' => 'Qtrly % Car Loans ->90d+ late for 18-29 yo', '30-39' => 'Qtrly % Car Loans ->90d+ late for 30-39 yo', '40-49' => 'Qtrly % Car Loans ->90d+ late for 40-49 yo', '50-59' => 'Qtrly % Car Loans ->90d+ late for 50-59 yo', '60-69' => 'Qtrly % Car Loans ->90d+ late for 60-69 yo', '70+' => 'Qtrly % Car Loans ->90d+ late for 70+ yo', 'all' => 'Qtrly % Car Loans ->90d+ late'},
+            'Transition into Serious Delinquency (90+) for Credit Cards by Age' => {'18-29' => 'Qtrly % CC ->90d+ late for 18-29 yo', '30-39' => 'Qtrly % CC ->90d+ late for 30-39 yo', '40-49' => 'Qtrly % CC ->90d+ late for 40-49 yo', '50-59' => 'Qtrly % CC ->90d+ late for 50-59 yo', '60-69' => 'Qtrly % CC ->90d+ late for 60-69 yo', '70+' => 'Qtrly % CC ->90d+ late for 70+ yo', 'all' => 'Qtrly % CC ->90d+ late'},
+            'Transition into Serious Delinquency (90+) for Student Loans by Age' => {'18-29' => 'Qtrly % Student Loan ->90d+ late for 18-29 yo', '30-39' => 'Qtrly % Student Loan ->90d+ late for 30-39 yo', '40-49' => 'Qtrly % Student Loan ->90d+ late for 40-49 yo', '50-59' => 'Qtrly % Student Loan ->90d+ late for 50-59 yo', 'all' => 'Qtrly % Student Loan ->90d+ late'},
+            'New Foreclosures By Age' => {'18-29' => 'New Foreclosures for 18-29 yo (000)', '30-39' => 'New Foreclosures for 30-39 yo (000)', '40-49' => 'New Foreclosures for 40-49 yo (000)', '50-59' => 'New Foreclosures for 50-59 yo (000)', '60-69' => 'New Foreclosures for 60-69 yo (000)', '70+' => 'New Foreclosures for 70+ yo (000)'},
+            'New Bankruptcy By Age' => {'18-29' => 'New Bankruptcies for 18-29 yo (000)', '30-39' => 'New Bankruptcies for 30-39 yo (000)', '40-49' => 'New Bankruptcies for 40-49 yo (000)', '50-59' => 'New Bankruptcies for 50-59 yo (000)', '60-69' => 'New Bankruptcies for 60-69 yo (000)', '70+' => 'New Bankruptcies for 70+ yo (000)'}
+          }
+
+        # Polars::Config.set_tbl_cols(-1)
+        Polars::Config.set_tbl_rows(-1)
+
+        done = false
+        xlsx_file.sheets.each do |sht|  # Begin per-sheet analysis
+          nm = sht.name
+
+          begin 
+            _ = sht.rows
+            pg_nm_text = sht.rows
+            pg_nm_text = pg_nm_text.first 
+          rescue 
+            begin
+              pg_nm_text = sht.rows.first
+            rescue 
+              done = true
+            end
+          end
+
+          break if done == true
+
+          pg_nm_text = pg_nm_text.first
+          next unless col_nm_hsh.keys.include?(pg_nm_text)
+
+          header_found = false
+          csv_txt = []
+
+          sht.rows.each do |row|   # Begin per-line analysis
+            r = row
+
+            if header_found == false 
+              if r.length > (r - col_nm_hsh[pg_nm_text].keys).length
+                header_found = true 
+                0.upto(r.length-1).to_a.each do |ndex|
+                  if ndex.zero?
+                    r[ndex] = 'Timestamps_'
+                  else
+                    col_nm_hsh[pg_nm_text].keys.each do |k|
+                      r[ndex] = col_nm_hsh[pg_nm_text][k] if r[ndex] == k
+                    end
+                  end
+                end
+              end
+
+              next if header_found == false
+            end 
+
+            break if r.compact.length.zero?
+
+            # puts "#{__FILE__}:#{__LINE__} r = >#{r.inspect}<"
+            if /(?<yr>\d{2}):Q(?<qtr>\d)/ =~ r[0].to_s
+              yr = yr.to_i + 2000
+              yr -= 100 if yr > Date.today.year
+              dte = Date.new(yr, qtr.to_i * 3 - 2, 1)
+              r[0] = dte
+            elsif (r[0].to_date rescue false)
+              r[0] = r[0].to_date.beginning_of_quarter
+            end
+
+            next if r.map{|rr| (rr.nil? ? '' : rr.to_s).length }.sum.zero?
+            rr = r.map{|c| c.nil? ? '' : c}
+            rr = rr.reverse.drop_while(&:nil?).reverse
+            next if rr.compact.length.zero?
+            csv_txt << rr.join(',')
+
+          end    # End per-line analysis
+
+          csv_dta = CSV.parse(csv_txt.join("\n"), headers: true, converters: [:numeric])
+          csv_dta.delete(nil)
+
+          keys = csv_dta.headers.compact.intersection(['Timestamps_'] + col_nm_hsh[pg_nm_text].values)
+          vals = csv_dta.to_a[1..-1].map{|r| (r + [''] * keys.length)[0..keys.length-1] }.transpose
+
+          tmp_df = {}; (0..(keys.length-1)).to_a.each{|i| tmp_df[keys[i]] = vals[i]}
+
+          tmp_df = Polars::DataFrame.new(tmp_df)
+          tmp_df = tmp_df.filter(Polars.col('Timestamps_') <= fin.to_date) unless start.nil?
+          tmp_df = tmp_df.filter(Polars.col('Timestamps_') >= start.to_date) unless fin.nil?
+          tmp_df.sort!('Timestamps_')
+
+          if df.nil?
+            df = tmp_df.dup 
+          else 
+            df = df.join(tmp_df, on: 'Timestamps_', how: 'full')
+
+            tmp_timestamps = df['Timestamps_'].to_a 
+            tmp_timestamps__ = df['Timestamps__right'].to_a
+            tmp_timestamps.length.times.each {|ndex| tmp_timestamps[ndex] ||= tmp_timestamps__[ndex] }
+            df.replace('Timestamps_', Polars::Series.new(tmp_timestamps))
+            df = df.drop(['Timestamps__right'])
+          end
+          df = df.sort('Timestamps_')
+ 
+        end    # End per-sheet analysis
+
+        s = Polars::Series.new('Timestamps', df['Timestamps_'].to_a)
+        df = df.insert_column(0,s).drop(['Timestamps_'])
+      end
+
+      df = df.drop(df.columns - ['Timestamps', @tag]) unless @tag.nil? || !df.columns.include?(@tag)
+      df
+    end
+  end
+end

data/lib/ny_frb_as_dataframe/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module NyFrbAsDataframe
+  VERSION = "0.1.0"
+end

data/lib/ny_frb_as_dataframe.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require_relative "ny_frb_as_dataframe/version"
+require_relative "ny_frb_as_dataframe/hhdc"
+
+# module NyFrbAsDataframe
+#   class Error < StandardError; end
+#   # Your code goes here...
+# end

data/ny_frb_as_dataframe.gemspec

@@ -0,0 +1,39 @@
+
+require_relative "lib/ny_frb_as_dataframe/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "ny_frb_as_dataframe"
+  spec.version = NyFrbAsDataframe::VERSION
+  spec.authors = ["Bill McKinnon"]
+  spec.email = ["bill.mckinnon@bmck.org"]
+
+  spec.summary       = "Alternative economic and financial data from the NY FRB"
+  spec.description   = "Alternative economic and financial data from the NY FRB"
+  spec.homepage      = "https://github.com/bmck/ny_frb_as_dataframe"
+  spec.license       = "MIT"
+  spec.required_ruby_version = Gem::Requirement.new(">= 3.0.0")
+
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/bmck/ny_frb_as_dataframe"
+  spec.metadata["changelog_uri"] = "https://github.com/bmck/ny_frb_as_dataframe/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  # Uncomment to register a new dependency of your gem
+  # spec.add_dependency "example-gem", "~> 1.0"
+
+  # For more information and examples about making a new gem, checkout our
+  # guide at: https://bundler.io/guides/creating_gem.html
+  spec.add_dependency 'polars-df'
+  spec.add_dependency 'httparty'  
+  spec.add_dependency 'simple_xlsx_reader'
+  spec.add_dependency 'activesupport'
+end

data/sig/ny_frb_as_dataframe.rbs

@@ -0,0 +1,4 @@
+module NyFrbAsDataframe
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,117 @@
+--- !ruby/object:Gem::Specification
+name: ny_frb_as_dataframe
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Bill McKinnon
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2024-08-17 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: polars-df
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: httparty
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: simple_xlsx_reader
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Alternative economic and financial data from the NY FRB
+email:
+- bill.mckinnon@bmck.org
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/ny_frb_as_dataframe.rb
+- lib/ny_frb_as_dataframe/hhdc.rb
+- lib/ny_frb_as_dataframe/version.rb
+- ny_frb_as_dataframe.gemspec
+- sig/ny_frb_as_dataframe.rbs
+homepage: https://github.com/bmck/ny_frb_as_dataframe
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/bmck/ny_frb_as_dataframe
+  source_code_uri: https://github.com/bmck/ny_frb_as_dataframe
+  changelog_uri: https://github.com/bmck/ny_frb_as_dataframe/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.19
+signing_key:
+specification_version: 4
+summary: Alternative economic and financial data from the NY FRB
+test_files: []