@yottagraph-app/data-model-skill 0.0.33 → 0.0.34

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yottagraph-app/data-model-skill",
-  "version": "0.0.33",
+  "version": "0.0.34",
   "description": "Data model skill documentation for AI agents - entity types, properties, and schemas from Lovelace fetch sources",
   "repository": {
     "type": "git",

package/skill/sbaloans/DATA_DICTIONARY.md

@@ -0,0 +1,240 @@
+# Data Dictionary: SBA Loans (FOIA)
+
+## Source Overview
+
+The SBA 7(a) & 504 FOIA dataset is published by the U.S. Small Business Administration, Office of Capital Access. It contains loan-level records for the two main SBA loan guarantee programs:
+
+- **7(a) Loan Program** -- the SBA's primary and most flexible loan program, covering working capital, equipment, real estate, and debt refinancing. Maximum loan amount is $5 million.
+- **504 Loan Program** -- provides long-term, fixed-rate financing for major fixed assets (real estate, heavy equipment) through Certified Development Companies (CDCs). Structured as a three-party arrangement: borrower (10%), CDC-backed SBA portion (40%), and a conventional third-party lender (50%).
+
+The data is distributed as CSV files via the CKAN open data platform at `https://data.sba.gov/dataset/7-a-504-foia`. File URLs change with each quarterly update (contain an "as of" date); the CKAN package API is used to discover current resource URLs dynamically.
+
+The dataset is updated quarterly, typically available one month after the quarter ends.
+
+| Program | CSV Files | Approx Size |
+|---------|-----------|-------------|
+| 7(a) | 4 files by decade: FY1991-1999, FY2000-2009, FY2010-2019, FY2020-Present | ~780MB |
+| 504 | 2 files: FY1991-2009, FY2010-Present | ~104MB |
+
+Both programs share a common column subset (borrower info, approval details, NAICS codes, loan status). 7(a) adds bank/lender fields and interest rate details; 504 adds CDC and third-party lender fields.
+
+| Pipeline | `Record.Source` |
+|----------|----------------|
+| All files | `sbaloans` |
+
+---
+
+## Entity Types
+
+### `financial_instrument`
+
+An SBA-backed loan under the 7(a) or 504 program, identified by its location ID (a unique loan identifier assigned by the SBA).
+
+- Primary key: Composite of program and location ID (e.g., `"7a_317954"`, `"504_188194"`)
+- Entity resolver: named entity. Strong ID = `sba_loan_id`. Not mergeable across sources.
+
+### `organization`
+
+A business, bank, CDC, or third-party lender involved in an SBA loan. Three roles:
+- **Borrower** -- the small business receiving the loan
+- **Bank/Primary Lender** (7(a)) -- the financial institution originating the loan, identified by FDIC certificate number when available
+- **CDC** (504) -- the Certified Development Company administering the 504 portion
+- **Third-Party Lender** (504) -- the conventional lender providing the remaining portion
+
+- Primary key: Organization name (borrowers), or FDIC certificate number (banks)
+- Entity resolver: named entity, mergeable. Banks with an FDIC certificate number use it as a strong ID (property `fdic_certificate_number`), enabling cross-source entity resolution with existing FDIC-sourced bank entities.
+
+### `location`
+
+A geographic location representing the project county and state where the loan-funded activity takes place.
+
+- Primary key: County + state combination (e.g., `"EAST BATON ROUGE, LA"`)
+- Entity resolver: named entity, mergeable. Disambiguation via county name + state code.
+
+---
+
+## Properties
+
+### Loan Properties (`financial_instrument`)
+
+#### Identity
+
+* `sba_loan_id`
+  * Definition: Composite SBA loan identifier combining program and location ID.
+  * Examples: `"7a_317954"`, `"504_188194"`
+  * Derivation: Constructed from `program` (normalized) + `"_"` + `locationid` CSV column.
+
+* `program`
+  * Definition: SBA loan program name.
+  * Examples: `"7a"`, `"504"`
+  * Derivation: `program` CSV column, normalized (leading/trailing whitespace trimmed, " 7A" becomes "7a").
+
+#### Financial Terms
+
+* `gross_approval`
+  * Definition: Total approved loan amount in dollars.
+  * Examples: `"450000"`, `"810000"`
+  * Derivation: `grossapproval` CSV column.
+
+* `sba_guaranteed_approval`
+  * Definition: SBA-guaranteed portion of the loan amount in dollars. 7(a) only.
+  * Examples: `"337500"`
+  * Derivation: `sbaguaranteedapproval` CSV column. Empty for 504 loans.
+
+* `initial_interest_rate`
+  * Definition: Initial interest rate of the loan as a percentage. 7(a) only.
+  * Examples: `"6"`, `"5.5"`
+  * Derivation: `initialinterestrate` CSV column.
+
+* `interest_type`
+  * Definition: Whether the interest rate is fixed or variable. 7(a) only.
+  * Examples: `"Fixed"`, `"Variable"`
+  * Derivation: `fixedorvariableinterestind` CSV column, decoded: F=Fixed, V=Variable.
+
+* `term_in_months`
+  * Definition: Loan term length in months.
+  * Examples: `"120"`, `"240"`, `"300"`
+  * Derivation: `terminmonths` CSV column.
+
+* `third_party_dollars`
+  * Definition: Third-party lender contribution in dollars. 504 only.
+  * Examples: `"1334500"`
+  * Derivation: `thirdpartydollars` CSV column.
+
+#### Dates
+
+* `approval_date`
+  * Definition: Date the loan was approved.
+  * Examples: `"12/5/2020"`, `"10/2/2009"`
+  * Derivation: `approvaldate` CSV column, format M/D/YYYY.
+
+* `approval_fiscal_year`
+  * Definition: Federal fiscal year of approval (Oct 1 - Sep 30).
+  * Examples: `"2021"`, `"2010"`
+  * Derivation: `approvalfy` CSV column.
+
+* `first_disbursement_date`
+  * Definition: Date of first loan disbursement.
+  * Examples: `"1/1/2021"`, `"9/15/2010"`
+  * Derivation: `firstdisbursementdate` CSV column.
+
+* `paid_in_full_date`
+  * Definition: Date the loan was paid in full.
+  * Derivation: `paidinfulldate` CSV column. Empty if loan is not paid off.
+
+* `chargeoff_date`
+  * Definition: Date the loan was charged off.
+  * Derivation: `chargeoffdate` CSV column. Empty if loan has not been charged off.
+
+#### Processing
+
+* `processing_method`
+  * Definition: SBA processing method used for the loan.
+  * Examples: `"Preferred Lenders Program"`, `"504 Basic"`
+  * Derivation: `processingmethod` CSV column.
+
+* `sub_program`
+  * Definition: SBA sub-program designation.
+  * Examples: `"Guaranty"`, `"Sec. 504 - Loan Guarantees - Private Sector Financed"`
+  * Derivation: `subprogram` CSV column.
+
+#### Industry and Business
+
+* `naics_code`
+  * Definition: North American Industry Classification System code for the borrower's business.
+  * Examples: `"531390"`, `"721110"`
+  * Derivation: `naicscode` CSV column.
+
+* `naics_description`
+  * Definition: Description of the NAICS industry classification.
+  * Examples: `"Other Activities Related to Real Estate"`, `"Hotels (except Casino Hotels) and Motels"`
+  * Derivation: `naicsdescription` CSV column.
+
+* `franchise_name`
+  * Definition: Name of the franchise if the borrower is a franchisee.
+  * Examples: `"CHOICE HOTELS INTERNATIONAL INC."`
+  * Derivation: `franchisename` CSV column. Empty if not a franchise.
+
+* `business_type`
+  * Definition: Legal structure of the borrowing business.
+  * Examples: `"CORPORATION"`, `"LLC"`, `"PARTNERSHIP"`, `"INDIVIDUAL"`
+  * Derivation: `businesstype` CSV column.
+
+* `business_age`
+  * Definition: Age category of the borrowing business at time of loan.
+  * Examples: `"Existing or more than 2 years old"`, `"New Business or 2 years or less"`, `"Less than 4 years old but at least 3"`
+  * Derivation: `businessage` CSV column.
+
+#### Status
+
+* `loan_status`
+  * Definition: Current status of the loan.
+  * Examples: `"LIQUID"` (active), `"P I F"` (paid in full), `"CHGOFF"` (charged off), `"EXEMPT"`, `"PREPAID IN FULL"`
+  * Derivation: `loanstatus` CSV column.
+
+* `gross_chargeoff_amount`
+  * Definition: Gross charge-off amount in dollars.
+  * Examples: `"0"`, `"125000"`
+  * Derivation: `grosschargeoffamount` CSV column.
+
+* `revolver_status`
+  * Definition: Whether the loan is a revolving line of credit. 7(a) only.
+  * Examples: `"TRUE"`, `"FALSE"`
+  * Derivation: `revolverstatus` CSV column.
+
+* `jobs_supported`
+  * Definition: Number of jobs supported by the loan.
+  * Examples: `"9"`, `"250"`, `"0"`
+  * Derivation: `jobssupported` CSV column.
+
+#### Geography
+
+* `sba_district_office`
+  * Definition: SBA district office that processed the loan.
+  * Examples: `"LOUISIANA DISTRICT OFFICE"`, `"ILLINOIS DISTRICT OFFICE"`
+  * Derivation: `sbadistrictoffice` CSV column.
+
+* `congressional_district`
+  * Definition: Congressional district of the project.
+  * Examples: `"6"`, `"13"`
+  * Derivation: `congressionaldistrict` CSV column.
+
+### Organization Properties
+
+* `address`
+  * Definition: Physical address of the organization formatted as "street, city, state zip".
+  * Examples: `"11851 Wentling Ave, BATON ROUGE, LA 70816"`, `"1981 Marcus Avenue, LAKE SUCCESS, NY 11042"`
+  * Derivation: Formatted from borrower address fields (`borrstreet`/`borrcity`/`borrstate`/`borrzip`) for borrowers, bank fields (`bankstreet`/`bankcity`/`bankstate`/`bankzip`) for 7(a) banks, or CDC fields (`cdc_street`/`cdc_city`/`cdc_state`/`cdc_zip`) for 504 CDCs.
+
+* `fdic_certificate_number`
+  * Definition: FDIC certificate number identifying a bank. Used as a strong ID for entity resolution with FDIC-sourced bank entities.
+  * Examples: `"59345"`
+  * Derivation: `bankfdicnumber` CSV column. Only present for 7(a) bank entities.
+  * Note: Enables cross-source entity resolution with the FDIC dataset (`fdic` source), which also uses `fdic_certificate_number` as a strong ID property on bank organizations.
+
+* `ncua_number`
+  * Definition: NCUA charter number identifying a credit union.
+  * Derivation: `bankncuanumber` CSV column. Only present for 7(a) entities where the lender is a credit union.
+  * Note: Sparse -- most lenders are banks, not credit unions.
+
+---
+
+## Entity Relationships
+
+```
+organization (borrower)       ──[borrower_of]──→ financial_instrument (loan)
+organization (bank/CDC)       ──[lender_of]────→ financial_instrument (loan)    attr: lender_role=primary|cdc
+organization (third-party)    ──[lender_of]────→ financial_instrument (loan)    attr: lender_role=third_party
+financial_instrument (loan)   ──[is_located_at]─→ location (project county/state)
+```
+
+---
+
+## Attributes
+
+* `lender_role` (on `lender_of` relationship)
+  * Definition: Role of the lending organization in the loan.
+  * Values: `"primary"` (7(a) bank), `"cdc"` (504 Certified Development Company), `"third_party"` (504 conventional third-party lender)
+  * Derivation: Set based on which organization record is being emitted and the loan program type.
+
+---

package/skill/sbaloans/schema.yaml

@@ -0,0 +1,380 @@
+# Dataset schema for SBA 7(a) & 504 FOIA loan data.
+#
+# Source: https://data.sba.gov/dataset/7-a-504-foia
+# Updated quarterly (typically one month after quarter end).
+#
+# Each CSV row represents a single SBA-backed loan. The atomizer produces:
+#   - A financial_instrument entity for the loan itself
+#   - An organization entity for the borrower
+#   - An organization entity for the bank/CDC/lender
+#   - Optionally, a third-party lender organization (504 loans)
+#   - A location entity for the project county/state
+name: "sbaloans"
+description: "SBA 7(a) and 504 FOIA loan data including loan terms, borrower and lender organizations, and project locations"
+
+extraction:
+  flavors: closed
+  properties: closed
+  relationships: closed
+  attributes: closed
+  events: closed
+
+flavors:
+  - name: "sba_loan"
+    description: "An SBA-backed loan under the 7(a) or 504 program"
+    display_name: "SBA Loan"
+    mergeability: not_mergeable
+    strong_id_properties: ["sba_loan_id"]
+    passive: true
+
+  - name: "organization"
+    description: "A particular business, institution, or organization such as a corporation, university, government agency, or non-profit"
+    display_name: "Organization"
+    mergeability: not_mergeable
+    strong_id_properties: ["fdic_certificate_number"]
+    passive: true
+
+  - name: "sba_program"
+    description: "An SBA loan program or sub-program (e.g., SBA 7a: Guaranty, SBA 504: Sec. 504 - Loan Guarantees)"
+    display_name: "SBA Program"
+    mergeability: not_mergeable
+    strong_id_properties: ["sba_program_id"]
+    passive: true
+
+  - name: "industry"
+    description: "An industry classification of economic activity (e.g. NAICS or SIC) identifying the line of business associated with an organization or award"
+    display_name: "Industry"
+    mergeability: not_mergeable
+    strong_id_properties: ["naics_code"]
+    passive: true
+
+  - name: "location"
+    description: "A specific named geographic location such as a city, country, region, or landmark"
+    display_name: "Location"
+    mergeability: not_mergeable
+    examples: ["EAST BATON ROUGE, LA", "LINN, IA"]
+    passive: true
+
+properties:
+  # Loan identity
+  - name: "sba_loan_id"
+    type: string
+    description: "Composite SBA loan identifier derived from program, borrower name, lender name, and approval date"
+    display_name: "SBA Loan ID"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["7a_Title2Land,_L.L.C._Newtek_Small_Business_Finance,_Inc._12_5_2020"]
+    passive: true
+
+
+  - name: "sba_program_id"
+    type: string
+    description: "Composite identifier for an SBA program/sub-program (lowercased program name)"
+    display_name: "SBA Program ID"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_program"]
+    examples: ["sba 7a: guaranty", "sba 504: sec. 504 - loan guarantees - private sector financed"]
+    passive: true
+
+  - name: "program"
+    type: string
+    description: "SBA loan program name"
+    display_name: "Program"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_program"]
+    examples: ["7a", "504"]
+    passive: true
+
+  - name: "gross_approval"
+    type: float
+    description: "Total approved loan amount in dollars"
+    display_name: "Gross Approval"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["450000", "810000"]
+    passive: true
+
+  - name: "sba_guaranteed_approval"
+    type: float
+    description: "SBA-guaranteed portion of the loan amount in dollars (7(a) only)"
+    display_name: "SBA Guaranteed Approval"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["337500"]
+    passive: true
+
+  - name: "approval_date"
+    type: string
+    description: "Date the loan was approved"
+    display_name: "Approval Date"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["12/5/2020"]
+    passive: true
+
+  - name: "approval_fiscal_year"
+    type: string
+    description: "Federal fiscal year of approval"
+    display_name: "Approval Fiscal Year"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["2021", "2010"]
+    passive: true
+
+  - name: "first_disbursement_date"
+    type: string
+    description: "Date of first loan disbursement"
+    display_name: "First Disbursement Date"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "term_in_months"
+    type: float
+    description: "Loan term length in months"
+    display_name: "Term in Months"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["120", "240", "300"]
+    passive: true
+
+  - name: "initial_interest_rate"
+    type: float
+    description: "Initial interest rate of the loan as a percentage (7(a) only)"
+    display_name: "Initial Interest Rate"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["6", "5.5"]
+    passive: true
+
+  - name: "interest_type"
+    type: string
+    description: "Whether the interest rate is fixed or variable (7(a) only)"
+    display_name: "Interest Type"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["Fixed", "Variable"]
+    passive: true
+
+  - name: "processing_method"
+    type: string
+    description: "SBA processing method used for the loan"
+    display_name: "Processing Method"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["Preferred Lenders Program", "504 Basic"]
+    passive: true
+
+  - name: "sub_program"
+    type: string
+    description: "SBA sub-program designation"
+    display_name: "Sub-Program"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_program"]
+    examples: ["Guaranty", "Sec. 504 - Loan Guarantees - Private Sector Financed"]
+    passive: true
+
+  - name: "naics_code"
+    type: string
+    description: "North American Industry Classification System code (typically 6 digits) identifying the industry of work performed under a contract (e.g., \"524114\" for Direct Health and Medical Insurance Carriers)"
+    display_name: "NAICS code"
+    mergeability: not_mergeable
+    domain_flavors: ["industry"]
+    examples: ["531390", "721110"]
+    passive: true
+
+  - name: "naics_description"
+    type: string
+    description: "Human-readable name of the NAICS industry (e.g., \"DIRECT HEALTH AND MEDICAL INSURANCE CARRIERS\")"
+    display_name: "NAICS description"
+    mergeability: not_mergeable
+    domain_flavors: ["industry"]
+    examples: ["Other Activities Related to Real Estate", "Hotels (except Casino Hotels) and Motels"]
+    passive: true
+
+  - name: "franchise_name"
+    type: string
+    description: "Name of the franchise if the borrower is a franchisee"
+    display_name: "Franchise Name"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "business_type"
+    type: string
+    description: "Legal structure of the borrowing business"
+    display_name: "Business Type"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["CORPORATION", "LLC", "PARTNERSHIP"]
+    passive: true
+
+  - name: "business_age"
+    type: string
+    description: "Age category of the borrowing business at time of loan"
+    display_name: "Business Age"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["Existing or more than 2 years old", "New Business or 2 years or less"]
+    passive: true
+
+  - name: "loan_status"
+    type: string
+    description: "Current status of the loan"
+    display_name: "Loan Status"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["LIQUID", "P I F", "CHGOFF", "EXEMPT"]
+    passive: true
+
+  - name: "paid_in_full_date"
+    type: string
+    description: "Date the loan was paid in full"
+    display_name: "Paid in Full Date"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "chargeoff_date"
+    type: string
+    description: "Date the loan was charged off"
+    display_name: "Charge-Off Date"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "gross_chargeoff_amount"
+    type: float
+    description: "Gross charge-off amount in dollars"
+    display_name: "Gross Charge-Off Amount"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "revolver_status"
+    type: string
+    description: "Whether the loan is a revolving line of credit (7(a) only)"
+    display_name: "Revolver Status"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "jobs_supported"
+    type: float
+    description: "Number of jobs supported by the loan"
+    display_name: "Jobs Supported"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    examples: ["9", "250"]
+    passive: true
+
+  - name: "third_party_dollars"
+    type: float
+    description: "Third-party lender contribution in dollars (504 only)"
+    display_name: "Third-Party Dollars"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "sba_district_office"
+    type: string
+    description: "SBA district office that processed the loan"
+    display_name: "SBA District Office"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  - name: "congressional_district"
+    type: string
+    description: "Congressional district of the project"
+    display_name: "Congressional District"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    passive: true
+
+  # Organization properties
+  - name: "address"
+    type: string
+    description: "Physical street address of the entity"
+    display_name: "Address"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    examples: ["11851 Wentling Ave, BATON ROUGE, LA 70816"]
+    passive: true
+
+  - name: "fdic_certificate_number"
+    type: string
+    description: "FDIC Certificate Number uniquely identifying an insured institution, derived from the CERT field in the Institutions API"
+    display_name: "FDIC Certificate Number"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    passive: true
+
+  - name: "ncua_number"
+    type: string
+    description: "NCUA charter number identifying a credit union"
+    display_name: "NCUA Number"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    passive: true
+
+relationships:
+  - name: "borrower_of"
+    description: "An organization is the borrower, applicant, or account party on a financial instrument such as a letter of credit, loan, or bond issue."
+    display_name: "Borrower Of"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    target_flavors: ["sba_loan"]
+    examples: ["Title2Land, L.L.C. is borrower of SBA 7a Loan 317954"]
+    passive: true
+
+  - name: "lender_of"
+    description: "An organization (bank, CDC, or third-party lender) is the lender on an SBA loan"
+    display_name: "Lender Of"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    target_flavors: ["sba_loan"]
+    examples: ["Newtek Small Business Finance, Inc. is lender of SBA 7a Loan 317954"]
+    passive: true
+
+  - name: "in_program"
+    description: "The SBA program/sub-program under which this loan was made"
+    display_name: "In Program"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    target_flavors: ["sba_program"]
+    passive: true
+
+  - name: "in_industry"
+    description: "The industry (NAICS) classification of the work performed under this contract or IDV"
+    display_name: "In industry (NAICS)"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    target_flavors: ["industry"]
+    passive: true
+
+  - name: "project_is_located_at"
+    description: "Geographic location of the SBA loan's project"
+    display_name: "Project Located At"
+    mergeability: not_mergeable
+    domain_flavors: ["sba_loan"]
+    target_flavors: ["location"]
+    examples: ["SBA 7a Loan 317954 is located at EAST BATON ROUGE, LA"]
+    passive: true
+
+attributes:
+  - properties: ["lender_of"]
+    name: "lender_type"
+    type: string
+    description: "Type of lender in the loan (primary, cdc, third_party)"
+    display_name: "Lender Type"
+    mergeability: not_mergeable
+    examples: ["primary", "cdc", "third_party"]
+
+  - properties: ["lender_of"]
+    name: "lender_amount"
+    type: float
+    description: "Dollar amount lent by this lender"
+    display_name: "Lender Amount"
+    mergeability: not_mergeable
+    examples: ["450000", "810000", "1334500"]