knowledgelm 4.0.0__tar.gz

knowledgelm-4.0.0/.agent/README.md

@@ -0,0 +1,16 @@
+# Agent Resources
+
+This directory contains resources for AI agents working with KnowledgeLM.
+
+## Structure
+
+- **skills/** - Claude Code skill definitions for KnowledgeLM functionality
+  - `knowledgelm-nse/` - Main skill for downloading NSE company filings
+    - `SKILL.md` - Skill definition
+    - `references/` - Reference documentation loaded as needed
+      - `notebooklm_audio_prompt.md` - Template for generating fundamental analysis audio overviews
+    - _(Future: scripts/, assets/ for additional bundled resources)_
+
+## Future Extensions
+
+This directory is designed to support additional agent protocol resources as the project evolves.

knowledgelm-4.0.0/.agent/skills/knowledgelm-nse/SKILL.md

@@ -0,0 +1,131 @@
+---
+name: knowledgelm-nse
+description: >
+  Batch download Indian company filings (transcripts, investor presentations,
+  credit ratings, annual reports) from NSE and optionally add to NotebookLM.
+  Use when user asks to: (1) Download investor materials for Indian publicly 
+  listed companies, (2) Research Indian stocks/companies, (3) Create research 
+  notebooks with company filings, or (4) Analyze NSE-listed company documents.
+---
+
+# KnowledgeLM NSE
+
+Batch download Indian company filings from NSE and optionally integrate with NotebookLM.
+
+## Installation
+
+Check if installed: `knowledgelm --version`
+
+If not: `uv tool install knowledgelm`
+
+To upgrade: `uv tool upgrade knowledgelm`
+
+## Skill Upgrade
+
+To upgrade this skill to the latest version:
+
+1. Download the latest skill directory from GitHub:
+   ```
+   https://github.com/eggmasonvalue/KnowledgeLM/tree/main/.agent/skills/knowledgelm-nse
+   ```
+
+2. Download all files in the `knowledgelm-nse/` directory:
+   - `SKILL.md`
+   - `references/notebooklm_audio_prompt.md`
+   - (Any future bundled resources)
+
+3. Replace the existing skill directory in your skills directory with the downloaded version.
+
+**Note:** The skill directory location depends on your AI agent vendor (e.g., `~/.claude/skills/knowledgelm-nse/`).
+
+## Command Discovery
+
+**Use `--help` extensively to discover current options and flags.**
+
+```bash
+knowledgelm --help
+knowledgelm download --help
+knowledgelm list-files --help
+```
+
+## Core Workflow
+
+### 1. Gather Required Information
+
+**NSE Symbol:** If not provided, use `web_search` to find it.
+
+**Date Range:** If not provided, ask for clarification. Accept various formats:
+- Explicit: `"2023-01-01 to 2025-01-26"`, `"2023 to 2025"`, `"from 2023"`
+- Relative: `"last 2 years"`
+- Milestones: `"Since IPO"`, `"since <event>"` (use `web_search` to resolve dates)
+
+Convert to `YYYY-MM-DD` for CLI.
+
+**Categories:** Default to all categories if not specified. Use `--annual-reports-all` by default.
+
+### 2. Download Filings
+
+Use `knowledgelm download` with appropriate flags. Files save to `./{SYMBOL}_knowledgeLM/`.
+
+### 3. List Files (if needed)
+
+Use `knowledgelm list-files` with `--json` flag to get file paths (excludes `.pkl` cookies).
+
+## NotebookLM Integration
+
+If user wants to create a NotebookLM notebook:
+
+### 1. Ensure Latest Package Version
+
+Check if installed and upgrade to latest:
+
+```bash
+notebooklm --version
+```
+
+If not installed:
+```bash
+uv tool install notebooklm-py
+```
+
+If installed, upgrade to latest:
+```bash
+uv tool upgrade notebooklm-py
+```
+
+**Browser extras (for first-time setup):** If user hasn't authenticated with NotebookLM before, they need browser login support:
+
+```bash
+uv tool install --reinstall "notebooklm-py[browser]"
+playwright install chromium
+```
+
+### 2. Update Skill to Latest Version
+
+**Always run** `notebooklm skill install` to ensure skill is current:
+
+```bash
+notebooklm skill install
+```
+
+This installs/updates to the default directory (typically `~/.claude/skills/notebooklm/`).
+
+**Important:** Do NOT delete this directory - it's used for version tracking.
+
+### 3. Copy to Your Skills Directory (if different)
+
+If your AI agent uses a different skills directory, copy the installed skill there. The install directory path is shown in the `skill install` output.
+
+### 4. Create Notebook
+
+Use the notebooklm skill to create notebook and add downloaded files as sources (exclude `.pkl` files).
+
+**Optional - Audio Overview Generation:**
+
+For generating audio overviews focused on fundamental analysis, use the prompt template at `references/notebooklm_audio_prompt.md` as a system prompt. This provides structured guidance for creating investor-focused audio summaries.
+
+## Exception Handling
+
+- **Invalid symbol:** CLI returns `"success": false` in JSON
+- **Network issues:** Retry once after 5 seconds
+- **Incomplete data:** May indicate newly listed company on the NSE mainboard or corporate action. Use `web_search` to verify.

knowledgelm-4.0.0/.agent/skills/knowledgelm-nse/references/notebooklm_audio_prompt.md

@@ -0,0 +1,103 @@
+# NotebookLM Audio Overview Prompt Template
+
+Use this as a system prompt when generating audio overviews for fundamental analysis of Indian publicly listed companies.
+
+## Core Instructions
+
+You are creating an audio overview for investors conducting fundamental analysis of an Indian publicly listed company. Your goal is to synthesize insights from earnings call transcripts, investor presentations, annual reports, credit rating reports and other investor material into a coherent, actionable narrative.
+
+## Structure
+
+### 1. Company Overview
+- Business model and core operations
+- Market position and competitive advantages
+- Key products/services and revenue streams
+- Geographic presence and market share
+
+### 2. Financial Performance Analysis
+- Revenue growth trends and drivers
+- Profitability metrics (margins, ROE, ROCE)
+- Cash flow generation and quality of earnings
+- Balance sheet strength (debt levels, working capital)
+- Key financial ratios and their trends
+
+### 3. Strategic Initiatives & Growth Drivers
+- Management's stated strategic priorities
+- Capital allocation plans (capex, acquisitions, dividends)
+- New product launches or market expansions
+- Digital transformation or operational improvements
+- Sustainability and ESG initiatives
+
+### 4. Risk Assessment
+- Industry-specific risks and headwinds
+- Regulatory or policy risks
+- Competitive threats
+- Operational or execution risks
+- Credit rating insights and concerns
+- Management's risk mitigation strategies
+
+### 5. Management Quality & Governance
+- Track record of execution against guidance
+- Capital allocation discipline
+- Transparency in communication
+- Corporate governance practices
+- Related party transactions (if material)
+
+### 6. Valuation Context
+- Current valuation multiples vs historical averages
+- Peer comparison (if data available)
+- Management's commentary on valuation
+- Return ratios relative to cost of capital
+
+### 7. Investment Thesis Summary
+- Bull case: Key reasons to be optimistic
+- Bear case: Key concerns and red flags
+- Critical questions for further research
+
+## Tone & Style
+
+- **Conversational but analytical**: Explain complex financial concepts clearly
+- **Balanced perspective**: Present both positives and concerns objectively
+- **Data-driven**: Reference specific numbers, percentages, and trends
+- **Forward-looking**: Focus on future prospects, not just historical performance
+- **Investor-centric**: Frame insights around investment decision-making
+
+## Key Principles
+
+1. **Synthesize, don't summarize**: Connect insights across documents to identify patterns
+2. **Highlight changes**: Emphasize quarter-over-quarter or year-over-year changes in strategy, guidance, or performance
+3. **Flag inconsistencies**: Note any contradictions between management commentary and actual results
+4. **Context matters**: Compare metrics to industry benchmarks and company's own historical performance
+5. **Management credibility**: Assess whether management delivers on commitments
+6. **Quality over quantity**: Focus on material insights, not exhaustive detail
+
+## Red Flags to Watch For
+
+- Deteriorating working capital or cash conversion
+- Frequent changes in accounting policies
+- Aggressive revenue recognition practices
+- Rising debt without corresponding EBITDA growth
+- Declining margins despite revenue growth
+- Management guidance misses becoming a pattern
+- High related party transactions
+- Frequent management turnover
+- Credit rating downgrades or negative outlooks
+
+## Green Flags to Highlight
+
+- Consistent execution against stated strategy
+- Improving return ratios (ROE, ROCE)
+- Strong free cash flow generation
+- Market share gains in core segments
+- Successful new product/service launches
+- Debt reduction or deleveraging
+- Credit rating upgrades or stable outlooks
+- Transparent and proactive communication
+- Prudent capital allocation
+
+## Customization Notes
+
+When using this template:
+- Emphasize sections most relevant to the specific company/industry
+- Incorporate sector-specific metrics (e.g., AUM for AMCs, NIM for banks, same-store sales for retail)
+- Reference specific management quotes when particularly insightful or concerning

knowledgelm-4.0.0/.context/ARCHITECTURE.md

@@ -0,0 +1,157 @@
+# Architecture
+
+## Module Structure
+
+```mermaid
+graph TD
+    subgraph Entry["Entry Points"]
+        CLI[src/knowledgelm/cli.py<br/>Click CLI]
+        APP[src/knowledgelm/app.py<br/>Streamlit UI]
+    end
+    
+    subgraph Logic["Core Logic"]
+        SRV[src/knowledgelm/core/service.py]
+    end
+    
+    subgraph Data["Data Layer"]
+        NSE_ADPT[src/knowledgelm/data/nse_adapter.py]
+        SCR_ADPT[src/knowledgelm/data/screener_adapter.py]
+    end
+    
+    subgraph Agent["Agent Resources"]
+        SKILL[.agent/skills/knowledgelm-nse/SKILL.md<br/>Agent Skill]
+    end
+    
+    subgraph Utils["Utilities"]
+        CONF[src/knowledgelm/config.py]
+        F_UTIL[src/knowledgelm/utils/file_utils.py]
+    end
+    
+    subgraph External["External Sources"]
+        NSE_LIB[NSE API<br/>nse library]
+        SCR_WEB[screener.in<br/>Credit Ratings]
+        NLMPY[notebooklm-py<br/>Optional Integration]
+    end
+    
+    CLI --> SRV
+    APP --> SRV
+    SRV --> NSE_ADPT
+    SRV --> SCR_ADPT
+    SRV --> F_UTIL
+    NSE_ADPT --> NSE_LIB
+    SCR_ADPT --> SCR_WEB
+    CLI .-> SKILL
+    SKILL .-> NLMPY
+    APP ..-> CONF
+    SRV ..-> CONF
+```
+
+## Project Structure
+
+```
+KnowledgeLM/
+├── src/
+│   └── knowledgelm/
+│       ├── __init__.py
+│       ├── cli.py                # Click CLI (v3.0)
+│       ├── app.py                # Streamlit UI
+│       ├── config.py             # Configuration
+│       ├── core/
+│       │   └── service.py        # Orchestration Logic
+│       ├── data/
+│       │   ├── nse_adapter.py    # NSE Library Wrapper
+│       │   └── screener_adapter.py # Screener Scraper
+│       └── utils/
+│           └── file_utils.py     # Sanitization & paths
+├── tests/
+│   └── test_placeholder.py
+├── .agent/
+│   └── skills/
+│       └── knowledgelm-nse/
+│           └── SKILL.md          # Agent Skill (v3.0)
+├── .context/
+├── pyproject.toml                # uv config
+└── README.md
+```
+
+## Component Responsibilities
+
+### cli.py
+- **CLI**: Click-based command interface (`download`, `list-categories`, `list-files`)
+- **JSON Output**: `--json` flag for agent parsing
+- **Help Discovery**: `--help` on all commands for agent self-discovery
+
+### app.py
+- **UI**: Streamlit forms for symbol, dates, category selection.
+- **Display**: Renders status and download tables.
+- **Validations**: Calls `KnowledgeService`.
+
+### core/service.py
+- **`KnowledgeService`**: Orchestrates fetching and downloading.
+- **Filters**: Applies business logic (category filters) on fetched data.
+
+### data/
+- **`nse_adapter.py`**: Wraps the external `nse` library.
+  - **Validation**: Checks symbol validity via `equityQuote`.
+- **`screener_adapter.py`**: Handles scraping from Screener.in.
+  - Resolves ICRA PDF links directly.
+  - Uses Selenium for high-fidelity HTML-to-PDF conversion.
+
+### .agent/skills/knowledgelm-nse/
+- **`SKILL.md`**: Agent skill following [Agent Skills](https://agentskills.io) standard.
+  - Instructs AI agents on CLI usage and NotebookLM integration.
+  - Self-upgradeable via GitHub raw URL.
+
+### utils/file_utils.py
+- **`sanitize_folder_name`**: Prevents path traversal security issues.
+
+## Data Flow
+
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant A as app.py
+    participant S as Service
+    participant N as NSEAdapter
+    participant SC as ScreenerAdapter
+
+    U->>A: Enter symbol, dates, folder name
+    A->>S: process_request()
+    S->>S: sanitize_folder_name()
+    
+    S->>N: validate_symbol()
+    alt Invalid Symbol
+        N-->>S: False
+        S--x A: Raise ValueError
+    end
+
+    S->>N: get_announcements()
+    N-->>S: JSON data
+    
+    par Download Categories
+        loop Each Category
+             S->>N: download_document()
+        end
+        alt Credit Ratings
+             S->>SC: download_credit_ratings()
+             SC-->>S: Count
+        end
+    end
+    
+    S-->>A: data, counts
+    A-->>U: Show status + updated tables
+```
+
+## Output Structure
+
+```
+{folder_name}/
+├── transcripts/
+├── investor_presentations/
+├── credit_rating/
+├── related_party_txns/
+├── annual_reports/
+├── resignations/      (Optional: Logical grouping in UI, physical folder relies on user download)
+├── updates/           (Optional: Logical grouping in UI)
+└── press_releases/    (Optional: Logical grouping in UI)
+```

knowledgelm-4.0.0/.context/CHANGELOG.md

@@ -0,0 +1,99 @@
+# Changelog
+
+## [Unreleased]
+
+## [4.0.0] - 2026-02-08
+
+### Breaking Changes
+- **Skill Location**: Moved agent skill from `src/knowledgelm/data/SKILL.md` to `.agent/skills/knowledgelm-nse/SKILL.md` following skill-creator best practices for directory-based structure with bundled resources
+
+### Features
+- **Self-Upgradeable Skill**: Skill can now self-upgrade by downloading latest version from GitHub raw URL
+- **Bundled Resources**: Added `references/` directory with NotebookLM audio overview prompt template for fundamental analysis
+- **Enhanced NotebookLM Integration**: 
+  - Comprehensive version management for both notebooklm-py package and skill
+  - Uses `uv tool upgrade` for package updates
+  - Always runs `notebooklm skill install` to ensure skill is current
+  - Handles browser extras conditionally for first-time authentication
+  - Vendor-agnostic approach for different AI agent skills directories
+- **Package Upgrade**: Added `uv tool upgrade knowledgelm` to Installation section
+
+### Documentation
+- **Skill Improvements**: Streamlined skill to be principle-based rather than task-like, emphasizing `--help` discovery over prescriptive commands
+- **Architecture Updates**: Updated `.context/ARCHITECTURE.md` to reflect new `.agent/` directory structure with separate "Agent Resources" subgraph in mermaid diagram
+- **Changelog Updates**: Updated skill path references throughout `.context/` artifacts
+
+### Cleanup
+- Removed legacy pip artifacts (root `__pycache__`, committed `.coverage`, and `nse_cookies_requests.pkl`)
+- Updated documentation and agent skill to prefer `uv` over `pip`
+
+## [3.0.0] - 2026-01-26
+
+### Features
+- **CLI Interface**: New `knowledgelm` CLI with commands:
+  - `download SYMBOL --from DATE --to DATE`: Batch download filings
+  - `list-categories`: Show available filing types
+  - `list-files DIRECTORY --json`: List downloaded files for NotebookLM integration
+- **AI Agent Skill**: Bundled `SKILL.md` at `.agent/skills/knowledgelm-nse/SKILL.md` following [Agent Skills](https://agentskills.io) open standard
+- **NotebookLM Integration**: Skill includes workflow for adding downloads to NotebookLM notebooks via notebooklm-py
+
+### Architecture
+- **CLI-First Design**: All operations accessible via CLI with `--help` discovery and `--json` output for agent parsing
+- **Skill Maintenance**: SKILL.md uses `--help` discovery instead of hardcoded commands (reduces sync burden)
+
+### Dependencies
+- Added `click>=8.1.0` for CLI
+
+### Documentation
+- Updated README with CLI usage, agent skill installation prompt (LLM-agnostic)
+
+
+## [2.0.0] - 2026-01-26
+
+### Security
+- **SSL Verification**: Enabled SSL verification for Screener.in requests to prevent MitM attacks.
+- **Input Sanitization**: Implemented `sanitize_folder_name` to prevent path traversal vulnerabilities in download folders.
+
+### Architecture
+- **Modular Design**: Split monolithic `filings_downloader.py` into:
+  - `core/service.py`: Business logic and orchestration.
+  - `data/nse_adapter.py`: Wrapper for NSE library.
+  - `data/screener_adapter.py`: Secure scraper for Screener.in.
+  - `utils/file_utils.py`: Shared utilities.
+  - `config.py`: Centralized configuration.
+- **UI Decoupling**: Refactored `app.py` to delegate logic to `KnowledgeService`.
+- **Validation**: Added early-exit symbol validation using `nse.equityQuote()` to prevent invalid API calls.
+- **Cleanup**: Removed intermediate `{symbol}_announcements.json` dump from output folder (data is ephemeral/in-memory).
+
+### Features
+- **Selenium Support**: Integrated `selenium` and `webdriver-manager` for headless Chrome operations to handle dynamic content.
+- **Robust Scraper**: Added fallback logic to handle various content types (PDF vs HTML) from Screener.in.
+- **Direct PDF Resolution**: implemented direct resolution for ICRA reports to bypass viewers.
+- **High-Fidelity HTML Conversion**: Replaced `markdownify` with Selenium-based "Print to PDF" for better report quality.
+
+### Documentation
+- **Walkthrough**: Added a browser recording demonstrating app functionalities to `README.md`.
+- **Assets**: Created `assets/` directory for media files.
+- **Context Artifacts**: Added `.context/` documentation artifacts (DESIGN, ARCHITECTURE, CHANGELOG).
+
+### Improvements
+- **Logging**: Redirected library logs (NSE, Selenium) to application logger and reduced noise.
+- **Project Structure**: Adopted `src/` layout.
+- **Dependency Management**: Migrated to `pyproject.toml` and `uv`.
+- **Code Quality**: Added `tests/` directory, configured `ruff`, and applied Google-style docstrings.
+
+### Fixed
+- **Empty ICRA Reports**: Resolved by bypassing JS viewers.
+- **Missing Dependencies**: Added `beautifulsoup4`, `markdownify`, `requests`.
+- **Duplicate Imports**: Cleaned up `FilingsDownloader.py` (now refactored).
+
+---
+
+## Initial Release
+
+### Features
+- Batch download NSE announcements by category
+- Credit rating dual-source (screener.in primary, NSE fallback)
+- Individual filing views (Resignations, Reg 30, Press Releases)
+- Annual report download (date range or all)
+- Streamlit UI with status feedback

knowledgelm-4.0.0/.context/CONVENTIONS.md

@@ -0,0 +1,74 @@
+# Code Conventions
+
+## Project Structure
+
+```
+KnowledgeLM/
+├── src/
+│   └── knowledgelm/
+│       ├── app.py
+│       ├── config.py
+│       ├── core/
+│       ├── data/
+│       └── utils/
+├── .context/
+└── pyproject.toml
+```
+
+## Naming
+
+| Element | Pattern | Example |
+|---------|---------|---------|
+| Constants | `UPPER_SNAKE_CASE` | `CREDIT_RATING_FOLDER` |
+| Functions | `snake_case` | `download_announcements()` |
+| Download folders | `snake_case` | `investor_presentations/` |
+
+## Patterns
+
+### Category Configuration
+Categories defined declaratively in `DOWNLOAD_CATEGORIES` dict:
+```python
+"transcripts": {
+    "enabled_arg": "download_transcripts",
+    "filter": lambda item: ...,
+    "label": "transcript"
+}
+```
+
+### Streamlit Session State
+All persistent data stored in `st.session_state`:
+```python
+if "data" not in st.session_state:
+    st.session_state.data = None
+```
+
+### Error Handling
+- Use `logging` module (not `print()`)
+- Catch specific exceptions (`requests.RequestException`, `ValueError`)
+- Fail gracefully in UI (show error message but don't crash)
+
+### Logging & Output
+- **No `print()`**: All output must go through `logger`.
+- **Third-Party Noise**: Use `log_utils.redirect_stdout_to_logger` for libraries that print to stdout (e.g., `nse`).
+- **Silence Warnings**: Explicitly suppress expected non-critical warnings (e.g., `InsecureRequestWarning` for specific legacy sites).
+
+## UI Components
+
+## Scraping & Automation
+
+- **Browser Automation**: Use Selenium (headless Chrome) when:
+  - Content is dynamically rendered via JS (e.g., PDF viewers).
+  - High-fidelity artifact generation (PDF) is required from HTML.
+- **Direct Downloads**: Prefer `requests` for static files (PDFs, CSVs) for efficiency.
+- **Resilience**: Always verify file integrity (size, content type) after download.
+
+
+- `st.columns()` for layout
+- `st.expander()` for collapsible sections
+- `st.spinner()` for loading feedback
+- Checkboxes for category toggles
+
+## Testing
+
+> [!CAUTION]
+> No automated tests exist. Manual testing via Streamlit UI.

knowledgelm-4.0.0/.context/DESIGN.md

@@ -0,0 +1,75 @@
+# Feature Design
+
+## [done] Core Download Categories
+
+Batch download filings by category with configurable filters:
+- Analyst Call Transcripts
+- Investor Presentations  
+- Credit Ratings (dual-source)
+- Related Party Transactions
+- Annual Reports (date range or all)
+
+## [done] CLI Interface (v3.0)
+
+Full programmatic access via `knowledgelm` command:
+- **Download**: Batch download with automated folder creation.
+- **Discovery**: `--help` on all levels for self-documenting interface.
+- **JSON Output**: `--json` flag for machine readability and AI agent parsing.
+
+## [done] Agent-First Design (v3.0)
+
+- **Standardized Skill**: `SKILL.md` compliant with [Agent Skills](https://agentskills.io) standard.
+- **Automation workflows**: Optimized for LLM tools (Claude Code, Gemini CLI, etc.).
+- **NotebookLM Synergy**: Purpose-built `list-files --json` command to facilitate source injection.
+
+## [done] Credit Rating Dual-Source
+    
+1. Primary: Scrape screener.in (all-time)
+   - **ICRA**: Direct PDF link resolution (bypassing JS viewer).
+   - **HTML Reports (CRISIL, etc.)**: Selenium-based HTML-to-PDF conversion.
+   - **PDF Reports**: Direct download.
+2. Fallback: NSE API (date-filtered)
+
+## [done] Input Validation
+
+- **Symbol Check**: Validates company symbol against NSE using `equityQuote` before attempting downloads.
+
+## [done] Individual Filing Views
+
+Expandable tables with per-row downloads:
+- Resignations
+- Regulation 30 Updates (experimental)
+- Press Releases
+
+## [done] Session State Management
+
+Persist data across Streamlit reruns:
+- Fetched announcements
+- Category counts
+- Status messages
+
+- DataFrames for view tables
+
+## [done] Modular Architecture
+
+Separation of concerns:
+- **UI Layer**: Streamlit (`app.py`) handles only presentation.
+- **Service Layer**: `KnowledgeService` orchestrates downloads.
+- **Data Adapters**: Isolated `NSEAdapter` and `ScreenerAdapter`.
+- **Config**: Centralized settings in `config.py`.
+
+## [done] Security Hardening
+
+- **SSL Verification**: Enabled for all external requests.
+- **Input Sanitization**: Strict validation of download folder names (`file_utils.sanitize_folder_name`).
+- **Logging**: Replaced console printing with structured logging.
+
+
+---
+
+## [idea] Future Enhancements
+
+- BSE support
+- Export to structured formats (CSV, Excel)
+- Configurable alert filters
+- Progress bars for bulk downloads