agentic-cli 0.3.1__tar.gz → 0.3.3__tar.gz

{agentic_cli-0.3.1 → agentic_cli-0.3.3}/.gitignore

@@ -46,3 +46,7 @@ CLAUDE.md
 plans/
 docs/
 changes/
+.research_demo/
+
+# Git worktrees
+.worktrees/

{agentic_cli-0.3.1 → agentic_cli-0.3.3}/CHANGELOG.md

@@ -5,6 +5,62 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.3] - 2026-02-04
+
+### Added
+
+- **Shell Security Architecture**: Modular shell executor with 8-layer defense-in-depth security
+  - Input preprocessing with encoding/obfuscation detection
+  - Command tokenization and classification
+  - Path analysis and sandboxing
+  - Risk assessment with HITL approval workflows
+  - Comprehensive audit logging
+  - **Note**: Shell tool is disabled by default pending security validation
+- **File Operation Tools**: New categorized file tools with permission levels
+  - READ tools (safe): `read_file`, `grep`, `glob`, `list_dir`, `diff_compare`
+  - WRITE tools (caution): `write_file`, `edit_file`
+- **Feature Demo Scripts**: New examples for arxiv, fileops, memory, planning, shell, and websearch
+
+### Changed
+
+- **File Operations Refactoring**: Replaced monolithic `file_manager` with distinct, categorized tools
+- **Tool Registry**: Added `PermissionLevel` enum for tool categorization (SAFE, CAUTION, DANGEROUS)
+
+### Removed
+
+- Deprecated `file_manager` tool (replaced by new file operation tools)
+
+## [0.3.2] - 2026-02-01
+
+### Added
+
+- **Web Fetch Tool**: Full-featured web content fetching with LLM summarization
+  - HTML-to-Markdown conversion with html2text
+  - Robots.txt compliance checking
+  - SSRF protection and URL validation
+  - Caching and redirect handling
+- **Web Search Tool**: Pluggable web search with backend abstraction
+- **arXiv Integration**: Enhanced arXiv search with rate limiting, caching, advanced query options, and paper analysis tools
+- **LLM Event Logging**: Debug logging for model interactions
+- **Task Progress Events**: `verbose_thinking` setting for detailed task progress display
+
+### Changed
+
+- **CLI Architecture Refactoring**:
+  - Extracted `WorkflowController` and `MessageProcessor` from `BaseCLIApp` for better separation of concerns
+  - Added `background_init` context manager to `WorkflowController`
+  - Added public query methods to managers to reduce command-workflow coupling
+- **BaseWorkflowManager**: Moved shared implementations from ADK/LangGraph managers to base class
+- **Code Quality**: Added enums for string literals, improved `TaskGraph` encapsulation
+- **Settings**: Use app-specific paths and generic settings application
+
+### Fixed
+
+- Circular import in workflow module
+- Web search tool integration issues
+- arXiv cache unbounded growth (added size limit)
+- Webfetch redirect response structure alignment with spec
+
 ## [0.3.1] - 2026-01-28
 
 ### Added

{agentic_cli-0.3.1 → agentic_cli-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentic-cli
-Version: 0.3.1
+Version: 0.3.3
 Summary: A framework for building domain-specific agentic CLI applications
 Project-URL: Homepage, https://github.com/shoom1/agentic-cli
 Project-URL: Repository, https://github.com/shoom1/agentic-cli
@@ -48,7 +48,7 @@ Agentic CLI provides the core infrastructure for building interactive CLI applic
 - **Pluggable Orchestration**: Choose between Google ADK or LangGraph for agent workflows
 - **Rich Terminal UI**: Thinking boxes, markdown rendering, and streaming responses via `thinking-prompt`
 - **Declarative Agents**: Define agents with simple configuration objects
-- **Built-in Tools**: Python execution, knowledge base search, web search
+- **Built-in Tools**: Python execution, file operations, knowledge base, web search, web fetch, arXiv search
 - **Session Persistence**: Save and restore conversation sessions
 - **Type-safe Configuration**: Settings management with pydantic-settings
 
@@ -251,6 +251,10 @@ class MySettings(BaseSettings):
 | `orchestrator` | `AGENTIC_ORCHESTRATOR` | "adk" | Orchestrator: adk or langgraph |
 | `workspace_dir` | `AGENTIC_WORKSPACE_DIR` | ~/.agentic | Storage directory |
 | `log_level` | `AGENTIC_LOG_LEVEL` | "warning" | Logging level |
+| `tavily_api_key` | `TAVILY_API_KEY` | None | Tavily API key for web search |
+| `brave_api_key` | `BRAVE_API_KEY` | None | Brave Search API key |
+| `search_backend` | `AGENTIC_SEARCH_BACKEND` | Auto | Web search provider (tavily/brave) |
+| `webfetch_model` | `AGENTIC_WEBFETCH_MODEL` | Auto | Model for web content summarization |
 
 ### Settings Context
 
@@ -365,6 +369,124 @@ np.mean(data)
 
 Allowed modules: numpy, pandas, scipy, math, json, datetime, collections, itertools, re, random
 
+#### Web Search
+
+Search the web using pluggable backends (Tavily or Brave):
+
+```python
+from agentic_cli.tools import web_search
+
+# Use as agent tool
+agent = AgentConfig(
+    name="researcher",
+    tools=[web_search],
+)
+
+# Or call directly
+results = web_search("Python async programming", max_results=5)
+# Returns: {"results": [{"title": "...", "url": "...", "snippet": "..."}], ...}
+```
+
+Backends auto-select based on available API keys. Set `TAVILY_API_KEY` or `BRAVE_API_KEY`.
+
+#### Web Fetch
+
+Fetch web content and summarize with LLM:
+
+```python
+from agentic_cli.tools import web_fetch
+
+result = web_fetch(
+    url="https://example.com/article",
+    prompt="Extract the main points from this article",
+)
+# Returns: {"url": "...", "summary": "...", "content_length": ...}
+```
+
+Features: URL validation, robots.txt compliance, SSRF protection, content caching.
+
+#### ArXiv Search
+
+Search and analyze academic papers:
+
+```python
+from agentic_cli.tools import search_arxiv, fetch_arxiv_paper, analyze_arxiv_paper
+
+# Search papers
+results = search_arxiv("transformer attention", max_results=10, categories=["cs.CL"])
+
+# Fetch paper details
+paper = fetch_arxiv_paper("1706.03762")  # "Attention Is All You Need"
+
+# Analyze with LLM
+analysis = await analyze_arxiv_paper("1706.03762", "Summarize the key contributions")
+```
+
+#### File Operations
+
+Categorized file tools with permission levels:
+
+**READ Tools (Safe)**
+
+```python
+from agentic_cli.tools import read_file, grep, glob, list_dir, diff_compare
+
+# Read file contents
+result = read_file("src/main.py", offset=0, limit=100)
+# Returns: {"success": True, "content": "...", "size": 1234, "lines_read": 100}
+
+# Search for patterns (ripgrep-like)
+result = grep("def.*async", path="src/", file_pattern="*.py", recursive=True)
+# Returns: {"success": True, "matches": [...], "file_count": 5}
+
+# Find files by pattern
+result = glob("**/*.py", path="src/", include_metadata=True)
+# Returns: {"success": True, "files": [...], "count": 42}
+
+# List directory contents
+result = list_dir("src/", include_hidden=False)
+# Returns: {"success": True, "entries": [...]}
+
+# Compare files or text
+result = diff_compare(source1="old.txt", source2="new.txt")
+# Returns: {"success": True, "diff": "...", "has_changes": True}
+```
+
+**WRITE Tools (Caution)**
+
+```python
+from agentic_cli.tools import write_file, edit_file
+
+# Write file (creates or overwrites)
+result = write_file("output.txt", content="Hello, World!", create_dirs=True)
+# Returns: {"success": True, "path": "...", "size": 13, "created": True}
+
+# Edit file (sed-like replacement)
+result = edit_file("config.py", old_text="DEBUG = True", new_text="DEBUG = False")
+# Returns: {"success": True, "replacements": 1}
+```
+
+#### Shell Executor
+
+> **Note**: Shell execution is currently **disabled by default** while security safeguards are being validated.
+
+The shell tool provides layered security with 8 defense layers:
+- Input preprocessing (encoding/obfuscation detection)
+- Command tokenization and classification
+- Path analysis and sandboxing
+- Risk assessment with approval workflows
+- Audit logging
+
+```python
+from agentic_cli.tools import shell_executor, is_shell_enabled
+
+# Check if shell is enabled
+if is_shell_enabled():
+    result = shell_executor("ls -la", working_dir="/project")
+else:
+    print("Shell tool disabled pending security validation")
+```
+
 #### KnowledgeBaseManager
 
 Semantic search over documents:
@@ -470,8 +592,20 @@ Status icons:
 
 See the `examples/` directory for complete working examples:
 
+**Getting Started**
 - **hello_agent.py** - Simple assistant using Google ADK
 - **hello_langgraph.py** - Same assistant using LangGraph orchestration
+
+**Feature Demos**
+- **arxiv_demo.py** - ArXiv paper search and analysis
+- **fileops_demo.py** - File operation tools (read, write, grep, glob)
+- **memory_demo.py** - Working and long-term memory management
+- **planning_demo.py** - Task graph and planning tools
+- **shell_demo.py** - Shell security pattern detection
+- **webfetch_demo.py** - Web fetching and summarization
+- **websearch_demo.py** - Web search with multiple backends
+
+**Full Applications**
 - **research_demo/** - Full-featured research assistant with memory, planning, and file operations
 
 Run examples:
@@ -480,6 +614,10 @@ Run examples:
 export GOOGLE_API_KEY="your-key"
 python examples/hello_agent.py
 
+# Feature demos (no API key needed for some)
+python examples/fileops_demo.py
+python examples/shell_demo.py
+
 # Or with LangGraph (requires langgraph extra)
 pip install agentic-cli[langgraph]
 python examples/hello_langgraph.py
@@ -512,7 +650,9 @@ agentic-cli/
 │   ├── config.py             # BaseSettings, SettingsContext
 │   ├── cli/
 │   │   ├── app.py            # BaseCLIApp
-│   │   └── commands.py       # Command, CommandRegistry
+│   │   ├── commands.py       # Command, CommandRegistry
+│   │   ├── workflow_controller.py  # Workflow orchestration
+│   │   └── message_processor.py    # Event stream processing
 │   ├── workflow/
 │   │   ├── events.py         # WorkflowEvent, EventType
 │   │   ├── config.py         # AgentConfig
@@ -520,15 +660,27 @@ agentic-cli/
 │   │   └── langgraph/        # LangGraph submodule
 │   │       ├── manager.py    # LangGraphWorkflowManager
 │   │       ├── state.py      # AgentState, CheckpointData
-│   │       ├── persistence/  # Checkpointers and stores
-│   │       └── tools/        # Shell, file search tools
+│   │       └── persistence/  # Checkpointers and stores
 │   ├── tools/
-│   │   └── executor.py       # SafePythonExecutor
+│   │   ├── executor.py       # SafePythonExecutor
+│   │   ├── file_read.py      # read_file, diff_compare
+│   │   ├── file_write.py     # write_file, edit_file
+│   │   ├── grep_tool.py      # grep (pattern search)
+│   │   ├── glob_tool.py      # glob, list_dir (file discovery)
+│   │   ├── search.py         # Web search (Tavily, Brave)
+│   │   ├── webfetch_tool.py  # Web content fetching
+│   │   └── shell/            # Shell executor with security
+│   │       ├── executor.py   # Main entry point (disabled by default)
+│   │       ├── tokenizer.py  # Command parsing
+│   │       ├── classifier.py # Risk classification
+│   │       ├── sandbox.py    # Execution sandboxing
+│   │       └── audit.py      # Security logging
 │   └── knowledge_base/
 │       └── manager.py        # KnowledgeBaseManager
 ├── examples/
-│   ├── hello_agent.py
-│   ├── hello_langgraph.py
+│   ├── hello_agent.py        # Basic ADK example
+│   ├── hello_langgraph.py    # Basic LangGraph example
+│   ├── *_demo.py             # Feature demonstration scripts
 │   └── research_demo/        # Full-featured example
 └── tests/
 ```

{agentic_cli-0.3.1 → agentic_cli-0.3.3}/README.md

@@ -9,7 +9,7 @@ Agentic CLI provides the core infrastructure for building interactive CLI applic
 - **Pluggable Orchestration**: Choose between Google ADK or LangGraph for agent workflows
 - **Rich Terminal UI**: Thinking boxes, markdown rendering, and streaming responses via `thinking-prompt`
 - **Declarative Agents**: Define agents with simple configuration objects
-- **Built-in Tools**: Python execution, knowledge base search, web search
+- **Built-in Tools**: Python execution, file operations, knowledge base, web search, web fetch, arXiv search
 - **Session Persistence**: Save and restore conversation sessions
 - **Type-safe Configuration**: Settings management with pydantic-settings
 
@@ -212,6 +212,10 @@ class MySettings(BaseSettings):
 | `orchestrator` | `AGENTIC_ORCHESTRATOR` | "adk" | Orchestrator: adk or langgraph |
 | `workspace_dir` | `AGENTIC_WORKSPACE_DIR` | ~/.agentic | Storage directory |
 | `log_level` | `AGENTIC_LOG_LEVEL` | "warning" | Logging level |
+| `tavily_api_key` | `TAVILY_API_KEY` | None | Tavily API key for web search |
+| `brave_api_key` | `BRAVE_API_KEY` | None | Brave Search API key |
+| `search_backend` | `AGENTIC_SEARCH_BACKEND` | Auto | Web search provider (tavily/brave) |
+| `webfetch_model` | `AGENTIC_WEBFETCH_MODEL` | Auto | Model for web content summarization |
 
 ### Settings Context
 
@@ -326,6 +330,124 @@ np.mean(data)
 
 Allowed modules: numpy, pandas, scipy, math, json, datetime, collections, itertools, re, random
 
+#### Web Search
+
+Search the web using pluggable backends (Tavily or Brave):
+
+```python
+from agentic_cli.tools import web_search
+
+# Use as agent tool
+agent = AgentConfig(
+    name="researcher",
+    tools=[web_search],
+)
+
+# Or call directly
+results = web_search("Python async programming", max_results=5)
+# Returns: {"results": [{"title": "...", "url": "...", "snippet": "..."}], ...}
+```
+
+Backends auto-select based on available API keys. Set `TAVILY_API_KEY` or `BRAVE_API_KEY`.
+
+#### Web Fetch
+
+Fetch web content and summarize with LLM:
+
+```python
+from agentic_cli.tools import web_fetch
+
+result = web_fetch(
+    url="https://example.com/article",
+    prompt="Extract the main points from this article",
+)
+# Returns: {"url": "...", "summary": "...", "content_length": ...}
+```
+
+Features: URL validation, robots.txt compliance, SSRF protection, content caching.
+
+#### ArXiv Search
+
+Search and analyze academic papers:
+
+```python
+from agentic_cli.tools import search_arxiv, fetch_arxiv_paper, analyze_arxiv_paper
+
+# Search papers
+results = search_arxiv("transformer attention", max_results=10, categories=["cs.CL"])
+
+# Fetch paper details
+paper = fetch_arxiv_paper("1706.03762")  # "Attention Is All You Need"
+
+# Analyze with LLM
+analysis = await analyze_arxiv_paper("1706.03762", "Summarize the key contributions")
+```
+
+#### File Operations
+
+Categorized file tools with permission levels:
+
+**READ Tools (Safe)**
+
+```python
+from agentic_cli.tools import read_file, grep, glob, list_dir, diff_compare
+
+# Read file contents
+result = read_file("src/main.py", offset=0, limit=100)
+# Returns: {"success": True, "content": "...", "size": 1234, "lines_read": 100}
+
+# Search for patterns (ripgrep-like)
+result = grep("def.*async", path="src/", file_pattern="*.py", recursive=True)
+# Returns: {"success": True, "matches": [...], "file_count": 5}
+
+# Find files by pattern
+result = glob("**/*.py", path="src/", include_metadata=True)
+# Returns: {"success": True, "files": [...], "count": 42}
+
+# List directory contents
+result = list_dir("src/", include_hidden=False)
+# Returns: {"success": True, "entries": [...]}
+
+# Compare files or text
+result = diff_compare(source1="old.txt", source2="new.txt")
+# Returns: {"success": True, "diff": "...", "has_changes": True}
+```
+
+**WRITE Tools (Caution)**
+
+```python
+from agentic_cli.tools import write_file, edit_file
+
+# Write file (creates or overwrites)
+result = write_file("output.txt", content="Hello, World!", create_dirs=True)
+# Returns: {"success": True, "path": "...", "size": 13, "created": True}
+
+# Edit file (sed-like replacement)
+result = edit_file("config.py", old_text="DEBUG = True", new_text="DEBUG = False")
+# Returns: {"success": True, "replacements": 1}
+```
+
+#### Shell Executor
+
+> **Note**: Shell execution is currently **disabled by default** while security safeguards are being validated.
+
+The shell tool provides layered security with 8 defense layers:
+- Input preprocessing (encoding/obfuscation detection)
+- Command tokenization and classification
+- Path analysis and sandboxing
+- Risk assessment with approval workflows
+- Audit logging
+
+```python
+from agentic_cli.tools import shell_executor, is_shell_enabled
+
+# Check if shell is enabled
+if is_shell_enabled():
+    result = shell_executor("ls -la", working_dir="/project")
+else:
+    print("Shell tool disabled pending security validation")
+```
+
 #### KnowledgeBaseManager
 
 Semantic search over documents:
@@ -431,8 +553,20 @@ Status icons:
 
 See the `examples/` directory for complete working examples:
 
+**Getting Started**
 - **hello_agent.py** - Simple assistant using Google ADK
 - **hello_langgraph.py** - Same assistant using LangGraph orchestration
+
+**Feature Demos**
+- **arxiv_demo.py** - ArXiv paper search and analysis
+- **fileops_demo.py** - File operation tools (read, write, grep, glob)
+- **memory_demo.py** - Working and long-term memory management
+- **planning_demo.py** - Task graph and planning tools
+- **shell_demo.py** - Shell security pattern detection
+- **webfetch_demo.py** - Web fetching and summarization
+- **websearch_demo.py** - Web search with multiple backends
+
+**Full Applications**
 - **research_demo/** - Full-featured research assistant with memory, planning, and file operations
 
 Run examples:
@@ -441,6 +575,10 @@ Run examples:
 export GOOGLE_API_KEY="your-key"
 python examples/hello_agent.py
 
+# Feature demos (no API key needed for some)
+python examples/fileops_demo.py
+python examples/shell_demo.py
+
 # Or with LangGraph (requires langgraph extra)
 pip install agentic-cli[langgraph]
 python examples/hello_langgraph.py
@@ -473,7 +611,9 @@ agentic-cli/
 │   ├── config.py             # BaseSettings, SettingsContext
 │   ├── cli/
 │   │   ├── app.py            # BaseCLIApp
-│   │   └── commands.py       # Command, CommandRegistry
+│   │   ├── commands.py       # Command, CommandRegistry
+│   │   ├── workflow_controller.py  # Workflow orchestration
+│   │   └── message_processor.py    # Event stream processing
 │   ├── workflow/
 │   │   ├── events.py         # WorkflowEvent, EventType
 │   │   ├── config.py         # AgentConfig
@@ -481,15 +621,27 @@ agentic-cli/
 │   │   └── langgraph/        # LangGraph submodule
 │   │       ├── manager.py    # LangGraphWorkflowManager
 │   │       ├── state.py      # AgentState, CheckpointData
-│   │       ├── persistence/  # Checkpointers and stores
-│   │       └── tools/        # Shell, file search tools
+│   │       └── persistence/  # Checkpointers and stores
 │   ├── tools/
-│   │   └── executor.py       # SafePythonExecutor
+│   │   ├── executor.py       # SafePythonExecutor
+│   │   ├── file_read.py      # read_file, diff_compare
+│   │   ├── file_write.py     # write_file, edit_file
+│   │   ├── grep_tool.py      # grep (pattern search)
+│   │   ├── glob_tool.py      # glob, list_dir (file discovery)
+│   │   ├── search.py         # Web search (Tavily, Brave)
+│   │   ├── webfetch_tool.py  # Web content fetching
+│   │   └── shell/            # Shell executor with security
+│   │       ├── executor.py   # Main entry point (disabled by default)
+│   │       ├── tokenizer.py  # Command parsing
+│   │       ├── classifier.py # Risk classification
+│   │       ├── sandbox.py    # Execution sandboxing
+│   │       └── audit.py      # Security logging
 │   └── knowledge_base/
 │       └── manager.py        # KnowledgeBaseManager
 ├── examples/
-│   ├── hello_agent.py
-│   ├── hello_langgraph.py
+│   ├── hello_agent.py        # Basic ADK example
+│   ├── hello_langgraph.py    # Basic LangGraph example
+│   ├── *_demo.py             # Feature demonstration scripts
 │   └── research_demo/        # Full-featured example
 └── tests/
 ```

{agentic_cli-0.3.1 → agentic_cli-0.3.3}/environment.yml

@@ -5,6 +5,7 @@ channels:
 dependencies:
   - python=3.12
   - pip
+  - html2text
   - pip:
     - -e .
     - -e .[dev]