cmdop 0.1.13__tar.gz → 0.1.16__tar.gz

{cmdop-0.1.13 → cmdop-0.1.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cmdop
-Version: 0.1.13
+Version: 0.1.16
 Summary: Python SDK for CMDOP agent interaction
 Project-URL: Homepage, https://cmdop.com
 Project-URL: Documentation, https://cmdop.com
@@ -8,7 +8,7 @@ Project-URL: Repository, https://github.com/markolofsen/cmdop-client
 Author: CMDOP Team
 License: MIT
 License-File: LICENSE
-Keywords: agent,automation,cmdop,grpc,terminal
+Keywords: agent,automation,cmdop,terminal
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -24,6 +24,8 @@ Requires-Dist: httpx>=0.27.0
 Requires-Dist: protobuf>=4.25.0
 Requires-Dist: pydantic-settings>=2.0.0
 Requires-Dist: pydantic>=2.5.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: toon-python>=0.1.2
 Provides-Extra: dev
 Requires-Dist: grpcio-tools>=1.60.0; extra == 'dev'
 Requires-Dist: mypy>=1.8.0; extra == 'dev'
@@ -63,37 +65,70 @@ Agent connects out. Your code connects to relay. Done.
 
 ---
 
-## What You Get
+## Install
+
+```bash
+pip install cmdop
+```
+
+```python
+from cmdop import CMDOPClient, AsyncCMDOPClient
+
+# Remote (via cloud relay)
+with CMDOPClient.remote(api_key="cmd_xxx") as client:
+    client.files.list("/home")
+
+# Local (direct IPC)
+with CMDOPClient.local() as client:
+    client.terminal.execute("ls -la")
+
+# Async
+async with AsyncCMDOPClient.remote(api_key="cmd_xxx") as client:
+    await client.files.read("/etc/hostname")
+```
+
+---
+
+# Part 1: Remote Control
+
+## Terminal
 
-**Terminal:**
 ```python
 session = server.terminal.create()
 server.terminal.send_input(session.session_id, "kubectl get pods\n")
 output = server.terminal.get_history(session.session_id)
 ```
 
-**Files:**
+| Method | Description |
+|--------|-------------|
+| `create(shell)` | Start session |
+| `send_input(id, data)` | Send commands |
+| `get_history(id)` | Get output |
+| `resize(id, cols, rows)` | Resize |
+| `send_signal(id, signal)` | SIGINT/SIGTERM |
+| `close(id)` | End session |
+
+## Files
+
 ```python
 server.files.list("/var/log")
 server.files.read("/etc/nginx/nginx.conf")
 server.files.write("/tmp/config.json", b'{"key": "value"}')
 ```
 
-**Browser — scrape without Selenium/Playwright bullshit:**
-```python
-with server.browser.create_session() as browser:
-    browser.navigate("https://shop.com/products")
+| Method | Description |
+|--------|-------------|
+| `list(path)` | List dir |
+| `read(path)` | Read file |
+| `write(path, content)` | Write file |
+| `delete(path)` | Delete |
+| `copy(src, dst)` | Copy |
+| `move(src, dst)` | Move |
+| `mkdir(path)` | Create dir |
+| `info(path)` | Metadata |
 
-    # One call → structured data
-    products = browser.extract_data(
-        ".product-card",
-        '{"name": "h2", "price": ".price", "url": {"selector": "a", "attr": "href"}}',
-        limit=100
-    )["items"]
-    # → [{"name": "iPhone", "price": "$999", "url": "/p/123"}, ...]
-```
+## Agent
 
-**AI Agent — typed responses:**
 ```python
 from pydantic import BaseModel
 
@@ -106,11 +141,15 @@ result = server.agent.run("Check server health", output_schema=Health)
 health: Health = result.output  # Typed!
 ```
 
----
+| Method | Description |
+|--------|-------------|
+| `run(prompt, output_schema)` | Run agent, get typed result |
+
+Types: `chat`, `terminal`, `command`, `router`, `planner`
 
-## Real World
+## Real World Examples
 
-**AI Agent + Typed Output:**
+**Deploy with typed result:**
 ```python
 class DeployResult(BaseModel):
     success: bool
@@ -125,7 +164,7 @@ if not result.output.success:
     rollback(result.output.errors)
 ```
 
-**Fleet Update (1000 devices):**
+**Fleet update (1000 devices):**
 ```python
 async def update_fleet(keys: list[str], config: bytes):
     async with asyncio.TaskGroup() as tg:
@@ -138,7 +177,7 @@ async def update_one(key: str, config: bytes):
         await dev.terminal.execute("systemctl restart app")
 ```
 
-**Debug Customer Machine:**
+**Debug customer machine:**
 ```python
 with CMDOPClient.remote(api_key=customer_key) as m:
     m.terminal.send_input(sid, "ps aux\n")
@@ -146,75 +185,60 @@ with CMDOPClient.remote(api_key=customer_key) as m:
     m.terminal.send_input(sid, "df -h\n")
 ```
 
-**Scrape Products:**
-```python
-class Product(SDKBaseModel):
-    __base_url__ = "https://amazon.com"
-    title: str = ""
-    price: int = 0   # "$1,299" → 1299
-    url: str = ""    # "/dp/..." → "https://amazon.com/dp/..."
-
-with client.browser.create_session(headless=True) as b:
-    b.navigate("https://amazon.com/s?k=laptop")
-    raw = b.extract_data(".s-result-item", '{"title": "h2", "price": ".a-price-whole", "url": {"selector": "a", "attr": "href"}}', limit=50)
-    products = Product.from_list(raw["items"])  # clean + dedupe + filter
-```
-
 ---
 
-## Install
-
-```bash
-pip install cmdop
-```
+# Part 2: Web Parsing
 
-## Usage
+## Browser
 
+**DOM extraction:**
 ```python
-from cmdop import CMDOPClient, AsyncCMDOPClient
-
-# Remote (via cloud relay)
-with CMDOPClient.remote(api_key="cmd_xxx") as client:
-    client.files.list("/home")
-
-# Local (direct IPC)
-with CMDOPClient.local() as client:
-    client.terminal.execute("ls -la")
+with server.browser.create_session() as b:
+    b.navigate("https://shop.com/products")
 
-# Async
-async with AsyncCMDOPClient.remote(api_key="cmd_xxx") as client:
-    await client.files.read("/etc/hostname")
+    products = b.extract_data(
+        ".product-card",
+        '{"name": "h2", "price": ".price", "url": {"selector": "a", "attr": "href"}}',
+        limit=100
+    )["items"]
+    # → [{"name": "iPhone", "price": "$999", "url": "/p/123"}, ...]
 ```
 
----
-
-## API
-
-### Terminal
-
-| Method | Description |
-|--------|-------------|
-| `create(shell)` | Start session |
-| `send_input(id, data)` | Send commands |
-| `get_history(id)` | Get output |
-| `resize(id, cols, rows)` | Resize |
-| `send_signal(id, signal)` | SIGINT/SIGTERM |
-| `close(id)` | End session |
+**JS fetch injection (bypass CORS):**
+```python
+with client.browser.create_session() as b:
+    b.navigate("https://site.com")  # Get cookies/session
+
+    # Single API call
+    data = b.fetch_json("https://api.site.com/v1/items")
+
+    # Parallel fetch with headers and credentials
+    results = b.fetch_all(
+        urls={
+            "users": "https://api.site.com/v1/users",
+            "orders": "https://api.site.com/v1/orders",
+        },
+        headers={"Accept": "application/json"},
+        credentials=True,  # Include cookies
+    )
+    # → {"users": {"data": [...], "error": None}, "orders": {"data": [...], "error": None}}
+```
 
-### Files
+**Custom JS execution:**
+```python
+with client.browser.create_session() as b:
+    b.navigate("https://site.com")
 
-| Method | Description |
-|--------|-------------|
-| `list(path)` | List dir |
-| `read(path)` | Read file |
-| `write(path, content)` | Write file |
-| `delete(path)` | Delete |
-| `copy(src, dst)` | Copy |
-| `move(src, dst)` | Move |
-| `mkdir(path)` | Create dir |
-| `info(path)` | Metadata |
+    # Execute async JS with auto-wrap and JSON parsing
+    result = b.execute_js("""
+        const resp = await fetch('/api/data');
+        return await resp.json();
+    """)
+    # → {"items": [...]}
 
-### Browser
+    # Raw mode (returns JSON string)
+    raw = b.execute_js("return document.title", raw=True)
+```
 
 | Method | Description |
 |--------|-------------|
@@ -226,30 +250,15 @@ async with AsyncCMDOPClient.remote(api_key="cmd_xxx") as client:
 | `extract(selector, attr)` | Get text/attr |
 | `extract_regex(pattern)` | Regex matches |
 | `validate_selectors(item, fields)` | Check selectors |
-| `extract_data(item, fields, limit)` | **Bulk extract → list[dict]** |
-| `execute_script(js)` | Run JS |
+| `extract_data(item, fields, limit)` | Bulk extract → list[dict] |
+| `fetch_json(url)` | JS fetch → dict |
+| `fetch_all(urls, headers, credentials)` | Parallel fetch → {id: {data, error}} |
+| `execute_js(code, raw)` | Async JS with auto-wrap → dict |
+| `execute_script(js)` | Raw JS → str |
 | `screenshot()` | PNG |
 | `get_cookies()` / `set_cookies()` | Cookies |
 
-**Scraping workflow:**
-```python
-with client.browser.create_session() as b:
-    b.navigate("https://cars.com/listings")
-
-    # 1. Validate (fail fast if site changed)
-    v = b.validate_selectors(".item", {"title": "h2", "price": ".price"})
-    if not v["valid"]:
-        raise Exception(v["errors"])  # also has counts, samples
-
-    # 2. Extract
-    cars = b.extract_data(
-        ".item",
-        '{"title": "h2", "price": {"selector": ".price", "regex": "\\\\d+"}}',
-        limit=200
-    )["items"]
-```
-
-### SDKBaseModel
+## SDKBaseModel
 
 Auto-cleaning Pydantic model for scraped data. No more manual `.strip()`, regex, URL joining.
 
@@ -257,7 +266,7 @@ Auto-cleaning Pydantic model for scraped data. No more manual `.strip()`, regex,
 from cmdop import SDKBaseModel
 
 class Product(SDKBaseModel):
-    __base_url__ = "https://shop.com"  # for relative URLs
+    __base_url__ = "https://shop.com"
 
     name: str = ""    # "  iPhone 15  \n" → "iPhone 15"
     price: int = 0    # "$1,299.00" → 1299
@@ -268,8 +277,6 @@ class Product(SDKBaseModel):
 products = Product.from_list(raw["items"])
 ```
 
-**What it does:**
-
 | Type | Input | Output |
 |------|-------|--------|
 | `str` | `"  text \n\t "` | `"text"` |
@@ -277,19 +284,162 @@ products = Product.from_list(raw["items"])
 | `float` | `"4.5 out of 5"` | `4.5` |
 | `str` (url field) | `"/path"` | `"https://base.com/path"` |
 
-**Usage with extract_data:**
+## Parsing Examples
+
+**Scrape with validation:**
 ```python
-raw = browser.extract_data(".product", fields, limit=100)
-products = Product.from_list(raw["items"])  # clean + dedupe by url + filter empty
+with client.browser.create_session() as b:
+    b.navigate("https://cars.com/listings")
+
+    # 1. Validate (fail fast if site changed)
+    v = b.validate_selectors(".item", {"title": "h2", "price": ".price"})
+    if not v["valid"]:
+        raise Exception(v["errors"])
+
+    # 2. Extract
+    cars = b.extract_data(".item", '{"title": "h2", "price": ".price"}', limit=200)["items"]
 ```
 
-### Agent
+**Scrape with SDKBaseModel:**
+```python
+class Product(SDKBaseModel):
+    __base_url__ = "https://amazon.com"
+    title: str = ""
+    price: int = 0   # "$1,299" → 1299
+    url: str = ""    # "/dp/..." → "https://amazon.com/dp/..."
 
-| Method | Description |
-|--------|-------------|
-| `run(prompt, output_schema)` | Run agent, get typed result |
+with client.browser.create_session(headless=True) as b:
+    b.navigate("https://amazon.com/s?k=laptop")
+    raw = b.extract_data(".s-result-item", '{"title": "h2", "price": ".a-price-whole", "url": {"selector": "a", "attr": "href"}}', limit=50)
+    products = Product.from_list(raw["items"])  # clean + dedupe + filter
+```
 
-Types: `chat`, `terminal`, `command`, `router`, `planner`
+**Parallel API fetching:**
+```python
+with client.browser.create_session() as b:
+    b.navigate("https://api.example.com")
+
+    # Fetch 10 pages in parallel
+    urls = {f"page_{i}": f"https://api.example.com/items?page={i}" for i in range(10)}
+    results = b.fetch_all(urls)
+
+    items = []
+    for key, res in results.items():
+        if res["data"]:  # {"data": {...}, "error": None}
+            items.extend(res["data"].get("items", []))
+        elif res["error"]:
+            print(f"{key} failed: {res['error']}")
+```
+
+**JS fetch for protected APIs:**
+```python
+with client.browser.create_session() as b:
+    b.navigate("https://site.com")  # Get session cookies
+
+    # Fetch JSON API (inherits cookies)
+    data = b.fetch_json("https://api.site.com/v1/data")
+
+    # Parallel fetch with custom headers
+    results = b.fetch_all(
+        urls={
+            "inspection": f"https://api.site.com/v1/car/{car_id}/inspection",
+            "options": f"https://api.site.com/v1/car/{car_id}/options",
+        },
+        headers={"Accept": "application/json", "Origin": "https://site.com"},
+        credentials=True,
+    )
+
+    inspection = results["inspection"]["data"]
+    options = results["options"]["data"]
+```
+
+---
+
+# Part 3: Utilities
+
+## Logging
+
+Rich-powered logger with automatic project root detection and file persistence.
+
+```python
+from cmdop import get_logger
+
+log = get_logger(__name__)
+log.info("Starting process")
+log.debug("Details: %s", data)
+log.warning("Something seems off")
+log.error("Failed!", exc_info=True)  # With traceback
+```
+
+**Features:**
+- Rich console output with colors and timestamps
+- Auto-saves to `logs/` folder in project root
+- Finds project root by `pyproject.toml`, `requirements.txt`, `.git`
+- Daily log rotation by filename
+
+**Custom settings:**
+```python
+from cmdop import get_logger, setup_logging
+
+# Set log level
+log = get_logger(__name__, level="DEBUG")
+
+# Custom app name for log files
+log = get_logger(__name__, app_name="myparser")
+# → logs/myparser_2024-01-17.log
+
+# Disable file logging
+log = get_logger(__name__, log_to_file=False)
+```
+
+**Log file format:**
+```
+2024-01-17 14:30:25 | INFO     | myapp.parser:42 - Starting process
+2024-01-17 14:30:26 | ERROR    | myapp.parser:58 - Failed to fetch
+```
+
+| Function | Description |
+|----------|-------------|
+| `get_logger(name, level, app_name)` | Get configured logger |
+| `setup_logging(level, log_to_file)` | Configure root logger |
+| `find_project_root()` | Find Python project root |
+| `get_log_dir(app_name)` | Get/create logs directory |
+
+## TOON Format (Token Optimization)
+
+Convert JSON to TOON format — saves 30-50% tokens for LLM consumption.
+
+```python
+from cmdop import json_to_toon, JsonCleaner
+
+# Simple conversion
+data = {"name": "Alice", "age": 25, "city": "Seoul"}
+toon = json_to_toon(data)
+# → "name: Alice\nage: 25\ncity: Seoul"
+
+# With cleaning (removes nulls, empty values, noise keys)
+cleaner = JsonCleaner(noise_keys={"internal_id", "metadata"})
+clean_toon = cleaner.to_toon({
+    "name": "BMW X5",
+    "price": 50000,
+    "internal_id": "abc123",  # removed
+    "specs": None,            # removed
+    "options": [],            # removed
+})
+# → "name: BMW X5\nprice: 50000"
+```
+
+**JsonCleaner features:**
+- Removes `None` values and empty `[]`, `{}`, `""`
+- Simplifies `{"code": "1", "title": "Good"}` → `"Good"`
+- Custom noise keys per domain
+
+| Function | Description |
+|----------|-------------|
+| `json_to_toon(data)` | Convert dict/list to TOON string |
+| `JsonCleaner(noise_keys)` | Cleaner with custom noise keys |
+| `cleaner.compact(data)` | Clean without converting |
+| `cleaner.to_toon(data)` | Clean + convert to TOON |
 
 ---