logtap 0.3.0__tar.gz → 0.4.0__tar.gz

{logtap-0.3.0 → logtap-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: logtap
-Version: 0.3.0
+Version: 0.4.0
 Summary: A CLI-first log access tool for Unix systems. Remote log file access without SSH.
 Project-URL: Homepage, https://github.com/cainky/logtap
 Project-URL: Repository, https://github.com/cainky/logtap

logtap-0.4.0/docs/plans/2026-02-02-gpu-cloud-pivot-design.md

@@ -0,0 +1,334 @@
+# logtap GPU Cloud Pivot Design
+
+**Date:** 2026-02-02
+**Status:** Approved
+**Version:** 0.4.0 target
+
+---
+
+## Product Thesis
+
+**One-line:** `tail -f` over HTTPS for ML training jobs — survives disconnects, supports multi-node aggregation, zero infra.
+
+**Job-to-be-done:** When training on a cheap GPU box, watch training logs and troubleshoot without babysitting an SSH session, with logs preserved even if the instance is ephemeral.
+
+**Primary user:** Solo ML engineers running training jobs (1-5 machines)
+**Viral amplifier:** Indie hackers / hobbyists
+**Explicitly not:** Teams/orgs (avoid MLOps platform gravity)
+
+---
+
+## Core Features (v0.4.0)
+
+1. **Ingest path** — `python train.py | logtap ingest run1`
+2. **Resume cursors** — reconnect with `?since=<cursor>`, no gaps
+3. **Collector mode** — `logtap collect` accepts ingested streams
+4. **Tags + filtering** — `--tag node=gpu1` and `--tag node=gpu1` on tail
+
+---
+
+## CLI Commands
+
+### Existing (unchanged)
+- `logtap serve` — serve static log directory (legacy)
+- `logtap query` — query a file
+- `logtap tail` — stream/tail a file or run
+- `logtap files` — list available files
+
+### New Commands
+
+#### `logtap collect`
+```
+logtap collect [OPTIONS]
+
+Start collector server to accept ingested runs.
+
+Options:
+  -p, --port INT          Port to listen on [default: 8000]
+  -H, --host TEXT         Host to bind to [default: 0.0.0.0]
+  -k, --api-key TEXT      API key for auth [env: LOGTAP_API_KEY]
+  -d, --data-dir PATH     Directory for run storage [default: ~/.logtap/runs]
+  --buffer-lines INT      In-memory cache size per run [default: 100000]
+  --max-disk-mb INT       Max disk usage across all runs [default: 1000]
+  --retention-hours INT   Hours to retain runs [default: 72]
+```
+
+#### `logtap ingest`
+```
+logtap ingest [RUN_ID] [OPTIONS]
+
+Pipe stdin to collector as a named run.
+
+Arguments:
+  RUN_ID                  Run name [default: run-YYYYMMDD-HHMMSS]
+
+Options:
+  -s, --server TEXT       Collector URL [default: http://localhost:8000] [env: LOGTAP_SERVER]
+  -k, --api-key TEXT      API key [env: LOGTAP_API_KEY]
+  -t, --tag TEXT          Tags as key=value (repeatable)
+  --quiet                 Suppress status messages
+```
+
+#### `logtap runs`
+```
+logtap runs [OPTIONS]
+
+List runs on a collector.
+
+Options:
+  -s, --server TEXT       Collector URL [env: LOGTAP_SERVER]
+  -k, --api-key TEXT      API key [env: LOGTAP_API_KEY]
+  --since-hours INT       Show runs active within N hours [default: 24]
+```
+
+#### `logtap tail` (updated)
+```
+logtap tail TARGET [OPTIONS]
+
+Tail a run or file.
+
+Arguments:
+  TARGET                  Run name or file path
+
+Options:
+  -s, --server TEXT       Server URL [env: LOGTAP_SERVER]
+  -k, --api-key TEXT      API key [env: LOGTAP_API_KEY]
+  -f, --follow            Keep streaming new lines
+  -n, --lines INT         Initial lines to show [default: 50]
+  --since INT             Resume from cursor (exclusive)
+  -m, --mode TEXT         auto|runs|files [default: auto]
+  -t, --tag TEXT          Filter by tag (repeatable, AND semantics)
+  --output TEXT           pretty|plain|jsonl [default: pretty]
+
+Resolution (mode=auto):
+  1. If --server set: query server for run, fall back to file
+  2. If --server unset: treat as local file path
+```
+
+---
+
+## API Endpoints
+
+### Collector Mode (`/runs/*`)
+
+#### `POST /runs/{run_id}/ingest`
+
+Chunked streaming ingest of log lines.
+
+**Request:**
+```http
+POST /runs/run1/ingest HTTP/1.1
+Content-Type: text/plain
+Transfer-Encoding: chunked
+X-API-Key: secret
+X-Logtap-Tag: node=gpu1
+X-Logtap-Tag: rank=0
+
+line 1\n
+line 2\n
+```
+
+**Behavior:**
+- `\n` is record delimiter
+- Partial line at close: flush as final line
+- Auto-creates run if doesn't exist (201 Created)
+- Tags: merge semantics, 409 on conflicting value for same key
+
+**Response:**
+```json
+{"run_id": "run1", "lines_ingested": 1848, "cursor_end": 1847}
+```
+
+#### `GET /runs`
+
+List runs.
+
+**Query params:**
+- `since_hours` — filter to runs active within N hours
+
+**Response:**
+```json
+{
+  "runs": [{
+    "id": "run1",
+    "lines": 12847,
+    "cursor_earliest": 0,
+    "cursor_latest": 12846,
+    "tags": {"node": "gpu1"},
+    "created_at": "2026-02-02T18:30:00Z",
+    "last_activity": "2026-02-02T19:45:12Z",
+    "active": true
+  }]
+}
+```
+
+#### `GET /runs/{run_id}`
+
+Get run details.
+
+**Response:**
+```json
+{
+  "id": "run1",
+  "lines": 12847,
+  "cursor_earliest": 0,
+  "cursor_latest": 12846,
+  "tags": {"node": "gpu1"},
+  "created_at": "2026-02-02T18:30:00Z",
+  "last_activity": "2026-02-02T19:45:12Z",
+  "active": true,
+  "bytes_on_disk": 524288
+}
+```
+
+#### `GET /runs/{run_id}/stream`
+
+SSE stream for tailing.
+
+**Query params:**
+- `since` — cursor to resume from (exclusive)
+- `tail` — last N lines if since omitted [default: 50]
+- `follow` — keep connection open [default: false]
+- `tag` — filter by tag (repeatable, AND semantics)
+
+**Headers:**
+- `Last-Event-ID` — alternative to `since` param
+
+**Response:**
+```
+HTTP/1.1 200 OK
+Content-Type: text/event-stream
+X-Logtap-Earliest-Cursor: 0
+X-Logtap-Latest-Cursor: 12846
+
+event: meta
+data: {"cursor_earliest": 0, "cursor_latest": 12846, "gap": false}
+
+id: 5001
+event: line
+data: {"cursor": 5001, "line": "Epoch 5: loss=0.089", "ts": "2026-02-02T19:30:01Z"}
+
+: heartbeat
+
+```
+
+**Gap detection:**
+If `since < cursor_earliest`, first meta event includes:
+```json
+{"cursor_earliest": 1000, "cursor_latest": 12846, "gap": true, "missed": 500}
+```
+
+#### `GET /runs/{run_id}/query`
+
+Non-streaming query for a range.
+
+**Query params:**
+- `from` / `to` — cursor range (inclusive)
+- `tail` — last N lines (alternative)
+- `limit` — max lines [default: 1000, max: 10000]
+- `search` — substring filter
+- `regex` — regex filter (mutually exclusive with search)
+- `output` — jsonl|plain [default: jsonl]
+
+**Response (jsonl):**
+```
+{"cursor": 1000, "line": "Epoch 1: loss=0.523", "ts": "..."}
+{"cursor": 1001, "line": "Epoch 2: loss=0.312", "ts": "..."}
+```
+
+#### `GET /health`
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "mode": "collect",
+  "features": ["runs"],
+  "runs": 5,
+  "uptime_seconds": 3600
+}
+```
+
+### Error Responses
+
+| Code | Error | When |
+|------|-------|------|
+| 400 | `invalid_tag` | Malformed tag |
+| 400 | `invalid_cursor` | Non-numeric cursor |
+| 400 | `invalid_query` | Both search and regex provided |
+| 401 | `unauthorized` | Missing/invalid API key |
+| 404 | `run_not_found` | Run doesn't exist |
+| 409 | `tag_conflict` | Conflicting tag value on ingest |
+| 507 | `insufficient_storage` | Disk cap exceeded |
+
+---
+
+## Storage Model
+
+**Append-only file per run + in-memory tail cache.**
+
+- Each run: `{data_dir}/{run_id}/log.txt` (append-only)
+- Cursor = line number (0-indexed)
+- In-memory ring buffer caches last N lines for fast tail
+- Retention enforced by `--retention-hours`
+- Disk cap enforced by `--max-disk-mb` with oldest-run eviction
+
+---
+
+## Protocol Details
+
+### Ingest Flow
+1. Client does preflight `GET /runs/{id}` to check auth (optional, fast-fail)
+2. Client opens chunked `POST /runs/{id}/ingest`
+3. Server assigns cursor per line (monotonic int64)
+4. On connection close: flush partial line, return summary
+
+### Resume Flow
+1. Client connects `GET /runs/{id}/stream?since=5000&follow=true`
+2. Server returns `X-Logtap-Earliest-Cursor` / `X-Logtap-Latest-Cursor`
+3. If `since < earliest`: meta event includes `gap: true, missed: N`
+4. Client prints: `reconnected (missed N lines)` or continues seamlessly
+
+### Tag Semantics
+- Keys: `[a-zA-Z0-9_.-]+`
+- Values: any string, max 256 chars
+- Ingest: merge, error on conflict
+- Filter: AND semantics, exact match
+
+---
+
+## Migration Strategy
+
+**Soft migration (backward compatible):**
+- Keep `/logs/*`, `/files/*` endpoints unchanged
+- Add `/runs/*` endpoints for new workflow
+- Docs lead with ML use case, legacy in appendix
+- Stay on `0.x` until features stabilize
+
+---
+
+## Build Order
+
+1. **Ingest path** — stdin to collector
+2. **Resume cursors** — gap detection, seamless reconnect
+3. **Collector mode** — `logtap collect` with file storage
+4. **Tags + filtering** — multi-node support
+
+---
+
+## What NOT to Build
+
+- Dashboards
+- Metrics/graphs
+- Persistence backends (S3/R2) — defer to v0.5+
+- RBAC / SSO
+- Experiment comparison
+- Anything that smells like "MLOps platform"
+
+---
+
+## Success Criteria
+
+- User can `pip install logtap` and stream training logs in <60 seconds
+- SSH disconnect + reconnect shows "missed 0 lines"
+- README demo is a 20-second GIF showing the magic moment

{logtap-0.3.0 → logtap-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "logtap"
-version = "0.3.0"
+version = "0.4.0"
 description = "A CLI-first log access tool for Unix systems. Remote log file access without SSH."
 authors = [{name = "cainky", email = "kylecain.me@gmail.com"}]
 readme = "README.md"

{logtap-0.3.0 → logtap-0.4.0}/src/logtap/__init__.py

@@ -4,5 +4,5 @@ logtap - A CLI-first log access tool for Unix systems.
 Remote log file access without SSH. No database. No complex setup.
 """
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 __author__ = "Kyle Cain"

logtap-0.4.0/src/logtap/api/app.py

@@ -0,0 +1,111 @@
+"""FastAPI application factory for logtap."""
+
+import os
+import time
+from pathlib import Path
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+from logtap import __version__
+from logtap.api.routes import files, health, logs, parsed, runs
+from logtap.core.runs import RunStore
+
+
+def create_app() -> FastAPI:
+    """
+    Create and configure the FastAPI application for serve mode.
+
+    Serves static log files from a directory (legacy mode).
+
+    Returns:
+        Configured FastAPI application instance.
+    """
+    app = FastAPI(
+        title="logtap",
+        description="A CLI-first log access tool for Unix systems.",
+        version=__version__,
+        docs_url="/docs",
+        redoc_url="/redoc",
+        openapi_url="/openapi.json",
+    )
+
+    # Store mode info
+    app.state.mode = "serve"
+    app.state.features = ["files"]
+
+    # Configure CORS
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    # Include routers
+    app.include_router(health.router, tags=["health"])
+    app.include_router(logs.router, prefix="/logs", tags=["logs"])
+    app.include_router(files.router, prefix="/files", tags=["files"])
+    app.include_router(parsed.router, prefix="/parsed", tags=["parsed"])
+
+    return app
+
+
+def create_collector_app() -> FastAPI:
+    """
+    Create and configure the FastAPI application for collector mode.
+
+    Accepts ingested log streams and serves them for tailing.
+    This is the recommended mode for ML training logs.
+
+    Returns:
+        Configured FastAPI application instance.
+    """
+    app = FastAPI(
+        title="logtap",
+        description="tail -f for GPU clouds. Survives disconnects, aggregates multi-node.",
+        version=__version__,
+        docs_url="/docs",
+        redoc_url="/redoc",
+        openapi_url="/openapi.json",
+    )
+
+    # Store mode info and start time
+    app.state.mode = "collect"
+    app.state.features = ["runs"]
+    app.state.start_time = time.time()
+
+    # Configure CORS
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+
+    # Initialize run store from environment
+    data_dir = Path(os.environ.get("LOGTAP_DATA_DIR", "~/.logtap/runs")).expanduser()
+    buffer_lines = int(os.environ.get("LOGTAP_BUFFER_LINES", "100000"))
+    max_disk_mb = int(os.environ.get("LOGTAP_MAX_DISK_MB", "1000"))
+    retention_hours = int(os.environ.get("LOGTAP_RETENTION_HOURS", "72"))
+
+    run_store = RunStore(
+        data_dir=data_dir,
+        buffer_lines=buffer_lines,
+        max_disk_mb=max_disk_mb,
+        retention_hours=retention_hours,
+    )
+    runs.set_run_store(run_store)
+    app.state.run_store = run_store
+
+    # Include routers
+    app.include_router(health.router, tags=["health"])
+    app.include_router(runs.router, prefix="/runs", tags=["runs"])
+
+    return app
+
+
+# Create default app instance for uvicorn (serve mode)
+app = create_app()

logtap-0.4.0/src/logtap/api/routes/health.py

@@ -0,0 +1,41 @@
+"""Health check endpoint for logtap."""
+
+import time
+
+from fastapi import APIRouter, Request
+
+from logtap import __version__
+from logtap.models.responses import HealthResponse
+
+router = APIRouter()
+
+
+@router.get("/health", response_model=HealthResponse)
+async def health_check(request: Request) -> HealthResponse:
+    """
+    Check the health of the logtap service.
+
+    Returns:
+        Health status, version, mode, and capability information.
+    """
+    mode = getattr(request.app.state, "mode", "serve")
+    features = getattr(request.app.state, "features", ["files"])
+
+    # Get run count if in collect mode
+    runs_count = None
+    if hasattr(request.app.state, "run_store"):
+        runs_count = len(request.app.state.run_store.list_runs())
+
+    # Get uptime if start_time is set
+    uptime = None
+    if hasattr(request.app.state, "start_time"):
+        uptime = int(time.time() - request.app.state.start_time)
+
+    return HealthResponse(
+        status="healthy",
+        version=__version__,
+        mode=mode,
+        features=features,
+        runs=runs_count,
+        uptime_seconds=uptime,
+    )