warp-server 0.2.3__tar.gz → 0.2.8__tar.gz

{warp_server-0.2.3 → warp_server-0.2.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: warp-server
-Version: 0.2.3
+Version: 0.2.8
 Summary: WARP signature server API
 Requires-Python: >=3.12
 Requires-Dist: aiosqlite>=0.20
@@ -144,6 +144,54 @@ Log in with your username and API key. The UI lets you:
 - Manage users (admin only)
 - Create API keys
 - Browse functions and commits
+- Upload, browse, and download BNDBs
+
+## BNDB Sharing
+
+The server supports uploading and downloading Binary Ninja Database (`.bndb`) files.
+Files are stored on disk as gzip-compressed blobs in a configurable directory, keeping
+the SQLite metadata database lightweight.
+
+Any authenticated user can upload or download BNDBs. Only the uploader (or an admin)
+can delete them.
+
+### API
+
+```bash
+# upload a BNDB
+curl -X POST http://localhost:8000/api/v1/bndbs \
+  -H "Authorization: Bearer <key>" \
+  -F "file=@firmware.bndb" \
+  -F "name=firmware.bndb" \
+  -F "description=Extracted from router firmware"
+
+# list / search BNDBs
+curl -X POST http://localhost:8000/api/v1/bndbs/query \
+  -H "Authorization: Bearer <key>" \
+  -H "Content-Type: application/json" \
+  -d '{"name": "firmware", "limit": 20, "page": 1}'
+
+# get metadata
+curl -H "Authorization: Bearer <key>" \
+  http://localhost:8000/api/v1/bndbs/<uuid>
+
+# download
+curl -H "Authorization: Bearer <key>" -o output.bndb \
+  http://localhost:8000/api/v1/bndbs/<uuid>/download
+
+# delete (owner or admin)
+curl -X DELETE -H "Authorization: Bearer <key>" \
+  http://localhost:8000/api/v1/bndbs/<uuid>
+```
+
+BNDBs can also be managed from the **BNDBs** tab in the web UI.
+
+### Storage
+
+BNDB files are stored as `<uuid>.bndb.gz` in the directory configured by
+`WARP_BNDB_STORAGE_DIR` (default `./bndb_storage`). The directory is created
+automatically on server startup. Compression is transparent — clients always
+upload and download raw `.bndb` files.
 
 ## Configuration
 
@@ -157,6 +205,7 @@ The server is configured via environment variables, all prefixed with `WARP_`.
 | `WARP_PORT` | `8000` | Listen port. |
 | `WARP_LOG_LEVEL` | `info` | Uvicorn log level (`debug`, `info`, `warning`, `error`). |
 | `WARP_RELOAD` | `false` | Enable auto-reload on file changes (dev only). |
+| `WARP_BNDB_STORAGE_DIR` | `./bndb_storage` | Directory for uploaded BNDB files (created automatically). |
 | `WARP_CORS_ORIGINS` | `[]` | JSON list of allowed CORS origins, e.g. `'["https://app.example.com"]'`. |
 | `WARP_TRUSTED_PROXIES` | `[]` | JSON list of trusted reverse-proxy IPs allowed to set `X-Forwarded-For`. |
 | `WARP_MAX_LOGIN_ATTEMPTS` | `5` | Failed attempts before an IP or user account is locked. |
@@ -202,3 +251,4 @@ Migrations are idempotent and safe to re-run. Current migrations:
 
 - **hash-keys** — hashes any plaintext API keys (SHA-256). After migration, previously issued raw keys become invalid; create new ones via `warp-ctl key create` or the web UI.
 - **lockout-columns** — adds `locked`, `failed_login_count`, and `locked_at` columns to the `users` table for account lockout support.
+- **purge-auto-names** — deletes functions and symbols with auto-generated names (`sub_`, `nullsub`, `j_sub_`, `outlined_sub_`, `_OUTLINED_FUNCTION`).

{warp_server-0.2.3 → warp_server-0.2.8}/README.md

@@ -128,6 +128,54 @@ Log in with your username and API key. The UI lets you:
 - Manage users (admin only)
 - Create API keys
 - Browse functions and commits
+- Upload, browse, and download BNDBs
+
+## BNDB Sharing
+
+The server supports uploading and downloading Binary Ninja Database (`.bndb`) files.
+Files are stored on disk as gzip-compressed blobs in a configurable directory, keeping
+the SQLite metadata database lightweight.
+
+Any authenticated user can upload or download BNDBs. Only the uploader (or an admin)
+can delete them.
+
+### API
+
+```bash
+# upload a BNDB
+curl -X POST http://localhost:8000/api/v1/bndbs \
+  -H "Authorization: Bearer <key>" \
+  -F "file=@firmware.bndb" \
+  -F "name=firmware.bndb" \
+  -F "description=Extracted from router firmware"
+
+# list / search BNDBs
+curl -X POST http://localhost:8000/api/v1/bndbs/query \
+  -H "Authorization: Bearer <key>" \
+  -H "Content-Type: application/json" \
+  -d '{"name": "firmware", "limit": 20, "page": 1}'
+
+# get metadata
+curl -H "Authorization: Bearer <key>" \
+  http://localhost:8000/api/v1/bndbs/<uuid>
+
+# download
+curl -H "Authorization: Bearer <key>" -o output.bndb \
+  http://localhost:8000/api/v1/bndbs/<uuid>/download
+
+# delete (owner or admin)
+curl -X DELETE -H "Authorization: Bearer <key>" \
+  http://localhost:8000/api/v1/bndbs/<uuid>
+```
+
+BNDBs can also be managed from the **BNDBs** tab in the web UI.
+
+### Storage
+
+BNDB files are stored as `<uuid>.bndb.gz` in the directory configured by
+`WARP_BNDB_STORAGE_DIR` (default `./bndb_storage`). The directory is created
+automatically on server startup. Compression is transparent — clients always
+upload and download raw `.bndb` files.
 
 ## Configuration
 
@@ -141,6 +189,7 @@ The server is configured via environment variables, all prefixed with `WARP_`.
 | `WARP_PORT` | `8000` | Listen port. |
 | `WARP_LOG_LEVEL` | `info` | Uvicorn log level (`debug`, `info`, `warning`, `error`). |
 | `WARP_RELOAD` | `false` | Enable auto-reload on file changes (dev only). |
+| `WARP_BNDB_STORAGE_DIR` | `./bndb_storage` | Directory for uploaded BNDB files (created automatically). |
 | `WARP_CORS_ORIGINS` | `[]` | JSON list of allowed CORS origins, e.g. `'["https://app.example.com"]'`. |
 | `WARP_TRUSTED_PROXIES` | `[]` | JSON list of trusted reverse-proxy IPs allowed to set `X-Forwarded-For`. |
 | `WARP_MAX_LOGIN_ATTEMPTS` | `5` | Failed attempts before an IP or user account is locked. |
@@ -186,3 +235,4 @@ Migrations are idempotent and safe to re-run. Current migrations:
 
 - **hash-keys** — hashes any plaintext API keys (SHA-256). After migration, previously issued raw keys become invalid; create new ones via `warp-ctl key create` or the web UI.
 - **lockout-columns** — adds `locked`, `failed_login_count`, and `locked_at` columns to the `users` table for account lockout support.
+- **purge-auto-names** — deletes functions and symbols with auto-generated names (`sub_`, `nullsub`, `j_sub_`, `outlined_sub_`, `_OUTLINED_FUNCTION`).

{warp_server-0.2.3 → warp_server-0.2.8}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "warp-server"
-version = "0.2.3"
+version = "0.2.8"
 description = "WARP signature server API"
 readme = "README.md"
 requires-python = ">=3.12"

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/cli.py

@@ -7,7 +7,7 @@ from datetime import datetime, timezone
 from pathlib import Path
 
 import click
-from sqlalchemy import inspect, select, text, update
+from sqlalchemy import delete, inspect, select, text, update
 
 from warp_server.auth import hash_api_key
 from warp_server.database import async_session, engine
@@ -15,10 +15,14 @@ from warp_server.models import (
     ApiKey,
     Base,
     Commit,
+    Function,
+    FunctionComment,
+    FunctionConstraint,
     LoginAttempt,
     Source,
     SourceTag,
     SourceUser,
+    Symbol,
     User,
 )
 
@@ -604,6 +608,66 @@ async def _migration_lockout_columns(conn, db, apply: bool) -> bool:
     return True
 
 
+_IGNORED_NAME_PREFIXES = (
+    "sub_",
+    "nullsub",
+    "j_sub_",
+    "outlined_sub_",
+    "_OUTLINED_FUNCTION",
+)
+
+
+async def _migration_purge_auto_names(db, apply: bool) -> bool:
+    """Delete functions/symbols with auto-generated names (sub_, nullsub, etc.)."""
+    sym_sub = (
+        select(Symbol.id)
+        .where(
+            Symbol.name.like("sub_%")
+            | Symbol.name.like("nullsub%")
+            | Symbol.name.like("j_sub_%")
+            | Symbol.name.like("outlined_sub_%")
+            | Symbol.name.like("_OUTLINED_FUNCTION%")
+        )
+        .scalar_subquery()
+    )
+
+    func_sub = (
+        select(Function.id).where(Function.symbol_id.in_(sym_sub)).scalar_subquery()
+    )
+
+    result = await db.execute(
+        select(
+            select(Symbol.id).where(Symbol.id.in_(sym_sub)).count(),
+            select(Function.id).where(Function.id.in_(func_sub)).count(),
+        )
+    )
+    n_symbols, n_functions = result.one()
+
+    if n_symbols == 0:
+        return False
+
+    click.echo(
+        f"  purge-auto-names: {n_symbols} symbol(s), {n_functions} function(s) to purge"
+    )
+    if apply:
+        await db.execute(
+            delete(FunctionComment).where(FunctionComment.function_id.in_(func_sub))
+        )
+        await db.execute(
+            delete(FunctionConstraint).where(
+                FunctionConstraint.function_id.in_(func_sub)
+            )
+        )
+        await db.execute(delete(Function).where(Function.symbol_id.in_(sym_sub)))
+        await db.execute(delete(Symbol).where(Symbol.id.in_(sym_sub)))
+        await db.commit()
+        click.echo(
+            f"  purge-auto-names: purged {n_functions} function(s) "
+            f"and {n_symbols} symbol(s)"
+        )
+    return True
+
+
 @app.command("migrate")
 @click.option(
     "--apply", is_flag=True, help="Actually write changes (default is dry-run)."
@@ -629,6 +693,9 @@ def migrate(apply):
                 if await _migration_lockout_columns(conn, db, apply):
                     pending += 1
 
+                if await _migration_purge_auto_names(db, apply):
+                    pending += 1
+
         if pending == 0:
             click.echo("Everything up to date — no migrations needed.")
         elif not apply:

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/config.py

@@ -11,6 +11,7 @@ class Settings(BaseSettings):
     lockout_duration_minutes: int = 30
     reload: bool = False
     debug: bool = False
+    bndb_storage_dir: str = "./bndb_storage"
     # IPs of trusted reverse proxies allowed to set X-Forwarded-For
     trusted_proxies: list[str] = []
 

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/main.py

@@ -1,3 +1,4 @@
+import gzip
 import logging
 import time
 from contextlib import asynccontextmanager
@@ -14,6 +15,7 @@ from warp_server.database import engine
 from warp_server.models import Base
 from warp_server.routers import (
     auth,
+    bndbs,
     commits,
     files,
     functions,
@@ -33,6 +35,7 @@ logger = logging.getLogger("warp_server")
 async def lifespan(app: FastAPI):
     async with engine.begin() as conn:
         await conn.run_sync(Base.metadata.create_all)
+    Path(settings.bndb_storage_dir).mkdir(parents=True, exist_ok=True)
     yield
     await engine.dispose()
 
@@ -47,6 +50,35 @@ app = FastAPI(
 )
 
 
+@app.middleware("http")
+async def decompress_gzip_request(request: Request, call_next):
+    if request.headers.get("content-encoding") == "gzip":
+        body = await request.body()
+        try:
+            decompressed = gzip.decompress(body)
+        except Exception:
+            pass
+        else:
+            scope = request.scope
+
+            # Replace body with decompressed data
+            async def receive():
+                return {
+                    "type": "http.request",
+                    "body": decompressed,
+                    "more_body": False,
+                }
+
+            request = Request(scope, receive)
+            # Remove content-encoding so downstream doesn't re-process
+            if "content-encoding" in request.headers.keys():
+                raw_headers = [
+                    (k, v) for k, v in scope["headers"] if k != b"content-encoding"
+                ]
+                scope["headers"] = raw_headers
+    return await call_next(request)
+
+
 @app.middleware("http")
 async def log_request_duration(request: Request, call_next):
     start = time.perf_counter()
@@ -73,6 +105,7 @@ app.include_router(functions.router)
 app.include_router(files.router)
 app.include_router(types.router)
 app.include_router(search.router)
+app.include_router(bndbs.router)
 
 # -- Static web UI ---------------------------------------------------------
 _static_dir = Path(__file__).parent / "static"

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/models.py

@@ -252,3 +252,24 @@ class Type(Base):
     created_at = Column(
         DateTime, default=lambda: datetime.now(timezone.utc), nullable=False
     )
+
+
+# ── BNDBs ────────────────────────────────────────────────────────────
+
+
+class Bndb(Base):
+    __tablename__ = "bndbs"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    uuid = Column(Uuid, unique=True, nullable=False, default=uuid.uuid4)
+    name = Column(String, nullable=False)
+    description = Column(Text, nullable=True)
+    user_id = Column(
+        Integer, ForeignKey("users.id", ondelete="SET NULL"), nullable=True
+    )
+    size_original = Column(Integer, nullable=False)
+    size_compressed = Column(Integer, nullable=False)
+    sha256 = Column(String(64), nullable=False)
+    created_at = Column(
+        DateTime, default=lambda: datetime.now(timezone.utc), nullable=False
+    )

warp_server-0.2.8/src/warp_server/routers/bndbs.py

@@ -0,0 +1,215 @@
+import asyncio
+import gzip
+import hashlib
+import logging
+import math
+from pathlib import Path
+from uuid import UUID, uuid4
+
+from fastapi import APIRouter, Depends, Form, HTTPException, UploadFile, status
+from fastapi.responses import StreamingResponse
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from warp_server.auth import get_current_user
+from warp_server.config import settings
+from warp_server.database import get_db
+from warp_server.models import Bndb, User
+from warp_server.schemas import BndbMdl, BndbQueryRequest, PaginatedResponse
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/v1/bndbs", tags=["bndbs"])
+
+CHUNK_SIZE = 256 * 1024  # 256 KB
+
+
+def _storage_dir() -> Path:
+    return Path(settings.bndb_storage_dir)
+
+
+def _bndb_path(bndb_uuid: UUID) -> Path:
+    return _storage_dir() / f"{bndb_uuid}.bndb.gz"
+
+
+def _bndb_to_response(bndb: Bndb, username: str | None = None) -> BndbMdl:
+    return BndbMdl(
+        id=bndb.id,
+        uuid=bndb.uuid,
+        name=bndb.name,
+        description=bndb.description,
+        user_id=bndb.user_id,
+        username=username,
+        size_original=bndb.size_original,
+        size_compressed=bndb.size_compressed,
+        sha256=bndb.sha256,
+        created_at=bndb.created_at,
+    )
+
+
+async def _stream_compress_upload(
+    upload: UploadFile, dest: Path
+) -> tuple[int, int, str]:
+    """Stream an upload through gzip to disk.
+
+    Returns (original_size, compressed_size, sha256_hex).
+    """
+    sha = hashlib.sha256()
+    original_size = 0
+
+    def _write_sync():
+        nonlocal original_size
+        with gzip.open(dest, "wb", compresslevel=6) as gz:
+            while True:
+                chunk = upload.file.read(CHUNK_SIZE)
+                if not chunk:
+                    break
+                sha.update(chunk)
+                original_size += len(chunk)
+                gz.write(chunk)
+
+    await asyncio.to_thread(_write_sync)
+
+    compressed_size = dest.stat().st_size
+    return original_size, compressed_size, sha.hexdigest()
+
+
+@router.post("", response_model=BndbMdl)
+async def upload_bndb(
+    file: UploadFile,
+    name: str = Form(...),
+    description: str | None = Form(None),
+    db: AsyncSession = Depends(get_db),
+    user: User = Depends(get_current_user),
+):
+    bndb = Bndb(
+        uuid=uuid4(),
+        name=name,
+        description=description,
+        user_id=user.id,
+        size_original=0,
+        size_compressed=0,
+        sha256="",
+    )
+    dest = _bndb_path(bndb.uuid)
+
+    try:
+        original, compressed, sha256_hex = await _stream_compress_upload(file, dest)
+        bndb.size_original = original
+        bndb.size_compressed = compressed
+        bndb.sha256 = sha256_hex
+
+        db.add(bndb)
+        await db.commit()
+        await db.refresh(bndb)
+    except Exception:
+        dest.unlink(missing_ok=True)
+        raise
+
+    logger.info(
+        "BNDB upload %s: %d → %d bytes (%.0f%% compression)",
+        name,
+        original,
+        compressed,
+        (1 - compressed / original) * 100 if original else 0,
+    )
+
+    return _bndb_to_response(bndb, user.username)
+
+
+@router.post("/query", response_model=PaginatedResponse[BndbMdl])
+async def query_bndbs(
+    body: BndbQueryRequest,
+    db: AsyncSession = Depends(get_db),
+    _user: User = Depends(get_current_user),
+):
+    limit = body.limit or 50
+    page = body.page or 1
+
+    query = select(Bndb, User.username).outerjoin(User, Bndb.user_id == User.id)
+    count_query = select(func.count(Bndb.id))
+
+    if body.name:
+        query = query.where(Bndb.name.ilike(f"%{body.name}%"))
+        count_query = count_query.where(Bndb.name.ilike(f"%{body.name}%"))
+
+    total = (await db.execute(count_query)).scalar() or 0
+    total_pages = max(1, math.ceil(total / limit))
+
+    query = query.order_by(Bndb.created_at.desc())
+    query = query.offset((page - 1) * limit).limit(limit)
+
+    result = await db.execute(query)
+    items = [_bndb_to_response(row.Bndb, row.username) for row in result.all()]
+
+    return PaginatedResponse(items=items, total_pages=total_pages, total_results=total)
+
+
+@router.get("/{bndb_uuid}", response_model=BndbMdl)
+async def get_bndb(
+    bndb_uuid: UUID,
+    db: AsyncSession = Depends(get_db),
+    _user: User = Depends(get_current_user),
+):
+    result = await db.execute(
+        select(Bndb, User.username)
+        .outerjoin(User, Bndb.user_id == User.id)
+        .where(Bndb.uuid == bndb_uuid)
+    )
+    row = result.one_or_none()
+    if row is None:
+        raise HTTPException(status_code=404, detail="BNDB not found")
+    return _bndb_to_response(row.Bndb, row.username)
+
+
+@router.get("/{bndb_uuid}/download")
+async def download_bndb(
+    bndb_uuid: UUID,
+    db: AsyncSession = Depends(get_db),
+    _user: User = Depends(get_current_user),
+):
+    result = await db.execute(select(Bndb).where(Bndb.uuid == bndb_uuid))
+    bndb = result.scalar_one_or_none()
+    if bndb is None:
+        raise HTTPException(status_code=404, detail="BNDB not found")
+
+    path = _bndb_path(bndb.uuid)
+    if not path.exists():
+        raise HTTPException(status_code=404, detail="BNDB file missing from storage")
+
+    def _decompress_stream():
+        with gzip.open(path, "rb") as gz:
+            while chunk := gz.read(CHUNK_SIZE):
+                yield chunk
+
+    filename = bndb.name if bndb.name.endswith(".bndb") else bndb.name + ".bndb"
+    safe_name = filename.replace('"', "'")
+    return StreamingResponse(
+        _decompress_stream(),
+        media_type="application/octet-stream",
+        headers={
+            "Content-Disposition": f'attachment; filename="{safe_name}"',
+            "Content-Length": str(bndb.size_original),
+        },
+    )
+
+
+@router.delete("/{bndb_uuid}", status_code=status.HTTP_204_NO_CONTENT)
+async def delete_bndb(
+    bndb_uuid: UUID,
+    db: AsyncSession = Depends(get_db),
+    user: User = Depends(get_current_user),
+):
+    result = await db.execute(select(Bndb).where(Bndb.uuid == bndb_uuid))
+    bndb = result.scalar_one_or_none()
+    if bndb is None:
+        raise HTTPException(status_code=404, detail="BNDB not found")
+
+    if bndb.user_id != user.id and user.role != "Admin":
+        raise HTTPException(status_code=403, detail="Only the owner or an admin can delete this BNDB")
+
+    path = _bndb_path(bndb.uuid)
+    await db.delete(bndb)
+    await db.commit()
+
+    path.unlink(missing_ok=True)

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/routers/files.py

@@ -158,6 +158,14 @@ async def push_file_json(
 
 BATCH_SIZE = 500
 
+_IGNORED_NAME_PREFIXES = (
+    "sub_",
+    "nullsub",
+    "j_sub_",
+    "outlined_sub_",
+    "_OUTLINED_FUNCTION",
+)
+
 
 async def _get_or_create_target(
     db: AsyncSession,
@@ -191,6 +199,8 @@ async def _resolve_symbols(
     needed: set[tuple[str, str, str]] = set()
     for chunk in chunks:
         for pf in chunk.functions:
+            if pf.symbol.name.startswith(_IGNORED_NAME_PREFIXES):
+                continue
             needed.add((pf.symbol.name, pf.symbol.symbol_class, pf.symbol.modifier))
 
     if not needed:
@@ -291,6 +301,8 @@ async def _ingest_chunks(
         )
 
         for pf in chunk.functions:
+            if pf.symbol.name.startswith(_IGNORED_NAME_PREFIXES):
+                continue
             sym_key = (pf.symbol.name, pf.symbol.symbol_class, pf.symbol.modifier)
             func_rows.append(
                 {

{warp_server-0.2.3 → warp_server-0.2.8}/src/warp_server/schemas.py

@@ -291,3 +291,25 @@ class SearchResponse(BaseModel):
     limit: int
     offset: int
     items: list[SearchIndexRow]
+
+
+# ── BNDBs ────────────────────────────────────────────────────────────
+
+
+class BndbMdl(BaseModel):
+    id: int
+    uuid: UUID
+    name: str
+    description: str | None = None
+    user_id: int | None = None
+    username: str | None = None
+    size_original: int
+    size_compressed: int
+    sha256: str
+    created_at: datetime
+
+
+class BndbQueryRequest(BaseModel):
+    name: str | None = None
+    limit: int | None = Field(None, ge=1, le=200)
+    page: int | None = Field(None, ge=1)