weed-cli 1.6.0__tar.gz → 1.6.7__tar.gz

{weed_cli-1.6.0/weed_cli.egg-info → weed_cli-1.6.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: weed-cli
-Version: 1.6.0
+Version: 1.6.7
 Summary: Censorship-resistant video PoC — discovery/hosting/download over signed relay events, a real Kademlia DHT, or a TLS-capable NAT-traversal tunnel
 License: MIT
 Keywords: p2p,video,censorship-resistant,discovery,dht,kademlia,nat-traversal
@@ -174,17 +174,28 @@ blind.
 ### Docker
 
 `Dockerfile.node` + `docker-compose.node.yml` package `web_ui.py` to run
-somewhere other than a laptop.
+somewhere other than a laptop — alongside its own local discovery relay
+(`Dockerfile.discovery-relay`), so there's always something real for it
+to announce to and discover from out of the box.
 
 ```bash
 make node                            # build + run, http://127.0.0.1:8080
 make node-down                       # stop it — data persists
+make node-shell                      # interactive weed shell, same container as the running node
 WEED_SHARE_DIR=~/Movies make node    # mount a real directory of .ott archives at /share
+docker compose -f docker-compose.node.yml exec node python3 weed.py discover  # or a one-off CLI command
 ```
 
 - Identity key, reputation store, and library manifest all persist in a
   named volume (`node-data`, mounted at `$HOME=/data`) — one node keeps
   the same pubkey (and everything vouching for it) across restarts.
+- The `relay` service is `node`'s default `WEED_RELAY` (`http://relay:9101`,
+  resolved over Compose's own network) — its events persist in their own
+  `relay-data` volume, and port 9101 is published to the host too, for
+  curling `/events` directly or pointing a non-Docker `weed`/shell session
+  at the exact same relay. Override `WEED_RELAY` (shell env or `.env`,
+  see `.env.sample`) to point at public infra instead, e.g. for a real
+  end-to-end test.
 - `/share` is a separate *bind* mount, not a named volume, so the actual
   `.ott` archives you're hosting are real files on the host you can see
   and manage directly. Point the web UI's Host form at `/share`.

{weed_cli-1.6.0 → weed_cli-1.6.7}/README.md

@@ -144,17 +144,28 @@ blind.
 ### Docker
 
 `Dockerfile.node` + `docker-compose.node.yml` package `web_ui.py` to run
-somewhere other than a laptop.
+somewhere other than a laptop — alongside its own local discovery relay
+(`Dockerfile.discovery-relay`), so there's always something real for it
+to announce to and discover from out of the box.
 
 ```bash
 make node                            # build + run, http://127.0.0.1:8080
 make node-down                       # stop it — data persists
+make node-shell                      # interactive weed shell, same container as the running node
 WEED_SHARE_DIR=~/Movies make node    # mount a real directory of .ott archives at /share
+docker compose -f docker-compose.node.yml exec node python3 weed.py discover  # or a one-off CLI command
 ```
 
 - Identity key, reputation store, and library manifest all persist in a
   named volume (`node-data`, mounted at `$HOME=/data`) — one node keeps
   the same pubkey (and everything vouching for it) across restarts.
+- The `relay` service is `node`'s default `WEED_RELAY` (`http://relay:9101`,
+  resolved over Compose's own network) — its events persist in their own
+  `relay-data` volume, and port 9101 is published to the host too, for
+  curling `/events` directly or pointing a non-Docker `weed`/shell session
+  at the exact same relay. Override `WEED_RELAY` (shell env or `.env`,
+  see `.env.sample`) to point at public infra instead, e.g. for a real
+  end-to-end test.
 - `/share` is a separate *bind* mount, not a named volume, so the actual
   `.ott` archives you're hosting are real files on the host you can see
   and manage directly. Point the web UI's Host form at `/share`.

{weed_cli-1.6.0 → weed_cli-1.6.7}/node.py

@@ -19,6 +19,7 @@ every previous script verified chunks locally or over a network, none of
 them reassembled a full file from a remote peer onto disk before this.
 """
 import base64
+import concurrent.futures
 import hashlib
 import json
 import os
@@ -44,8 +45,12 @@ TAGLINE = 'we do in 4 what others do in 5'
 
 
 def weed_version():
-    """Installed package version (pyproject.toml's source of truth), with a
-    fallback for running straight from source without `pip install -e .`."""
+    """Installed package version (pyproject.toml's source of truth), with
+    fallbacks for the two other real ways this runs: straight from a git
+    checkout without `pip install -e .` (read pyproject.toml directly —
+    Dockerfile.node copies it in for exactly this, since it copies loose
+    .py files rather than pip-installing the package), or neither file
+    present at all (genuinely no version info available)."""
     try:
         from importlib.metadata import version, PackageNotFoundError
         try:
@@ -54,6 +59,15 @@ def weed_version():
             pass
     except ImportError:
         pass
+    pyproject_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'pyproject.toml')
+    try:
+        with open(pyproject_path) as f:
+            for line in f:
+                line = line.strip()
+                if line.startswith('version'):
+                    return line.split('=', 1)[1].strip().strip('"\'')
+    except (OSError, IndexError):
+        pass
     return '0.0.0-dev'
 
 
@@ -65,9 +79,36 @@ def _git_commit_hash():
     behind a real debugging session: two checkouts reporting the same
     version, one of them missing a fix the other had. None for anything
     that isn't a git checkout at all (installed from a built wheel/sdist,
-    no .git present) — a version number is all there is to go on there."""
+    no .git present) — a version number is all there is to go on there.
+
+    $WEED_GIT_COMMIT checked first — Dockerfile.node has no .git directory
+    at all (never copied in, on purpose: bloats the image and ships the
+    full history for no reason), so a container build has no way to
+    answer this live. docker-compose.node.yml passes the *host's* commit
+    hash in as a build arg at image-build time instead, baked in as this
+    env var, same intent as the live git lookup below just computed once,
+    earlier, somewhere that actually has the repo.
+
+    `git rev-parse` walks *up* parent directories looking for a .git —
+    a real `pip install .` (not `-e`) copies this file into site-packages,
+    which usually isn't a checkout at all, but if it happens to be nested
+    anywhere under some unrelated git-tracked ancestor directory (a
+    dotfiles repo, a pyenv install tracked in git, anything above it),
+    this would otherwise silently report *that* repo's own unrelated
+    commit — worse than showing nothing, since it looks plausible. Only
+    trust the hash once --show-toplevel confirms this file's own
+    directory really is that repo's root."""
+    env_commit = os.environ.get('WEED_GIT_COMMIT')
+    if env_commit:
+        return env_commit
     repo_dir = os.path.dirname(os.path.abspath(__file__))
     try:
+        toplevel = subprocess.run(['git', 'rev-parse', '--show-toplevel'], cwd=repo_dir,
+                                   capture_output=True, text=True, timeout=2)
+        if toplevel.returncode != 0:
+            return None
+        if os.path.realpath(toplevel.stdout.strip()) != os.path.realpath(repo_dir):
+            return None
         result = subprocess.run(['git', 'rev-parse', '--short', 'HEAD'], cwd=repo_dir,
                                  capture_output=True, text=True, timeout=2)
         if result.returncode == 0:
@@ -337,8 +378,24 @@ def serve_session(conn, entries_by_hash, default_hash, price, ln_node=None):
         _graceful_close(conn)
 
 
+def bind_host_port(port, bind_host='0.0.0.0'):
+    """Split out of run_host_server so a caller (see web_ui.py's
+    _run_host_job) can bind the real, permanent listening socket early —
+    right where a port-collision needs to fail fast, before announcing
+    anything — and hand that exact socket to run_host_server later
+    instead of it binding a second, separate one. A bind-then-close probe
+    followed by a *second*, later bind of the same port isn't atomic
+    across two hosts starting concurrently: both can pass the probe
+    before either does the real bind. Binding once, early, and reusing
+    the same socket removes that race entirely instead of narrowing it."""
+    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+    srv.bind((bind_host, port))
+    return srv
+
+
 def run_host_server(archive_dir, file_name, port, bind_host='0.0.0.0', quiet=False, price=0,
-                     ln_node=None):
+                     ln_node=None, sock=None):
     archive_dir = os.path.expanduser(archive_dir)
     entries = load_manifest_entries(archive_dir, file_name)
     entries_by_hash = {}
@@ -350,9 +407,7 @@ def run_host_server(archive_dir, file_name, port, bind_host='0.0.0.0', quiet=Fal
         entries_by_hash[entry['sha256']] = (entry, leaves, file_path)
     default_hash = next(iter(entries_by_hash)) if len(entries_by_hash) == 1 else None
 
-    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    srv.bind((bind_host, port))
+    srv = sock if sock is not None else bind_host_port(port, bind_host)
     srv.listen(8)
     if not quiet:
         # a background thread's print() races with cmd.Cmd's input()-driven
@@ -979,19 +1034,61 @@ def _relay_url_hint(relay_url):
     return None
 
 
+def _normalize_relay_url(relay_url):
+    """A bare host:port with no scheme (e.g. '127.0.0.1:9101', the most
+    natural way to type a local relay) isn't a URL urllib can parse at
+    all -- everything before the first ':' becomes the *scheme*, so this
+    reads as scheme '127.0.0.1' and raises "unknown url type", not a
+    connection error. That already lands on post_event/fetch_events'
+    existing except clauses (an unreachable relay is routine, not
+    exceptional), which is exactly the trap: hosting against a
+    scheme-less relay used to report status: running with zero actual
+    announcements ever reaching it, and discover against one silently
+    came back empty -- nothing hinted that the URL itself, not the
+    relay, was the problem. Every relay this project actually ships or
+    documents is plain HTTP(S), so defaulting a missing scheme to
+    http:// turns the common typo into the thing the user obviously
+    meant instead of a silent no-op."""
+    if '://' not in relay_url:
+        return 'http://' + relay_url
+    return relay_url
+
+
+_dns_bound_pool = concurrent.futures.ThreadPoolExecutor(max_workers=8, thread_name_prefix='urlopen')
+
+
+def _urlopen_bounded(req_or_url, timeout=5):
+    """urlopen's own timeout= only bounds the connect/read phases — the
+    getaddrinfo() call underneath (DNS resolution) is a blocking libc call
+    with no timeout of its own, so a relay hostname that's slow or
+    unreachable to resolve can hang here indefinitely no matter what
+    timeout= says. That's exactly the situation right after a container
+    boots and its network/DNS isn't fully settled yet: _resume_persisted_hosts
+    (web_ui.py) resuming a host at that moment used to get stuck at
+    "(starting…)" forever, since nothing downstream ever times out either.
+    Running the call in a worker thread and bounding *that* with
+    .result(timeout=) catches DNS hangs the same as connect/read hangs —
+    the stuck worker thread leaks (a blocking C call can't be cancelled),
+    but it's a daemon-pool thread and this process was going to give up and
+    move on regardless."""
+    future = _dns_bound_pool.submit(urllib.request.urlopen, req_or_url, timeout=timeout)
+    return future.result(timeout=timeout + 1)
+
+
 def post_event(relay_url, event):
     hint = _relay_url_hint(relay_url)
     if hint:
         return {'ok': False, 'error': hint}
+    relay_url = _normalize_relay_url(relay_url)
     req = urllib.request.Request(
         f'{relay_url}/event', data=json.dumps(event).encode(),
         headers={'Content-Type': 'application/json'}, method='POST')
     try:
-        with urllib.request.urlopen(req, timeout=5) as resp:
+        with _urlopen_bounded(req, timeout=5) as resp:
             return json.loads(resp.read())
     except urllib.error.HTTPError as e:
         return json.loads(e.read())
-    except (urllib.error.URLError, ConnectionRefusedError) as e:
+    except (urllib.error.URLError, ConnectionRefusedError, concurrent.futures.TimeoutError) as e:
         # download_with_auction posts the post-download attestation to every
         # relay in relay_urls, which by default includes 127.0.0.1:9101
         # whether or not anything is actually listening there (same default
@@ -1006,11 +1103,12 @@ def fetch_events(relay_url, event_type=None):
     if hint:
         print(f"  {relay_url}: {hint}")
         return None
+    relay_url = _normalize_relay_url(relay_url)
     url = f'{relay_url}/events' + (f'?type={event_type}' if event_type else '')
     try:
-        with urllib.request.urlopen(url, timeout=5) as resp:
+        with _urlopen_bounded(url, timeout=5) as resp:
             return json.loads(resp.read())
-    except (urllib.error.URLError, ConnectionRefusedError):
+    except (urllib.error.URLError, ConnectionRefusedError, concurrent.futures.TimeoutError):
         return None
 
 

{weed_cli-1.6.0 → weed_cli-1.6.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "weed-cli"
-version = "1.6.0"
+version = "1.6.7"
 description = "Censorship-resistant video PoC — discovery/hosting/download over signed relay events, a real Kademlia DHT, or a TLS-capable NAT-traversal tunnel"
 readme = "README.md"
 license = { text = "MIT" }

{weed_cli-1.6.0 → weed_cli-1.6.7}/web_ui.py

@@ -187,9 +187,35 @@ def _ott_status(archive_dir):
 
 
 def _resume_persisted_hosts():
+    """Sequential on purpose, not fire-and-forget-them-all-at-once: two
+    persisted configs sharing a port (the default in the Host form is
+    always 9201, so this is the common case, not an edge case) could
+    both bind successfully if started concurrently, since node.bind_host_port
+    for the second one wouldn't even run until the first's already had time
+    to occupy the port — but nothing serializes *when* each thread gets
+    there. Waiting for each host to reach 'bound' (or 'error', if the bind
+    itself failed) before starting the next one closes that window: by the
+    time the next one binds, the previous one already has the port or
+    already knows it can't.
+
+    Only waits for the bind, not all the way to 'running' — 'running'
+    doesn't happen until after each host's full publish loop (real network
+    I/O, up to 5s timeout per relay), which several persisted hosts
+    resuming in sequence have no reason to serialize on. That was a real
+    regression the first version of this function had: booting with a
+    handful of persisted hosts got noticeably slower, for zero extra
+    correctness, since the publish delay doesn't affect port ownership at
+    all."""
     for cfg in _persisted_hosts.values():
-        _start_host_job(cfg['archive_dir'], cfg['file_name'], cfg['port'], cfg['price'],
-                         cfg['relay'], cfg['advertise_host'], cfg['tunnel'], cfg.get('lightning_node'))
+        host_id = _start_host_job(cfg['archive_dir'], cfg['file_name'], cfg['port'], cfg['price'],
+                                   cfg['relay'], cfg['advertise_host'], cfg['tunnel'],
+                                   cfg.get('lightning_node'))
+        for _ in range(100):  # up to ~10s per host before moving on regardless
+            with _lock:
+                status = _hosts[host_id]['status']
+            if status in ('bound', 'running', 'error'):
+                break
+            time.sleep(0.1)
 
 
 def _unpublish_all_hosts():
@@ -284,29 +310,6 @@ def _detect_lan_ip():
         s.close()
 
 
-def _probe_port_free(port, bind_host='0.0.0.0'):
-    """Bind-and-immediately-release, purely to fail fast on a port
-    collision *before* announcing anything — see _run_host_job's real
-    incident: the web UI's Host form always defaults to port 9201, so
-    hosting more than one file (including multiple entries auto-resumed
-    at once on startup, see _resume_persisted_hosts) without picking a
-    distinct port each time meant every loser published a real, signed
-    announcement and only discovered it couldn't actually bind *after*
-    — a permanent, unreachable zombie listing nothing could ever clean
-    up, since a host that never reaches 'running' never gets files/
-    announced_on populated for the graceful-shutdown unpublish to use
-    either. Still has a small window (this socket closes before
-    run_host_server's own real bind), not a hard guarantee — but it
-    catches the common case (concurrent startup, the actual incident)
-    instead of the current zero attempt to catch it at all."""
-    probe = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    try:
-        probe.bind((bind_host, port))
-    finally:
-        probe.close()
-
-
 def _run_host_job(host_id, archive_dir, file_name, port, price, relay_urls, advertise_host, tunnel,
                    ln_node=None):
     captured = io.StringIO()
@@ -325,17 +328,44 @@ def _run_host_job(host_id, archive_dir, file_name, port, price, relay_urls, adve
             # and only fail later, deep in a background thread with no way
             # for the UI to ever find out
             all_leaves = {e['sha256']: node.load_leaves(archive_dir, e['sha256']) for e in entries}
-            # also fail fast on a port collision, before announcing anything
-            # — see _probe_port_free's docstring for the real incident this
-            # prevents
-            _probe_port_free(port)
+            # bind the real, permanent listening socket now — before
+            # announcing anything — and hand it to run_host_server below
+            # instead of it binding a second one later. A separate
+            # bind-then-close probe here, followed by a *later* real bind,
+            # isn't atomic across two hosts starting concurrently (see
+            # _resume_persisted_hosts): both could pass the probe before
+            # either did the real bind. One bind, kept alive and reused,
+            # has no such window — the port either really is free right
+            # now or this raises immediately, before any publish.
+            bound_sock = node.bind_host_port(port)
+            # a distinct status the instant the bind (the only part
+            # _resume_persisted_hosts actually needs to wait for) is done
+            # — 'running' doesn't happen until after the publish loop
+            # below, which is real network I/O (up to 5s timeout *per*
+            # relay, see post_event) that resuming several hosts
+            # sequentially has no reason to serialize on
+            with _lock:
+                _hosts[host_id]['status'] = 'bound'
             ott_status = node.ott_commit_status(archive_dir)
             announced = []
             for entry in entries:
                 for relay_url in relay_urls:
                     host_addr = f'{advertise_host}:{port}'
-                    node.publish(identity, relay_url, entry['sha256'], entry['name'], host_addr,
-                                  tunnel=tunnel, ott_status=ott_status)
+                    result = node.publish(identity, relay_url, entry['sha256'], entry['name'], host_addr,
+                                           tunnel=tunnel, ott_status=ott_status)
+                    # publish()/post_event() report a failed announce as a
+                    # normal {'ok': False, ...} return, not an exception (an
+                    # unreachable or malformed relay is routine, not
+                    # exceptional -- see post_event's own docstring) -- which
+                    # this loop used to just ignore entirely, so a host could
+                    # sit at status: running, "announced" on a relay it never
+                    # actually reached, with nothing in the UI to say so.
+                    # do_host (shell.py) already prints every result
+                    # unconditionally; this only prints the failures; a
+                    # working host's log staying free of routine "ok: True"
+                    # noise matters more here than in a CLI's own scrollback.
+                    if isinstance(result, dict) and not result.get('ok', True):
+                        print(f'  ✗ announce to {relay_url} failed: {result.get("error")}')
                 announced = relay_urls
             files = [{'name': e['name'], 'content_hash': e['sha256']} for e in entries]
             with _lock:
@@ -354,7 +384,8 @@ def _run_host_job(host_id, archive_dir, file_name, port, price, relay_urls, adve
                                             all_leaves[entry['sha256']], file_path, price),
                                       kwargs={'use_tls': use_tls, 'quiet': True, 'ln_node': ln_node},
                                       daemon=True).start()
-            node.run_host_server(archive_dir, file_name, port, quiet=True, price=price, ln_node=ln_node)
+            node.run_host_server(archive_dir, file_name, port, quiet=True, price=price, ln_node=ln_node,
+                                  sock=bound_sock)
     except SystemExit as e:
         with _lock:
             _hosts[host_id].update(status='error', error=_with_captured_detail(str(e), captured))
@@ -374,6 +405,11 @@ def _run_download_job(job_id, content_hash, relay_urls, out_path, k, use_lightni
     try:
         t0 = time.time()
         with _quiet() as captured:
+            # registered as soon as it's the real live buffer (not the
+            # placeholder above) so /api/download/<job_id> can read
+            # node.py's real prints as they happen, not just after the
+            # job finishes — see the GET handler's use of _job_logs
+            _job_logs[job_id] = captured
             path = node.download_with_auction(content_hash, relay_urls, out_path=out_path, k=k,
                                                use_lightning=use_lightning, lightning_node=lightning_node,
                                                on_progress=on_progress)
@@ -458,9 +494,16 @@ class Handler(BaseHTTPRequestHandler):
                     'subscriptions': list(_library['subscriptions']),
                 })
         if path.startswith('/api/download/'):
+            job_id = path[len('/api/download/'):]
             with _lock:
-                job = _jobs.get(path[len('/api/download/'):])
-            return self._json(job) if job else self._json({'error': 'no such job'}, status=404)
+                job = _jobs.get(job_id)
+                job = dict(job) if job else None
+            if not job:
+                return self._json({'error': 'no such job'}, status=404)
+            log_buf = _job_logs.get(job_id)
+            if log_buf is not None:
+                job['log'] = log_buf.getvalue()
+            return self._json(job)
         if path.startswith('/api/reputation/'):
             from poc_reputation import ReputationStore
             pubkey = path[len('/api/reputation/'):]
@@ -755,6 +798,13 @@ def run_web_ui(port=8080, bind_host='127.0.0.1', quiet=False, advertise_host=Non
         _lan_url = f'http://{reachable_host}:{port}/'
 
     if not quiet:
+        # answers "is this container actually running the code I think it
+        # is" directly in `docker compose logs`/`make node`'s own output —
+        # see node.weed_banner()'s own docstring for exactly the debugging
+        # session that motivated adding a commit hash to it in the first
+        # place; printing it here means the same question doesn't need a
+        # docker exec + grep to answer for the web UI specifically
+        print(f"[web:{port}] {node.weed_banner()}", flush=True)
         print(f"[web:{port}] weed control UI at http://{bind_host}:{port}/", flush=True)
         if bind_host == '127.0.0.1':
             print("  bound to localhost only -- pass --bind 0.0.0.0 to reach this from your "

{weed_cli-1.6.0 → weed_cli-1.6.7/weed_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: weed-cli
-Version: 1.6.0
+Version: 1.6.7
 Summary: Censorship-resistant video PoC — discovery/hosting/download over signed relay events, a real Kademlia DHT, or a TLS-capable NAT-traversal tunnel
 License: MIT
 Keywords: p2p,video,censorship-resistant,discovery,dht,kademlia,nat-traversal
@@ -174,17 +174,28 @@ blind.
 ### Docker
 
 `Dockerfile.node` + `docker-compose.node.yml` package `web_ui.py` to run
-somewhere other than a laptop.
+somewhere other than a laptop — alongside its own local discovery relay
+(`Dockerfile.discovery-relay`), so there's always something real for it
+to announce to and discover from out of the box.
 
 ```bash
 make node                            # build + run, http://127.0.0.1:8080
 make node-down                       # stop it — data persists
+make node-shell                      # interactive weed shell, same container as the running node
 WEED_SHARE_DIR=~/Movies make node    # mount a real directory of .ott archives at /share
+docker compose -f docker-compose.node.yml exec node python3 weed.py discover  # or a one-off CLI command
 ```
 
 - Identity key, reputation store, and library manifest all persist in a
   named volume (`node-data`, mounted at `$HOME=/data`) — one node keeps
   the same pubkey (and everything vouching for it) across restarts.
+- The `relay` service is `node`'s default `WEED_RELAY` (`http://relay:9101`,
+  resolved over Compose's own network) — its events persist in their own
+  `relay-data` volume, and port 9101 is published to the host too, for
+  curling `/events` directly or pointing a non-Docker `weed`/shell session
+  at the exact same relay. Override `WEED_RELAY` (shell env or `.env`,
+  see `.env.sample`) to point at public infra instead, e.g. for a real
+  end-to-end test.
 - `/share` is a separate *bind* mount, not a named volume, so the actual
   `.ott` archives you're hosting are real files on the host you can see
   and manage directly. Point the web UI's Host form at `/share`.