hindsight-embed 1.2.0__tar.gz → 1.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/PKG-INFO +1 -1
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/__init__.py +1 -1
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/daemon_embed_manager.py +178 -6
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/env.example +117 -11
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/profile_manager.py +2 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/pyproject.toml +1 -1
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_daemon_client.py +159 -1
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_embed_manager.py +3 -1
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_profile_daemon_config.py +2 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/.gitignore +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/LICENSE +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/README.md +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/_http_probe.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/cli.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/__init__.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/lifecycle.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/providers.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/server.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/service.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/assets/index-Bti1AEAP.css +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/assets/index-D7TwAsED.js +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/favicon.png +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/Inter-400.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/Inter-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/Inter-600.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/JetBrainsMono-400.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/JetBrainsMono-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/SpaceGrotesk-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/SpaceGrotesk-600.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/fonts/SpaceGrotesk-700.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/index.html +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/logo.png +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/README.md +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/index.html +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/package-lock.json +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/package.json +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/favicon.png +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/Inter-400.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/Inter-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/Inter-600.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/JetBrainsMono-400.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/JetBrainsMono-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/SpaceGrotesk-500.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/SpaceGrotesk-600.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/fonts/SpaceGrotesk-700.woff2 +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/logo.png +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/api.js +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/app.jsx +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/main.jsx +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/service-panel.jsx +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/style.css +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/vite.config.js +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/daemon_client.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/embed_manager.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/env_template.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/py.typed +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/test.sh +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/__init__.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/http_stub.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_cli_forwarding.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_control_center.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_env_template.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_http_probe.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_profile_integration.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_profile_lock.py +0 -0
- {hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/tests/test_profile_manager.py +0 -0
|
@@ -11,6 +11,8 @@ import os
|
|
|
11
11
|
import platform
|
|
12
12
|
import re
|
|
13
13
|
import signal
|
|
14
|
+
import socket
|
|
15
|
+
import struct
|
|
14
16
|
import subprocess
|
|
15
17
|
import sys
|
|
16
18
|
import sysconfig
|
|
@@ -28,7 +30,7 @@ from rich.text import Text
|
|
|
28
30
|
|
|
29
31
|
from ._http_probe import ProbeResponse, probe_get
|
|
30
32
|
from .embed_manager import EmbedManager
|
|
31
|
-
from .profile_manager import ProfileLockTimeout, ProfileManager, lock_file, unlock_file
|
|
33
|
+
from .profile_manager import ProfileLockTimeout, ProfileManager, ProfilePaths, lock_file, unlock_file
|
|
32
34
|
|
|
33
35
|
logger = logging.getLogger(__name__)
|
|
34
36
|
console = Console(stderr=True)
|
|
@@ -71,6 +73,13 @@ def _parse_non_negative_int(value: str | None, default: int, name: str) -> int:
|
|
|
71
73
|
# runners than POSIX.
|
|
72
74
|
DAEMON_STARTUP_TIMEOUT = int(os.getenv("HINDSIGHT_EMBED_DAEMON_STARTUP_TIMEOUT", "180"))
|
|
73
75
|
DAEMON_STARTUP_TERMINATE_TIMEOUT = 10
|
|
76
|
+
# How long a start waits for the daemon it replaces to finish exiting. A stopping
|
|
77
|
+
# daemon frees its port first, then drains in-flight work (up to
|
|
78
|
+
# HINDSIGHT_API_WORKER_SHUTDOWN_DRAIN_SECONDS, 20s by default), and only then stops
|
|
79
|
+
# the profile's embedded Postgres. A replacement started inside that window ran its
|
|
80
|
+
# migrations against a Postgres the old daemon then shut down: "Database migration
|
|
81
|
+
# failed" on restart (#6540). Generous on purpose: waiting is cheap, the race is not.
|
|
82
|
+
PREVIOUS_DAEMON_EXIT_TIMEOUT = 60.0
|
|
74
83
|
DEFAULT_DAEMON_IDLE_TIMEOUT = 0 # 0 = disabled (no auto-exit)
|
|
75
84
|
ENV_DAEMON_LOG_MAX_BYTES = "HINDSIGHT_EMBED_DAEMON_LOG_MAX_BYTES"
|
|
76
85
|
ENV_DAEMON_LOG_BACKUP_COUNT = "HINDSIGHT_EMBED_DAEMON_LOG_BACKUP_COUNT"
|
|
@@ -186,6 +195,39 @@ def _strip_parent_interpreter_env(env: dict[str, str]) -> dict[str, str]:
|
|
|
186
195
|
return {key: value for key, value in env.items() if key not in _PARENT_INTERPRETER_ENV}
|
|
187
196
|
|
|
188
197
|
|
|
198
|
+
_AF_INET = 2
|
|
199
|
+
_AF_INET6 = 23
|
|
200
|
+
_ERROR_INSUFFICIENT_BUFFER = 122
|
|
201
|
+
|
|
202
|
+
|
|
203
|
+
@dataclass(frozen=True)
|
|
204
|
+
class _TcpRowLayout:
|
|
205
|
+
"""Where the fields sit in one GetExtendedTcpTable row (TCP_TABLE_OWNER_PID_*)."""
|
|
206
|
+
|
|
207
|
+
size: int
|
|
208
|
+
port_offset: int
|
|
209
|
+
pid_offset: int
|
|
210
|
+
|
|
211
|
+
|
|
212
|
+
# MIB_TCPROW_OWNER_PID is six DWORDs; MIB_TCP6ROW_OWNER_PID puts a 16-byte address
|
|
213
|
+
# and a scope id before the port.
|
|
214
|
+
_TCP_ROW_LAYOUT = {_AF_INET: _TcpRowLayout(24, 8, 20), _AF_INET6: _TcpRowLayout(56, 20, 52)}
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
def _listener_pids_from_tcp_table(table: bytes, family: int, port: int) -> list[int]:
|
|
218
|
+
"""Owning PIDs of the rows bound to `port` in a raw GetExtendedTcpTable buffer."""
|
|
219
|
+
layout = _TCP_ROW_LAYOUT[family]
|
|
220
|
+
(count,) = struct.unpack_from("<I", table, 0)
|
|
221
|
+
pids: list[int] = []
|
|
222
|
+
for index in range(count):
|
|
223
|
+
row = 4 + index * layout.size
|
|
224
|
+
(raw_port,) = struct.unpack_from("<I", table, row + layout.port_offset)
|
|
225
|
+
# The port sits in network byte order in the DWORD's low 16 bits.
|
|
226
|
+
if socket.ntohs(raw_port & 0xFFFF) == port:
|
|
227
|
+
pids.append(struct.unpack_from("<I", table, row + layout.pid_offset)[0])
|
|
228
|
+
return pids
|
|
229
|
+
|
|
230
|
+
|
|
189
231
|
def _detach_popen_kwargs(log_handle: IO[bytes]) -> dict:
|
|
190
232
|
"""Cross-platform kwargs to spawn a subprocess detached from the caller.
|
|
191
233
|
|
|
@@ -239,6 +281,25 @@ def _signal_startup_process(process: subprocess.Popen, *, force: bool) -> None:
|
|
|
239
281
|
os.killpg(process.pid, signal.SIGKILL if force else signal.SIGTERM)
|
|
240
282
|
|
|
241
283
|
|
|
284
|
+
def _kill_process_tree(pid: int) -> None:
|
|
285
|
+
"""Force-kill a daemon we launched and everything it started.
|
|
286
|
+
|
|
287
|
+
Like `_signal_startup_process`: the pid may be the ``uvx`` launcher, and
|
|
288
|
+
killing only it would leave the real API holding the embedded Postgres.
|
|
289
|
+
"""
|
|
290
|
+
if platform.system() == "Windows":
|
|
291
|
+
DaemonEmbedManager._run_probe(["taskkill", "/T", "/F", "/PID", str(pid)])
|
|
292
|
+
return
|
|
293
|
+
try:
|
|
294
|
+
os.killpg(pid, signal.SIGKILL)
|
|
295
|
+
except OSError:
|
|
296
|
+
# Not a group leader after all: kill what there is.
|
|
297
|
+
try:
|
|
298
|
+
os.kill(pid, signal.SIGKILL)
|
|
299
|
+
except OSError:
|
|
300
|
+
pass
|
|
301
|
+
|
|
302
|
+
|
|
242
303
|
def _terminate_startup_process(process: subprocess.Popen) -> None:
|
|
243
304
|
"""Stop the daemon child owned by a failed startup attempt, escalating to kill."""
|
|
244
305
|
if process.poll() is not None:
|
|
@@ -584,8 +645,8 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
584
645
|
host that has plenty.
|
|
585
646
|
|
|
586
647
|
`errors="replace"` is the half that makes it deterministic; `encoding` alone
|
|
587
|
-
still raises.
|
|
588
|
-
|
|
648
|
+
still raises. A replaced character can land in the netstat state column too
|
|
649
|
+
("ABHÖREN"), which is why `_windows_listening_pids` does not read that column.
|
|
589
650
|
"""
|
|
590
651
|
try:
|
|
591
652
|
result = subprocess.run(
|
|
@@ -603,15 +664,25 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
603
664
|
|
|
604
665
|
@staticmethod
|
|
605
666
|
def _windows_listening_pids(port: int) -> list[int]:
|
|
606
|
-
"""PIDs listening on `port
|
|
667
|
+
"""PIDs listening on `port`: asked from the kernel, netstat only as a fallback.
|
|
668
|
+
|
|
669
|
+
netstat prints its state column in the display language ("ABHÖREN" on a
|
|
670
|
+
German Windows) and is a subprocess that a loaded machine can push past the
|
|
671
|
+
probe deadline. Either way the caller got no PID for a daemon that was up,
|
|
672
|
+
and `daemon stop` / the stale-daemon restart refused to act (#6602).
|
|
673
|
+
"""
|
|
674
|
+
native = DaemonEmbedManager._windows_native_listening_pids(port)
|
|
675
|
+
if native is not None:
|
|
676
|
+
return native
|
|
607
677
|
output = DaemonEmbedManager._run_probe(["netstat", "-ano", "-p", "TCP"])
|
|
608
678
|
if output is None:
|
|
609
679
|
return []
|
|
610
680
|
pids: list[int] = []
|
|
611
681
|
for line in output.splitlines():
|
|
612
682
|
fields = line.split()
|
|
613
|
-
# proto, local address, foreign address, state, pid
|
|
614
|
-
|
|
683
|
+
# proto, local address, foreign address, state, pid. The state is
|
|
684
|
+
# localized ("ABHÖREN"), so a listener is told by its foreign port 0.
|
|
685
|
+
if len(fields) < 5 or fields[0] != "TCP" or not fields[2].endswith(":0"):
|
|
615
686
|
continue
|
|
616
687
|
# Match on the local address' port only: the daemon binds 127.0.0.1
|
|
617
688
|
# but the UI can bind 0.0.0.0 or [::1] (issue #3527), and an
|
|
@@ -625,6 +696,48 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
625
696
|
continue
|
|
626
697
|
return pids
|
|
627
698
|
|
|
699
|
+
@staticmethod
|
|
700
|
+
def _windows_native_listening_pids(port: int) -> list[int] | None:
|
|
701
|
+
"""PIDs listening on `port` from GetExtendedTcpTable, or None if it can't be read.
|
|
702
|
+
|
|
703
|
+
The table carries the state as a number and the owning PID, so nothing here
|
|
704
|
+
depends on the display language, and it answers without a subprocess.
|
|
705
|
+
"""
|
|
706
|
+
import ctypes
|
|
707
|
+
from ctypes import wintypes
|
|
708
|
+
|
|
709
|
+
win_dll = getattr(ctypes, "WinDLL", None) # only defined on Windows
|
|
710
|
+
if win_dll is None:
|
|
711
|
+
return None
|
|
712
|
+
try:
|
|
713
|
+
get_table = win_dll("iphlpapi").GetExtendedTcpTable
|
|
714
|
+
except (OSError, AttributeError):
|
|
715
|
+
return None
|
|
716
|
+
get_table.restype = wintypes.DWORD
|
|
717
|
+
get_table.argtypes = [
|
|
718
|
+
ctypes.c_void_p,
|
|
719
|
+
ctypes.POINTER(wintypes.DWORD),
|
|
720
|
+
wintypes.BOOL,
|
|
721
|
+
wintypes.ULONG,
|
|
722
|
+
ctypes.c_int, # TCP_TABLE_CLASS
|
|
723
|
+
wintypes.ULONG,
|
|
724
|
+
]
|
|
725
|
+
tcp_table_owner_pid_listener = 3
|
|
726
|
+
pids: list[int] = []
|
|
727
|
+
for family in (_AF_INET, _AF_INET6):
|
|
728
|
+
size = wintypes.DWORD(0)
|
|
729
|
+
# The first call only reports the size the table needs. The table can
|
|
730
|
+
# grow between the two calls, so retry while it says the buffer is short.
|
|
731
|
+
for _attempt in range(3):
|
|
732
|
+
buffer = ctypes.create_string_buffer(max(size.value, 4))
|
|
733
|
+
status = get_table(buffer, ctypes.byref(size), False, family, tcp_table_owner_pid_listener, 0)
|
|
734
|
+
if status != _ERROR_INSUFFICIENT_BUFFER:
|
|
735
|
+
break
|
|
736
|
+
if status != 0:
|
|
737
|
+
return None
|
|
738
|
+
pids.extend(_listener_pids_from_tcp_table(buffer.raw, family, port))
|
|
739
|
+
return pids
|
|
740
|
+
|
|
628
741
|
@staticmethod
|
|
629
742
|
def _posix_listening_pids(port: int) -> list[int]:
|
|
630
743
|
"""PIDs listening on `port`, via lsof, falling back to ss.
|
|
@@ -819,6 +932,57 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
819
932
|
return True # Already gone
|
|
820
933
|
return False
|
|
821
934
|
|
|
935
|
+
@staticmethod
|
|
936
|
+
def _daemon_pid_file(paths: ProfilePaths) -> Path:
|
|
937
|
+
"""Where the start path records the PID of the daemon it launched."""
|
|
938
|
+
return paths.log.with_suffix(".pid")
|
|
939
|
+
|
|
940
|
+
@staticmethod
|
|
941
|
+
def _is_profile_daemon(pid: int, port: int) -> bool:
|
|
942
|
+
"""True while `pid` is alive and is the daemon launched for `port`.
|
|
943
|
+
|
|
944
|
+
The port check guards against PID reuse (another profile's daemon, or
|
|
945
|
+
anything else) and rules out an exited daemon left as a zombie, whose
|
|
946
|
+
command line no longer carries its arguments.
|
|
947
|
+
"""
|
|
948
|
+
cmdline = DaemonEmbedManager._process_command_line(pid)
|
|
949
|
+
if cmdline is None:
|
|
950
|
+
return False
|
|
951
|
+
lowered = cmdline.lower()
|
|
952
|
+
return any(marker in lowered for marker in DAEMON_PROCESS_MARKERS) and (
|
|
953
|
+
re.search(rf"--port[ =]{port}(\s|$)", lowered) is not None
|
|
954
|
+
)
|
|
955
|
+
|
|
956
|
+
def _wait_for_previous_daemon_exit(self, paths: ProfilePaths) -> None:
|
|
957
|
+
"""Block until the daemon this profile last launched has exited.
|
|
958
|
+
|
|
959
|
+
Callers reach this with the port already free, so a recorded daemon that
|
|
960
|
+
is still alive is one that is shutting down and has yet to stop the
|
|
961
|
+
embedded Postgres a replacement would share. If it outlives the wait it
|
|
962
|
+
is hung; it is killed so the start can proceed.
|
|
963
|
+
"""
|
|
964
|
+
pid_file = self._daemon_pid_file(paths)
|
|
965
|
+
try:
|
|
966
|
+
pid = int(pid_file.read_text().strip())
|
|
967
|
+
except (OSError, ValueError):
|
|
968
|
+
return
|
|
969
|
+
|
|
970
|
+
def _exited_within(seconds: float) -> bool:
|
|
971
|
+
deadline = time.monotonic() + seconds
|
|
972
|
+
while self._is_profile_daemon(pid, paths.port):
|
|
973
|
+
if time.monotonic() >= deadline:
|
|
974
|
+
return False
|
|
975
|
+
time.sleep(0.2)
|
|
976
|
+
return True
|
|
977
|
+
|
|
978
|
+
if self._is_profile_daemon(pid, paths.port):
|
|
979
|
+
logger.info(f"Waiting for the previous daemon (PID {pid}) to finish shutting down")
|
|
980
|
+
if not _exited_within(PREVIOUS_DAEMON_EXIT_TIMEOUT):
|
|
981
|
+
logger.warning(f"Previous daemon (PID {pid}) did not exit in time; killing it")
|
|
982
|
+
_kill_process_tree(pid)
|
|
983
|
+
_exited_within(DAEMON_STARTUP_TERMINATE_TIMEOUT)
|
|
984
|
+
pid_file.unlink(missing_ok=True)
|
|
985
|
+
|
|
822
986
|
@staticmethod
|
|
823
987
|
def _port_health_ok(port: int) -> bool:
|
|
824
988
|
"""Return True when the listener on port responds like initialized Hindsight."""
|
|
@@ -964,6 +1128,10 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
964
1128
|
self._register_profile(profile, port, config)
|
|
965
1129
|
return True
|
|
966
1130
|
|
|
1131
|
+
# A free port does not mean the previous daemon is gone (see
|
|
1132
|
+
# PREVIOUS_DAEMON_EXIT_TIMEOUT).
|
|
1133
|
+
self._wait_for_previous_daemon_exit(paths)
|
|
1134
|
+
|
|
967
1135
|
# Load profile's .env file and merge with provided config
|
|
968
1136
|
# This fixes issue #305 where profile env vars were ignored
|
|
969
1137
|
profile_config = self._profile_manager.load_profile_config(profile)
|
|
@@ -1053,6 +1221,10 @@ class DaemonEmbedManager(EmbedManager):
|
|
|
1053
1221
|
# can close its handle as soon as Popen returns.
|
|
1054
1222
|
with open(daemon_log, "ab") as daemon_log_handle:
|
|
1055
1223
|
daemon_process = subprocess.Popen(cmd, env=env, **_detach_popen_kwargs(daemon_log_handle))
|
|
1224
|
+
try:
|
|
1225
|
+
self._daemon_pid_file(paths).write_text(str(daemon_process.pid))
|
|
1226
|
+
except OSError:
|
|
1227
|
+
logger.debug("Could not record daemon PID for %s", profile_label, exc_info=True)
|
|
1056
1228
|
|
|
1057
1229
|
# Wait for daemon to be ready with rich UI
|
|
1058
1230
|
start_time = time.time()
|
|
@@ -68,6 +68,12 @@ HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
|
|
68
68
|
# pointed at a local server that can't constrain output: the schema then goes in
|
|
69
69
|
# the prompt only.
|
|
70
70
|
# HINDSIGHT_API_LLM_OPENAI_COMPATIBLE_JSON_MODE=
|
|
71
|
+
# On that path, have the backend enforce the response schema itself instead of
|
|
72
|
+
# json_object (any JSON object). For vLLM / llama.cpp; set true for Gemma 4 on vLLM.
|
|
73
|
+
# HINDSIGHT_API_LLM_OPENAI_COMPATIBLE_ENFORCE_SCHEMA=false
|
|
74
|
+
# JSON_MODE for retain's fact extraction only (unset = the global value). Set false
|
|
75
|
+
# for Gemma 4 on vLLM: constrained, it answers retain with an empty facts list.
|
|
76
|
+
# HINDSIGHT_API_RETAIN_LLM_OPENAI_COMPATIBLE_JSON_MODE=
|
|
71
77
|
|
|
72
78
|
# Constrain retain's occurred_start/occurred_end to an ISO timestamp with a JSON
|
|
73
79
|
# Schema pattern. Stops grammar-constrained models reasoning inside the timestamp
|
|
@@ -105,6 +111,9 @@ HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
|
|
105
111
|
# outright -- e.g. Bedrock Claude in ap-southeast-2 ("Extra inputs are not permitted");
|
|
106
112
|
# the same model in us-east-1 accepts response_format and needs nothing here.
|
|
107
113
|
# HINDSIGHT_API_LLM_STRUCTURED_OUTPUT_IN_PROMPT=false
|
|
114
|
+
# LiteLLM/Bedrock: mark the prompt for the provider's prompt cache. Unset = on for models
|
|
115
|
+
# named "claude"; set true for a Claude model behind an ARN or alias, false to turn it off.
|
|
116
|
+
# HINDSIGHT_API_LLM_LITELLM_PROMPT_CACHE=
|
|
108
117
|
|
|
109
118
|
# Transport-level output cap for reflect's final synthesis call. Unset = uncapped:
|
|
110
119
|
# the model runs to a natural stop and the reflect/mental-model max_tokens governs
|
|
@@ -154,7 +163,7 @@ HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
|
|
154
163
|
# Example: z.ai configuration (Zhipu GLM series, https://z.ai)
|
|
155
164
|
# HINDSIGHT_API_LLM_PROVIDER=zai
|
|
156
165
|
# HINDSIGHT_API_LLM_API_KEY=your-zai-api-key
|
|
157
|
-
# HINDSIGHT_API_LLM_MODEL=glm-
|
|
166
|
+
# HINDSIGHT_API_LLM_MODEL=glm-5.3-flash # or glm-4.5-flash for the free tier
|
|
158
167
|
|
|
159
168
|
# Example: Atlas Cloud configuration (OpenAI-compatible, https://www.atlascloud.ai)
|
|
160
169
|
# HINDSIGHT_API_LLM_PROVIDER=atlas
|
|
@@ -310,6 +319,7 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
310
319
|
# HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
|
|
311
320
|
# HINDSIGHT_API_READ_DATABASE_URL= # Optional read-replica URL. When set, recall queries (semantic, BM25, graph, temporal) flow through a separate pool against this URL, offloading the primary. Typically points to a read-only endpoint (CNPG's <cluster>-ro service or Aurora reader endpoint).
|
|
312
321
|
# HINDSIGHT_API_MIGRATION_DATABASE_URL= # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
|
|
322
|
+
# HINDSIGHT_API_DATABASE_CONNECT_HOOK= # module.path:function — async callable with asyncpg.connect's signature, used to open every new database connection (e.g. to fetch rotated credentials). Unset = plain asyncpg.connect.
|
|
313
323
|
# HINDSIGHT_API_DATABASE_SCHEMA=public # PostgreSQL schema name (default: public)
|
|
314
324
|
# HINDSIGHT_API_DB_POOL_SLOW_ACQUIRE_THRESHOLD_SECONDS=0.05 # Warn when a pool connection takes longer than this to acquire; 0 disables the warning
|
|
315
325
|
# HINDSIGHT_API_DB_MAX_PARALLEL_WORKERS_PER_GATHER= # Optional cap on Postgres planner parallelism for this process's pool connections. Unset leaves the server default; 0 makes background/bulk queries run serially (useful on worker processes sharing a primary with latency-sensitive traffic).
|
|
@@ -330,6 +340,7 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
330
340
|
# not. On deployments with thousands of tenants these intervals are the knob to raise.
|
|
331
341
|
# HINDSIGHT_API_RETENTION_SWEEP_INTERVAL_SECONDS=3600 # How often expired audit_log / llm_requests rows are deleted across all tenant schemas. Retention is counted in days, so this only sets how promptly they disappear; 0 disables the sweeps.
|
|
332
342
|
# HINDSIGHT_API_OPERATION_CLEANUP_INTERVAL_SECONDS=900 # How often expired terminal operation rows are pruned; with the batch size above this sets the drain rate for a backlog. 0 disables the job.
|
|
343
|
+
# HINDSIGHT_API_ATTACHMENT_SWEEP_INTERVAL_SECONDS=86400 # How often images and files of failed retains (attachments a day or more old that their document does not reference) are deleted. 0 disables the job.
|
|
333
344
|
# HINDSIGHT_API_MAINTENANCE_START_JITTER_SECONDS=60 # Upper bound on a random delay before a process runs its FIRST maintenance tick. Every job is due on that tick, so without an offset a fleet started together runs every sweep in every process at once. 0 disables the jitter.
|
|
334
345
|
|
|
335
346
|
# Vector Extension (Optional - uses pgvector by default)
|
|
@@ -445,6 +456,13 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
445
456
|
# not bigger requests, is what saturates an embedding service (applies to every remote
|
|
446
457
|
# provider; the in-process local/onnx backends are unaffected).
|
|
447
458
|
# HINDSIGHT_API_EMBEDDINGS_MAX_CONCURRENT_REQUESTS=8
|
|
459
|
+
# Separate query / passage embeddings (optional). Each side starts from the embeddings
|
|
460
|
+
# config above; any HINDSIGHT_API_QUERY_EMBEDDINGS_<X> / HINDSIGHT_API_PASSAGE_EMBEDDINGS_<X>
|
|
461
|
+
# replaces HINDSIGHT_API_EMBEDDINGS_<X> for that side only (recall queries vs stored text).
|
|
462
|
+
# Both sides must share a vector space; startup fails if their dimensions differ.
|
|
463
|
+
# HINDSIGHT_API_QUERY_EMBEDDINGS_OPENAI_BASE_URL=http://tei-small:8080/v1
|
|
464
|
+
# HINDSIGHT_API_QUERY_EMBEDDINGS_PROVIDER=onnx
|
|
465
|
+
# HINDSIGHT_API_QUERY_EMBEDDINGS_ONNX_MODEL_ID=Qdrant/constella-nano
|
|
448
466
|
# For TEI provider:
|
|
449
467
|
# HINDSIGHT_API_EMBEDDINGS_TEI_URL=http://localhost:8080
|
|
450
468
|
# Max texts per TEI /embed request, and the unit the client fans out over (see the
|
|
@@ -537,32 +555,44 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
537
555
|
# Explicit refreshes always run immediately.
|
|
538
556
|
# HINDSIGHT_API_MENTAL_MODEL_MIN_REFRESH_INTERVAL_SECONDS=0
|
|
539
557
|
|
|
540
|
-
# Default trigger for newly created
|
|
558
|
+
# Default trigger for newly created pages, as a JSON object merged over the
|
|
541
559
|
# built-in default ({"mode": "delta", "fact_types": ["observation"],
|
|
542
560
|
# "exclude_mental_models": true, "refresh_after_consolidation": true}). A trigger sent
|
|
543
561
|
# with the create request still wins. Hierarchical (overridable per bank).
|
|
544
|
-
#
|
|
562
|
+
# The old name HINDSIGHT_API_KNOWLEDGE_PAGE_DEFAULT_TRIGGER still works (deprecated).
|
|
563
|
+
# HINDSIGHT_API_PAGE_DEFAULT_TRIGGER={"refresh_cron": "0 * * * *"}
|
|
545
564
|
|
|
546
|
-
# Default reflect options, as a JSON object applied whenever a reflect request
|
|
547
|
-
#
|
|
565
|
+
# Default reflect options, as a JSON object applied whenever a reflect request leaves the
|
|
566
|
+
# option unset. Fields: reflect_search_observations_max_tokens
|
|
548
567
|
# (budget for the search_observations tool, default 5000),
|
|
549
568
|
# reflect_search_observations_include_entities (attach resolved entity names, default true; they can be
|
|
550
569
|
# over half the tool payload) and reflect_search_mental_models_max_tokens (budget for the mental models
|
|
551
570
|
# search_mental_models returns in full, default 16000; the best match is always in full, the others that
|
|
552
571
|
# do not fit come back as a snippet). Hierarchical (overridable per bank). Mental-model refreshes do not
|
|
553
|
-
# read this — they take these settings from their own trigger (
|
|
572
|
+
# read this — they take these settings from their own trigger (page_default_trigger).
|
|
554
573
|
# A refresh always gets only the best mental model in full and the others as snippets.
|
|
555
574
|
# HINDSIGHT_API_REFLECT_DEFAULT_OPTIONS={"reflect_search_observations_max_tokens": 3000}
|
|
556
575
|
# With a decision model set up for reflect (HINDSIGHT_API_DECISION_MODEL_* below), reflect is
|
|
557
576
|
# decision-orchestrated: the layers run in parallel rounds and the decision model prunes the
|
|
558
577
|
# evidence and judges when it is enough, in place of one LLM turn per layer. Fewer LLM calls.
|
|
559
|
-
# Set false to keep agent turns, for reflect and for
|
|
578
|
+
# Set false to keep agent turns, for reflect and for page refreshes. Hierarchical (per bank).
|
|
560
579
|
# HINDSIGHT_API_REFLECT_USE_DECISION_MODEL=true
|
|
561
580
|
# HINDSIGHT_API_MENTAL_MODEL_REFRESH_USE_DECISION_MODEL=true
|
|
562
581
|
# Delta page refresh asks the decision model which new memories could change the page, instead of reflect
|
|
563
582
|
# HINDSIGHT_API_MENTAL_MODEL_DELTA_USE_DECISION_MODEL=true
|
|
564
583
|
# Above this many new memories a delta refresh searches instead of asking the decision model about each (server-level)
|
|
565
584
|
# HINDSIGHT_API_MENTAL_MODEL_DELTA_GATE_MAX_MEMORIES=200
|
|
585
|
+
# How an interactive reflect searches the knowledge banks a memory bank links.
|
|
586
|
+
# Media types the search may return (comma-separated: document,image,audio,video,text; unset = all; per bank)
|
|
587
|
+
# HINDSIGHT_API_REFLECT_KB_SEARCH_MEDIA_TYPES=
|
|
588
|
+
# Show the model the picture behind an image or PDF-page hit, when it is known to read images (server-level)
|
|
589
|
+
# HINDSIGHT_API_REFLECT_KB_SEARCH_IMAGES=true
|
|
590
|
+
# The most pictures one search hands the model (0 = none; server-level)
|
|
591
|
+
# HINDSIGHT_API_REFLECT_KB_SEARCH_MAX_IMAGES=4
|
|
592
|
+
# How deep the search ranks before its token budget picks: low, mid or high (server-level)
|
|
593
|
+
# HINDSIGHT_API_REFLECT_KB_SEARCH_BUDGET=mid
|
|
594
|
+
# Filter the search by the reflect's tags/tags_match/tag_groups, as its memories are (per bank)
|
|
595
|
+
# HINDSIGHT_API_REFLECT_KB_SEARCH_APPLY_TAGS=true
|
|
566
596
|
|
|
567
597
|
# Admission control. The *_MAX_CONCURRENT caps above bound how much runs at once and
|
|
568
598
|
# let an unbounded queue form behind them; these bound how long a request may WAIT
|
|
@@ -590,7 +620,7 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
590
620
|
# Keyword (BM25) arm. false leaves pure vector search: the arm is left out of the
|
|
591
621
|
# query entirely rather than filtered to nothing, so its SQL, its query tokenization
|
|
592
622
|
# and its pg_stats term-selection lookup are all skipped. Also drops the keyword arm
|
|
593
|
-
# from
|
|
623
|
+
# from page search:
|
|
594
624
|
# HINDSIGHT_API_ENABLE_TEXT_SEARCH=true
|
|
595
625
|
# Temporal retrieval arm, plus the date-aware query analysis that feeds it:
|
|
596
626
|
# HINDSIGHT_API_ENABLE_TEMPORAL_RETRIEVAL=true
|
|
@@ -683,7 +713,12 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
683
713
|
# A typed-judgment model operations use for quick decisions instead of an LLM call.
|
|
684
714
|
# Set up once; shared by every operation that supports one, unless it overrides or
|
|
685
715
|
# opts out. Replaces the deprecated typesafe reranker above.
|
|
686
|
-
#
|
|
716
|
+
# Reflect and page refresh support only hindsight and typesafe for now; recall
|
|
717
|
+
# and consolidation support all of them. See https://hindsight.vectorize.io/developer/decision-models
|
|
718
|
+
# hindsight is Hindsight Inference (https://inference.hindsight.vectorize.io, hs-decide-1-flash): run
|
|
719
|
+
# `hindsight-admin cloud login` or set the API key (hsk_...). Recall keeps its reranker
|
|
720
|
+
# unless HINDSIGHT_API_RECALL_DECISION_MODEL_PROVIDER=hindsight.
|
|
721
|
+
# HINDSIGHT_API_DECISION_MODEL_PROVIDER=typesafe # none (default) | hindsight | typesafe | cloudflare | openai
|
|
687
722
|
# HINDSIGHT_API_DECISION_MODEL_API_KEY=your-api-key
|
|
688
723
|
# Model and URL default per provider: typesafe jev-latest @ https://api.typesafe.ai,
|
|
689
724
|
# openai gpt-6-luna @ https://api.openai.com, cloudflare clef @ (required) your account URL:
|
|
@@ -692,10 +727,15 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
692
727
|
# HINDSIGHT_API_DECISION_MODEL_TIMEOUT=60
|
|
693
728
|
# HINDSIGHT_API_DECISION_MODEL_MAX_CONCURRENT=24
|
|
694
729
|
# Most tokens one request may carry, counted on the whole body; larger asks are split
|
|
695
|
-
# across requests. Follows the served model's context window (32k for jev-latest), less a margin
|
|
730
|
+
# across requests. Follows the served model's context window (32k for jev-latest), less a margin.
|
|
731
|
+
# Not needed for hindsight on Hindsight Inference, which fits each request itself (no token split);
|
|
732
|
+
# hindsight on a custom BASE_URL (your own vLLM) splits at 54000:
|
|
696
733
|
# HINDSIGHT_API_DECISION_MODEL_MAX_REQUEST_TOKENS=26000
|
|
734
|
+
# Show the decision model pictures (reflect's knowledge-bank pictures when it asks "is this enough?").
|
|
735
|
+
# System One providers only, on a model that reads images (Jev 1.13+); a preview, so off by default:
|
|
736
|
+
# HINDSIGHT_API_DECISION_MODEL_VISION=false
|
|
697
737
|
# The vocabulary that ceiling and recall's caps are counted in: the served model's own (jev for typesafe, else
|
|
698
|
-
# HINDSIGHT_API_TOKENIZER_ENCODING). Set it when the endpoint serves another model, e.g. qwen3.5:
|
|
738
|
+
# HINDSIGHT_API_TOKENIZER_ENCODING). Set it when the endpoint serves another model, e.g. qwen3.5 or hs-decide-1-flash:
|
|
699
739
|
# HINDSIGHT_API_DECISION_MODEL_TOKENIZER=jev
|
|
700
740
|
# Recall: when it has a decision model, it replaces the reranker above. Each candidate is
|
|
701
741
|
# judged on whether it helps answer the question, told the question's date (else today),
|
|
@@ -705,6 +745,8 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
705
745
|
# request that omits `prune` / `dedup`, per bank; both off by default:
|
|
706
746
|
# HINDSIGHT_API_RECALL_DECISION_MODEL_PRUNE=false
|
|
707
747
|
# HINDSIGHT_API_RECALL_DECISION_MODEL_DEDUP=false
|
|
748
|
+
# Rank a recall the decision model fails with the HINDSIGHT_API_RERANKER_* reranker instead of failing it
|
|
749
|
+
# HINDSIGHT_API_RECALL_DECISION_MODEL_FALLBACK=false
|
|
708
750
|
# Per-operation overrides (fall back to the global value while on the global provider;
|
|
709
751
|
# PROVIDER=none opts out):
|
|
710
752
|
# HINDSIGHT_API_{RECALL,REFLECT,CONSOLIDATION}_DECISION_MODEL_{PROVIDER,API_KEY,MODEL,BASE_URL}
|
|
@@ -765,6 +807,66 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
765
807
|
# HINDSIGHT_API_LLM_TRACE_MAX_CHARS=50000 # truncate stored input/output
|
|
766
808
|
# HINDSIGHT_API_LLM_TRACE_DELETE_WITH_DOCUMENT=true # deleting a document deletes its trace rows (default: false)
|
|
767
809
|
|
|
810
|
+
# -----------------------------------------------------------------------------
|
|
811
|
+
# Knowledge banks (Optional; off by default while the feature is built)
|
|
812
|
+
# -----------------------------------------------------------------------------
|
|
813
|
+
|
|
814
|
+
# Turn knowledge banks on (documents a memory bank links, read by reflect)
|
|
815
|
+
# HINDSIGHT_API_ENABLE_KB=false
|
|
816
|
+
# The one embedder for every knowledge bank: local, gemini, vertex, voyage, cohere, openai or text
|
|
817
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_PROVIDER=local
|
|
818
|
+
# Model (unset: the provider's own), key, and /v1 URL (openai: required)
|
|
819
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_MODEL=
|
|
820
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_API_KEY=
|
|
821
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_BASE_URL=
|
|
822
|
+
# Vector size for a model that can shorten them (unset: the model's own)
|
|
823
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_DIMENSIONS=
|
|
824
|
+
# openai provider: what the remote model embeds
|
|
825
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_MODALITIES=text,image
|
|
826
|
+
# local provider: torch device (cpu, cuda, mps; unset picks one)
|
|
827
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_DEVICE=
|
|
828
|
+
# Passages per embedder request, and remote requests in flight per process (0 = no cap)
|
|
829
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_BATCH_SIZE=32
|
|
830
|
+
# HINDSIGHT_API_KB_EMBEDDINGS_MAX_IN_FLIGHT=16
|
|
831
|
+
# text provider: its own embeddings setup, any HINDSIGHT_API_EMBEDDINGS_* with this prefix
|
|
832
|
+
# HINDSIGHT_API_KB_TEXT_EMBEDDINGS_PROVIDER=
|
|
833
|
+
# Knowledge-bank search's reranker: any Cohere-style /rerank endpoint (full URL; unset: no rerank).
|
|
834
|
+
# The best hits go in one request, as many as MAX_TOKENS holds (a low search budget: at most 26000),
|
|
835
|
+
# counted with TOKENIZER (unset: HINDSIGHT_API_TOKENIZER_ENCODING); videos keep their search place.
|
|
836
|
+
# IMAGES sends image and PDF page hits as pictures with their text (false: a text-only reranker).
|
|
837
|
+
# MAX_TOKENS is sized for a long-window reranker: lower it to your reranker's window.
|
|
838
|
+
# HINDSIGHT_API_KB_RERANKER_URL=
|
|
839
|
+
# HINDSIGHT_API_KB_RERANKER_MODEL=
|
|
840
|
+
# HINDSIGHT_API_KB_RERANKER_API_KEY=
|
|
841
|
+
# HINDSIGHT_API_KB_RERANKER_MAX_TOKENS=120000
|
|
842
|
+
# HINDSIGHT_API_KB_RERANKER_TOKENIZER=
|
|
843
|
+
# HINDSIGHT_API_KB_RERANKER_IMAGES=true
|
|
844
|
+
# HINDSIGHT_API_KB_RERANKER_QUERY_IMAGES=false
|
|
845
|
+
# Ingest limits (the per-bank ones can be narrowed per knowledge bank)
|
|
846
|
+
# HINDSIGHT_API_KB_DISABLED_MEDIA_TYPES=
|
|
847
|
+
# HINDSIGHT_API_KB_MAX_FILE_MB=500
|
|
848
|
+
# HINDSIGHT_API_KB_MAX_MEDIA_SECONDS=7200
|
|
849
|
+
# HINDSIGHT_API_KB_MAX_PDF_PAGES=2000
|
|
850
|
+
# How files are split into passages (per bank)
|
|
851
|
+
# HINDSIGHT_API_KB_TEXT_PASSAGE_TOKENS=256
|
|
852
|
+
# HINDSIGHT_API_KB_IMAGE_TOKENS=280
|
|
853
|
+
# HINDSIGHT_API_KB_PDF_PAGE_PIXELS=1536
|
|
854
|
+
# HINDSIGHT_API_KB_PDF_BANDS=false
|
|
855
|
+
# HINDSIGHT_API_KB_AUDIO_WINDOW_SECONDS=60
|
|
856
|
+
# HINDSIGHT_API_KB_VIDEO_WINDOW_SECONDS=30
|
|
857
|
+
# HINDSIGHT_API_KB_MEDIA_OVERLAP_SECONDS=5
|
|
858
|
+
# Search weights: PDF text vs page image, and keyword matches
|
|
859
|
+
# HINDSIGHT_API_KB_PDF_TEXT_WEIGHT=0.7
|
|
860
|
+
# HINDSIGHT_API_KB_KEYWORD_WEIGHT=0.2
|
|
861
|
+
# Async ingest: worker task size in bytes, and memory one batch may hold (0 = off)
|
|
862
|
+
# HINDSIGHT_API_KB_INGEST_CHILD_BYTES=1048576
|
|
863
|
+
# HINDSIGHT_API_KB_INGEST_MEMORY_BUDGET_MB=128
|
|
864
|
+
# Named ingest strategies (JSON) and the default one
|
|
865
|
+
# HINDSIGHT_API_KB_INGEST_STRATEGIES=
|
|
866
|
+
# HINDSIGHT_API_KB_INGEST_DEFAULT_STRATEGY=
|
|
867
|
+
# Where documents and passages live (module.path:ClassName; unset: PostgreSQL)
|
|
868
|
+
# HINDSIGHT_API_KB_STORE_EXTENSION=
|
|
869
|
+
|
|
768
870
|
# -----------------------------------------------------------------------------
|
|
769
871
|
# Operations (Optional)
|
|
770
872
|
# -----------------------------------------------------------------------------
|
|
@@ -780,6 +882,10 @@ HINDSIGHT_API_LOG_LEVEL=info
|
|
|
780
882
|
# Deferrals do not count against HINDSIGHT_API_WORKER_MAX_RETRIES.
|
|
781
883
|
# HINDSIGHT_API_BACKPRESSURE_DEFER_SECONDS=120
|
|
782
884
|
|
|
885
|
+
# Seconds shutdown waits for in-flight tasks, counted from SIGTERM, before cancelling
|
|
886
|
+
# them and handing them back to 'pending'. Keep it ~10s under the platform's grace period.
|
|
887
|
+
# HINDSIGHT_API_WORKER_SHUTDOWN_DRAIN_SECONDS=20
|
|
888
|
+
|
|
783
889
|
# Legacy MCP bearer token, checked before the TenantExtension's own auth.
|
|
784
890
|
# HINDSIGHT_API_MCP_AUTH_TOKEN=your-mcp-token
|
|
785
891
|
|
|
@@ -487,6 +487,8 @@ class ProfileManager:
|
|
|
487
487
|
stale_log.unlink(missing_ok=True)
|
|
488
488
|
except OSError as exc:
|
|
489
489
|
logger.warning("Could not remove log %s while deleting profile '%s': %s", stale_log, name, exc)
|
|
490
|
+
# The PID the start path recorded for the profile's daemon.
|
|
491
|
+
log_path.with_suffix(".pid").unlink(missing_ok=True)
|
|
490
492
|
|
|
491
493
|
# Update metadata
|
|
492
494
|
metadata = self._load_metadata()
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
"""Tests for daemon_client module."""
|
|
2
2
|
|
|
3
|
+
import os
|
|
3
4
|
import sys
|
|
5
|
+
import time
|
|
4
6
|
from unittest.mock import MagicMock, Mock, patch
|
|
5
7
|
|
|
6
8
|
import pytest
|
|
@@ -401,6 +403,9 @@ class TestStartDaemonSerialization:
|
|
|
401
403
|
|
|
402
404
|
with (
|
|
403
405
|
patch.object(DaemonEmbedManager, "_clear_port", side_effect=lambda _: calls.append("clear") or True),
|
|
406
|
+
patch.object(
|
|
407
|
+
DaemonEmbedManager, "_wait_for_previous_daemon_exit", side_effect=lambda _: calls.append("wait")
|
|
408
|
+
),
|
|
404
409
|
patch.object(DaemonEmbedManager, "is_running", side_effect=[False, True, True]),
|
|
405
410
|
patch.object(
|
|
406
411
|
DaemonEmbedManager, "_rotate_daemon_log", side_effect=lambda *_args, **_kwargs: calls.append("rotate")
|
|
@@ -416,7 +421,106 @@ class TestStartDaemonSerialization:
|
|
|
416
421
|
):
|
|
417
422
|
assert manager._start_daemon_locked({}, "codex", paths) is True
|
|
418
423
|
|
|
419
|
-
assert calls == ["clear", "rotate", "spawn"]
|
|
424
|
+
assert calls == ["clear", "wait", "rotate", "spawn"]
|
|
425
|
+
|
|
426
|
+
def test_start_waits_for_the_previous_daemon_to_exit(self, tmp_path):
|
|
427
|
+
"""A free port is not a stopped daemon (#6540).
|
|
428
|
+
|
|
429
|
+
A stopping daemon frees its port first and stops the profile's embedded
|
|
430
|
+
Postgres last. A replacement spawned in between ran its migrations against
|
|
431
|
+
a Postgres that then went away. The stand-in below is "still shutting down"
|
|
432
|
+
for a moment after the port is free; the wait must outlast it.
|
|
433
|
+
"""
|
|
434
|
+
import subprocess
|
|
435
|
+
|
|
436
|
+
from hindsight_embed.profile_manager import ProfilePaths
|
|
437
|
+
|
|
438
|
+
paths = ProfilePaths(
|
|
439
|
+
config=tmp_path / "embed", lock=tmp_path / "daemon.lock", log=tmp_path / "daemon.log", port=9600
|
|
440
|
+
)
|
|
441
|
+
exiting = subprocess.Popen(
|
|
442
|
+
[sys.executable, "-c", "import time; time.sleep(1.5)", "hindsight-api", "--port", "9600"]
|
|
443
|
+
)
|
|
444
|
+
try:
|
|
445
|
+
DaemonEmbedManager._daemon_pid_file(paths).write_text(str(exiting.pid))
|
|
446
|
+
DaemonEmbedManager()._wait_for_previous_daemon_exit(paths)
|
|
447
|
+
# The process is our child, so it lingered as a zombie until this
|
|
448
|
+
# poll() reaped it; the wait must have read that zombie as gone.
|
|
449
|
+
assert exiting.poll() is not None, "the wait returned while the previous daemon was alive"
|
|
450
|
+
assert not DaemonEmbedManager._daemon_pid_file(paths).exists()
|
|
451
|
+
finally:
|
|
452
|
+
exiting.kill()
|
|
453
|
+
exiting.wait()
|
|
454
|
+
|
|
455
|
+
def test_previous_daemon_pid_reused_by_another_process_is_ignored(self, tmp_path, monkeypatch):
|
|
456
|
+
"""The recorded PID only counts while it is still a daemon on this profile's port."""
|
|
457
|
+
from hindsight_embed.profile_manager import ProfilePaths
|
|
458
|
+
|
|
459
|
+
paths = ProfilePaths(
|
|
460
|
+
config=tmp_path / "embed", lock=tmp_path / "daemon.lock", log=tmp_path / "daemon.log", port=9600
|
|
461
|
+
)
|
|
462
|
+
DaemonEmbedManager._daemon_pid_file(paths).write_text("4321")
|
|
463
|
+
monkeypatch.setattr(
|
|
464
|
+
DaemonEmbedManager,
|
|
465
|
+
"_process_command_line",
|
|
466
|
+
# 96000 starts with 9600: a prefix match would take it for ours.
|
|
467
|
+
staticmethod(lambda _pid: "python -m hindsight_api.main --idle-timeout 0 --port 96000"),
|
|
468
|
+
)
|
|
469
|
+
killed: list[int] = []
|
|
470
|
+
monkeypatch.setattr("hindsight_embed.daemon_embed_manager._kill_process_tree", killed.append)
|
|
471
|
+
|
|
472
|
+
DaemonEmbedManager()._wait_for_previous_daemon_exit(paths)
|
|
473
|
+
|
|
474
|
+
assert killed == []
|
|
475
|
+
|
|
476
|
+
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX process groups")
|
|
477
|
+
def test_a_hung_previous_daemon_is_killed_with_what_it_started(self, tmp_path, monkeypatch):
|
|
478
|
+
"""Past the wait, the kill reaches the whole group: the recorded PID may be
|
|
479
|
+
the uvx launcher, and the real API under it holds the embedded Postgres."""
|
|
480
|
+
import subprocess
|
|
481
|
+
|
|
482
|
+
from hindsight_embed.profile_manager import ProfilePaths
|
|
483
|
+
|
|
484
|
+
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.PREVIOUS_DAEMON_EXIT_TIMEOUT", 0.2)
|
|
485
|
+
paths = ProfilePaths(
|
|
486
|
+
config=tmp_path / "embed", lock=tmp_path / "daemon.lock", log=tmp_path / "daemon.log", port=9600
|
|
487
|
+
)
|
|
488
|
+
grandchild_pid_file = tmp_path / "grandchild.pid"
|
|
489
|
+
launcher = subprocess.Popen(
|
|
490
|
+
[
|
|
491
|
+
sys.executable,
|
|
492
|
+
"-c",
|
|
493
|
+
"import subprocess, sys, time; "
|
|
494
|
+
"p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)']); "
|
|
495
|
+
f"open({str(grandchild_pid_file)!r}, 'w').write(str(p.pid)); time.sleep(60)",
|
|
496
|
+
"hindsight-api",
|
|
497
|
+
"--port",
|
|
498
|
+
"9600",
|
|
499
|
+
],
|
|
500
|
+
start_new_session=True,
|
|
501
|
+
)
|
|
502
|
+
try:
|
|
503
|
+
deadline = time.monotonic() + 10
|
|
504
|
+
while not grandchild_pid_file.exists() or not grandchild_pid_file.read_text():
|
|
505
|
+
assert time.monotonic() < deadline, "launcher never started its child"
|
|
506
|
+
time.sleep(0.05)
|
|
507
|
+
grandchild = int(grandchild_pid_file.read_text())
|
|
508
|
+
DaemonEmbedManager._daemon_pid_file(paths).write_text(str(launcher.pid))
|
|
509
|
+
|
|
510
|
+
DaemonEmbedManager()._wait_for_previous_daemon_exit(paths)
|
|
511
|
+
|
|
512
|
+
assert launcher.wait(timeout=5) is not None
|
|
513
|
+
deadline = time.monotonic() + 5
|
|
514
|
+
while True:
|
|
515
|
+
try:
|
|
516
|
+
os.kill(grandchild, 0)
|
|
517
|
+
except OSError:
|
|
518
|
+
break
|
|
519
|
+
assert time.monotonic() < deadline, "the process the launcher started survived the kill"
|
|
520
|
+
time.sleep(0.05)
|
|
521
|
+
finally:
|
|
522
|
+
launcher.kill()
|
|
523
|
+
launcher.wait()
|
|
420
524
|
|
|
421
525
|
|
|
422
526
|
class TestDaemonLogRotation:
|
|
@@ -656,6 +760,7 @@ class TestListeningPidDiscovery:
|
|
|
656
760
|
" TCP 127.0.0.1:29177 0.0.0.0:0 LISTENING 9999\n"
|
|
657
761
|
)
|
|
658
762
|
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.platform.system", lambda: "Windows")
|
|
763
|
+
_force_netstat(monkeypatch)
|
|
659
764
|
monkeypatch.setattr(
|
|
660
765
|
"hindsight_embed.daemon_embed_manager.subprocess.run",
|
|
661
766
|
lambda cmd, **kwargs: Mock(returncode=0, stdout=netstat),
|
|
@@ -663,6 +768,51 @@ class TestListeningPidDiscovery:
|
|
|
663
768
|
|
|
664
769
|
assert DaemonEmbedManager._listening_pids(19177) == [4321, 4322]
|
|
665
770
|
|
|
771
|
+
def test_windows_netstat_in_another_language_still_finds_the_listener(self, monkeypatch):
|
|
772
|
+
"""A German netstat says ABHÖREN, not LISTENING; the listener must still be found (#6602)."""
|
|
773
|
+
netstat = (
|
|
774
|
+
" TCP 127.0.0.1:9177 0.0.0.0:0 ABHÖREN 4321\n"
|
|
775
|
+
" TCP 127.0.0.1:9177 127.0.0.1:50000 HERGESTELLT 7777\n"
|
|
776
|
+
)
|
|
777
|
+
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.platform.system", lambda: "Windows")
|
|
778
|
+
_force_netstat(monkeypatch)
|
|
779
|
+
monkeypatch.setattr(
|
|
780
|
+
"hindsight_embed.daemon_embed_manager.subprocess.run",
|
|
781
|
+
lambda cmd, **kwargs: Mock(returncode=0, stdout=netstat),
|
|
782
|
+
)
|
|
783
|
+
|
|
784
|
+
assert DaemonEmbedManager._listening_pids(9177) == [4321]
|
|
785
|
+
|
|
786
|
+
def test_reads_listener_pids_from_a_kernel_tcp_table(self):
|
|
787
|
+
"""The GetExtendedTcpTable rows are decoded at the right offsets, for both families."""
|
|
788
|
+
import socket
|
|
789
|
+
import struct
|
|
790
|
+
|
|
791
|
+
from hindsight_embed.daemon_embed_manager import _AF_INET, _AF_INET6, _listener_pids_from_tcp_table
|
|
792
|
+
|
|
793
|
+
def v4_row(port: int, pid: int) -> bytes:
|
|
794
|
+
return struct.pack("<6I", 2, 0x0100007F, socket.htons(port), 0, 0, pid)
|
|
795
|
+
|
|
796
|
+
def v6_row(port: int, pid: int) -> bytes:
|
|
797
|
+
return bytes(16) + struct.pack("<2I", 0, socket.htons(port)) + bytes(16) + struct.pack("<4I", 0, 0, 2, pid)
|
|
798
|
+
|
|
799
|
+
v4 = struct.pack("<I", 2) + v4_row(9177, 4321) + v4_row(8888, 9999)
|
|
800
|
+
v6 = struct.pack("<I", 2) + v6_row(8888, 9999) + v6_row(9177, 4322)
|
|
801
|
+
|
|
802
|
+
assert _listener_pids_from_tcp_table(v4, _AF_INET, 9177) == [4321]
|
|
803
|
+
assert _listener_pids_from_tcp_table(v6, _AF_INET6, 9177) == [4322]
|
|
804
|
+
assert _listener_pids_from_tcp_table(struct.pack("<I", 0), _AF_INET, 9177) == []
|
|
805
|
+
|
|
806
|
+
@pytest.mark.skipif(sys.platform != "win32", reason="reads the Windows kernel")
|
|
807
|
+
def test_windows_native_lookup_finds_a_live_listener(self):
|
|
808
|
+
import socket
|
|
809
|
+
|
|
810
|
+
with socket.socket() as server:
|
|
811
|
+
server.bind(("127.0.0.1", 0))
|
|
812
|
+
server.listen()
|
|
813
|
+
port = server.getsockname()[1]
|
|
814
|
+
assert DaemonEmbedManager._windows_native_listening_pids(port) == [os.getpid()]
|
|
815
|
+
|
|
666
816
|
|
|
667
817
|
class TestProcessOwnership:
|
|
668
818
|
"""Regression coverage for #3520: never signal a process we cannot identify."""
|
|
@@ -742,6 +892,11 @@ class TestProcessOwnership:
|
|
|
742
892
|
assert DaemonEmbedManager._process_matches(1, DAEMON_PROCESS_MARKERS) is True
|
|
743
893
|
|
|
744
894
|
|
|
895
|
+
def _force_netstat(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
896
|
+
"""Send the Windows lookup to its netstat fallback. A real Windows runner would ask the kernel."""
|
|
897
|
+
monkeypatch.setattr(DaemonEmbedManager, "_windows_native_listening_pids", staticmethod(lambda _port: None))
|
|
898
|
+
|
|
899
|
+
|
|
745
900
|
class TestStartDaemonLockTimeout:
|
|
746
901
|
"""A start that cannot get the profile lock reports failure, not an OSError."""
|
|
747
902
|
|
|
@@ -799,6 +954,9 @@ class TestProcessCommandLine:
|
|
|
799
954
|
|
|
800
955
|
monkeypatch.setattr(dem.platform, "system", lambda: "Windows")
|
|
801
956
|
monkeypatch.setattr(DaemonEmbedManager, "_run_probe", staticmethod(_record))
|
|
957
|
+
# Force the probe fallback: on a real Windows runner the native read would
|
|
958
|
+
# otherwise answer for whatever process holds this PID (#6396).
|
|
959
|
+
monkeypatch.setattr(DaemonEmbedManager, "_windows_native_command_line", staticmethod(lambda _pid: None))
|
|
802
960
|
|
|
803
961
|
assert DaemonEmbedManager._process_command_line(1234) is None
|
|
804
962
|
assert timeouts, "expected the Windows branch to run at least one probe"
|
|
@@ -680,6 +680,7 @@ def test_listening_pids_windows_hides_netstat_console(monkeypatch):
|
|
|
680
680
|
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.platform.system", lambda: "Windows")
|
|
681
681
|
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.subprocess.CREATE_NO_WINDOW", 0x08000000, raising=False)
|
|
682
682
|
monkeypatch.setattr("hindsight_embed.daemon_embed_manager.subprocess.run", fake_run)
|
|
683
|
+
monkeypatch.setattr(DaemonEmbedManager, "_windows_native_listening_pids", staticmethod(lambda _port: None))
|
|
683
684
|
|
|
684
685
|
assert DaemonEmbedManager._listening_pids(9177) == [4321]
|
|
685
686
|
assert calls[0][1]["creationflags"] == 0x08000000
|
|
@@ -940,12 +941,13 @@ def test_run_probe_reads_output_that_is_not_utf8():
|
|
|
940
941
|
|
|
941
942
|
|
|
942
943
|
def test_windows_listening_pids_parses_a_localized_netstat(monkeypatch):
|
|
943
|
-
"""
|
|
944
|
+
"""A localized header does not stop the netstat fallback from reading the rows."""
|
|
944
945
|
decoded = _CP936_NETSTAT.decode("utf-8", errors="replace")
|
|
945
946
|
monkeypatch.setattr(
|
|
946
947
|
"hindsight_embed.daemon_embed_manager.DaemonEmbedManager._run_probe",
|
|
947
948
|
staticmethod(lambda cmd, timeout=None: decoded),
|
|
948
949
|
)
|
|
950
|
+
monkeypatch.setattr(DaemonEmbedManager, "_windows_native_listening_pids", staticmethod(lambda _port: None))
|
|
949
951
|
|
|
950
952
|
assert DaemonEmbedManager._windows_listening_pids(8642) == [4242]
|
|
951
953
|
assert DaemonEmbedManager._windows_listening_pids(445) == [4]
|
|
@@ -930,6 +930,8 @@ def _capture_daemon_env(manager, profile: str, config: dict) -> dict[str, str]:
|
|
|
930
930
|
patch("hindsight_embed.daemon_embed_manager.subprocess.Popen", side_effect=fake_popen),
|
|
931
931
|
patch("hindsight_embed.daemon_embed_manager.time.sleep"),
|
|
932
932
|
patch.object(manager, "_clear_port", return_value=True),
|
|
933
|
+
# Popen is stubbed, so the process lookup this would run must not be.
|
|
934
|
+
patch.object(manager, "_wait_for_previous_daemon_exit"),
|
|
933
935
|
patch.object(manager, "_find_api_command", return_value=["hindsight-api"]),
|
|
934
936
|
patch.object(manager, "is_running", side_effect=fake_is_running),
|
|
935
937
|
):
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/favicon.png
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/index.html
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/static/logo.png
RENAMED
|
File without changes
|
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/index.html
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/package-lock.json
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/package.json
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/favicon.png
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/public/logo.png
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/api.js
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/app.jsx
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/main.jsx
RENAMED
|
File without changes
|
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/src/style.css
RENAMED
|
File without changes
|
{hindsight_embed-1.2.0 → hindsight_embed-1.2.2}/hindsight_embed/control_center/ui/vite.config.js
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|