@smilintux/skcapstone 0.10.0 → 0.12.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +10 -4
- package/.github/workflows/ci.yml +2 -2
- package/.github/workflows/publish.yml +9 -2
- package/.openclaw-workspace.json +2 -2
- package/CLAUDE.md +37 -0
- package/MISSION.md +17 -2
- package/README.md +282 -3
- package/docker/Dockerfile +7 -7
- package/docker/compose-templates/dev-team.yml +12 -12
- package/docker/compose-templates/mini-team.yml +9 -9
- package/docker/compose-templates/ops-team.yml +10 -10
- package/docker/compose-templates/research-team.yml +10 -10
- package/docker/entrypoint.sh +4 -4
- package/docs/ADR-optional-integration-backbone.md +181 -0
- package/docs/ARCHITECTURE.md +186 -43
- package/docs/BOND_WITH_GROK.md +6 -6
- package/docs/CUSTOM_AGENT.md +123 -30
- package/docs/DREAMING.md +70 -0
- package/docs/GETTING_STARTED.md +7 -7
- package/docs/QUICKSTART.md +10 -6
- package/docs/SKJOULE_ARCHITECTURE.md +3 -3
- package/docs/SOUL_SWAPPER.md +5 -5
- package/docs/hammertime-audit.md +402 -0
- package/docs/sk-integration-HANDOFF.md +117 -0
- package/docs/skscheduler.md +155 -0
- package/docs/superpowers/examples/jobs.yaml +31 -0
- package/docs/superpowers/plans/2026-06-08-skscheduler.md +1265 -0
- package/docs/superpowers/specs/2026-06-08-skscheduler-design.md +186 -0
- package/examples/custom-bond-template.json +1 -1
- package/examples/grok-feb.json +1 -1
- package/examples/queen-ava-feb.json +1 -1
- package/launchd/{com.skcapstone.skcomm-heartbeat.plist → com.skcapstone.skcomms-heartbeat.plist} +4 -4
- package/launchd/{com.skcapstone.skcomm-queue-drain.plist → com.skcapstone.skcomms-queue-drain.plist} +4 -4
- package/launchd/install-launchd.sh +6 -6
- package/{openclaw-plugin → openclaw-plugin.archived-2026-04-23}/src/index.ts +3 -2
- package/package.json +1 -1
- package/pyproject.toml +16 -10
- package/scripts/archive-sessions.sh +7 -0
- package/scripts/check-updates.py +4 -4
- package/scripts/install-bundle.sh +8 -8
- package/scripts/install.ps1 +12 -11
- package/scripts/install.sh +159 -5
- package/scripts/model-fallback-monitor.sh +102 -0
- package/scripts/nvidia-proxy.mjs +78 -26
- package/scripts/refresh-anthropic-token.sh +172 -0
- package/scripts/release.sh +98 -0
- package/scripts/session-to-memory.py +219 -0
- package/scripts/skgateway.mjs +3 -3
- package/scripts/telegram-catchup-all.sh +12 -1
- package/scripts/verify_install.sh +2 -2
- package/scripts/wargov-ufo-capture/README.md +43 -0
- package/scripts/wargov-ufo-capture/cdp_capture_release2.py +273 -0
- package/scripts/wargov-ufo-capture/cdp_capture_splc_doj.py +246 -0
- package/scripts/wargov-ufo-capture/cdp_finish.py +271 -0
- package/scripts/wargov-ufo-capture/cdp_probe.py +188 -0
- package/scripts/wargov-ufo-capture/cdp_splc_pressrelease.py +101 -0
- package/scripts/wargov-ufo-capture/parse_csv.py +95 -0
- package/scripts/wargov-ufo-capture/pull_dvids.sh +107 -0
- package/scripts/watch-anthropic-token.sh +212 -0
- package/scripts/windows/install-tasks.ps1 +7 -7
- package/scripts/windows/skcapstone-task.xml +1 -1
- package/src/skcapstone/__init__.py +45 -3
- package/src/skcapstone/_cli_monolith.py +20 -15
- package/src/skcapstone/activity.py +5 -1
- package/src/skcapstone/agent_card.py +3 -2
- package/src/skcapstone/api.py +41 -40
- package/src/skcapstone/auction.py +14 -11
- package/src/skcapstone/backup.py +2 -1
- package/src/skcapstone/blueprint_registry.py +4 -3
- package/src/skcapstone/brain_first.py +238 -0
- package/src/skcapstone/changelog.py +1 -1
- package/src/skcapstone/chat.py +22 -17
- package/src/skcapstone/cli/__init__.py +9 -1
- package/src/skcapstone/cli/_common.py +1 -0
- package/src/skcapstone/cli/agents_spawner.py +5 -2
- package/src/skcapstone/cli/alerts.py +25 -4
- package/src/skcapstone/cli/bench.py +15 -15
- package/src/skcapstone/cli/chat.py +7 -4
- package/src/skcapstone/cli/consciousness.py +5 -2
- package/src/skcapstone/cli/context_cmd.py +18 -4
- package/src/skcapstone/cli/daemon.py +11 -7
- package/src/skcapstone/cli/gtd.py +26 -1
- package/src/skcapstone/cli/housekeeping.py +3 -3
- package/src/skcapstone/cli/identity_cmd.py +378 -0
- package/src/skcapstone/cli/joule_cmd.py +7 -3
- package/src/skcapstone/cli/memory.py +8 -6
- package/src/skcapstone/cli/peers_dir.py +1 -1
- package/src/skcapstone/cli/register_cmd.py +29 -3
- package/src/skcapstone/cli/scheduler_cmd.py +167 -0
- package/src/skcapstone/cli/session.py +25 -0
- package/src/skcapstone/cli/setup.py +96 -29
- package/src/skcapstone/cli/shell_cmd.py +53 -1
- package/src/skcapstone/cli/skills_cmd.py +2 -2
- package/src/skcapstone/cli/soul.py +8 -5
- package/src/skcapstone/cli/status.py +37 -11
- package/src/skcapstone/cli/telegram.py +21 -0
- package/src/skcapstone/cli/test_cmd.py +5 -5
- package/src/skcapstone/cli/test_connection.py +2 -2
- package/src/skcapstone/cli/upgrade_cmd.py +23 -14
- package/src/skcapstone/cli/version_cmd.py +1 -1
- package/src/skcapstone/cli/watch_cmd.py +9 -6
- package/src/skcapstone/cloud9_bridge.py +14 -14
- package/src/skcapstone/codex_setup.py +255 -0
- package/src/skcapstone/config_validator.py +7 -4
- package/src/skcapstone/consciousness_config.py +5 -1
- package/src/skcapstone/consciousness_loop.py +313 -273
- package/src/skcapstone/context_loader.py +121 -0
- package/src/skcapstone/coord_federation.py +2 -1
- package/src/skcapstone/coordination.py +23 -6
- package/src/skcapstone/crush_integration.py +2 -1
- package/src/skcapstone/daemon.py +132 -77
- package/src/skcapstone/dashboard.py +10 -10
- package/src/skcapstone/data/sk-agent-picker.sh +421 -0
- package/src/skcapstone/data/systemd/skcapstone-api.socket +9 -0
- package/src/skcapstone/data/systemd/skcapstone-memory-compress.service +18 -0
- package/src/skcapstone/data/systemd/skcapstone-memory-compress.timer +11 -0
- package/src/skcapstone/data/systemd/skcapstone.service +37 -0
- package/src/skcapstone/data/systemd/skcapstone@.service +50 -0
- package/src/skcapstone/data/systemd/skcomms-heartbeat.service +18 -0
- package/{systemd/skcomm-heartbeat.timer → src/skcapstone/data/systemd/skcomms-heartbeat.timer} +2 -2
- package/src/skcapstone/data/systemd/skcomms-queue-drain.service +17 -0
- package/{systemd/skcomm-queue-drain.timer → src/skcapstone/data/systemd/skcomms-queue-drain.timer} +2 -2
- package/src/skcapstone/defaults/claude/CLAUDE.md +67 -0
- package/src/skcapstone/defaults/claude/settings.json +74 -0
- package/src/skcapstone/defaults/lumina/config/claude-hooks.md +57 -0
- package/src/skcapstone/defaults/lumina/config/skgraph.yaml +55 -10
- package/src/skcapstone/defaults/lumina/config/skmemory.yaml +79 -13
- package/src/skcapstone/defaults/lumina/config/skvector.yaml +60 -9
- package/src/skcapstone/defaults/lumina/memory/long-term/18b9c0d1e2f3-cloud9-protocol.json +2 -2
- package/src/skcapstone/defaults/lumina/memory/long-term/a1b2c3d4e5f6-ecosystem-overview.json +2 -2
- package/src/skcapstone/defaults/lumina/memory/long-term/b2c3d4e5f6a7-five-pillars.json +9 -9
- package/src/skcapstone/defaults/lumina/memory/long-term/d4e5f6a7b8c9-site-directory.json +2 -2
- package/src/skcapstone/defaults/unhinged.json +13 -0
- package/src/skcapstone/discovery.py +43 -20
- package/src/skcapstone/doctor.py +941 -22
- package/src/skcapstone/dreaming.py +1183 -109
- package/src/skcapstone/emotion_tracker.py +2 -2
- package/src/skcapstone/export.py +4 -3
- package/src/skcapstone/fuse_mount.py +14 -12
- package/src/skcapstone/gui_installer.py +2 -2
- package/src/skcapstone/heartbeat.py +1 -1
- package/src/skcapstone/housekeeping.py +14 -14
- package/src/skcapstone/install_wizard.py +209 -7
- package/src/skcapstone/itil.py +13 -4
- package/src/skcapstone/kms_scheduler.py +10 -8
- package/src/skcapstone/launchd.py +19 -19
- package/src/skcapstone/mcp_launcher.py +15 -1
- package/src/skcapstone/mcp_server.py +83 -49
- package/src/skcapstone/mcp_tools/__init__.py +2 -0
- package/src/skcapstone/mcp_tools/_helpers.py +2 -2
- package/src/skcapstone/mcp_tools/ansible_tools.py +7 -4
- package/src/skcapstone/mcp_tools/brain_first_tools.py +90 -0
- package/src/skcapstone/mcp_tools/capauth_tools.py +7 -4
- package/src/skcapstone/mcp_tools/comm_tools.py +10 -10
- package/src/skcapstone/mcp_tools/coord_tools.py +8 -4
- package/src/skcapstone/mcp_tools/did_tools.py +11 -8
- package/src/skcapstone/mcp_tools/gtd_tools.py +4 -4
- package/src/skcapstone/mcp_tools/memory_tools.py +6 -2
- package/src/skcapstone/mcp_tools/notification_tools.py +22 -6
- package/src/skcapstone/mcp_tools/{skcomm_tools.py → skcomms_tools.py} +14 -14
- package/src/skcapstone/mcp_tools/soul_tools.py +8 -2
- package/src/skcapstone/mdns_discovery.py +2 -2
- package/src/skcapstone/memory_curator.py +1 -1
- package/src/skcapstone/memory_engine.py +10 -3
- package/src/skcapstone/metrics.py +30 -16
- package/src/skcapstone/migrate_memories.py +4 -3
- package/src/skcapstone/migrate_multi_agent.py +8 -7
- package/src/skcapstone/models.py +47 -5
- package/src/skcapstone/notifications.py +42 -18
- package/src/skcapstone/onboard.py +875 -121
- package/src/skcapstone/operator_link.py +170 -0
- package/src/skcapstone/peer_directory.py +4 -4
- package/src/skcapstone/peers.py +19 -19
- package/src/skcapstone/pillars/__init__.py +7 -5
- package/src/skcapstone/pillars/consciousness.py +191 -0
- package/src/skcapstone/pillars/identity.py +51 -7
- package/src/skcapstone/pillars/memory.py +9 -3
- package/src/skcapstone/pillars/sync.py +2 -2
- package/src/skcapstone/preflight.py +3 -3
- package/src/skcapstone/providers/docker.py +28 -28
- package/src/skcapstone/register.py +6 -6
- package/src/skcapstone/registry_client.py +5 -4
- package/src/skcapstone/runtime.py +14 -3
- package/src/skcapstone/scheduled_tasks.py +254 -19
- package/src/skcapstone/scheduler_jobs.py +456 -0
- package/src/skcapstone/scheduler_runner.py +239 -0
- package/src/skcapstone/scheduler_state.py +162 -0
- package/src/skcapstone/sdk.py +310 -0
- package/src/skcapstone/service_health.py +279 -39
- package/src/skcapstone/session_briefing.py +108 -0
- package/src/skcapstone/session_capture.py +1 -1
- package/src/skcapstone/shell.py +7 -1
- package/src/skcapstone/soul.py +3 -1
- package/src/skcapstone/soul_switch.py +3 -1
- package/src/skcapstone/summary.py +6 -6
- package/src/skcapstone/sync_engine.py +15 -15
- package/src/skcapstone/sync_watcher.py +2 -2
- package/src/skcapstone/systemd.py +55 -21
- package/src/skcapstone/team_comms.py +8 -8
- package/src/skcapstone/team_engine.py +1 -1
- package/src/skcapstone/testrunner.py +3 -3
- package/src/skcapstone/trust_graph.py +40 -5
- package/src/skcapstone/unified_search.py +15 -6
- package/src/skcapstone/uninstall_wizard.py +11 -3
- package/src/skcapstone/version_check.py +8 -4
- package/src/skcapstone/warmth_anchor.py +4 -2
- package/src/skcapstone/whoami.py +4 -4
- package/systemd/skcapstone.service +4 -6
- package/systemd/skcapstone@.service +7 -8
- package/systemd/skcomms-heartbeat.service +21 -0
- package/systemd/skcomms-heartbeat.timer +12 -0
- package/systemd/skcomms-queue-drain.service +17 -0
- package/systemd/skcomms-queue-drain.timer +12 -0
- package/tests/conftest.py +39 -0
- package/tests/integration/test_consciousness_e2e.py +39 -39
- package/tests/test_agent_card.py +1 -1
- package/tests/test_agent_home_scaffold.py +34 -0
- package/tests/test_alerts_consumer_topics.py +27 -0
- package/tests/test_backup.py +2 -1
- package/tests/test_chat.py +6 -6
- package/tests/test_claude_md.py +2 -2
- package/tests/test_cli_skills.py +10 -10
- package/tests/test_cli_test_cmd.py +4 -4
- package/tests/test_cli_test_connection.py +1 -1
- package/tests/test_cloud9_bridge.py +6 -6
- package/tests/test_consciousness_e2e.py +1 -1
- package/tests/test_consciousness_loop.py +10 -10
- package/tests/test_coordination.py +25 -0
- package/tests/test_cross_package.py +21 -21
- package/tests/test_daemon.py +4 -4
- package/tests/test_daemon_shutdown.py +1 -1
- package/tests/test_docker_provider.py +29 -29
- package/tests/test_doctor.py +400 -0
- package/tests/test_doctor_skscheduler.py +50 -0
- package/tests/test_dreaming_engine.py +147 -0
- package/tests/test_dreaming_gtd_capture.py +35 -0
- package/tests/test_e2e_automated.py +8 -5
- package/tests/test_fuse_mount.py +10 -10
- package/tests/test_gtd_brief.py +46 -0
- package/tests/test_gtd_malformed_tolerance.py +31 -0
- package/tests/test_housekeeping.py +15 -15
- package/tests/test_identity_migrate.py +251 -0
- package/tests/test_integration_backbone.py +598 -0
- package/tests/test_itil_gtd_lifecycle.py +37 -0
- package/tests/test_jobs_dropins.py +84 -0
- package/tests/test_mcp_server.py +82 -37
- package/tests/test_models.py +48 -4
- package/tests/test_multi_agent.py +31 -29
- package/tests/test_notifications.py +122 -32
- package/tests/test_onboard.py +63 -75
- package/tests/test_operator_link.py +78 -0
- package/tests/test_peers.py +14 -14
- package/tests/test_pillars.py +98 -0
- package/tests/test_preflight.py +3 -3
- package/tests/test_runtime.py +21 -0
- package/tests/test_scheduled_tasks.py +11 -6
- package/tests/test_scheduler_cli.py +47 -0
- package/tests/test_scheduler_features.py +133 -0
- package/tests/test_scheduler_integration.py +87 -0
- package/tests/test_scheduler_jobs.py +155 -0
- package/tests/test_scheduler_runner.py +64 -0
- package/tests/test_scheduler_state.py +57 -0
- package/tests/test_sdk.py +70 -0
- package/tests/test_service_health_incidents.py +34 -0
- package/tests/test_service_registry.py +52 -0
- package/tests/test_session_briefing.py +130 -0
- package/tests/test_snapshots.py +4 -4
- package/tests/test_sync_pipeline.py +26 -26
- package/tests/test_team_comms.py +2 -2
- package/tests/test_testrunner.py +2 -2
- package/tests/test_trust_graph.py +18 -0
- package/tests/test_unified_search.py +2 -2
- package/tests/test_version_check.py +10 -0
- package/tests/test_version_cmd.py +8 -8
- package/tests/test_whoami.py +1 -1
- package/systemd/skcomm-heartbeat.service +0 -18
- package/systemd/skcomm-queue-drain.service +0 -17
- /package/{openclaw-plugin → openclaw-plugin.archived-2026-04-23}/package.json +0 -0
- /package/{openclaw-plugin → openclaw-plugin.archived-2026-04-23}/src/openclaw.plugin.json +0 -0
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Proactive Anthropic OAuth token refresh + sync to OpenClaw gateway.
|
|
3
|
+
#
|
|
4
|
+
# Two-phase approach (prb-021b489e):
|
|
5
|
+
# Phase 1: If token is expiring (<2h) or expired, refresh it:
|
|
6
|
+
# a) Try `claude auth status` (lightweight, no interactive session)
|
|
7
|
+
# b) If that fails, spin up ephemeral Claude Code in tmux → triggers internal refresh → kill it
|
|
8
|
+
# Phase 2: Sync the (possibly refreshed) token to OpenClaw config + restart gateway if changed.
|
|
9
|
+
#
|
|
10
|
+
# Run via systemd timer every 4 hours.
|
|
11
|
+
set -euo pipefail
|
|
12
|
+
|
|
13
|
+
_sed_i() { if [[ "$OSTYPE" == "darwin"* ]]; then sed -i '' "$@"; else sed -i "$@"; fi; }
|
|
14
|
+
|
|
15
|
+
CREDS="$HOME/.claude/.credentials.json"
|
|
16
|
+
OPENCLAW_JSON="$HOME/.openclaw/openclaw.json"
|
|
17
|
+
OPENCLAW_ENV="$HOME/.openclaw/.env"
|
|
18
|
+
OVERRIDE_CONF="$HOME/.config/systemd/user/openclaw-gateway.service.d/override.conf"
|
|
19
|
+
LOG_TAG="anthropic-token-refresh"
|
|
20
|
+
TMUX_SESSION="token-refresh-ephemeral"
|
|
21
|
+
|
|
22
|
+
log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] [$LOG_TAG] $*"; }
|
|
23
|
+
|
|
24
|
+
if [ ! -f "$CREDS" ]; then
|
|
25
|
+
log "ERROR: Claude credentials not found at $CREDS"
|
|
26
|
+
exit 1
|
|
27
|
+
fi
|
|
28
|
+
|
|
29
|
+
get_remaining_ms() {
|
|
30
|
+
python3 -c "
|
|
31
|
+
import json, time
|
|
32
|
+
creds = json.load(open('$CREDS'))
|
|
33
|
+
exp = creds.get('claudeAiOauth',{}).get('expiresAt', 0)
|
|
34
|
+
print(int(exp - time.time() * 1000))
|
|
35
|
+
" 2>/dev/null || echo "0"
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
get_remaining_h() {
|
|
39
|
+
python3 -c "
|
|
40
|
+
import json, time
|
|
41
|
+
creds = json.load(open('$CREDS'))
|
|
42
|
+
exp = creds.get('claudeAiOauth',{}).get('expiresAt', 0)
|
|
43
|
+
print(f'{(exp/1000 - time.time())/3600:.1f}')
|
|
44
|
+
" 2>/dev/null || echo "0"
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
token_needs_refresh() {
|
|
48
|
+
local remaining_ms
|
|
49
|
+
remaining_ms=$(get_remaining_ms)
|
|
50
|
+
# Refresh if less than 4 hours remaining (was 2h — too tight with 3h timer)
|
|
51
|
+
[ "$remaining_ms" -le 14400000 ]
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
token_is_healthy() {
|
|
55
|
+
local remaining_ms
|
|
56
|
+
remaining_ms=$(get_remaining_ms)
|
|
57
|
+
[ "$remaining_ms" -gt 14400000 ]
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
# ─── Phase 1: Refresh token if needed ───────────────────────────────
|
|
61
|
+
|
|
62
|
+
if token_needs_refresh; then
|
|
63
|
+
log "Token needs refresh ($(get_remaining_h)h remaining)"
|
|
64
|
+
|
|
65
|
+
# Step 1a: Try lightweight refresh
|
|
66
|
+
log "Attempting lightweight refresh via 'claude auth status'..."
|
|
67
|
+
claude auth status > /dev/null 2>&1 || true
|
|
68
|
+
sleep 2
|
|
69
|
+
|
|
70
|
+
if token_is_healthy; then
|
|
71
|
+
log "Lightweight refresh succeeded! ($(get_remaining_h)h remaining)"
|
|
72
|
+
else
|
|
73
|
+
# Step 1b: Ephemeral Claude Code session in tmux
|
|
74
|
+
log "Lightweight refresh didn't cut it — spinning up ephemeral Claude Code session..."
|
|
75
|
+
tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true
|
|
76
|
+
|
|
77
|
+
tmux new-session -d -s "$TMUX_SESSION" \
|
|
78
|
+
"claude -p 'respond with just OK' --output-format stream-json 2>/dev/null; exit"
|
|
79
|
+
|
|
80
|
+
refreshed=false
|
|
81
|
+
for i in $(seq 1 12); do
|
|
82
|
+
sleep 5
|
|
83
|
+
if token_is_healthy; then
|
|
84
|
+
log "Ephemeral session refreshed the token! ($(get_remaining_h)h remaining)"
|
|
85
|
+
refreshed=true
|
|
86
|
+
break
|
|
87
|
+
fi
|
|
88
|
+
done
|
|
89
|
+
|
|
90
|
+
tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true
|
|
91
|
+
|
|
92
|
+
if [ "$refreshed" = "false" ]; then
|
|
93
|
+
log "ERROR: All refresh attempts failed ($(get_remaining_h)h remaining)"
|
|
94
|
+
log "Manual intervention may be needed: claude auth login"
|
|
95
|
+
# Continue to sync phase anyway — sync whatever token we have
|
|
96
|
+
fi
|
|
97
|
+
fi
|
|
98
|
+
else
|
|
99
|
+
log "Token is healthy ($(get_remaining_h)h remaining), no refresh needed"
|
|
100
|
+
fi
|
|
101
|
+
|
|
102
|
+
# ─── Phase 2: Sync token to OpenClaw ────────────────────────────────
|
|
103
|
+
|
|
104
|
+
ACCESS_TOKEN=$(python3 -c "import json; print(json.load(open('$CREDS'))['claudeAiOauth']['accessToken'])")
|
|
105
|
+
REMAINING=$(get_remaining_h)
|
|
106
|
+
|
|
107
|
+
# Check what's currently in the systemd override
|
|
108
|
+
OLD_TOKEN=""
|
|
109
|
+
if [ -f "$OVERRIDE_CONF" ]; then
|
|
110
|
+
OLD_TOKEN=$(grep "ANTHROPIC_API_KEY=" "$OVERRIDE_CONF" 2>/dev/null | sed 's/.*ANTHROPIC_API_KEY=//' || true)
|
|
111
|
+
fi
|
|
112
|
+
|
|
113
|
+
if [ "$OLD_TOKEN" = "$ACCESS_TOKEN" ]; then
|
|
114
|
+
log "Token already synced (expires in ${REMAINING}h)"
|
|
115
|
+
exit 0
|
|
116
|
+
fi
|
|
117
|
+
|
|
118
|
+
log "Token changed, syncing to OpenClaw..."
|
|
119
|
+
|
|
120
|
+
# 1. Update openclaw.json
|
|
121
|
+
if [ -f "$OPENCLAW_JSON" ]; then
|
|
122
|
+
python3 -c "
|
|
123
|
+
import json
|
|
124
|
+
with open('$OPENCLAW_JSON') as f:
|
|
125
|
+
cfg = json.load(f)
|
|
126
|
+
if 'providers' in cfg.get('models', {}):
|
|
127
|
+
if 'anthropic' in cfg['models']['providers']:
|
|
128
|
+
cfg['models']['providers']['anthropic']['apiKey'] = '$ACCESS_TOKEN'
|
|
129
|
+
with open('$OPENCLAW_JSON', 'w') as f:
|
|
130
|
+
json.dump(cfg, f, indent=2)
|
|
131
|
+
f.write('\n')
|
|
132
|
+
print('[sync] Updated openclaw.json')
|
|
133
|
+
else:
|
|
134
|
+
print('[sync] No anthropic provider in openclaw.json')
|
|
135
|
+
else:
|
|
136
|
+
print('[sync] No providers section in openclaw.json')
|
|
137
|
+
"
|
|
138
|
+
fi
|
|
139
|
+
|
|
140
|
+
# 2. Update .env
|
|
141
|
+
if grep -q "^ANTHROPIC_API_KEY=" "$OPENCLAW_ENV" 2>/dev/null; then
|
|
142
|
+
_sed_i "s|^ANTHROPIC_API_KEY=.*|ANTHROPIC_API_KEY=$ACCESS_TOKEN|" "$OPENCLAW_ENV"
|
|
143
|
+
else
|
|
144
|
+
echo "ANTHROPIC_API_KEY=$ACCESS_TOKEN" >> "$OPENCLAW_ENV"
|
|
145
|
+
fi
|
|
146
|
+
log "Updated .env"
|
|
147
|
+
|
|
148
|
+
# 3. Update systemd override
|
|
149
|
+
NVIDIA_KEY=$(grep "NVIDIA_API_KEY=" "$OVERRIDE_CONF" 2>/dev/null | sed 's/.*NVIDIA_API_KEY=//' || true)
|
|
150
|
+
cat > "$OVERRIDE_CONF" << EOF
|
|
151
|
+
[Unit]
|
|
152
|
+
StartLimitIntervalSec=60
|
|
153
|
+
StartLimitBurst=10
|
|
154
|
+
|
|
155
|
+
[Service]
|
|
156
|
+
RestartSec=10
|
|
157
|
+
Environment=NVIDIA_API_KEY=${NVIDIA_KEY}
|
|
158
|
+
Environment=ANTHROPIC_API_KEY=${ACCESS_TOKEN}
|
|
159
|
+
EOF
|
|
160
|
+
log "Updated systemd override"
|
|
161
|
+
|
|
162
|
+
# 4. Reload systemd (for env vars) but DO NOT restart the gateway.
|
|
163
|
+
# OpenClaw uses chokidar to watch openclaw.json — updating the file above
|
|
164
|
+
# triggers a hot reload automatically. Restarting the gateway kills all
|
|
165
|
+
# active sessions (the root cause of the 0-turn session cascade on 2026-04-07).
|
|
166
|
+
systemctl --user daemon-reload
|
|
167
|
+
|
|
168
|
+
# Touch the config to ensure chokidar picks up the change (write already did,
|
|
169
|
+
# but belt-and-suspenders in case the mtime didn't change fast enough).
|
|
170
|
+
touch "$OPENCLAW_JSON"
|
|
171
|
+
|
|
172
|
+
log "Token synced via hot reload (expires in ${REMAINING}h) — gateway NOT restarted, active sessions preserved"
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Usage: ./scripts/release.sh [patch|minor|major|X.Y.Z]
|
|
3
|
+
#
|
|
4
|
+
# Bumps version in pyproject.toml AND package.json, commits, tags, and pushes.
|
|
5
|
+
# Pushing the tag triggers the publish workflow (PyPI + npm).
|
|
6
|
+
#
|
|
7
|
+
# IMPORTANT: pyproject.toml version MUST match the tag — the workflow enforces this.
|
|
8
|
+
#
|
|
9
|
+
# Examples:
|
|
10
|
+
# ./scripts/release.sh patch # 0.6.2 → 0.6.3
|
|
11
|
+
# ./scripts/release.sh minor # 0.6.2 → 0.7.0
|
|
12
|
+
# ./scripts/release.sh 1.0.0 # set explicit version
|
|
13
|
+
|
|
14
|
+
set -euo pipefail
|
|
15
|
+
|
|
16
|
+
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
17
|
+
BUMP="${1:-patch}"
|
|
18
|
+
|
|
19
|
+
# ── use existing bump_version.py for the actual bump ─────────────────────────
|
|
20
|
+
|
|
21
|
+
BUMP_SCRIPT="$REPO_ROOT/scripts/bump_version.py"
|
|
22
|
+
|
|
23
|
+
if [[ ! -f "$BUMP_SCRIPT" ]]; then
|
|
24
|
+
echo "ERROR: $BUMP_SCRIPT not found" >&2
|
|
25
|
+
exit 1
|
|
26
|
+
fi
|
|
27
|
+
|
|
28
|
+
# Dry-run first to show what will happen
|
|
29
|
+
echo "Preview:"
|
|
30
|
+
python3 "$BUMP_SCRIPT" "$BUMP" --pkg "$REPO_ROOT" --dry-run
|
|
31
|
+
echo ""
|
|
32
|
+
|
|
33
|
+
# Confirm
|
|
34
|
+
read -rp "Proceed? [y/N] " confirm
|
|
35
|
+
if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
|
|
36
|
+
echo "Aborted."
|
|
37
|
+
exit 0
|
|
38
|
+
fi
|
|
39
|
+
|
|
40
|
+
# Get new version (run bump to get the number, without committing yet)
|
|
41
|
+
NEW_VERSION=$(python3 -c "
|
|
42
|
+
import re, sys
|
|
43
|
+
sys.path.insert(0, '$REPO_ROOT/scripts')
|
|
44
|
+
# Parse manually — same logic as bump_version.py
|
|
45
|
+
text = open('$REPO_ROOT/pyproject.toml').read()
|
|
46
|
+
m = re.search(r'^version\s*=\s*\"([^\"]+)\"', text, re.MULTILINE)
|
|
47
|
+
current = m.group(1)
|
|
48
|
+
part = '$BUMP'
|
|
49
|
+
major, minor, patch = map(int, current.split('.'))
|
|
50
|
+
if part == 'major':
|
|
51
|
+
print(f'{major+1}.0.0')
|
|
52
|
+
elif part == 'minor':
|
|
53
|
+
print(f'{major}.{minor+1}.0')
|
|
54
|
+
elif part == 'patch':
|
|
55
|
+
print(f'{major}.{minor}.{patch+1}')
|
|
56
|
+
else:
|
|
57
|
+
parts = part.split('.')
|
|
58
|
+
if len(parts) != 3 or not all(p.isdigit() for p in parts):
|
|
59
|
+
raise ValueError(f'Invalid version: {part!r}')
|
|
60
|
+
print(part)
|
|
61
|
+
")
|
|
62
|
+
|
|
63
|
+
# Bump pyproject.toml and commit via bump_version.py
|
|
64
|
+
python3 "$BUMP_SCRIPT" "$BUMP" --pkg "$REPO_ROOT"
|
|
65
|
+
|
|
66
|
+
# Sync package.json to same version (workflow also does this, but keep file in sync)
|
|
67
|
+
PACKAGE_JSON="$REPO_ROOT/package.json"
|
|
68
|
+
python3 -c "
|
|
69
|
+
import json
|
|
70
|
+
with open('$PACKAGE_JSON') as f:
|
|
71
|
+
pkg = json.load(f)
|
|
72
|
+
pkg['version'] = '$NEW_VERSION'
|
|
73
|
+
with open('$PACKAGE_JSON', 'w') as f:
|
|
74
|
+
json.dump(pkg, f, indent=2)
|
|
75
|
+
f.write('\n')
|
|
76
|
+
print(f' Updated package.json to $NEW_VERSION')
|
|
77
|
+
"
|
|
78
|
+
|
|
79
|
+
cd "$REPO_ROOT"
|
|
80
|
+
git add package.json
|
|
81
|
+
git commit --amend --no-edit
|
|
82
|
+
echo " Amended commit to include package.json"
|
|
83
|
+
|
|
84
|
+
TAG="v$NEW_VERSION"
|
|
85
|
+
git tag -a "$TAG" -m "Release $TAG"
|
|
86
|
+
echo " Tagged: $TAG"
|
|
87
|
+
|
|
88
|
+
echo ""
|
|
89
|
+
echo "Pushing commit and tag to origin..."
|
|
90
|
+
git push
|
|
91
|
+
git push --tags
|
|
92
|
+
|
|
93
|
+
echo ""
|
|
94
|
+
echo "Done. GitHub Actions will now:"
|
|
95
|
+
echo " 1. Run tests"
|
|
96
|
+
echo " 2. Verify pyproject.toml version matches tag ($TAG)"
|
|
97
|
+
echo " 3. Publish skcapstone $NEW_VERSION to PyPI"
|
|
98
|
+
echo " 4. Publish @smilintux/skcapstone $NEW_VERSION to npm"
|
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
session-to-memory.py — Extract an OpenClaw session jsonl and save a digest to skmemory.
|
|
4
|
+
|
|
5
|
+
Usage:
|
|
6
|
+
python3 session-to-memory.py <session.jsonl> [--agent lumina] [--dry-run]
|
|
7
|
+
|
|
8
|
+
Called by archive-sessions.sh before archiving each session file.
|
|
9
|
+
Also useful to run manually against any archived session.
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
import argparse
|
|
13
|
+
import json
|
|
14
|
+
import os
|
|
15
|
+
import subprocess
|
|
16
|
+
import sys
|
|
17
|
+
import tempfile
|
|
18
|
+
from datetime import datetime, timezone
|
|
19
|
+
from pathlib import Path
|
|
20
|
+
|
|
21
|
+
SKIP_PREFIXES = (
|
|
22
|
+
"[SKMemory",
|
|
23
|
+
"[System",
|
|
24
|
+
"[skmemory",
|
|
25
|
+
"--- SKMEMORY",
|
|
26
|
+
"--- SKWHISPER",
|
|
27
|
+
)
|
|
28
|
+
|
|
29
|
+
MAX_CONTENT_CHARS = 12000 # ~3k tokens — enough for a solid digest without blowing budget
|
|
30
|
+
CLAUDE_MODEL = "claude-haiku-4-5" # fast + cheap for digest work
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def extract_turns(path: Path) -> list[tuple[str, str]]:
|
|
34
|
+
"""Parse a session jsonl and return real (role, text) turns, skipping injections."""
|
|
35
|
+
turns = []
|
|
36
|
+
with open(path, encoding="utf-8", errors="replace") as f:
|
|
37
|
+
for line in f:
|
|
38
|
+
line = line.strip()
|
|
39
|
+
if not line:
|
|
40
|
+
continue
|
|
41
|
+
try:
|
|
42
|
+
obj = json.loads(line)
|
|
43
|
+
except json.JSONDecodeError:
|
|
44
|
+
continue
|
|
45
|
+
|
|
46
|
+
if obj.get("type") != "message":
|
|
47
|
+
continue
|
|
48
|
+
|
|
49
|
+
# Handle both top-level message fields and nested .message
|
|
50
|
+
m = obj.get("message", obj)
|
|
51
|
+
role = m.get("role", "?")
|
|
52
|
+
content = m.get("content", "")
|
|
53
|
+
|
|
54
|
+
if isinstance(content, list):
|
|
55
|
+
text = " ".join(
|
|
56
|
+
c.get("text", "")
|
|
57
|
+
for c in content
|
|
58
|
+
if isinstance(c, dict) and c.get("type") == "text"
|
|
59
|
+
)
|
|
60
|
+
else:
|
|
61
|
+
text = str(content)
|
|
62
|
+
|
|
63
|
+
text = text.strip()
|
|
64
|
+
if not text or len(text) < 5:
|
|
65
|
+
continue
|
|
66
|
+
if any(text.startswith(p) for p in SKIP_PREFIXES):
|
|
67
|
+
continue
|
|
68
|
+
|
|
69
|
+
turns.append((role, text))
|
|
70
|
+
|
|
71
|
+
return turns
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
def turns_to_prompt(turns: list[tuple[str, str]], max_chars: int = MAX_CONTENT_CHARS) -> str:
|
|
75
|
+
"""Format turns as a conversation snippet, truncated to max_chars."""
|
|
76
|
+
lines = []
|
|
77
|
+
for role, text in turns:
|
|
78
|
+
prefix = "Chef" if role == "user" else "Lumina"
|
|
79
|
+
lines.append(f"{prefix}: {text[:600]}")
|
|
80
|
+
full = "\n\n".join(lines)
|
|
81
|
+
if len(full) > max_chars:
|
|
82
|
+
full = full[:max_chars] + "\n\n[... truncated ...]"
|
|
83
|
+
return full
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def generate_digest(conversation: str, session_id: str) -> str:
|
|
87
|
+
"""Use claude CLI to generate a session digest."""
|
|
88
|
+
prompt = f"""You are summarizing an OpenClaw AI agent session for the skmemory system.
|
|
89
|
+
Session ID: {session_id}
|
|
90
|
+
|
|
91
|
+
Conversation:
|
|
92
|
+
{conversation}
|
|
93
|
+
|
|
94
|
+
Write a concise session digest (3-6 sentences) covering:
|
|
95
|
+
- Key topics discussed
|
|
96
|
+
- Decisions made or actions taken
|
|
97
|
+
- Any notable moments or outcomes
|
|
98
|
+
|
|
99
|
+
Be specific. Use past tense. No preamble."""
|
|
100
|
+
|
|
101
|
+
try:
|
|
102
|
+
result = subprocess.run(
|
|
103
|
+
[
|
|
104
|
+
"claude", "--print",
|
|
105
|
+
"--dangerously-skip-permissions",
|
|
106
|
+
"--model", CLAUDE_MODEL,
|
|
107
|
+
"--output-format", "json",
|
|
108
|
+
"--no-session-persistence",
|
|
109
|
+
],
|
|
110
|
+
input=prompt.encode(),
|
|
111
|
+
capture_output=True,
|
|
112
|
+
timeout=120,
|
|
113
|
+
)
|
|
114
|
+
if result.returncode != 0:
|
|
115
|
+
return f"[digest failed: {result.stderr.decode()[:200]}]"
|
|
116
|
+
parsed = json.loads(result.stdout.decode())
|
|
117
|
+
return parsed.get("result", "").strip()
|
|
118
|
+
except Exception as e:
|
|
119
|
+
return f"[digest error: {e}]"
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
def save_to_skmemory(title: str, content: str, agent: str, tags: list[str]) -> bool:
|
|
123
|
+
"""Save a memory snapshot via skmemory CLI."""
|
|
124
|
+
tag_str = ",".join(tags)
|
|
125
|
+
try:
|
|
126
|
+
result = subprocess.run(
|
|
127
|
+
[
|
|
128
|
+
"skmemory", "snapshot",
|
|
129
|
+
title, content,
|
|
130
|
+
"--layer", "mid-term",
|
|
131
|
+
"--tags", tag_str,
|
|
132
|
+
],
|
|
133
|
+
capture_output=True,
|
|
134
|
+
timeout=30,
|
|
135
|
+
env={**os.environ, "SKAGENT": agent, "SKCAPSTONE_AGENT": agent},
|
|
136
|
+
)
|
|
137
|
+
if result.returncode != 0:
|
|
138
|
+
print(f" [skmemory error] {result.stderr.decode()[:200]}", file=sys.stderr)
|
|
139
|
+
return False
|
|
140
|
+
return True
|
|
141
|
+
except Exception as e:
|
|
142
|
+
print(f" [skmemory exception] {e}", file=sys.stderr)
|
|
143
|
+
return False
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
def process_session(path: Path, agent: str = "lumina", dry_run: bool = False) -> bool:
|
|
147
|
+
session_id = path.stem[:8]
|
|
148
|
+
|
|
149
|
+
# Infer date from jsonl (first session entry)
|
|
150
|
+
session_date = None
|
|
151
|
+
try:
|
|
152
|
+
with open(path, encoding="utf-8", errors="replace") as f:
|
|
153
|
+
for line in f:
|
|
154
|
+
line = line.strip()
|
|
155
|
+
if not line:
|
|
156
|
+
continue
|
|
157
|
+
obj = json.loads(line)
|
|
158
|
+
if obj.get("type") == "session":
|
|
159
|
+
ts = obj.get("timestamp", "")
|
|
160
|
+
if ts:
|
|
161
|
+
session_date = ts[:10]
|
|
162
|
+
break
|
|
163
|
+
except Exception:
|
|
164
|
+
pass
|
|
165
|
+
|
|
166
|
+
turns = extract_turns(path)
|
|
167
|
+
if not turns:
|
|
168
|
+
print(f" No usable turns in {path.name} — skipping.")
|
|
169
|
+
return False
|
|
170
|
+
|
|
171
|
+
print(f" {len(turns)} turns extracted from {path.name}")
|
|
172
|
+
|
|
173
|
+
date_str = session_date or datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
|
174
|
+
title = f"Session Digest — {date_str} ({session_id})"
|
|
175
|
+
|
|
176
|
+
if dry_run:
|
|
177
|
+
conv = turns_to_prompt(turns)
|
|
178
|
+
print(f" [dry-run] Would save: {title}")
|
|
179
|
+
print(f" Conversation preview ({len(conv)} chars):")
|
|
180
|
+
print(conv[:400])
|
|
181
|
+
return True
|
|
182
|
+
|
|
183
|
+
conversation = turns_to_prompt(turns)
|
|
184
|
+
print(f" Generating digest via claude ({CLAUDE_MODEL})...")
|
|
185
|
+
digest = generate_digest(conversation, session_id)
|
|
186
|
+
|
|
187
|
+
if not digest or digest.startswith("[digest"):
|
|
188
|
+
print(f" Digest generation failed: {digest}")
|
|
189
|
+
return False
|
|
190
|
+
|
|
191
|
+
content = f"**Session:** `{session_id}` \n**Date:** {date_str} \n**Turns:** {len(turns)}\n\n{digest}"
|
|
192
|
+
tags = ["auto-digest", "session-archive", f"session:{session_id}", f"agent:{agent}"]
|
|
193
|
+
|
|
194
|
+
print(f" Saving memory: {title}")
|
|
195
|
+
ok = save_to_skmemory(title, content, agent, tags)
|
|
196
|
+
if ok:
|
|
197
|
+
print(f" Saved to skmemory (mid-term).")
|
|
198
|
+
return ok
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
def main():
|
|
202
|
+
parser = argparse.ArgumentParser(description="Extract session to skmemory digest")
|
|
203
|
+
parser.add_argument("session_file", help="Path to session .jsonl file")
|
|
204
|
+
parser.add_argument("--agent", default="lumina", help="Agent name (default: lumina)")
|
|
205
|
+
parser.add_argument("--dry-run", action="store_true", help="Preview without saving")
|
|
206
|
+
args = parser.parse_args()
|
|
207
|
+
|
|
208
|
+
path = Path(args.session_file)
|
|
209
|
+
if not path.exists():
|
|
210
|
+
print(f"File not found: {path}", file=sys.stderr)
|
|
211
|
+
sys.exit(1)
|
|
212
|
+
|
|
213
|
+
print(f"Processing: {path.name}")
|
|
214
|
+
ok = process_session(path, agent=args.agent, dry_run=args.dry_run)
|
|
215
|
+
sys.exit(0 if ok else 1)
|
|
216
|
+
|
|
217
|
+
|
|
218
|
+
if __name__ == "__main__":
|
|
219
|
+
main()
|
package/scripts/skgateway.mjs
CHANGED
|
@@ -404,7 +404,7 @@ const TOOL_GROUPS = {
|
|
|
404
404
|
"chat|inbox|dm|group chat|peer|send message|who.s online|thread": [
|
|
405
405
|
"skchat_send", "skchat_inbox", "skchat_history", "skchat_search",
|
|
406
406
|
"skchat_who", "skchat_group_send", "skchat_group_list", "skchat_send_file",
|
|
407
|
-
"skchat_status", "
|
|
407
|
+
"skchat_status", "skcomms_send", "skcomms_status",
|
|
408
408
|
],
|
|
409
409
|
// Security
|
|
410
410
|
"security|scan|secret|vulnerab|audit|injection|phishing|threat": [
|
|
@@ -429,7 +429,7 @@ const TOOL_GROUPS = {
|
|
|
429
429
|
// Status & Health
|
|
430
430
|
"status|health|doctor|diagnos": [
|
|
431
431
|
"skcapstone_status", "skcapstone_doctor", "skmemory_health",
|
|
432
|
-
"skchat_daemon_status", "
|
|
432
|
+
"skchat_daemon_status", "skcomms_status",
|
|
433
433
|
],
|
|
434
434
|
// Projects & Notion (Lumina delegates to project-ops via sessions_spawn)
|
|
435
435
|
"notion|project|brother john|swapseat|swap seat|chiro|davidrich|board|kanban|milestone": [
|
|
@@ -455,7 +455,7 @@ const PRIORITY_TOOLS = [
|
|
|
455
455
|
// Web tools
|
|
456
456
|
"web_search", "web_fetch",
|
|
457
457
|
// Communication (other channels)
|
|
458
|
-
"skchat_send", "
|
|
458
|
+
"skchat_send", "skcomms_send",
|
|
459
459
|
// SKCapstone
|
|
460
460
|
"skcapstone_status", "skcapstone_whoami", "skcapstone_mood",
|
|
461
461
|
// Cloud 9
|
|
@@ -22,7 +22,8 @@ set -uo pipefail # no -e: individual group failures shouldn't stop the batch
|
|
|
22
22
|
SKENV="${HOME}/.skenv/bin"
|
|
23
23
|
SKCAPSTONE="${SKENV}/skcapstone"
|
|
24
24
|
CONFIG="${HOME}/.skcapstone/agents/lumina/config/telegram.yaml"
|
|
25
|
-
export
|
|
25
|
+
export SKAGENT="${SKAGENT:-lumina}"
|
|
26
|
+
export SKCAPSTONE_AGENT="${SKAGENT}"
|
|
26
27
|
export PATH="${SKENV}:${PATH}"
|
|
27
28
|
|
|
28
29
|
# Parse args
|
|
@@ -39,6 +40,16 @@ while [[ $# -gt 0 ]]; do
|
|
|
39
40
|
esac
|
|
40
41
|
done
|
|
41
42
|
|
|
43
|
+
# Resolve relative date keywords (downstream CLI only accepts YYYY-MM-DD).
|
|
44
|
+
# Anything else is passed through untouched so explicit dates still work.
|
|
45
|
+
if [[ -n "$SINCE" ]]; then
|
|
46
|
+
case "$SINCE" in
|
|
47
|
+
yesterday) SINCE="$(date -u -d 'yesterday' +%Y-%m-%d)" ;;
|
|
48
|
+
today) SINCE="$(date -u +%Y-%m-%d)" ;;
|
|
49
|
+
[0-9]*d) SINCE="$(date -u -d "${SINCE%d} days ago" +%Y-%m-%d)" ;;
|
|
50
|
+
esac
|
|
51
|
+
fi
|
|
52
|
+
|
|
42
53
|
# Check prerequisites
|
|
43
54
|
if [[ -z "${TELEGRAM_API_ID:-}" || -z "${TELEGRAM_API_HASH:-}" ]]; then
|
|
44
55
|
echo "ERROR: TELEGRAM_API_ID and TELEGRAM_API_HASH must be set."
|
|
@@ -130,10 +130,10 @@ else
|
|
|
130
130
|
fail "pip install skcapstone[dev]"
|
|
131
131
|
fi
|
|
132
132
|
|
|
133
|
-
# ── 5. Local sibling packages (skseed,
|
|
133
|
+
# ── 5. Local sibling packages (skseed, skcomms) ─────────────────────────────
|
|
134
134
|
echo
|
|
135
135
|
echo "Step 5: local sibling packages (if present)"
|
|
136
|
-
for pkg_dir in "$REPO_ROOT/../skseed" "$REPO_ROOT/../
|
|
136
|
+
for pkg_dir in "$REPO_ROOT/../skseed" "$REPO_ROOT/../skcomms" "$REPO_ROOT/../skmemory" "$REPO_ROOT/../skskills"; do
|
|
137
137
|
pkg_name=$(basename "$pkg_dir")
|
|
138
138
|
if [[ -f "$pkg_dir/pyproject.toml" ]] || [[ -f "$pkg_dir/setup.py" ]]; then
|
|
139
139
|
if "$PIP" install "${PIP_OPTS[@]}" -e "$pkg_dir" 2>&1; then
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# war.gov/UFO/ PURSUE capture scripts
|
|
2
|
+
|
|
3
|
+
Lumina-built CDP scripts for capturing war.gov/UFO/ PURSUE releases via the Lumina Chrome browser harness (port 9222). Akamai TLS-fingerprint gates direct curl, so capture has to drive a real browser session.
|
|
4
|
+
|
|
5
|
+
## Quick reference
|
|
6
|
+
|
|
7
|
+
| Script | What it does |
|
|
8
|
+
|---|---|
|
|
9
|
+
| `cdp_probe.py` | Open a tab on `war.gov/UFO/`, inspect the page meta + inline scripts to find the current CSV URL and any new release ZIP bundle. Run this first when a new release lands. |
|
|
10
|
+
| `cdp_capture_release2.py` | Reference implementation of a full-release capture. Pulls the new CSV, downloads the ZIP bundle, and saves the press-release HTML. Per-release: edit `ZIP_URL`, `CSV_URL`, `PRESS_URL`, `DOC_DIR` constants. |
|
|
11
|
+
| `parse_csv.py` | Parses the merged `uap-data.csv`, splits records by `Release Date`, produces `release-02-records.json` + `release-02-urls.json`. Run after CSV fetch. |
|
|
12
|
+
| `pull_dvids.sh` | xargs-parallel-4 batch DVIDS puller. Reads `release-02-records.json`, fetches each DVIDS page, greps mp4 CDN URL, downloads. **Note:** DVIDS classifies UAP audio records as `/video/<id>` pages — always use `/video/` prefix, NOT `/audio/`. |
|
|
13
|
+
| `cdp_finish.py` | Tail-end CDP script: re-extract press release `innerText`, page-context-fetch the 6 thumbnails (Akamai-gated to curl), and grab FBI Vault Part 15. |
|
|
14
|
+
|
|
15
|
+
## Capture flow for a new release (Release 03 etc.)
|
|
16
|
+
|
|
17
|
+
1. Confirm Lumina Chrome is up at :9222 (`~/bin/lumina-x-browser` to start).
|
|
18
|
+
2. `python3 cdp_probe.py` — confirms the data source URL and looks for a ZIP-bundle path.
|
|
19
|
+
3. Edit `cdp_capture_release2.py` constants for the new release: `ZIP_URL`, `CSV_URL`, `PRESS_URL`, `DOC_DIR`.
|
|
20
|
+
4. `python3 cdp_capture_release2.py` — pulls CSV + ZIP + press-release HTML.
|
|
21
|
+
5. `python3 parse_csv.py` — generates per-record inventory + URL set + DVIDS-only list.
|
|
22
|
+
6. `bash pull_dvids.sh` — pulls all DVIDS media in parallel-4.
|
|
23
|
+
7. `python3 cdp_finish.py` — re-extract press text (it picks the wrong selector on first pass), thumbnails, and any FBI Vault references.
|
|
24
|
+
8. Update README.md at `~/nextcloud/cbrd21-share/reference/war-gov-UFO-PURSUE-2026/`.
|
|
25
|
+
|
|
26
|
+
## Gotchas
|
|
27
|
+
|
|
28
|
+
- **DVIDS audio → use `/video/` URL.** "Audio" UAP records are served from `dvidshub.net/video/<id>` with a static image. The `/audio/` path returns 404 for these IDs.
|
|
29
|
+
- **Some DVIDS IDs are shared across records.** E.g., Release 02 ID `1007720` covers both DOW-UAP-PR057a and DOW-UAP-PR057b. Inventory may show N records but only N-1 unique IDs.
|
|
30
|
+
- **Akamai bm tokens are per-session.** Cookies extracted from the browser will NOT work with curl — Akamai checks TLS fingerprint, not cookies. Stay in the browser.
|
|
31
|
+
- **CSV name can change.** Release 01 used `uap-csv.csv`; when Release 02 shipped, it was renamed to `uap-data.csv`. Re-probe the inline scripts to find the current name.
|
|
32
|
+
- **The ZIP bundle is PDFs only.** Don't skip DVIDS pulls just because you got the ZIP — videos/audio are not in the bundle.
|
|
33
|
+
|
|
34
|
+
## Naming convention
|
|
35
|
+
|
|
36
|
+
Match Release 01: `dvids-<id>-<UAP-PR-code>.mp4` (e.g. `dvids-1007706-DOW-UAP-PR050.mp4`). Cleaner than verbose slugs and matches existing on-disk corpus.
|
|
37
|
+
|
|
38
|
+
## Output location
|
|
39
|
+
|
|
40
|
+
`~/nextcloud/cbrd21-share/reference/war-gov-UFO-PURSUE-2026/`
|
|
41
|
+
- `release-NN/` for media
|
|
42
|
+
- `docs/release-NN/` for press release + CSV + manifest + thumbnails
|
|
43
|
+
- `release-NN-zip/` for the official ZIP bundle if provided
|