typeclaw 0.46.0 → 0.46.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/package.json +2 -2
- package/src/agent/multimodal/looker.ts +11 -1
- package/src/agent/network/safe-http.ts +17 -8
- package/src/agent/plugin-tools.ts +1 -3
- package/src/agent/tool-file-safety.ts +396 -136
- package/src/agent/tools/webfetch/fetch.ts +22 -2
- package/src/agent/tools-without-local-file-operands.ts +9 -6
- package/src/bundled-plugins/backup/README.md +23 -10
- package/src/bundled-plugins/backup/git-auth.ts +6 -4
- package/src/bundled-plugins/backup/index.ts +83 -43
- package/src/bundled-plugins/backup/runner.ts +83 -0
- package/src/bundled-plugins/github-cli-auth/git-askpass.ts +74 -26
- package/src/bundled-plugins/github-cli-auth/git-command.ts +224 -36
- package/src/bundled-plugins/github-cli-auth/graphql-auth-nudge.ts +9 -9
- package/src/bundled-plugins/github-cli-auth/index.ts +110 -31
- package/src/bundled-plugins/github-cli-auth/token-class.ts +4 -6
- package/src/bundled-plugins/memory/README.md +9 -0
- package/src/bundled-plugins/memory/vector/embed-threads.ts +36 -0
- package/src/bundled-plugins/memory/vector/embedder.ts +19 -1
- package/src/bundled-plugins/researcher/write-report.ts +3 -3
- package/src/bundled-plugins/security/policies/private-surface-read.ts +610 -50
- package/src/bundled-plugins/security/policies/ssrf.ts +119 -34
- package/src/channels/adapters/github/inbound.ts +11 -0
- package/src/channels/adapters/github/index.ts +48 -54
- package/src/channels/adapters/github/reconcile-cooldown-store.ts +178 -0
- package/src/channels/adapters/github/reconcile-open-prs.ts +31 -0
- package/src/channels/adapters/slack-bot-edit.ts +85 -20
- package/src/channels/adapters/slack-bot.ts +7 -2
- package/src/channels/adapters/slack-format.ts +245 -0
- package/src/channels/github-token-bridge.ts +2 -3
- package/src/container/docker-app.ts +42 -13
- package/src/container/start.ts +13 -0
- package/src/cron/bridge.ts +9 -2
- package/src/doctor/checks.ts +5 -2
- package/src/git/bound-gc-config.ts +82 -0
- package/src/hostd/spawn.ts +133 -47
- package/src/init/dockerfile.ts +13 -0
- package/src/inspect/replay.ts +21 -9
- package/src/inspect/session-list.ts +71 -33
- package/src/mcp/tools.ts +1 -0
- package/src/network/internal-destinations.ts +267 -0
- package/src/plugin/types.ts +3 -2
- package/src/sandbox/env-exposure.ts +18 -15
- package/src/sandbox/policy.ts +5 -3
- package/src/sandbox/privileged-runtime.ts +1 -0
- package/src/skills/typeclaw-config/SKILL.md +22 -6
- package/src/skills/typeclaw-permissions/SKILL.md +22 -9
- package/src/skills/typeclaw-plugins/SKILL.md +37 -0
- package/src/usage/scan.ts +11 -2
- package/typeclaw.schema.json +5 -0
- package/src/git/secret-history.ts +0 -371
|
@@ -160,6 +160,15 @@ The embedding model (`Xenova/multilingual-e5-base`) caps inputs at **512 tokens*
|
|
|
160
160
|
1. **Deterministic bound at embed time (primary, always on).** `embed()` calls `boundEmbeddableText` on every input before the tokenizer sees it, trimming to a character budget derived from the same conservative per-script token estimate (CJK counted 1:1, Latin ~3.5 chars/token; the tokenizer is never loaded, preserving the embedder's lazy-load). The cut is now explicit and owned, not a hidden tokenizer side effect, and the leading heading/belief sentence — the load-bearing retrieval signal — always survives because it comes first. This guarantees no silent loss even for inputs nothing ever rewrites (raw legacy prose, long queries).
|
|
161
161
|
2. **Compaction routed to dreaming (remediation).** Bounding keeps retrieval correct but the bounded tail is still not embedded. The real fix is to make the shard fit. On each run the dreaming handler computes which topic shards are over budget (measuring `topicPassage(...).text` — the same citation-stripped string the embedder bounds — so the flag matches what is actually truncated) and renders an "Over the embedding budget" table into the subagent's prompt, alongside the strength-signal table. Dreaming rule 8 instructs it to compact those shards into the one-belief-sentence form **while preserving every `fragments:`/`superseded:` citation** — shrinking prose, never dropping a citation (a dropped citation would trip the citation-superset revert, leaving the shard both reverted and still over budget). The signal is gated on the vector index (`memory/.vectors/index.db`) existing; before the first index exists, the table is suppressed. The `over_budget=N` count rides the `[dreaming] done` line.
|
|
162
162
|
|
|
163
|
+
## Embedder resource tuning
|
|
164
|
+
|
|
165
|
+
The embedding model is a single onnxruntime-node session shared process-wide (`vector/embedder.ts`, `getEmbedder()` singleton). Two session options bound its memory footprint, since the container runs with no docker `--cpus` limit:
|
|
166
|
+
|
|
167
|
+
- **Adaptive intra-op threads** (`vector/embed-threads.ts`). ORT's default (`intraOpNumThreads: 0`) sizes the shared session's intra-op thread pool to one worker per visible host core, each holding its own activation workspace for the singleton's lifetime. `resolveEmbedThreadCount()` caps this at half the visible cores, clamped to `[1, 4]`. The pool is session-level and shared — the transformers.js Node path calls `session.run()` directly (concurrent `Run()` is allowed; the serialization only applies to the web backend), so concurrent embeds contend for it. Capping trades some potential intra-op parallelism (per embed and across overlapping embeds) for RSS bounded by the cap rather than by host core count; for this workload of short per-turn query embeds that trade is worth it.
|
|
168
|
+
- **CPU mem arena off** (`enableCpuMemArena: false`). ORT's arena never returns memory to the OS; for the intermittent per-turn query embed it would inflate steady-state RSS. Disabling frees per-embed at a negligible per-call malloc cost.
|
|
169
|
+
|
|
170
|
+
Neither is configurable — the adaptive default is intended to be correct on any host.
|
|
171
|
+
|
|
163
172
|
## How `session.idle` works
|
|
164
173
|
|
|
165
174
|
Core fires `session.idle` immediately after every `session.prompt()` completion. The plugin owns the debounce: a `Map<sessionId, Timeout>` reset on every event. When the timer fires, the plugin spawns `memory-logger` for that session — unless the min-delta gate suppresses the spawn (see below).
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
import { availableParallelism, cpus } from 'node:os'
|
|
2
|
+
|
|
3
|
+
// The container is launched with no docker `--cpus` limit, so onnxruntime-node's
|
|
4
|
+
// default `intraOpNumThreads` (0 = auto) sizes the shared session's intra-op
|
|
5
|
+
// thread pool to one worker PER visible host core — each with its own activation
|
|
6
|
+
// workspace, held for the life of the embedder singleton. On an 11-core host
|
|
7
|
+
// that is ~11 thread-local buffers, a real contributor to the container's RSS.
|
|
8
|
+
// The pool is session-level and shared: the transformers.js Node path calls
|
|
9
|
+
// `session.run()` directly (concurrent `Run()` is allowed — serialization only
|
|
10
|
+
// applies to the web backend), so concurrent embeds from overlapping turns
|
|
11
|
+
// contend for it. Capping is a deliberate trade: it caps intra-op parallelism
|
|
12
|
+
// (per embed and across overlapping embeds) in exchange for RSS bounded by the
|
|
13
|
+
// cap instead of by host core count. The workload is short per-turn query embeds
|
|
14
|
+
// (tens of ms each), so bounding memory is the priority over peak parallelism.
|
|
15
|
+
export const DEFAULT_MAX_EMBED_THREADS = 4
|
|
16
|
+
|
|
17
|
+
// Halve the visible cores (leaves headroom for the event loop and concurrent
|
|
18
|
+
// sessions), then clamp into [1, cap]. `min 1` handles a single-core box; the
|
|
19
|
+
// cap stops the many-core blowup.
|
|
20
|
+
export function resolveEmbedThreadCount(input?: { cap?: number; parallelism?: number }): number {
|
|
21
|
+
const cap = input?.cap ?? DEFAULT_MAX_EMBED_THREADS
|
|
22
|
+
const parallelism = input?.parallelism ?? detectParallelism()
|
|
23
|
+
const halved = Math.floor(parallelism / 2)
|
|
24
|
+
return Math.min(Math.max(halved, 1), cap)
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
function detectParallelism(): number {
|
|
28
|
+
try {
|
|
29
|
+
const n = availableParallelism()
|
|
30
|
+
if (Number.isInteger(n) && n >= 1) return n
|
|
31
|
+
} catch {
|
|
32
|
+
// availableParallelism is Node 18.14+/20+; fall back to cpus() on older runtimes.
|
|
33
|
+
}
|
|
34
|
+
const count = cpus().length
|
|
35
|
+
return count >= 1 ? count : 1
|
|
36
|
+
}
|
|
@@ -16,6 +16,7 @@ import {
|
|
|
16
16
|
import { getResolvedTransformersVersion } from '@/models/transformers-version'
|
|
17
17
|
|
|
18
18
|
import { homeRoot } from '../../../hostd/paths'
|
|
19
|
+
import { resolveEmbedThreadCount } from './embed-threads'
|
|
19
20
|
import { type BoundedText, boundEmbeddableText, MAX_MODEL_TOKENS } from './truncation'
|
|
20
21
|
|
|
21
22
|
// Re-exported for the vector subsystem's existing imports. The canonical
|
|
@@ -99,7 +100,24 @@ export class Embedder {
|
|
|
99
100
|
// miss, or silently loads a stale variant. Fails loudly with a refresh hint.
|
|
100
101
|
await verifyModelCache()
|
|
101
102
|
configureTransformers(env)
|
|
102
|
-
const extractor = await pipeline('feature-extraction', MODEL_NAME, {
|
|
103
|
+
const extractor = await pipeline('feature-extraction', MODEL_NAME, {
|
|
104
|
+
local_files_only: true,
|
|
105
|
+
dtype: MODEL_DTYPE,
|
|
106
|
+
// onnxruntime-node session tuning (transformers.js forwards session_options
|
|
107
|
+
// straight into the native InferenceSession). Two memory levers:
|
|
108
|
+
// - intraOpNumThreads: capped adaptively instead of ORT's default (0 =
|
|
109
|
+
// one worker per visible host core), each of which holds its own
|
|
110
|
+
// activation workspace for the singleton's lifetime.
|
|
111
|
+
// - enableCpuMemArena: off — ORT's arena never returns memory to the OS,
|
|
112
|
+
// so for the intermittent per-turn query embed it inflates steady-state
|
|
113
|
+
// RSS with no benefit. Disabling frees per-embed, at a small per-call
|
|
114
|
+
// malloc cost that is negligible next to the ~tens-of-ms forward pass.
|
|
115
|
+
session_options: {
|
|
116
|
+
intraOpNumThreads: resolveEmbedThreadCount(),
|
|
117
|
+
interOpNumThreads: 1,
|
|
118
|
+
enableCpuMemArena: false,
|
|
119
|
+
},
|
|
120
|
+
})
|
|
103
121
|
return new Embedder(extractor)
|
|
104
122
|
}
|
|
105
123
|
|
|
@@ -31,8 +31,8 @@ const writtenBySession = new Map<string, true>()
|
|
|
31
31
|
// - the parent dir's realpath must equal the real workspace/public dir, which
|
|
32
32
|
// blocks `workspace -> /agent/.env` style symlink escapes that a lexical
|
|
33
33
|
// check would follow,
|
|
34
|
-
// - the
|
|
35
|
-
// final-path symlink is rejected
|
|
34
|
+
// - the runtime descriptor-anchors the parent, then the file is created with
|
|
35
|
+
// O_EXCL, so an existing file or planted final-path symlink is rejected,
|
|
36
36
|
// - a second write in the same session is rejected (one report per spawn),
|
|
37
37
|
// - the schema is strict, so an `acknowledgeGuards` field is rejected, not
|
|
38
38
|
// silently stripped.
|
|
@@ -52,7 +52,7 @@ Write to \`public/\` instead of \`workspace/\` when your resolved role lacks \`f
|
|
|
52
52
|
.describe('Absolute path: <agent>/workspace/research-<slug>.md or <agent>/public/research-<slug>.md'),
|
|
53
53
|
content: z.string().describe('The full markdown report body.'),
|
|
54
54
|
}),
|
|
55
|
-
fileOperands: {
|
|
55
|
+
fileOperands: { create: ['path'] },
|
|
56
56
|
async execute(args: WriteReportArgs, ctx: ToolContext) {
|
|
57
57
|
if (writtenBySession.has(ctx.sessionId)) {
|
|
58
58
|
throw new Error('A report has already been written for this session. You may write exactly one report.')
|