@kinosec/cli 0.10.3 → 0.11.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kinosec/cli",
-  "version": "0.10.3",
+  "version": "0.11.1",
   "description": "Official KinoSec command-line interface",
   "type": "module",
   "bin": {
@@ -8,6 +8,7 @@
   },
   "files": [
     "dist",
+    "vendor",
     "LICENSE.md",
     "README.md"
   ],

package/vendor/README.md

@@ -0,0 +1,134 @@
+<!--
+  VENDORED FILE — do not edit here.
+  Source of truth: xbowbench/tools/ctxpack/ (packer + README + test suite).
+  `kinosec pentest --source` shells out to ctxpack.py to pack a target's source
+  into the graybox `source_context` bundle. Stdlib-only Python 3, zero deps.
+  To update: re-copy ctxpack.py + README.md from the upstream tool.
+-->
+
+# ctxpack
+
+A **source-context ingestion** layer that turns a black-box CTF/pentest agent into a
+**graybox** one by feeding it the challenge's SOURCE CODE. Built for DEF CON-style
+targets (usually a small web/pwn app, occasionally a large codebase) under event time
+pressure: **one automatic, dependency-light step**, no mid-round setup.
+
+It does *not* build a code graph, embeddings/RAG store, or a whitebox analysis agent. The
+exploit-relevant logic in these targets is kilobytes and fits in context with room to
+spare, so the right move is to **paste the source** — the only real problem is that
+archives are padded with non-logic files (CSS frameworks, images, fonts, vendored deps,
+lockfiles, editor junk). ctxpack is the cheap **extension/path filter** that makes "just
+paste the source" work, with a mandatory manifest so nothing is dropped silently.
+
+## The one idea: the metric and the threshold
+
+- **Metric — kept-source tokens.** Every file is partitioned KEEP vs DROP; ctxpack sums
+  the estimated tokens of the KEEP set only. Token estimate = **`bytes / 3.7`** (no
+  tokenizer or API dependency). It routes on *logic-bearing source*, not raw archive bytes.
+- **Threshold — `--budget` (default `120000`).** If kept-source tokens **≤ budget**, emit a
+  **paste** bundle (the source itself). If **> budget**, emit a **retrieve** outline (a file
+  tree + paths, no bodies) so the agent greps/reads on demand.
+
+## Usage
+
+```bash
+python3 ctxpack.py <SOURCE> [-o OUT] [--budget N] [--bytes-per-token F] [--debug] \
+        [--stdout] [--emit-context-file PATH] [--stage-retrieve] [--stage-dir DIR] \
+        [--ctags] [--fork-diff [UPSTREAM_REF]]
+```
+
+`<SOURCE>` is a directory **or** an archive (`.zip` / `.tar[.gz]`, safely extracted).
+Outputs land in `OUT` (default `./ctxpack-out`):
+
+- `bundle.txt` — the inject-ready context (paste bundle, or retrieve outline).
+- `manifest.json` + `manifest.md` — every INCLUDED file (path, bytes, est-tokens, tier),
+  every EXCLUDED file (path, bytes, **reason**), totals, mode, budget, and signals.
+
+```bash
+python3 ctxpack.py ./challenge-src          # -> ./ctxpack-out/{bundle.txt,manifest.*}
+python3 ctxpack.py handout.zip -o /tmp/ctx --budget 120000
+python3 ctxpack.py ./big-repo --stage-retrieve --stage-dir /tmp/ctxpack_src
+python3 ctxpack.py . --fork-diff origin/main   # emit only the planted delta vs upstream
+```
+
+### Example (real challenge)
+
+`XBEN-039-24/app` → **paste**, 6 KEEP files (~2.4k tokens), 3 DROP (500 KB of JPGs + CSS):
+
+| kept | tier | dropped | reason |
+|---|---|---|---|
+| `html/index.php` | entrypoint | `html/images/*.jpg` | ext:image |
+| `html/admin.php`, `entrypoint.sh` | source | `html/styles.css` | ext:style |
+| `Dockerfile`, `000-default.conf` | config | | |
+| `db_init.sql` | other (logged) | | |
+
+## Tune knobs
+
+1. **The config block at the top of `ctxpack.py`** — the DROP/KEEP lists are plain
+   constants, easy to edit:
+   - `DROP_EXT` (images, fonts, styles, minified/maps, media, binaries) → reason codes.
+   - `DROP_FILENAMES` (lockfiles, editor junk) and `DROP_PATH_SEGMENTS`
+     (`node_modules/ vendor/ dist/ build/ __pycache__/ third_party/ .git/`).
+   - `KEEP_SOURCE_EXT`, `KEEP_TEMPLATE_EXT`, `KEEP_CONFIG_*` (Dockerfile, docker-compose,
+     `*.conf`, `requirements*.txt`, `package.json`, nginx/supervisor, `*.env.example`).
+   - `ENTRYPOINT_HINTS` — which names sort first (logic-first ordering).
+   - **Anything not dropped is KEPT** and, if it matched no KEEP hint, logged with
+     `note: kept:default-unknown-ext` ("when unsure, KEEP and log it").
+2. **`--budget N`** — the paste↔retrieve threshold (kept-source tokens).
+3. **`--bytes-per-token F`** — the estimate divisor (default `3.7`; the wider repo habit is
+   `4.0` — pass `--bytes-per-token 4`).
+4. **`--debug`** — trace every KEEP/DROP decision (and its reason) to stderr, so the filter
+   is auditable mid-event.
+
+## Manifest & signals (no silent truncation)
+
+`manifest.json`/`manifest.md` list **every** file on both sides with reasons, and totals
+that equal the sum of the included files (byte and token). `signals` flags, without acting:
+`git_present` (fork-of-OSS diff candidate), `binaries_only` (decompile handoff — out of
+scope for a source layer), and `known_ctf_markers` (e.g. `flag*`, `benchmark.json`, or a
+`FLAG{`/`CTF{`/`HTB{` string in a kept file).
+
+## Integration — wiring into `ctf_agent_defcon.sh` (default OFF)
+
+The bundle is folded into the agent's **system prompt** via the `CTXPACK_BUNDLE` env
+channel, so a small target never needs disk access. **Everything is default-OFF**, so the
+benchmark stays black-box (its anti-cheat measurement is preserved) unless you opt in.
+
+- **One command (real events):**
+  ```bash
+  ./graybox-run.sh <source-or-archive> <target-url>
+  ```
+  Packs the source, exports `CTXPACK_BUNDLE`, and launches the agent grayboxed.
+- **Benchmark launcher:** `GRAYBOX=1 ./run-one-defcon.sh XBEN-039-24 <out>` packs
+  `<challenge>/app` first. Unset → black-box (unchanged).
+- **Agent side:** a default-OFF block (before the `CTF_PROMPT` write) appends the bundle
+  under an *authoritative* `## TARGET SOURCE CODE` header. The bundle is inserted via
+  command substitution **inside** a double-quoted assignment, so shell metacharacters in
+  the source (`"`, `$`, backticks) stay literal and never execute — verified by
+  `test_integration.sh`. Trace a run with `grep GRAYBOX <debug-log>`.
+- **Retrieve mode (large codebase):** `--stage-retrieve --stage-dir /tmp/ctxpack_src` puts
+  KEEP files under `/tmp` (which the agent's RULE 1 permits it to read) for grep-on-demand.
+
+## Testing
+
+```bash
+python3 -m unittest -v test_ctxpack.py   # fixtures A/B/C + edge-case matrix + invariants + zero-dep proof
+python3 -I ctxpack.py <fixture>          # runs isolated: no site-packages, no env -> stdlib-only
+bash test_integration.sh                 # graybox injection is verbatim-safe + OFF stays black-box
+```
+
+Invariants asserted: no excluded file appears in the bundle; manifest byte/token totals
+equal the sum of the included files; the tool imports **only** the Python standard library.
+
+## Stretch flags (optional, off by default)
+
+- `--ctags` — in retrieve mode, run `ctags -R` over the KEEP files for jump-to-definition
+  (no-op with a manifest note if `ctags` is absent).
+- `--fork-diff [UPSTREAM_REF]` — if a `.git/` is present (or a ref is supplied), emit
+  `git log`/`git diff` vs upstream instead of the full bundle, to isolate a planted delta.
+
+## Requirements
+
+Python 3 standard library only — **zero third-party dependencies**. No network, no API key,
+no tokenizer. (Tested on CPython 3.14; uses `sys.stdlib_module_names` for the dep check and
+`tarfile`'s `filter='data'` safe extraction where available.)