ctx-packr 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ctx_packr-0.1.1/.gitignore +18 -0
- ctx_packr-0.1.1/LICENSE +21 -0
- ctx_packr-0.1.1/PKG-INFO +166 -0
- ctx_packr-0.1.1/README.md +142 -0
- ctx_packr-0.1.1/SPEC.md +158 -0
- ctx_packr-0.1.1/pyproject.toml +211 -0
- ctx_packr-0.1.1/sample/docs/architecture.md +14 -0
- ctx_packr-0.1.1/sample/src/api.py +14 -0
- ctx_packr-0.1.1/sample/src/auth.py +16 -0
- ctx_packr-0.1.1/sample/src/main.py +19 -0
- ctx_packr-0.1.1/sample/tests/test_auth.py +14 -0
- ctx_packr-0.1.1/src/ctxpack/__init__.py +5 -0
- ctx_packr-0.1.1/src/ctxpack/__main__.py +6 -0
- ctx_packr-0.1.1/src/ctxpack/cli.py +170 -0
- ctx_packr-0.1.1/src/ctxpack/errors.py +5 -0
- ctx_packr-0.1.1/src/ctxpack/manifest.py +16 -0
- ctx_packr-0.1.1/src/ctxpack/packer.py +101 -0
- ctx_packr-0.1.1/src/ctxpack/ranker.py +90 -0
- ctx_packr-0.1.1/src/ctxpack/render.py +112 -0
- ctx_packr-0.1.1/src/ctxpack/tokens.py +5 -0
- ctx_packr-0.1.1/src/ctxpack/walker.py +181 -0
- ctx_packr-0.1.1/tests/test_hidden.py +243 -0
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.pyc
|
|
3
|
+
|
|
4
|
+
# Packaging build artifacts
|
|
5
|
+
build/
|
|
6
|
+
dist/
|
|
7
|
+
*.egg-info/
|
|
8
|
+
|
|
9
|
+
# Virtual environments
|
|
10
|
+
.venv/
|
|
11
|
+
|
|
12
|
+
# uv's resolved environment lock. Not committed: this is a library, so users
|
|
13
|
+
# resolve their own versions. Delete this line if you ever want it pinned.
|
|
14
|
+
uv.lock
|
|
15
|
+
|
|
16
|
+
# Local run output
|
|
17
|
+
bundle*.md
|
|
18
|
+
manifest.json
|
ctx_packr-0.1.1/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Muhammad Shaheer
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
ctx_packr-0.1.1/PKG-INFO
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: ctx-packr
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: A stdlib-only CLI that packs the most relevant files of a codebase into a token-budgeted markdown bundle
|
|
5
|
+
Project-URL: Homepage, https://github.com/shaheersaifi07-prog/ctxpack
|
|
6
|
+
Project-URL: Repository, https://github.com/shaheersaifi07-prog/ctxpack
|
|
7
|
+
Project-URL: Issues, https://github.com/shaheersaifi07-prog/ctxpack/issues
|
|
8
|
+
Author: shaheersaifi07-prog
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: cli,codebase,context,llm,packing,tokens
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Software Development :: Pre-processors
|
|
22
|
+
Requires-Python: >=3.10
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
|
|
25
|
+
# `ctx-packr`
|
|
26
|
+
|
|
27
|
+
A stdlib-only Python CLI that selects the most relevant files from a codebase for a given task and packs them into a token-budgeted markdown bundle.
|
|
28
|
+
|
|
29
|
+
## Why `ctx-packr`?
|
|
30
|
+
|
|
31
|
+
Several tools occupy nearby space. The difference is not quality — it is which part of
|
|
32
|
+
the problem each one solves.
|
|
33
|
+
|
|
34
|
+
| Tool | Approach | What it lacks vs `ctx-packr` |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| `ctxpack` | Config-driven file selection, with `init` and `watch` commands | No relevance ranking against a stated task |
|
|
37
|
+
| `context-packer` | Smart filtering of build artifacts and binaries, plus a visual project tree | No relevance ranking against a stated task |
|
|
38
|
+
| `llm-ctxpack` | Priority-based packing into a token budget | Does not walk a codebase; the caller supplies the chunks |
|
|
39
|
+
| `contextcram` | Priority-based packing into a token budget | Does not walk a codebase; the caller supplies the chunks |
|
|
40
|
+
| `ctx-pack` (Rust) | Versioned bidirectional patch/replace | Solves a different problem — not context packing |
|
|
41
|
+
|
|
42
|
+
`ctx-packr` combines four things that otherwise appear separately: it walks a real
|
|
43
|
+
codebase, ranks what it finds against a specific stated task, enforces a hard token
|
|
44
|
+
budget, and has zero runtime dependencies. Ranking uses camelCase-aware keyword
|
|
45
|
+
scoring, with bonuses for entry-point filenames and for files whose `def`/`class` names
|
|
46
|
+
match the task, offset by a penalty for directory depth. When the budget is too small
|
|
47
|
+
for the full bundle, the output degrades step by step — dropping the contents table,
|
|
48
|
+
then the summary header — rather than failing.
|
|
49
|
+
|
|
50
|
+
*Comparison reflects these projects as of September 2026.*
|
|
51
|
+
|
|
52
|
+
## Requirements
|
|
53
|
+
|
|
54
|
+
- Python 3.10+
|
|
55
|
+
- No third-party packages — `ctx-packr` has zero runtime dependencies
|
|
56
|
+
|
|
57
|
+
## Install
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
pip install ctx-packr
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
That puts a `ctx-packr` command on your PATH:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
ctx-packr --path sample --task "understand the auth flow" --budget 2000 --out bundle.md --manifest manifest.json
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
> **Note on naming.** You install `ctx-packr`, but the importable module is `ctxpack`:
|
|
70
|
+
>
|
|
71
|
+
> ```python
|
|
72
|
+
> import ctxpack
|
|
73
|
+
> ```
|
|
74
|
+
>
|
|
75
|
+
> The two differ because a hyphen is not legal in a Python identifier. This is the
|
|
76
|
+
> same split used by `beautifulsoup4` → `bs4` and `python-dateutil` → `dateutil`.
|
|
77
|
+
|
|
78
|
+
## Quick Start (from a clone, no install)
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
git clone https://github.com/shaheersaifi07-prog/ctxpack.git
|
|
82
|
+
cd ctxpack
|
|
83
|
+
|
|
84
|
+
# Run straight from the source tree
|
|
85
|
+
python -m ctxpack --path sample --task "understand the auth flow" --budget 2000 --out bundle.md --manifest manifest.json
|
|
86
|
+
|
|
87
|
+
# View results
|
|
88
|
+
cat bundle.md
|
|
89
|
+
cat manifest.json
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
The source lives in `src/`, so running from a clone needs that directory importable — either `pip install -e .` once, or prefix the command with `PYTHONPATH=src` (`$env:PYTHONPATH="src"` in PowerShell).
|
|
93
|
+
|
|
94
|
+
If `--out` is omitted, the bundle prints to stdout. If `--manifest` is omitted, a one-line summary like `used 521/2000 tokens, 5 included, 0 excluded` goes to stderr.
|
|
95
|
+
|
|
96
|
+
## CLI Usage
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
ctx-packr --path <folder> --task "<desc>" --budget <int> [--out <file>] [--manifest <file>]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
| Flag | Required | Description |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| `--path` | Yes | Target folder (must exist and be readable) |
|
|
105
|
+
| `--task` | Yes | Free-text description of what you're looking for |
|
|
106
|
+
| `--budget` | Yes | Max tokens for the entire rendered bundle (integer > 0) |
|
|
107
|
+
| `--out` | No | Write bundle to this file; omitted = stdout |
|
|
108
|
+
| `--manifest` | No | Write JSON manifest; omitted = one-line stderr |
|
|
109
|
+
|
|
110
|
+
**Exit codes:** `0` success, `1` invalid args, `2` bad path. Errors are a single human-readable line on stderr — no tracebacks.
|
|
111
|
+
|
|
112
|
+
## Example
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
ctx-packr --path sample --task "auth and api" --budget 2000
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Produces a markdown bundle like this (truncated for display):
|
|
119
|
+
|
|
120
|
+
````
|
|
121
|
+
## src/auth.py
|
|
122
|
+
```
|
|
123
|
+
def login(username: str, password: str) -> bool:
|
|
124
|
+
if not username or not password:
|
|
125
|
+
return False
|
|
126
|
+
return _verify_credentials(username, password)
|
|
127
|
+
```
|
|
128
|
+
````
|
|
129
|
+
|
|
130
|
+
And if `--manifest` is given, a JSON file:
|
|
131
|
+
|
|
132
|
+
```json
|
|
133
|
+
{
|
|
134
|
+
"budget": 2000,
|
|
135
|
+
"used": 521,
|
|
136
|
+
"included": [
|
|
137
|
+
{"path": "tests/test_auth.py", "tokens": 69, "reason": "fits within budget"},
|
|
138
|
+
{"path": "src/auth.py", "tokens": 104, "reason": "fits within budget"},
|
|
139
|
+
{"path": "src/api.py", "tokens": 111, "reason": "fits within budget"}
|
|
140
|
+
],
|
|
141
|
+
"excluded": [
|
|
142
|
+
{"path": "package-lock.json", "reason": "lockfile shape detected"}
|
|
143
|
+
]
|
|
144
|
+
}
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Every file under `--path` appears exactly once — included with reason and token count, or excluded with reason.
|
|
148
|
+
|
|
149
|
+
## How It Works
|
|
150
|
+
|
|
151
|
+
1. **Walk** — recursive directory walk with 4 structural noise detectors (binary content, lockfile shape, dependency trees, build artifacts). No name blacklists.
|
|
152
|
+
2. **Rank** — tokenizes the task (camelCase-aware), scores files by keyword overlap with path and content, applies an entrypoint bonus and a depth penalty.
|
|
153
|
+
3. **Pack** — greedily fills the token budget. Files that don't fully fit get a 70/30 head-and-tail truncation with a marker line; files that can't be meaningfully truncated (<200 tokens) are excluded cleanly.
|
|
154
|
+
4. **Render** — assembles markdown sections. Optionally includes a project tree overview for folders with 15+ relevant files (capped at 5% of budget).
|
|
155
|
+
5. **Manifest** — JSON accounting for every file. `used` is the exact token count of the rendered bundle, verified by re-tokenizing before writing.
|
|
156
|
+
|
|
157
|
+
See [SPEC.md](SPEC.md) for the full spec, [CLARIFY.md](.specify/memory/clarify.md) for design decisions, and [PLAN.md](.specify/memory/plan.md) for the implementation plan.
|
|
158
|
+
|
|
159
|
+
## Known Limitations
|
|
160
|
+
|
|
161
|
+
- **No semantic understanding.** Matching is keyword-based with camelCase splitting — a task about "authentication" won't match a file about "login" unless the words overlap. This is an accepted tradeoff for an offline, stdlib-only, deterministic tool.
|
|
162
|
+
- **No language-aware parsing.** No import graphs, AST analysis, or per-language heuristics. Path and content keyword overlap is the only signal.
|
|
163
|
+
- **No incremental runs.** Every invocation does a fresh walk from scratch — no caching between runs.
|
|
164
|
+
- **No `.gitignore` support.** Hidden test folders may contain files a developer would typically exclude. The tool treats all readable files equally; structural noise is caught by content-shape detection, not ignore files.
|
|
165
|
+
|
|
166
|
+
This is a **Spec-Driven Development** project. The spec was written before any code, and all decisions are documented in the spec chain under `.specify/memory/`.
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
# `ctx-packr`
|
|
2
|
+
|
|
3
|
+
A stdlib-only Python CLI that selects the most relevant files from a codebase for a given task and packs them into a token-budgeted markdown bundle.
|
|
4
|
+
|
|
5
|
+
## Why `ctx-packr`?
|
|
6
|
+
|
|
7
|
+
Several tools occupy nearby space. The difference is not quality — it is which part of
|
|
8
|
+
the problem each one solves.
|
|
9
|
+
|
|
10
|
+
| Tool | Approach | What it lacks vs `ctx-packr` |
|
|
11
|
+
|---|---|---|
|
|
12
|
+
| `ctxpack` | Config-driven file selection, with `init` and `watch` commands | No relevance ranking against a stated task |
|
|
13
|
+
| `context-packer` | Smart filtering of build artifacts and binaries, plus a visual project tree | No relevance ranking against a stated task |
|
|
14
|
+
| `llm-ctxpack` | Priority-based packing into a token budget | Does not walk a codebase; the caller supplies the chunks |
|
|
15
|
+
| `contextcram` | Priority-based packing into a token budget | Does not walk a codebase; the caller supplies the chunks |
|
|
16
|
+
| `ctx-pack` (Rust) | Versioned bidirectional patch/replace | Solves a different problem — not context packing |
|
|
17
|
+
|
|
18
|
+
`ctx-packr` combines four things that otherwise appear separately: it walks a real
|
|
19
|
+
codebase, ranks what it finds against a specific stated task, enforces a hard token
|
|
20
|
+
budget, and has zero runtime dependencies. Ranking uses camelCase-aware keyword
|
|
21
|
+
scoring, with bonuses for entry-point filenames and for files whose `def`/`class` names
|
|
22
|
+
match the task, offset by a penalty for directory depth. When the budget is too small
|
|
23
|
+
for the full bundle, the output degrades step by step — dropping the contents table,
|
|
24
|
+
then the summary header — rather than failing.
|
|
25
|
+
|
|
26
|
+
*Comparison reflects these projects as of September 2026.*
|
|
27
|
+
|
|
28
|
+
## Requirements
|
|
29
|
+
|
|
30
|
+
- Python 3.10+
|
|
31
|
+
- No third-party packages — `ctx-packr` has zero runtime dependencies
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pip install ctx-packr
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
That puts a `ctx-packr` command on your PATH:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
ctx-packr --path sample --task "understand the auth flow" --budget 2000 --out bundle.md --manifest manifest.json
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
> **Note on naming.** You install `ctx-packr`, but the importable module is `ctxpack`:
|
|
46
|
+
>
|
|
47
|
+
> ```python
|
|
48
|
+
> import ctxpack
|
|
49
|
+
> ```
|
|
50
|
+
>
|
|
51
|
+
> The two differ because a hyphen is not legal in a Python identifier. This is the
|
|
52
|
+
> same split used by `beautifulsoup4` → `bs4` and `python-dateutil` → `dateutil`.
|
|
53
|
+
|
|
54
|
+
## Quick Start (from a clone, no install)
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
git clone https://github.com/shaheersaifi07-prog/ctxpack.git
|
|
58
|
+
cd ctxpack
|
|
59
|
+
|
|
60
|
+
# Run straight from the source tree
|
|
61
|
+
python -m ctxpack --path sample --task "understand the auth flow" --budget 2000 --out bundle.md --manifest manifest.json
|
|
62
|
+
|
|
63
|
+
# View results
|
|
64
|
+
cat bundle.md
|
|
65
|
+
cat manifest.json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
The source lives in `src/`, so running from a clone needs that directory importable — either `pip install -e .` once, or prefix the command with `PYTHONPATH=src` (`$env:PYTHONPATH="src"` in PowerShell).
|
|
69
|
+
|
|
70
|
+
If `--out` is omitted, the bundle prints to stdout. If `--manifest` is omitted, a one-line summary like `used 521/2000 tokens, 5 included, 0 excluded` goes to stderr.
|
|
71
|
+
|
|
72
|
+
## CLI Usage
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
ctx-packr --path <folder> --task "<desc>" --budget <int> [--out <file>] [--manifest <file>]
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
| Flag | Required | Description |
|
|
79
|
+
|---|---|---|
|
|
80
|
+
| `--path` | Yes | Target folder (must exist and be readable) |
|
|
81
|
+
| `--task` | Yes | Free-text description of what you're looking for |
|
|
82
|
+
| `--budget` | Yes | Max tokens for the entire rendered bundle (integer > 0) |
|
|
83
|
+
| `--out` | No | Write bundle to this file; omitted = stdout |
|
|
84
|
+
| `--manifest` | No | Write JSON manifest; omitted = one-line stderr |
|
|
85
|
+
|
|
86
|
+
**Exit codes:** `0` success, `1` invalid args, `2` bad path. Errors are a single human-readable line on stderr — no tracebacks.
|
|
87
|
+
|
|
88
|
+
## Example
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
ctx-packr --path sample --task "auth and api" --budget 2000
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Produces a markdown bundle like this (truncated for display):
|
|
95
|
+
|
|
96
|
+
````
|
|
97
|
+
## src/auth.py
|
|
98
|
+
```
|
|
99
|
+
def login(username: str, password: str) -> bool:
|
|
100
|
+
if not username or not password:
|
|
101
|
+
return False
|
|
102
|
+
return _verify_credentials(username, password)
|
|
103
|
+
```
|
|
104
|
+
````
|
|
105
|
+
|
|
106
|
+
And if `--manifest` is given, a JSON file:
|
|
107
|
+
|
|
108
|
+
```json
|
|
109
|
+
{
|
|
110
|
+
"budget": 2000,
|
|
111
|
+
"used": 521,
|
|
112
|
+
"included": [
|
|
113
|
+
{"path": "tests/test_auth.py", "tokens": 69, "reason": "fits within budget"},
|
|
114
|
+
{"path": "src/auth.py", "tokens": 104, "reason": "fits within budget"},
|
|
115
|
+
{"path": "src/api.py", "tokens": 111, "reason": "fits within budget"}
|
|
116
|
+
],
|
|
117
|
+
"excluded": [
|
|
118
|
+
{"path": "package-lock.json", "reason": "lockfile shape detected"}
|
|
119
|
+
]
|
|
120
|
+
}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Every file under `--path` appears exactly once — included with reason and token count, or excluded with reason.
|
|
124
|
+
|
|
125
|
+
## How It Works
|
|
126
|
+
|
|
127
|
+
1. **Walk** — recursive directory walk with 4 structural noise detectors (binary content, lockfile shape, dependency trees, build artifacts). No name blacklists.
|
|
128
|
+
2. **Rank** — tokenizes the task (camelCase-aware), scores files by keyword overlap with path and content, applies an entrypoint bonus and a depth penalty.
|
|
129
|
+
3. **Pack** — greedily fills the token budget. Files that don't fully fit get a 70/30 head-and-tail truncation with a marker line; files that can't be meaningfully truncated (<200 tokens) are excluded cleanly.
|
|
130
|
+
4. **Render** — assembles markdown sections. Optionally includes a project tree overview for folders with 15+ relevant files (capped at 5% of budget).
|
|
131
|
+
5. **Manifest** — JSON accounting for every file. `used` is the exact token count of the rendered bundle, verified by re-tokenizing before writing.
|
|
132
|
+
|
|
133
|
+
See [SPEC.md](SPEC.md) for the full spec, [CLARIFY.md](.specify/memory/clarify.md) for design decisions, and [PLAN.md](.specify/memory/plan.md) for the implementation plan.
|
|
134
|
+
|
|
135
|
+
## Known Limitations
|
|
136
|
+
|
|
137
|
+
- **No semantic understanding.** Matching is keyword-based with camelCase splitting — a task about "authentication" won't match a file about "login" unless the words overlap. This is an accepted tradeoff for an offline, stdlib-only, deterministic tool.
|
|
138
|
+
- **No language-aware parsing.** No import graphs, AST analysis, or per-language heuristics. Path and content keyword overlap is the only signal.
|
|
139
|
+
- **No incremental runs.** Every invocation does a fresh walk from scratch — no caching between runs.
|
|
140
|
+
- **No `.gitignore` support.** Hidden test folders may contain files a developer would typically exclude. The tool treats all readable files equally; structural noise is caught by content-shape detection, not ignore files.
|
|
141
|
+
|
|
142
|
+
This is a **Spec-Driven Development** project. The spec was written before any code, and all decisions are documented in the spec chain under `.specify/memory/`.
|
ctx_packr-0.1.1/SPEC.md
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
# specify.md — `ctxpack`
|
|
2
|
+
|
|
3
|
+
**Written before implementation. Governed by CONSTITUTION.md. This is a graded deliverable, not documentation written after the fact.**
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 1. CLI Contract
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
ctxpack --path <folder> --task "<task description>" --budget <int> [--out <file>] [--manifest <file>]
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
| Flag | Required | Behavior |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| `--path` | Yes | Folder to pack. Must exist and be readable. |
|
|
16
|
+
| `--task` | Yes | Free-text description of developer intent. Used for ranking. |
|
|
17
|
+
| `--budget` | Yes | Max tokens for the **entire rendered bundle**, integer > 0. |
|
|
18
|
+
| `--out` | No | Write bundle here. Omitted → stdout. |
|
|
19
|
+
| `--manifest` | No | Write manifest JSON here. Omitted → one-line summary to stderr. |
|
|
20
|
+
|
|
21
|
+
### Exit Codes
|
|
22
|
+
|
|
23
|
+
| Code | Meaning |
|
|
24
|
+
|---|---|
|
|
25
|
+
| `0` | Success. Bundle produced within budget. |
|
|
26
|
+
| `1` | Invalid arguments — missing required flag, non-integer/non-positive `--budget`, malformed `--task`. |
|
|
27
|
+
| `2` | `--path` does not exist, is not a directory, or is unreadable (permissions). |
|
|
28
|
+
|
|
29
|
+
Errors print a single human-readable line to stderr. No tracebacks, per Constitution Principle IV.
|
|
30
|
+
|
|
31
|
+
### Argument Validation Order
|
|
32
|
+
1. Parse flags → missing required flag = exit 1 immediately, before touching the filesystem.
|
|
33
|
+
2. Validate `--budget` is a positive integer → exit 1 if not.
|
|
34
|
+
3. Validate `--path` exists and is a readable directory → exit 2 if not.
|
|
35
|
+
4. Only then begin the walk.
|
|
36
|
+
|
|
37
|
+
This order matters: a bad `--budget` should never be masked by a slower path-not-found check.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## 2. Token Counting
|
|
42
|
+
|
|
43
|
+
```python
|
|
44
|
+
tokens = math.ceil(len(text) / 4)
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Applied to:
|
|
48
|
+
- Each individual file's content (for ranking/truncation decisions)
|
|
49
|
+
- The **fully rendered bundle** — headers, file path labels, separators, the optional tree diagram, everything that reaches `--out` or stdout
|
|
50
|
+
|
|
51
|
+
The budget check happens against the final rendered string, not against a running sum of file contents. This catches the case where formatting overhead itself would push the bundle over budget.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## 3. Ranking Strategy — Weighted Keyword Overlap
|
|
56
|
+
|
|
57
|
+
**Decision:** Rank files by a weighted score combining (a) keyword overlap between `--task` and file content/path, (b) a small structural prior, and (c) a path-depth penalty.
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
score(file) = 3 × path_match_count
|
|
61
|
+
+ 1 × content_match_count (capped at 20)
|
|
62
|
+
+ 2 × (file is entrypoint-like: main.py, index.js, __init__.py, README)
|
|
63
|
+
- 0.5 × directory_depth
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
- `--task` is lowercased, split on non-alphanumeric characters, stopwords removed (a small hardcoded stopword list of ~30 English function words — this is not "noise detection," it's tokenization hygiene, and is documented as such).
|
|
67
|
+
- `path_match_count`: task keywords found in the file's path/filename (case-insensitive substring match).
|
|
68
|
+
- `content_match_count`: task keywords found in file content, capped so one keyword-stuffed file can't dominate purely on repetition.
|
|
69
|
+
- Entrypoint bonus: files conventionally likely to explain "what this project does" get a small nudge, because task descriptions are often about behavior, and behavior is usually reachable from an entrypoint.
|
|
70
|
+
- Depth penalty: deeply nested files are slightly deprioritized, since top-level and near-top-level files are more often architecturally central.
|
|
71
|
+
|
|
72
|
+
**Why this over the alternatives considered:**
|
|
73
|
+
|
|
74
|
+
| Alternative | Why rejected |
|
|
75
|
+
|---|---|
|
|
76
|
+
| Import graph analysis | Requires language-aware parsing per file type (Python imports ≠ JS imports ≠ Go imports). Violates stdlib-only cleanly only for Python; brief implies mixed-language repos. Too much complexity for the time budget, and it silently fails on non-code files (README, config, docs) that are often exactly what's relevant to a task like "understand the auth flow." |
|
|
77
|
+
| Pure filename matching | Too brittle — a file named `utils.py` could be the single most task-relevant file and would never surface. |
|
|
78
|
+
| File recency (mtime) | Recency correlates with "recently touched," not "relevant to this task." A judge's test folder is likely a static snapshot with meaningless or identical mtimes — this signal would be near-random noise on the actual test set. |
|
|
79
|
+
| Pure content keyword count, no path weighting | Rewards keyword-stuffed files (tests, changelogs, generated docs) over structurally central ones. Path weighting is a cheap, dependency-free proxy for "this file is what it says it is." |
|
|
80
|
+
|
|
81
|
+
**Known weakness, stated up front:** this method has no semantic understanding — a task about "authentication" won't match a file about "login" unless the words overlap. This is an accepted tradeoff for a stdlib-only, offline, deterministic tool. It is not treated as a hidden flaw to discover later; it is the stated boundary of the approach.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## 4. Truncation Policy — Smart Head-and-Tail Slice
|
|
86
|
+
|
|
87
|
+
**Decision:** When a file doesn't fit in the remaining budget, `ctxpack` does not skip it outright and does not blindly take the head. It takes a **head + tail slice** sized to the remaining budget, with a visible truncation marker, provided the file scored high enough to be worth partial inclusion (see threshold below).
|
|
88
|
+
|
|
89
|
+
- Take roughly 70% of the available token allowance from the start of the file, 30% from the end, joined by a marker line: `... [truncated: N tokens omitted] ...`
|
|
90
|
+
- Rationale: file heads usually carry imports, class/function signatures, and top-of-file docstrings; tails often carry `main()` blocks, exports, or closing logic. A pure head-slice biases against exactly the code that shows what a file *does* at runtime.
|
|
91
|
+
- **Threshold:** partial inclusion is only attempted if the remaining budget can fit at least 200 tokens of a truncated file. Below that, a partial slice is judgment-free noise (a few import lines and a truncation marker convey nothing) — the file is excluded instead, with reason `"insufficient remaining budget for meaningful partial inclusion (needed ≥200 tokens, had N)"`.
|
|
92
|
+
|
|
93
|
+
**Why this over the alternatives considered:**
|
|
94
|
+
|
|
95
|
+
| Alternative | Why rejected |
|
|
96
|
+
|---|---|
|
|
97
|
+
| Always exclude if it doesn't fully fit | Throws away a highly relevant large file entirely in favor of lower-ranked small files, which can produce a worse bundle for the stated task even though it "used the budget efficiently." |
|
|
98
|
+
| Always take the head | Cheap and deterministic, but systematically biased against seeing how a file's logic concludes — a real cost for exactly the kind of "understand what this does" tasks the tool exists for. |
|
|
99
|
+
| Summarize the file (extractive/heuristic) | Non-trivial to make deterministic and language-agnostic without third-party NLP tooling; risks misrepresenting the file's actual content, which conflicts with Constitution Principle II (silence/misrepresentation is a bug). |
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## 5. Noise Detection — Structural Heuristics, Not a Name Blacklist
|
|
104
|
+
|
|
105
|
+
**Decision:** Noise is detected by structural signal, not a hardcoded list of filenames, so the approach generalizes beyond `node_modules`/`.git` to noise `ctxpack` has never seen named before.
|
|
106
|
+
|
|
107
|
+
A file/directory is flagged as noise if **any** of:
|
|
108
|
+
1. **Path contains a VCS or dependency directory marker**: detected by checking for a small set of *structural* markers — a directory containing a `.git` HEAD-like structure, or a directory whose immediate contents look like an installed package tree (a `package.json`-sibling `node_modules`-shaped nesting, i.e., a directory containing hundreds of subdirectories each containing their own manifest-like file). This is a structural detector, not a name check — `node_modules` is caught because of what's inside it, not because of its name (a differently-named vendor directory with the same shape is caught too).
|
|
109
|
+
2. **File is a lockfile by content shape**: single-file, machine-generated JSON/text with an extremely high ratio of punctuation/digits to words, and a repeating flat structure (long lists of `name@version` or hash-like tokens). Caught by content shape, not filename.
|
|
110
|
+
3. **Binary or non-text content**: detected by attempting a UTF-8 decode on a sample of the file; failure (or a high proportion of non-printable bytes) marks it as non-text. Handled under Requirement 5 (never crash) regardless of noise status.
|
|
111
|
+
4. **Build artifact directories**: detected structurally — a directory whose contents are overwhelmingly files with no source-like extension sitting alongside a sibling source directory of the same name (e.g., `dist/` next to `src/`), or a directory containing only compiled/minified-looking files (very long single-line files with minimal whitespace ratio — a cheap, dependency-free proxy for "this was generated, not written").
|
|
112
|
+
|
|
113
|
+
Each noise exclusion in the manifest states **which structural signal fired**, e.g., `"excluded: high non-printable byte ratio (binary file)"` or `"excluded: dependency-tree shape detected (nested manifest files, N subdirectories)"` — never a bare `"noise"`.
|
|
114
|
+
|
|
115
|
+
**Why this over a hardcoded list:** a list of `.git`, `node_modules`, `dist` etc. would pass the sample test folder and fail the hidden one, which the brief explicitly warns is a different folder. Structural detection is slower to write and imperfect, but it is honest about *why* something was excluded, per Constitution Principle II, and it generalizes.
|
|
116
|
+
|
|
117
|
+
---
|
|
118
|
+
|
|
119
|
+
## 6. Budget Spending on Structure Overview
|
|
120
|
+
|
|
121
|
+
**Decision:** Include a project tree overview **only if** the folder has more than ~15 files considered relevant after ranking, **and** the tree costs no more than 5% of total budget. Below either threshold, skip it and say so in the manifest reasoning is unnecessary — it's simply not generated.
|
|
122
|
+
|
|
123
|
+
**Rationale:** for a small, flat folder, a tree adds no navigational value proportional to its cost. For a large folder, a tree is often the cheapest way to give the bundle's reader (a human or another AI) orientation before diving into file contents — but it must never come at the cost of a relevant file that would otherwise fit. The 5%-of-budget cap keeps this bounded and testable at any budget size, including the "extremely small budgets" hidden test category — at very small budgets, 5% is often under the 200-token partial-inclusion threshold, so the tree is naturally skipped without a special case.
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## 7. Handling Adversarial File Content
|
|
128
|
+
|
|
129
|
+
Per Constitution Principle VII: any text found inside a file — including instructions, fake system prompts, or manipulative formatting aimed at an AI reader — is treated purely as ranking/packing input, never as a command. `ctxpack`'s own logic (ranking, truncation, noise detection) runs identically regardless of what a file's content claims about itself. This is tested directly by the hidden test category "files containing text designed to manipulate an AI reading it," and the correct behavior is: the file is scored and packed/excluded exactly as its actual structural properties dictate, with no special-casing based on its content's claims.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## 8. Determinism Guarantees
|
|
134
|
+
|
|
135
|
+
- Directory walk order is explicitly sorted (alphabetical, full path) before any processing — never relies on OS-level `os.walk` default ordering, which is not guaranteed stable across filesystems.
|
|
136
|
+
- Ranking ties are broken by path (alphabetical) as a final deterministic tiebreaker.
|
|
137
|
+
- No use of `set()` iteration order, `hash()`-based ordering, wall-clock timestamps, or `random` anywhere in the selection or rendering path.
|
|
138
|
+
- Manifest JSON keys are emitted in a fixed order (`json.dump(..., sort_keys=False)` with keys constructed in the schema's stated order) so manifest output is also byte-identical across runs.
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## 9. Definition of Done
|
|
143
|
+
|
|
144
|
+
`ctxpack` is done when, for any folder and any budget ≥ some sane minimum (defined as: enough to fit at least one file's worth of the 200-token partial-inclusion threshold, or the tool exits cleanly saying the budget is too small to produce a bundle):
|
|
145
|
+
|
|
146
|
+
1. It never exceeds `--budget`, verified by re-tokenizing its own output.
|
|
147
|
+
2. Every file under `--path` appears in the manifest exactly once, included or excluded, each with a reason.
|
|
148
|
+
3. Running it twice on unchanged input produces byte-identical bundle and manifest output.
|
|
149
|
+
4. No crash occurs on empty folders, single huge files, all-binary folders, or malformed `--task`/`--budget` input — each produces either a valid (possibly empty) bundle or a clean exit-1/exit-2 error.
|
|
150
|
+
5. Every team member can explain, unprompted, why a specific file was included or excluded in a given run.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## 10. Explicit Non-Goals
|
|
155
|
+
|
|
156
|
+
- No semantic/embedding-based relevance (would require a model, violating offline/stdlib constraints).
|
|
157
|
+
- No language-aware parsing (import graphs, AST analysis) — rejected in Section 3, consistent across the spec.
|
|
158
|
+
- No caching or incremental re-runs between invocations — every run is a fresh, from-scratch walk, to keep the determinism guarantee simple to verify.
|