ctxpack-cli 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ctxpack_cli-1.0.0/LICENSE +21 -0
- ctxpack_cli-1.0.0/PKG-INFO +353 -0
- ctxpack_cli-1.0.0/README.md +325 -0
- ctxpack_cli-1.0.0/ctxpack.py +1037 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/PKG-INFO +353 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/SOURCES.txt +11 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/dependency_links.txt +1 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/entry_points.txt +2 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/requires.txt +4 -0
- ctxpack_cli-1.0.0/ctxpack_cli.egg-info/top_level.txt +1 -0
- ctxpack_cli-1.0.0/pyproject.toml +51 -0
- ctxpack_cli-1.0.0/setup.cfg +4 -0
- ctxpack_cli-1.0.0/tests/test_ctxpack.py +2594 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Billy Box
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ctxpack-cli
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Dependency-free repo-to-prompt pack builder for AI workflows
|
|
5
|
+
Author-email: Billy Box <billybox1926@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/billybox1926-jpg/ctxpack
|
|
8
|
+
Project-URL: Repository, https://github.com/billybox1926-jpg/ctxpack
|
|
9
|
+
Project-URL: Issues, https://github.com/billybox1926-jpg/ctxpack/issues
|
|
10
|
+
Project-URL: Documentation, https://github.com/billybox1926-jpg/ctxpack#readme
|
|
11
|
+
Keywords: ai,context,llm,prompt,cli,rag
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: pytest; extra == "dev"
|
|
26
|
+
Requires-Dist: ruff; extra == "dev"
|
|
27
|
+
Dynamic: license-file
|
|
28
|
+
|
|
29
|
+
# ctxpack
|
|
30
|
+
|
|
31
|
+

|
|
32
|
+
|
|
33
|
+
[](LICENSE)
|
|
34
|
+
[](https://www.python.org/)
|
|
35
|
+
[](https://github.com/billybox1926-jpg/ctxpack/releases)
|
|
36
|
+
[](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml)
|
|
37
|
+
[](https://github.com/astral-sh/ruff)
|
|
38
|
+
|
|
39
|
+
**Dependency-free repo-to-prompt pack builder.**
|
|
40
|
+
|
|
41
|
+
`ctxpack` is the missing bridge between raw repository scanning and AI-ready context. It takes a local project, respects ignore rules, token-budgets the output, and emits two clean artifacts:
|
|
42
|
+
|
|
43
|
+
- `ctxpack.context.json` â machine-readable inventory for agents/tools
|
|
44
|
+
- `ctxpack.context.md` â human-readable prompt pack for pasting into an LLM
|
|
45
|
+
|
|
46
|
+
No dependencies. No network. **Secrets excluded by default.**
|
|
47
|
+
|
|
48
|
+
## Features
|
|
49
|
+
|
|
50
|
+
- ð Recursively scans the current directory
|
|
51
|
+
- ðŦ Respects `.ctxignore` (gitignore-style patterns)
|
|
52
|
+
- ðŠķ Skips binary and overly large files by default
|
|
53
|
+
- ð **Excludes secrets by default**: `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`
|
|
54
|
+
- ð§Ū Estimates token usage (approx `chars / 4` â **not** a model tokenizer count)
|
|
55
|
+
- âïļ Respects a max estimated token budget (`--budget`) and truncates gracefully
|
|
56
|
+
- âïļ Simple configuration via optional `ctxpack.json`
|
|
57
|
+
|
|
58
|
+
## Installation
|
|
59
|
+
|
|
60
|
+
No installation required. Just download the single file:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
curl -O https://raw.githubusercontent.com/billybox1926-jpg/ctxpack/main/ctxpack.py
|
|
64
|
+
chmod +x ctxpack.py
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Or install it to get the `ctxpack` command on your PATH:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pip install ctxpack-cli
|
|
71
|
+
ctxpack --version
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
> The PyPI distribution is named **`ctxpack-cli`** because `ctxpack` was already
|
|
75
|
+
> taken by an unrelated project. The installed command is still `ctxpack`.
|
|
76
|
+
|
|
77
|
+
## Usage
|
|
78
|
+
|
|
79
|
+
### Initialize a project
|
|
80
|
+
Create default `.ctxignore` and `ctxpack.json` files in your current directory:
|
|
81
|
+
```bash
|
|
82
|
+
python ctxpack.py init
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Pack a repository
|
|
86
|
+
Scan the current directory and generate context files:
|
|
87
|
+
```bash
|
|
88
|
+
python ctxpack.py pack
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Advanced options
|
|
92
|
+
```bash
|
|
93
|
+
# Set a specific token budget
|
|
94
|
+
python ctxpack.py pack --budget 12000
|
|
95
|
+
|
|
96
|
+
# Ignore ctxpack.json settings and use CLI defaults/flags only
|
|
97
|
+
python ctxpack.py pack --no-config --budget 4000
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Configuration
|
|
101
|
+
|
|
102
|
+
### `.ctxignore`
|
|
103
|
+
|
|
104
|
+
Uses ctxignore patterns â a tested subset of gitignore syntax. Lines starting with `#` are comments. Blank lines are ignored.
|
|
105
|
+
|
|
106
|
+
**Supported pattern types:**
|
|
107
|
+
|
|
108
|
+
| Pattern | Matches | Example |
|
|
109
|
+
|---------|---------|---------|
|
|
110
|
+
| `foo` | Exact path at any depth | `build/` matches `build/`, `src/build/` |
|
|
111
|
+
| `foo/` | Directory and everything inside | `venv/` skips `venv/lib/x.py` |
|
|
112
|
+
| `foo/**` | Directory and everything inside | `node_modules/**` skips `node_modules/pkg/x.js` |
|
|
113
|
+
| `*.ext` | Files with extension at any depth | `*.log` skips `debug.log` and `logs/debug.log` |
|
|
114
|
+
| `/foo` | Exact path at the scan root only | `/build/` skips `build/` but NOT `src/build/` |
|
|
115
|
+
| `**` | Spans path segments | `**/.aws/**` skips `.aws/` and `nested/.aws/` |
|
|
116
|
+
| `!foo` | Negation â re-includes a previous exclusion | `*.pem` then `!fixture.pem` |
|
|
117
|
+
| `\*`, `\[`, etc. | Escaped wildcard/bracket (literal) | `file\*.txt` matches the literal `file*.txt` |
|
|
118
|
+
|
|
119
|
+
**Pattern precedence:** Patterns are processed in order. The last matching pattern wins â so `!foo` can override an earlier `foo`.
|
|
120
|
+
|
|
121
|
+
**Include vs. exclude:** `--include` patterns restrict to specific files. `--exclude` patterns remove files. Excludes always override includes.
|
|
122
|
+
|
|
123
|
+
**Not supported:** Character classes (`[...]`), trailing whitespace significance, or full regex.
|
|
124
|
+
|
|
125
|
+
```text
|
|
126
|
+
# Ignore virtual environments
|
|
127
|
+
venv/
|
|
128
|
+
.venv/
|
|
129
|
+
|
|
130
|
+
# Ignore build artifacts at the root only
|
|
131
|
+
/build/
|
|
132
|
+
/dist/
|
|
133
|
+
|
|
134
|
+
# Ignore all log files anywhere
|
|
135
|
+
*.log
|
|
136
|
+
|
|
137
|
+
# But keep the main log file
|
|
138
|
+
!important.log
|
|
139
|
+
|
|
140
|
+
# Allow template env files
|
|
141
|
+
!.env.example
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
**Default secret exclusions:** ctxpack excludes these by default (not shown in generated `.ctxignore`):
|
|
145
|
+
`.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`, `*.jks`, `*.keystore`, `*.gpg`, `*.asc`, `**/.aws/**`, `**/.ssh/**`, `**/.netrc`, `**/.npmrc`, `**/.pypirc`
|
|
146
|
+
|
|
147
|
+
### `ctxpack.json`
|
|
148
|
+
|
|
149
|
+
Optional configuration file. Created via `python ctxpack.py init`.
|
|
150
|
+
|
|
151
|
+
```json
|
|
152
|
+
{
|
|
153
|
+
"budget_tokens": 8000,
|
|
154
|
+
"ignore_file": ".ctxignore",
|
|
155
|
+
"include_binary": false
|
|
156
|
+
}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
## Token Budget Semantics
|
|
160
|
+
|
|
161
|
+
### How token estimation works
|
|
162
|
+
|
|
163
|
+
ctxpack uses a script-aware heuristic to estimate token count:
|
|
164
|
+
|
|
165
|
+
- **Latin/ASCII text**: **~4 characters per token**, approximating typical LLM tokenization for English text and code.
|
|
166
|
+
- **CJK text** (Chinese, Japanese kana, Korean hangul): **~1.5 characters per token**, since CJK scripts tokenize far less densely than Latin text (typically 1-2 tokens per character). Mixed text is estimated per-script and summed.
|
|
167
|
+
|
|
168
|
+
Key details:
|
|
169
|
+
|
|
170
|
+
- **Empty content = 0 tokens**: Files with no content contribute zero tokens
|
|
171
|
+
- **Minimum 1 token**: Any non-empty file gets at least 1 token estimate
|
|
172
|
+
- **Truncation marker overhead**: When files are truncated, the truncation message (`...[TRUNCATED by ctxpack to fit budget]...`) accounts for ~11 tokens
|
|
173
|
+
|
|
174
|
+
### Budget enforcement behavior
|
|
175
|
+
|
|
176
|
+
When the total estimated tokens exceed the budget:
|
|
177
|
+
|
|
178
|
+
1. Files are processed in sorted path order
|
|
179
|
+
2. Files that fit entirely within remaining budget are included as-is
|
|
180
|
+
3. The first file that would exceed the budget is **truncated** (not dropped), with a truncation marker appended
|
|
181
|
+
4. Remaining files are marked as **omitted** (empty content, listed in output)
|
|
182
|
+
|
|
183
|
+
This ensures:
|
|
184
|
+
- **No silent drops**: Every discovered file appears in the output (either full, truncated, or omitted)
|
|
185
|
+
- **Budget never exceeded**: The truncation marker's token cost is reserved before slicing
|
|
186
|
+
- **Transparent about missing content**: Omitted files are listed with their original size/token estimates
|
|
187
|
+
|
|
188
|
+
### Edge cases
|
|
189
|
+
|
|
190
|
+
| Scenario | Behavior |
|
|
191
|
+
|----------|----------|
|
|
192
|
+
| Empty repository | Outputs header only, 0 tokens used |
|
|
193
|
+
| Single file > budget | File truncated to fit budget + marker |
|
|
194
|
+
| Exact budget match | All files included without truncation |
|
|
195
|
+
| Very small budget (< 20 tokens) | First file may be truncated immediately or omitted |
|
|
196
|
+
|
|
197
|
+
### Limitations
|
|
198
|
+
|
|
199
|
+
- This is a **rough estimate**, not an exact token count. Actual LLM tokenizers (e.g., tiktoken, sentencepiece) may vary by Âą20-30%
|
|
200
|
+
- Code with many symbols, non-English text, or unusual formatting may have different actual token counts
|
|
201
|
+
- For critical workflows, verify actual token usage with your target model's tokenizer
|
|
202
|
+
|
|
203
|
+
## Generated Artifacts: To Commit or Not?
|
|
204
|
+
|
|
205
|
+
**Short answer:** Generally **no**. Generated `*.context.*` files are ephemeral artifacts meant for immediate use, not long-term storage.
|
|
206
|
+
|
|
207
|
+
### Recommended practice
|
|
208
|
+
|
|
209
|
+
- â **Do not commit** `*.context.json` or `*.context.md` files to your repository
|
|
210
|
+
- â
**Do add** them to `.gitignore` (they're already in the default template from `ctxpack init`)
|
|
211
|
+
- â
**Do regenerate** them fresh whenever you need to share context with an LLM
|
|
212
|
+
|
|
213
|
+
### Why not commit generated packs?
|
|
214
|
+
|
|
215
|
+
1. **Stale content**: Context packs become outdated as soon as your code changes
|
|
216
|
+
2. **Noise in history**: Frequent regeneration creates churn in git history
|
|
217
|
+
3. **Repository bloat**: Large context packs can significantly increase repo size
|
|
218
|
+
4. **False sense of accuracy**: Old packs may misrepresent current project state
|
|
219
|
+
|
|
220
|
+
### When might you commit a pack?
|
|
221
|
+
|
|
222
|
+
Rare exceptions where committing *might* make sense:
|
|
223
|
+
|
|
224
|
+
- ðĶ **Release artifacts**: Including a context pack with a tagged release to capture exact state at release time
|
|
225
|
+
- ð **Debugging aid**: Committing a specific pack to help reproduce and debug an issue
|
|
226
|
+
- ð **Documentation example**: Sample packs in `examples/` directories (like this repo's `examples/sample.context.*`)
|
|
227
|
+
|
|
228
|
+
If you do commit a generated pack, consider:
|
|
229
|
+
- Adding a timestamp/generation note in comments
|
|
230
|
+
- Using git LFS for large files
|
|
231
|
+
- Setting up automated cleanup for stale packs
|
|
232
|
+
|
|
233
|
+
### Default behavior
|
|
234
|
+
|
|
235
|
+
The `ctxpack init` command adds these patterns to your `.gitignore`:
|
|
236
|
+
```text
|
|
237
|
+
# Generated context packs (ephemeral artifacts)
|
|
238
|
+
*.context.json
|
|
239
|
+
*.context.md
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
This keeps your repository clean while allowing you to generate fresh packs on demand.
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## Output Examples
|
|
247
|
+
|
|
248
|
+
The `examples/` directory contains static sample outputs generated by ctxpack. These are committed as documentation references:
|
|
249
|
+
|
|
250
|
+
- [`examples/sample.context.md`](./examples/sample.context.md)
|
|
251
|
+
- [`examples/sample.context.json`](./examples/sample.context.json)
|
|
252
|
+
|
|
253
|
+
To generate your own artifacts from the current project:
|
|
254
|
+
|
|
255
|
+
```bash
|
|
256
|
+
python ctxpack.py pack --output-dir ./out
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
This writes `ctxpack.context.md` and `ctxpack.context.json` into `./out/`. Generated artifacts are gitignored by default.
|
|
260
|
+
|
|
261
|
+
## Generated Artifact Policy
|
|
262
|
+
|
|
263
|
+
**Generated context artifacts are not committed to the repository.** Files matching `*.context.md` and `*.context.json` are excluded via `.gitignore`.
|
|
264
|
+
|
|
265
|
+
This policy prevents:
|
|
266
|
+
- Stale snapshots that diverge from current source
|
|
267
|
+
- Noisy diffs from regenerated output
|
|
268
|
+
- Accidental publication of sensitive material captured at generation time
|
|
269
|
+
|
|
270
|
+
The `examples/` directory holds static reference fixtures that are intentionally committed.
|
|
271
|
+
|
|
272
|
+
## Security
|
|
273
|
+
|
|
274
|
+
### Secret-safe by default
|
|
275
|
+
|
|
276
|
+
`ctxpack` excludes credential-bearing files by default so they never reach the generated pack. The built-in ignore policy covers:
|
|
277
|
+
|
|
278
|
+
- **Local env files**: `.env`, `.env.*` (but not `.env.example`, the conventional template)
|
|
279
|
+
- **Private keys & certificates**: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`
|
|
280
|
+
- **Keystores**: `*.jks`, `*.keystore`
|
|
281
|
+
- **GPG / signing material**: `*.gpg`, `*.asc`
|
|
282
|
+
- **Credential directories**: `**/.aws/**`, `**/.ssh/**`
|
|
283
|
+
- **Auth dotfiles**: `**/.netrc`, `**/.npmrc`, `**/.pypirc`
|
|
284
|
+
|
|
285
|
+
These defaults are applied automatically; you do not need to list them in `.ctxignore`. Custom `.ctxignore` entries are merged with these defaults and can add further exclusions.
|
|
286
|
+
|
|
287
|
+
**Security boundary**: Default secret exclusions are just patterns. Explicit `.ctxignore` negation patterns (e.g., `!.env`) can override these defaults. Users should treat custom re-inclusion rules as an explicit security opt-in â the tool cannot protect against intentional overrides in project configuration files.
|
|
288
|
+
|
|
289
|
+
To opt a specific secret file back in (e.g., a test fixture), add a negation pattern to `.ctxignore`:
|
|
290
|
+
|
|
291
|
+
```text
|
|
292
|
+
!important/test-fixture.pem
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
#### Strict mode
|
|
296
|
+
|
|
297
|
+
If you want the default secret exclusions to be non-overridable, pass `--strict-secrets`:
|
|
298
|
+
|
|
299
|
+
```bash
|
|
300
|
+
python ctxpack.py pack --strict-secrets
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
In strict mode, `.ctxignore` and CLI negation patterns **cannot** re-include any secret file (`.env`, `*.pem`, `*.key`, `*.gpg`, credential directories, etc.). The built-in `.env.example` template carve-out still applies, so conventional example files remain packable.
|
|
304
|
+
|
|
305
|
+
### Path privacy
|
|
306
|
+
|
|
307
|
+
Generated packs use a privacy-preserving `.` representation for the project root by default. To include the absolute path (e.g., for debugging), use:
|
|
308
|
+
|
|
309
|
+
```bash
|
|
310
|
+
python ctxpack.py pack --show-absolute-paths
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
This prevents accidentally leaking local usernames, directory structures, or project locations when sharing context packs.
|
|
314
|
+
|
|
315
|
+
## License
|
|
316
|
+
|
|
317
|
+
MIT License. See [LICENSE](LICENSE) for details.
|
|
318
|
+
|
|
319
|
+
## Release Process
|
|
320
|
+
|
|
321
|
+
### Versioning
|
|
322
|
+
|
|
323
|
+
`ctxpack` uses [Semantic Versioning](https://semver.org/):
|
|
324
|
+
- **MAJOR**: incompatible API changes
|
|
325
|
+
- **MINOR**: backwards-compatible functionality additions
|
|
326
|
+
- **PATCH**: backwards-compatible bug fixes
|
|
327
|
+
|
|
328
|
+
### Building a Release
|
|
329
|
+
|
|
330
|
+
```bash
|
|
331
|
+
# Clean build
|
|
332
|
+
rm -rf dist/
|
|
333
|
+
python -m build
|
|
334
|
+
|
|
335
|
+
# Verify artifacts
|
|
336
|
+
twine check dist/*
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
### Release Checklist
|
|
340
|
+
|
|
341
|
+
- [ ] All tests pass (`pytest -v`)
|
|
342
|
+
- [ ] Lint checks pass (`ruff check .`)
|
|
343
|
+
- [ ] Version bumped in `pyproject.toml`
|
|
344
|
+
- [ ] CHANGELOG updated (if applicable)
|
|
345
|
+
- [ ] Git tag created (`git tag v0.2.0`)
|
|
346
|
+
- [ ] CI green on all jobs (test, lint, security, packaging)
|
|
347
|
+
|
|
348
|
+
### CI Release Validation
|
|
349
|
+
|
|
350
|
+
The `packaging` CI job validates:
|
|
351
|
+
- sdist and wheel build successfully
|
|
352
|
+
- CLI works after installation from wheel
|
|
353
|
+
- All tests pass against installed package
|
|
@@ -0,0 +1,325 @@
|
|
|
1
|
+
# ctxpack
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
[](https://www.python.org/)
|
|
7
|
+
[](https://github.com/billybox1926-jpg/ctxpack/releases)
|
|
8
|
+
[](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml)
|
|
9
|
+
[](https://github.com/astral-sh/ruff)
|
|
10
|
+
|
|
11
|
+
**Dependency-free repo-to-prompt pack builder.**
|
|
12
|
+
|
|
13
|
+
`ctxpack` is the missing bridge between raw repository scanning and AI-ready context. It takes a local project, respects ignore rules, token-budgets the output, and emits two clean artifacts:
|
|
14
|
+
|
|
15
|
+
- `ctxpack.context.json` â machine-readable inventory for agents/tools
|
|
16
|
+
- `ctxpack.context.md` â human-readable prompt pack for pasting into an LLM
|
|
17
|
+
|
|
18
|
+
No dependencies. No network. **Secrets excluded by default.**
|
|
19
|
+
|
|
20
|
+
## Features
|
|
21
|
+
|
|
22
|
+
- ð Recursively scans the current directory
|
|
23
|
+
- ðŦ Respects `.ctxignore` (gitignore-style patterns)
|
|
24
|
+
- ðŠķ Skips binary and overly large files by default
|
|
25
|
+
- ð **Excludes secrets by default**: `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`
|
|
26
|
+
- ð§Ū Estimates token usage (approx `chars / 4` â **not** a model tokenizer count)
|
|
27
|
+
- âïļ Respects a max estimated token budget (`--budget`) and truncates gracefully
|
|
28
|
+
- âïļ Simple configuration via optional `ctxpack.json`
|
|
29
|
+
|
|
30
|
+
## Installation
|
|
31
|
+
|
|
32
|
+
No installation required. Just download the single file:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
curl -O https://raw.githubusercontent.com/billybox1926-jpg/ctxpack/main/ctxpack.py
|
|
36
|
+
chmod +x ctxpack.py
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Or install it to get the `ctxpack` command on your PATH:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
pip install ctxpack-cli
|
|
43
|
+
ctxpack --version
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
> The PyPI distribution is named **`ctxpack-cli`** because `ctxpack` was already
|
|
47
|
+
> taken by an unrelated project. The installed command is still `ctxpack`.
|
|
48
|
+
|
|
49
|
+
## Usage
|
|
50
|
+
|
|
51
|
+
### Initialize a project
|
|
52
|
+
Create default `.ctxignore` and `ctxpack.json` files in your current directory:
|
|
53
|
+
```bash
|
|
54
|
+
python ctxpack.py init
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Pack a repository
|
|
58
|
+
Scan the current directory and generate context files:
|
|
59
|
+
```bash
|
|
60
|
+
python ctxpack.py pack
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
### Advanced options
|
|
64
|
+
```bash
|
|
65
|
+
# Set a specific token budget
|
|
66
|
+
python ctxpack.py pack --budget 12000
|
|
67
|
+
|
|
68
|
+
# Ignore ctxpack.json settings and use CLI defaults/flags only
|
|
69
|
+
python ctxpack.py pack --no-config --budget 4000
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Configuration
|
|
73
|
+
|
|
74
|
+
### `.ctxignore`
|
|
75
|
+
|
|
76
|
+
Uses ctxignore patterns â a tested subset of gitignore syntax. Lines starting with `#` are comments. Blank lines are ignored.
|
|
77
|
+
|
|
78
|
+
**Supported pattern types:**
|
|
79
|
+
|
|
80
|
+
| Pattern | Matches | Example |
|
|
81
|
+
|---------|---------|---------|
|
|
82
|
+
| `foo` | Exact path at any depth | `build/` matches `build/`, `src/build/` |
|
|
83
|
+
| `foo/` | Directory and everything inside | `venv/` skips `venv/lib/x.py` |
|
|
84
|
+
| `foo/**` | Directory and everything inside | `node_modules/**` skips `node_modules/pkg/x.js` |
|
|
85
|
+
| `*.ext` | Files with extension at any depth | `*.log` skips `debug.log` and `logs/debug.log` |
|
|
86
|
+
| `/foo` | Exact path at the scan root only | `/build/` skips `build/` but NOT `src/build/` |
|
|
87
|
+
| `**` | Spans path segments | `**/.aws/**` skips `.aws/` and `nested/.aws/` |
|
|
88
|
+
| `!foo` | Negation â re-includes a previous exclusion | `*.pem` then `!fixture.pem` |
|
|
89
|
+
| `\*`, `\[`, etc. | Escaped wildcard/bracket (literal) | `file\*.txt` matches the literal `file*.txt` |
|
|
90
|
+
|
|
91
|
+
**Pattern precedence:** Patterns are processed in order. The last matching pattern wins â so `!foo` can override an earlier `foo`.
|
|
92
|
+
|
|
93
|
+
**Include vs. exclude:** `--include` patterns restrict to specific files. `--exclude` patterns remove files. Excludes always override includes.
|
|
94
|
+
|
|
95
|
+
**Not supported:** Character classes (`[...]`), trailing whitespace significance, or full regex.
|
|
96
|
+
|
|
97
|
+
```text
|
|
98
|
+
# Ignore virtual environments
|
|
99
|
+
venv/
|
|
100
|
+
.venv/
|
|
101
|
+
|
|
102
|
+
# Ignore build artifacts at the root only
|
|
103
|
+
/build/
|
|
104
|
+
/dist/
|
|
105
|
+
|
|
106
|
+
# Ignore all log files anywhere
|
|
107
|
+
*.log
|
|
108
|
+
|
|
109
|
+
# But keep the main log file
|
|
110
|
+
!important.log
|
|
111
|
+
|
|
112
|
+
# Allow template env files
|
|
113
|
+
!.env.example
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
**Default secret exclusions:** ctxpack excludes these by default (not shown in generated `.ctxignore`):
|
|
117
|
+
`.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`, `*.jks`, `*.keystore`, `*.gpg`, `*.asc`, `**/.aws/**`, `**/.ssh/**`, `**/.netrc`, `**/.npmrc`, `**/.pypirc`
|
|
118
|
+
|
|
119
|
+
### `ctxpack.json`
|
|
120
|
+
|
|
121
|
+
Optional configuration file. Created via `python ctxpack.py init`.
|
|
122
|
+
|
|
123
|
+
```json
|
|
124
|
+
{
|
|
125
|
+
"budget_tokens": 8000,
|
|
126
|
+
"ignore_file": ".ctxignore",
|
|
127
|
+
"include_binary": false
|
|
128
|
+
}
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Token Budget Semantics
|
|
132
|
+
|
|
133
|
+
### How token estimation works
|
|
134
|
+
|
|
135
|
+
ctxpack uses a script-aware heuristic to estimate token count:
|
|
136
|
+
|
|
137
|
+
- **Latin/ASCII text**: **~4 characters per token**, approximating typical LLM tokenization for English text and code.
|
|
138
|
+
- **CJK text** (Chinese, Japanese kana, Korean hangul): **~1.5 characters per token**, since CJK scripts tokenize far less densely than Latin text (typically 1-2 tokens per character). Mixed text is estimated per-script and summed.
|
|
139
|
+
|
|
140
|
+
Key details:
|
|
141
|
+
|
|
142
|
+
- **Empty content = 0 tokens**: Files with no content contribute zero tokens
|
|
143
|
+
- **Minimum 1 token**: Any non-empty file gets at least 1 token estimate
|
|
144
|
+
- **Truncation marker overhead**: When files are truncated, the truncation message (`...[TRUNCATED by ctxpack to fit budget]...`) accounts for ~11 tokens
|
|
145
|
+
|
|
146
|
+
### Budget enforcement behavior
|
|
147
|
+
|
|
148
|
+
When the total estimated tokens exceed the budget:
|
|
149
|
+
|
|
150
|
+
1. Files are processed in sorted path order
|
|
151
|
+
2. Files that fit entirely within remaining budget are included as-is
|
|
152
|
+
3. The first file that would exceed the budget is **truncated** (not dropped), with a truncation marker appended
|
|
153
|
+
4. Remaining files are marked as **omitted** (empty content, listed in output)
|
|
154
|
+
|
|
155
|
+
This ensures:
|
|
156
|
+
- **No silent drops**: Every discovered file appears in the output (either full, truncated, or omitted)
|
|
157
|
+
- **Budget never exceeded**: The truncation marker's token cost is reserved before slicing
|
|
158
|
+
- **Transparent about missing content**: Omitted files are listed with their original size/token estimates
|
|
159
|
+
|
|
160
|
+
### Edge cases
|
|
161
|
+
|
|
162
|
+
| Scenario | Behavior |
|
|
163
|
+
|----------|----------|
|
|
164
|
+
| Empty repository | Outputs header only, 0 tokens used |
|
|
165
|
+
| Single file > budget | File truncated to fit budget + marker |
|
|
166
|
+
| Exact budget match | All files included without truncation |
|
|
167
|
+
| Very small budget (< 20 tokens) | First file may be truncated immediately or omitted |
|
|
168
|
+
|
|
169
|
+
### Limitations
|
|
170
|
+
|
|
171
|
+
- This is a **rough estimate**, not an exact token count. Actual LLM tokenizers (e.g., tiktoken, sentencepiece) may vary by Âą20-30%
|
|
172
|
+
- Code with many symbols, non-English text, or unusual formatting may have different actual token counts
|
|
173
|
+
- For critical workflows, verify actual token usage with your target model's tokenizer
|
|
174
|
+
|
|
175
|
+
## Generated Artifacts: To Commit or Not?
|
|
176
|
+
|
|
177
|
+
**Short answer:** Generally **no**. Generated `*.context.*` files are ephemeral artifacts meant for immediate use, not long-term storage.
|
|
178
|
+
|
|
179
|
+
### Recommended practice
|
|
180
|
+
|
|
181
|
+
- â **Do not commit** `*.context.json` or `*.context.md` files to your repository
|
|
182
|
+
- â
**Do add** them to `.gitignore` (they're already in the default template from `ctxpack init`)
|
|
183
|
+
- â
**Do regenerate** them fresh whenever you need to share context with an LLM
|
|
184
|
+
|
|
185
|
+
### Why not commit generated packs?
|
|
186
|
+
|
|
187
|
+
1. **Stale content**: Context packs become outdated as soon as your code changes
|
|
188
|
+
2. **Noise in history**: Frequent regeneration creates churn in git history
|
|
189
|
+
3. **Repository bloat**: Large context packs can significantly increase repo size
|
|
190
|
+
4. **False sense of accuracy**: Old packs may misrepresent current project state
|
|
191
|
+
|
|
192
|
+
### When might you commit a pack?
|
|
193
|
+
|
|
194
|
+
Rare exceptions where committing *might* make sense:
|
|
195
|
+
|
|
196
|
+
- ðĶ **Release artifacts**: Including a context pack with a tagged release to capture exact state at release time
|
|
197
|
+
- ð **Debugging aid**: Committing a specific pack to help reproduce and debug an issue
|
|
198
|
+
- ð **Documentation example**: Sample packs in `examples/` directories (like this repo's `examples/sample.context.*`)
|
|
199
|
+
|
|
200
|
+
If you do commit a generated pack, consider:
|
|
201
|
+
- Adding a timestamp/generation note in comments
|
|
202
|
+
- Using git LFS for large files
|
|
203
|
+
- Setting up automated cleanup for stale packs
|
|
204
|
+
|
|
205
|
+
### Default behavior
|
|
206
|
+
|
|
207
|
+
The `ctxpack init` command adds these patterns to your `.gitignore`:
|
|
208
|
+
```text
|
|
209
|
+
# Generated context packs (ephemeral artifacts)
|
|
210
|
+
*.context.json
|
|
211
|
+
*.context.md
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
This keeps your repository clean while allowing you to generate fresh packs on demand.
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## Output Examples
|
|
219
|
+
|
|
220
|
+
The `examples/` directory contains static sample outputs generated by ctxpack. These are committed as documentation references:
|
|
221
|
+
|
|
222
|
+
- [`examples/sample.context.md`](./examples/sample.context.md)
|
|
223
|
+
- [`examples/sample.context.json`](./examples/sample.context.json)
|
|
224
|
+
|
|
225
|
+
To generate your own artifacts from the current project:
|
|
226
|
+
|
|
227
|
+
```bash
|
|
228
|
+
python ctxpack.py pack --output-dir ./out
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
This writes `ctxpack.context.md` and `ctxpack.context.json` into `./out/`. Generated artifacts are gitignored by default.
|
|
232
|
+
|
|
233
|
+
## Generated Artifact Policy
|
|
234
|
+
|
|
235
|
+
**Generated context artifacts are not committed to the repository.** Files matching `*.context.md` and `*.context.json` are excluded via `.gitignore`.
|
|
236
|
+
|
|
237
|
+
This policy prevents:
|
|
238
|
+
- Stale snapshots that diverge from current source
|
|
239
|
+
- Noisy diffs from regenerated output
|
|
240
|
+
- Accidental publication of sensitive material captured at generation time
|
|
241
|
+
|
|
242
|
+
The `examples/` directory holds static reference fixtures that are intentionally committed.
|
|
243
|
+
|
|
244
|
+
## Security
|
|
245
|
+
|
|
246
|
+
### Secret-safe by default
|
|
247
|
+
|
|
248
|
+
`ctxpack` excludes credential-bearing files by default so they never reach the generated pack. The built-in ignore policy covers:
|
|
249
|
+
|
|
250
|
+
- **Local env files**: `.env`, `.env.*` (but not `.env.example`, the conventional template)
|
|
251
|
+
- **Private keys & certificates**: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`
|
|
252
|
+
- **Keystores**: `*.jks`, `*.keystore`
|
|
253
|
+
- **GPG / signing material**: `*.gpg`, `*.asc`
|
|
254
|
+
- **Credential directories**: `**/.aws/**`, `**/.ssh/**`
|
|
255
|
+
- **Auth dotfiles**: `**/.netrc`, `**/.npmrc`, `**/.pypirc`
|
|
256
|
+
|
|
257
|
+
These defaults are applied automatically; you do not need to list them in `.ctxignore`. Custom `.ctxignore` entries are merged with these defaults and can add further exclusions.
|
|
258
|
+
|
|
259
|
+
**Security boundary**: Default secret exclusions are just patterns. Explicit `.ctxignore` negation patterns (e.g., `!.env`) can override these defaults. Users should treat custom re-inclusion rules as an explicit security opt-in â the tool cannot protect against intentional overrides in project configuration files.
|
|
260
|
+
|
|
261
|
+
To opt a specific secret file back in (e.g., a test fixture), add a negation pattern to `.ctxignore`:
|
|
262
|
+
|
|
263
|
+
```text
|
|
264
|
+
!important/test-fixture.pem
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
#### Strict mode
|
|
268
|
+
|
|
269
|
+
If you want the default secret exclusions to be non-overridable, pass `--strict-secrets`:
|
|
270
|
+
|
|
271
|
+
```bash
|
|
272
|
+
python ctxpack.py pack --strict-secrets
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
In strict mode, `.ctxignore` and CLI negation patterns **cannot** re-include any secret file (`.env`, `*.pem`, `*.key`, `*.gpg`, credential directories, etc.). The built-in `.env.example` template carve-out still applies, so conventional example files remain packable.
|
|
276
|
+
|
|
277
|
+
### Path privacy
|
|
278
|
+
|
|
279
|
+
Generated packs use a privacy-preserving `.` representation for the project root by default. To include the absolute path (e.g., for debugging), use:
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
python ctxpack.py pack --show-absolute-paths
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
This prevents accidentally leaking local usernames, directory structures, or project locations when sharing context packs.
|
|
286
|
+
|
|
287
|
+
## License
|
|
288
|
+
|
|
289
|
+
MIT License. See [LICENSE](LICENSE) for details.
|
|
290
|
+
|
|
291
|
+
## Release Process
|
|
292
|
+
|
|
293
|
+
### Versioning
|
|
294
|
+
|
|
295
|
+
`ctxpack` uses [Semantic Versioning](https://semver.org/):
|
|
296
|
+
- **MAJOR**: incompatible API changes
|
|
297
|
+
- **MINOR**: backwards-compatible functionality additions
|
|
298
|
+
- **PATCH**: backwards-compatible bug fixes
|
|
299
|
+
|
|
300
|
+
### Building a Release
|
|
301
|
+
|
|
302
|
+
```bash
|
|
303
|
+
# Clean build
|
|
304
|
+
rm -rf dist/
|
|
305
|
+
python -m build
|
|
306
|
+
|
|
307
|
+
# Verify artifacts
|
|
308
|
+
twine check dist/*
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
### Release Checklist
|
|
312
|
+
|
|
313
|
+
- [ ] All tests pass (`pytest -v`)
|
|
314
|
+
- [ ] Lint checks pass (`ruff check .`)
|
|
315
|
+
- [ ] Version bumped in `pyproject.toml`
|
|
316
|
+
- [ ] CHANGELOG updated (if applicable)
|
|
317
|
+
- [ ] Git tag created (`git tag v0.2.0`)
|
|
318
|
+
- [ ] CI green on all jobs (test, lint, security, packaging)
|
|
319
|
+
|
|
320
|
+
### CI Release Validation
|
|
321
|
+
|
|
322
|
+
The `packaging` CI job validates:
|
|
323
|
+
- sdist and wheel build successfully
|
|
324
|
+
- CLI works after installation from wheel
|
|
325
|
+
- All tests pass against installed package
|