ctxpack-cli 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Billy Box
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,353 @@
1
+ Metadata-Version: 2.4
2
+ Name: ctxpack-cli
3
+ Version: 1.0.0
4
+ Summary: Dependency-free repo-to-prompt pack builder for AI workflows
5
+ Author-email: Billy Box <billybox1926@gmail.com>
6
+ License: MIT
7
+ Project-URL: Homepage, https://github.com/billybox1926-jpg/ctxpack
8
+ Project-URL: Repository, https://github.com/billybox1926-jpg/ctxpack
9
+ Project-URL: Issues, https://github.com/billybox1926-jpg/ctxpack/issues
10
+ Project-URL: Documentation, https://github.com/billybox1926-jpg/ctxpack#readme
11
+ Keywords: ai,context,llm,prompt,cli,rag
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Environment :: Console
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: License :: OSI Approved :: MIT License
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.9
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Requires-Python: >=3.9
22
+ Description-Content-Type: text/markdown
23
+ License-File: LICENSE
24
+ Provides-Extra: dev
25
+ Requires-Dist: pytest; extra == "dev"
26
+ Requires-Dist: ruff; extra == "dev"
27
+ Dynamic: license-file
28
+
29
+ # ctxpack
30
+
31
+ ![ctxpack icon](assets/icon.png)
32
+
33
+ [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
34
+ [![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)
35
+ [![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)](https://github.com/billybox1926-jpg/ctxpack/releases)
36
+ [![Tests](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml/badge.svg)](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml)
37
+ [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
38
+
39
+ **Dependency-free repo-to-prompt pack builder.**
40
+
41
+ `ctxpack` is the missing bridge between raw repository scanning and AI-ready context. It takes a local project, respects ignore rules, token-budgets the output, and emits two clean artifacts:
42
+
43
+ - `ctxpack.context.json` — machine-readable inventory for agents/tools
44
+ - `ctxpack.context.md` — human-readable prompt pack for pasting into an LLM
45
+
46
+ No dependencies. No network. **Secrets excluded by default.**
47
+
48
+ ## Features
49
+
50
+ - 📁 Recursively scans the current directory
51
+ - ðŸšŦ Respects `.ctxignore` (gitignore-style patterns)
52
+ - ðŸŠķ Skips binary and overly large files by default
53
+ - 🔒 **Excludes secrets by default**: `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`
54
+ - ðŸ§Ū Estimates token usage (approx `chars / 4` — **not** a model tokenizer count)
55
+ - ✂ïļ Respects a max estimated token budget (`--budget`) and truncates gracefully
56
+ - ⚙ïļ Simple configuration via optional `ctxpack.json`
57
+
58
+ ## Installation
59
+
60
+ No installation required. Just download the single file:
61
+
62
+ ```bash
63
+ curl -O https://raw.githubusercontent.com/billybox1926-jpg/ctxpack/main/ctxpack.py
64
+ chmod +x ctxpack.py
65
+ ```
66
+
67
+ Or install it to get the `ctxpack` command on your PATH:
68
+
69
+ ```bash
70
+ pip install ctxpack-cli
71
+ ctxpack --version
72
+ ```
73
+
74
+ > The PyPI distribution is named **`ctxpack-cli`** because `ctxpack` was already
75
+ > taken by an unrelated project. The installed command is still `ctxpack`.
76
+
77
+ ## Usage
78
+
79
+ ### Initialize a project
80
+ Create default `.ctxignore` and `ctxpack.json` files in your current directory:
81
+ ```bash
82
+ python ctxpack.py init
83
+ ```
84
+
85
+ ### Pack a repository
86
+ Scan the current directory and generate context files:
87
+ ```bash
88
+ python ctxpack.py pack
89
+ ```
90
+
91
+ ### Advanced options
92
+ ```bash
93
+ # Set a specific token budget
94
+ python ctxpack.py pack --budget 12000
95
+
96
+ # Ignore ctxpack.json settings and use CLI defaults/flags only
97
+ python ctxpack.py pack --no-config --budget 4000
98
+ ```
99
+
100
+ ## Configuration
101
+
102
+ ### `.ctxignore`
103
+
104
+ Uses ctxignore patterns — a tested subset of gitignore syntax. Lines starting with `#` are comments. Blank lines are ignored.
105
+
106
+ **Supported pattern types:**
107
+
108
+ | Pattern | Matches | Example |
109
+ |---------|---------|---------|
110
+ | `foo` | Exact path at any depth | `build/` matches `build/`, `src/build/` |
111
+ | `foo/` | Directory and everything inside | `venv/` skips `venv/lib/x.py` |
112
+ | `foo/**` | Directory and everything inside | `node_modules/**` skips `node_modules/pkg/x.js` |
113
+ | `*.ext` | Files with extension at any depth | `*.log` skips `debug.log` and `logs/debug.log` |
114
+ | `/foo` | Exact path at the scan root only | `/build/` skips `build/` but NOT `src/build/` |
115
+ | `**` | Spans path segments | `**/.aws/**` skips `.aws/` and `nested/.aws/` |
116
+ | `!foo` | Negation — re-includes a previous exclusion | `*.pem` then `!fixture.pem` |
117
+ | `\*`, `\[`, etc. | Escaped wildcard/bracket (literal) | `file\*.txt` matches the literal `file*.txt` |
118
+
119
+ **Pattern precedence:** Patterns are processed in order. The last matching pattern wins — so `!foo` can override an earlier `foo`.
120
+
121
+ **Include vs. exclude:** `--include` patterns restrict to specific files. `--exclude` patterns remove files. Excludes always override includes.
122
+
123
+ **Not supported:** Character classes (`[...]`), trailing whitespace significance, or full regex.
124
+
125
+ ```text
126
+ # Ignore virtual environments
127
+ venv/
128
+ .venv/
129
+
130
+ # Ignore build artifacts at the root only
131
+ /build/
132
+ /dist/
133
+
134
+ # Ignore all log files anywhere
135
+ *.log
136
+
137
+ # But keep the main log file
138
+ !important.log
139
+
140
+ # Allow template env files
141
+ !.env.example
142
+ ```
143
+
144
+ **Default secret exclusions:** ctxpack excludes these by default (not shown in generated `.ctxignore`):
145
+ `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`, `*.jks`, `*.keystore`, `*.gpg`, `*.asc`, `**/.aws/**`, `**/.ssh/**`, `**/.netrc`, `**/.npmrc`, `**/.pypirc`
146
+
147
+ ### `ctxpack.json`
148
+
149
+ Optional configuration file. Created via `python ctxpack.py init`.
150
+
151
+ ```json
152
+ {
153
+ "budget_tokens": 8000,
154
+ "ignore_file": ".ctxignore",
155
+ "include_binary": false
156
+ }
157
+ ```
158
+
159
+ ## Token Budget Semantics
160
+
161
+ ### How token estimation works
162
+
163
+ ctxpack uses a script-aware heuristic to estimate token count:
164
+
165
+ - **Latin/ASCII text**: **~4 characters per token**, approximating typical LLM tokenization for English text and code.
166
+ - **CJK text** (Chinese, Japanese kana, Korean hangul): **~1.5 characters per token**, since CJK scripts tokenize far less densely than Latin text (typically 1-2 tokens per character). Mixed text is estimated per-script and summed.
167
+
168
+ Key details:
169
+
170
+ - **Empty content = 0 tokens**: Files with no content contribute zero tokens
171
+ - **Minimum 1 token**: Any non-empty file gets at least 1 token estimate
172
+ - **Truncation marker overhead**: When files are truncated, the truncation message (`...[TRUNCATED by ctxpack to fit budget]...`) accounts for ~11 tokens
173
+
174
+ ### Budget enforcement behavior
175
+
176
+ When the total estimated tokens exceed the budget:
177
+
178
+ 1. Files are processed in sorted path order
179
+ 2. Files that fit entirely within remaining budget are included as-is
180
+ 3. The first file that would exceed the budget is **truncated** (not dropped), with a truncation marker appended
181
+ 4. Remaining files are marked as **omitted** (empty content, listed in output)
182
+
183
+ This ensures:
184
+ - **No silent drops**: Every discovered file appears in the output (either full, truncated, or omitted)
185
+ - **Budget never exceeded**: The truncation marker's token cost is reserved before slicing
186
+ - **Transparent about missing content**: Omitted files are listed with their original size/token estimates
187
+
188
+ ### Edge cases
189
+
190
+ | Scenario | Behavior |
191
+ |----------|----------|
192
+ | Empty repository | Outputs header only, 0 tokens used |
193
+ | Single file > budget | File truncated to fit budget + marker |
194
+ | Exact budget match | All files included without truncation |
195
+ | Very small budget (< 20 tokens) | First file may be truncated immediately or omitted |
196
+
197
+ ### Limitations
198
+
199
+ - This is a **rough estimate**, not an exact token count. Actual LLM tokenizers (e.g., tiktoken, sentencepiece) may vary by Âą20-30%
200
+ - Code with many symbols, non-English text, or unusual formatting may have different actual token counts
201
+ - For critical workflows, verify actual token usage with your target model's tokenizer
202
+
203
+ ## Generated Artifacts: To Commit or Not?
204
+
205
+ **Short answer:** Generally **no**. Generated `*.context.*` files are ephemeral artifacts meant for immediate use, not long-term storage.
206
+
207
+ ### Recommended practice
208
+
209
+ - ❌ **Do not commit** `*.context.json` or `*.context.md` files to your repository
210
+ - ✅ **Do add** them to `.gitignore` (they're already in the default template from `ctxpack init`)
211
+ - ✅ **Do regenerate** them fresh whenever you need to share context with an LLM
212
+
213
+ ### Why not commit generated packs?
214
+
215
+ 1. **Stale content**: Context packs become outdated as soon as your code changes
216
+ 2. **Noise in history**: Frequent regeneration creates churn in git history
217
+ 3. **Repository bloat**: Large context packs can significantly increase repo size
218
+ 4. **False sense of accuracy**: Old packs may misrepresent current project state
219
+
220
+ ### When might you commit a pack?
221
+
222
+ Rare exceptions where committing *might* make sense:
223
+
224
+ - ðŸ“Ķ **Release artifacts**: Including a context pack with a tagged release to capture exact state at release time
225
+ - 🔍 **Debugging aid**: Committing a specific pack to help reproduce and debug an issue
226
+ - 📚 **Documentation example**: Sample packs in `examples/` directories (like this repo's `examples/sample.context.*`)
227
+
228
+ If you do commit a generated pack, consider:
229
+ - Adding a timestamp/generation note in comments
230
+ - Using git LFS for large files
231
+ - Setting up automated cleanup for stale packs
232
+
233
+ ### Default behavior
234
+
235
+ The `ctxpack init` command adds these patterns to your `.gitignore`:
236
+ ```text
237
+ # Generated context packs (ephemeral artifacts)
238
+ *.context.json
239
+ *.context.md
240
+ ```
241
+
242
+ This keeps your repository clean while allowing you to generate fresh packs on demand.
243
+
244
+ ---
245
+
246
+ ## Output Examples
247
+
248
+ The `examples/` directory contains static sample outputs generated by ctxpack. These are committed as documentation references:
249
+
250
+ - [`examples/sample.context.md`](./examples/sample.context.md)
251
+ - [`examples/sample.context.json`](./examples/sample.context.json)
252
+
253
+ To generate your own artifacts from the current project:
254
+
255
+ ```bash
256
+ python ctxpack.py pack --output-dir ./out
257
+ ```
258
+
259
+ This writes `ctxpack.context.md` and `ctxpack.context.json` into `./out/`. Generated artifacts are gitignored by default.
260
+
261
+ ## Generated Artifact Policy
262
+
263
+ **Generated context artifacts are not committed to the repository.** Files matching `*.context.md` and `*.context.json` are excluded via `.gitignore`.
264
+
265
+ This policy prevents:
266
+ - Stale snapshots that diverge from current source
267
+ - Noisy diffs from regenerated output
268
+ - Accidental publication of sensitive material captured at generation time
269
+
270
+ The `examples/` directory holds static reference fixtures that are intentionally committed.
271
+
272
+ ## Security
273
+
274
+ ### Secret-safe by default
275
+
276
+ `ctxpack` excludes credential-bearing files by default so they never reach the generated pack. The built-in ignore policy covers:
277
+
278
+ - **Local env files**: `.env`, `.env.*` (but not `.env.example`, the conventional template)
279
+ - **Private keys & certificates**: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`
280
+ - **Keystores**: `*.jks`, `*.keystore`
281
+ - **GPG / signing material**: `*.gpg`, `*.asc`
282
+ - **Credential directories**: `**/.aws/**`, `**/.ssh/**`
283
+ - **Auth dotfiles**: `**/.netrc`, `**/.npmrc`, `**/.pypirc`
284
+
285
+ These defaults are applied automatically; you do not need to list them in `.ctxignore`. Custom `.ctxignore` entries are merged with these defaults and can add further exclusions.
286
+
287
+ **Security boundary**: Default secret exclusions are just patterns. Explicit `.ctxignore` negation patterns (e.g., `!.env`) can override these defaults. Users should treat custom re-inclusion rules as an explicit security opt-in — the tool cannot protect against intentional overrides in project configuration files.
288
+
289
+ To opt a specific secret file back in (e.g., a test fixture), add a negation pattern to `.ctxignore`:
290
+
291
+ ```text
292
+ !important/test-fixture.pem
293
+ ```
294
+
295
+ #### Strict mode
296
+
297
+ If you want the default secret exclusions to be non-overridable, pass `--strict-secrets`:
298
+
299
+ ```bash
300
+ python ctxpack.py pack --strict-secrets
301
+ ```
302
+
303
+ In strict mode, `.ctxignore` and CLI negation patterns **cannot** re-include any secret file (`.env`, `*.pem`, `*.key`, `*.gpg`, credential directories, etc.). The built-in `.env.example` template carve-out still applies, so conventional example files remain packable.
304
+
305
+ ### Path privacy
306
+
307
+ Generated packs use a privacy-preserving `.` representation for the project root by default. To include the absolute path (e.g., for debugging), use:
308
+
309
+ ```bash
310
+ python ctxpack.py pack --show-absolute-paths
311
+ ```
312
+
313
+ This prevents accidentally leaking local usernames, directory structures, or project locations when sharing context packs.
314
+
315
+ ## License
316
+
317
+ MIT License. See [LICENSE](LICENSE) for details.
318
+
319
+ ## Release Process
320
+
321
+ ### Versioning
322
+
323
+ `ctxpack` uses [Semantic Versioning](https://semver.org/):
324
+ - **MAJOR**: incompatible API changes
325
+ - **MINOR**: backwards-compatible functionality additions
326
+ - **PATCH**: backwards-compatible bug fixes
327
+
328
+ ### Building a Release
329
+
330
+ ```bash
331
+ # Clean build
332
+ rm -rf dist/
333
+ python -m build
334
+
335
+ # Verify artifacts
336
+ twine check dist/*
337
+ ```
338
+
339
+ ### Release Checklist
340
+
341
+ - [ ] All tests pass (`pytest -v`)
342
+ - [ ] Lint checks pass (`ruff check .`)
343
+ - [ ] Version bumped in `pyproject.toml`
344
+ - [ ] CHANGELOG updated (if applicable)
345
+ - [ ] Git tag created (`git tag v0.2.0`)
346
+ - [ ] CI green on all jobs (test, lint, security, packaging)
347
+
348
+ ### CI Release Validation
349
+
350
+ The `packaging` CI job validates:
351
+ - sdist and wheel build successfully
352
+ - CLI works after installation from wheel
353
+ - All tests pass against installed package
@@ -0,0 +1,325 @@
1
+ # ctxpack
2
+
3
+ ![ctxpack icon](assets/icon.png)
4
+
5
+ [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
6
+ [![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)
7
+ [![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)](https://github.com/billybox1926-jpg/ctxpack/releases)
8
+ [![Tests](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml/badge.svg)](https://github.com/billybox1926-jpg/ctxpack/actions/workflows/test.yml)
9
+ [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
10
+
11
+ **Dependency-free repo-to-prompt pack builder.**
12
+
13
+ `ctxpack` is the missing bridge between raw repository scanning and AI-ready context. It takes a local project, respects ignore rules, token-budgets the output, and emits two clean artifacts:
14
+
15
+ - `ctxpack.context.json` — machine-readable inventory for agents/tools
16
+ - `ctxpack.context.md` — human-readable prompt pack for pasting into an LLM
17
+
18
+ No dependencies. No network. **Secrets excluded by default.**
19
+
20
+ ## Features
21
+
22
+ - 📁 Recursively scans the current directory
23
+ - ðŸšŦ Respects `.ctxignore` (gitignore-style patterns)
24
+ - ðŸŠķ Skips binary and overly large files by default
25
+ - 🔒 **Excludes secrets by default**: `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`
26
+ - ðŸ§Ū Estimates token usage (approx `chars / 4` — **not** a model tokenizer count)
27
+ - ✂ïļ Respects a max estimated token budget (`--budget`) and truncates gracefully
28
+ - ⚙ïļ Simple configuration via optional `ctxpack.json`
29
+
30
+ ## Installation
31
+
32
+ No installation required. Just download the single file:
33
+
34
+ ```bash
35
+ curl -O https://raw.githubusercontent.com/billybox1926-jpg/ctxpack/main/ctxpack.py
36
+ chmod +x ctxpack.py
37
+ ```
38
+
39
+ Or install it to get the `ctxpack` command on your PATH:
40
+
41
+ ```bash
42
+ pip install ctxpack-cli
43
+ ctxpack --version
44
+ ```
45
+
46
+ > The PyPI distribution is named **`ctxpack-cli`** because `ctxpack` was already
47
+ > taken by an unrelated project. The installed command is still `ctxpack`.
48
+
49
+ ## Usage
50
+
51
+ ### Initialize a project
52
+ Create default `.ctxignore` and `ctxpack.json` files in your current directory:
53
+ ```bash
54
+ python ctxpack.py init
55
+ ```
56
+
57
+ ### Pack a repository
58
+ Scan the current directory and generate context files:
59
+ ```bash
60
+ python ctxpack.py pack
61
+ ```
62
+
63
+ ### Advanced options
64
+ ```bash
65
+ # Set a specific token budget
66
+ python ctxpack.py pack --budget 12000
67
+
68
+ # Ignore ctxpack.json settings and use CLI defaults/flags only
69
+ python ctxpack.py pack --no-config --budget 4000
70
+ ```
71
+
72
+ ## Configuration
73
+
74
+ ### `.ctxignore`
75
+
76
+ Uses ctxignore patterns — a tested subset of gitignore syntax. Lines starting with `#` are comments. Blank lines are ignored.
77
+
78
+ **Supported pattern types:**
79
+
80
+ | Pattern | Matches | Example |
81
+ |---------|---------|---------|
82
+ | `foo` | Exact path at any depth | `build/` matches `build/`, `src/build/` |
83
+ | `foo/` | Directory and everything inside | `venv/` skips `venv/lib/x.py` |
84
+ | `foo/**` | Directory and everything inside | `node_modules/**` skips `node_modules/pkg/x.js` |
85
+ | `*.ext` | Files with extension at any depth | `*.log` skips `debug.log` and `logs/debug.log` |
86
+ | `/foo` | Exact path at the scan root only | `/build/` skips `build/` but NOT `src/build/` |
87
+ | `**` | Spans path segments | `**/.aws/**` skips `.aws/` and `nested/.aws/` |
88
+ | `!foo` | Negation — re-includes a previous exclusion | `*.pem` then `!fixture.pem` |
89
+ | `\*`, `\[`, etc. | Escaped wildcard/bracket (literal) | `file\*.txt` matches the literal `file*.txt` |
90
+
91
+ **Pattern precedence:** Patterns are processed in order. The last matching pattern wins — so `!foo` can override an earlier `foo`.
92
+
93
+ **Include vs. exclude:** `--include` patterns restrict to specific files. `--exclude` patterns remove files. Excludes always override includes.
94
+
95
+ **Not supported:** Character classes (`[...]`), trailing whitespace significance, or full regex.
96
+
97
+ ```text
98
+ # Ignore virtual environments
99
+ venv/
100
+ .venv/
101
+
102
+ # Ignore build artifacts at the root only
103
+ /build/
104
+ /dist/
105
+
106
+ # Ignore all log files anywhere
107
+ *.log
108
+
109
+ # But keep the main log file
110
+ !important.log
111
+
112
+ # Allow template env files
113
+ !.env.example
114
+ ```
115
+
116
+ **Default secret exclusions:** ctxpack excludes these by default (not shown in generated `.ctxignore`):
117
+ `.env`, `.env.*`, `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`, `*.jks`, `*.keystore`, `*.gpg`, `*.asc`, `**/.aws/**`, `**/.ssh/**`, `**/.netrc`, `**/.npmrc`, `**/.pypirc`
118
+
119
+ ### `ctxpack.json`
120
+
121
+ Optional configuration file. Created via `python ctxpack.py init`.
122
+
123
+ ```json
124
+ {
125
+ "budget_tokens": 8000,
126
+ "ignore_file": ".ctxignore",
127
+ "include_binary": false
128
+ }
129
+ ```
130
+
131
+ ## Token Budget Semantics
132
+
133
+ ### How token estimation works
134
+
135
+ ctxpack uses a script-aware heuristic to estimate token count:
136
+
137
+ - **Latin/ASCII text**: **~4 characters per token**, approximating typical LLM tokenization for English text and code.
138
+ - **CJK text** (Chinese, Japanese kana, Korean hangul): **~1.5 characters per token**, since CJK scripts tokenize far less densely than Latin text (typically 1-2 tokens per character). Mixed text is estimated per-script and summed.
139
+
140
+ Key details:
141
+
142
+ - **Empty content = 0 tokens**: Files with no content contribute zero tokens
143
+ - **Minimum 1 token**: Any non-empty file gets at least 1 token estimate
144
+ - **Truncation marker overhead**: When files are truncated, the truncation message (`...[TRUNCATED by ctxpack to fit budget]...`) accounts for ~11 tokens
145
+
146
+ ### Budget enforcement behavior
147
+
148
+ When the total estimated tokens exceed the budget:
149
+
150
+ 1. Files are processed in sorted path order
151
+ 2. Files that fit entirely within remaining budget are included as-is
152
+ 3. The first file that would exceed the budget is **truncated** (not dropped), with a truncation marker appended
153
+ 4. Remaining files are marked as **omitted** (empty content, listed in output)
154
+
155
+ This ensures:
156
+ - **No silent drops**: Every discovered file appears in the output (either full, truncated, or omitted)
157
+ - **Budget never exceeded**: The truncation marker's token cost is reserved before slicing
158
+ - **Transparent about missing content**: Omitted files are listed with their original size/token estimates
159
+
160
+ ### Edge cases
161
+
162
+ | Scenario | Behavior |
163
+ |----------|----------|
164
+ | Empty repository | Outputs header only, 0 tokens used |
165
+ | Single file > budget | File truncated to fit budget + marker |
166
+ | Exact budget match | All files included without truncation |
167
+ | Very small budget (< 20 tokens) | First file may be truncated immediately or omitted |
168
+
169
+ ### Limitations
170
+
171
+ - This is a **rough estimate**, not an exact token count. Actual LLM tokenizers (e.g., tiktoken, sentencepiece) may vary by Âą20-30%
172
+ - Code with many symbols, non-English text, or unusual formatting may have different actual token counts
173
+ - For critical workflows, verify actual token usage with your target model's tokenizer
174
+
175
+ ## Generated Artifacts: To Commit or Not?
176
+
177
+ **Short answer:** Generally **no**. Generated `*.context.*` files are ephemeral artifacts meant for immediate use, not long-term storage.
178
+
179
+ ### Recommended practice
180
+
181
+ - ❌ **Do not commit** `*.context.json` or `*.context.md` files to your repository
182
+ - ✅ **Do add** them to `.gitignore` (they're already in the default template from `ctxpack init`)
183
+ - ✅ **Do regenerate** them fresh whenever you need to share context with an LLM
184
+
185
+ ### Why not commit generated packs?
186
+
187
+ 1. **Stale content**: Context packs become outdated as soon as your code changes
188
+ 2. **Noise in history**: Frequent regeneration creates churn in git history
189
+ 3. **Repository bloat**: Large context packs can significantly increase repo size
190
+ 4. **False sense of accuracy**: Old packs may misrepresent current project state
191
+
192
+ ### When might you commit a pack?
193
+
194
+ Rare exceptions where committing *might* make sense:
195
+
196
+ - ðŸ“Ķ **Release artifacts**: Including a context pack with a tagged release to capture exact state at release time
197
+ - 🔍 **Debugging aid**: Committing a specific pack to help reproduce and debug an issue
198
+ - 📚 **Documentation example**: Sample packs in `examples/` directories (like this repo's `examples/sample.context.*`)
199
+
200
+ If you do commit a generated pack, consider:
201
+ - Adding a timestamp/generation note in comments
202
+ - Using git LFS for large files
203
+ - Setting up automated cleanup for stale packs
204
+
205
+ ### Default behavior
206
+
207
+ The `ctxpack init` command adds these patterns to your `.gitignore`:
208
+ ```text
209
+ # Generated context packs (ephemeral artifacts)
210
+ *.context.json
211
+ *.context.md
212
+ ```
213
+
214
+ This keeps your repository clean while allowing you to generate fresh packs on demand.
215
+
216
+ ---
217
+
218
+ ## Output Examples
219
+
220
+ The `examples/` directory contains static sample outputs generated by ctxpack. These are committed as documentation references:
221
+
222
+ - [`examples/sample.context.md`](./examples/sample.context.md)
223
+ - [`examples/sample.context.json`](./examples/sample.context.json)
224
+
225
+ To generate your own artifacts from the current project:
226
+
227
+ ```bash
228
+ python ctxpack.py pack --output-dir ./out
229
+ ```
230
+
231
+ This writes `ctxpack.context.md` and `ctxpack.context.json` into `./out/`. Generated artifacts are gitignored by default.
232
+
233
+ ## Generated Artifact Policy
234
+
235
+ **Generated context artifacts are not committed to the repository.** Files matching `*.context.md` and `*.context.json` are excluded via `.gitignore`.
236
+
237
+ This policy prevents:
238
+ - Stale snapshots that diverge from current source
239
+ - Noisy diffs from regenerated output
240
+ - Accidental publication of sensitive material captured at generation time
241
+
242
+ The `examples/` directory holds static reference fixtures that are intentionally committed.
243
+
244
+ ## Security
245
+
246
+ ### Secret-safe by default
247
+
248
+ `ctxpack` excludes credential-bearing files by default so they never reach the generated pack. The built-in ignore policy covers:
249
+
250
+ - **Local env files**: `.env`, `.env.*` (but not `.env.example`, the conventional template)
251
+ - **Private keys & certificates**: `*.pem`, `*.key`, `*.p12`, `*.pfx`, `*.crt`, `*.cer`
252
+ - **Keystores**: `*.jks`, `*.keystore`
253
+ - **GPG / signing material**: `*.gpg`, `*.asc`
254
+ - **Credential directories**: `**/.aws/**`, `**/.ssh/**`
255
+ - **Auth dotfiles**: `**/.netrc`, `**/.npmrc`, `**/.pypirc`
256
+
257
+ These defaults are applied automatically; you do not need to list them in `.ctxignore`. Custom `.ctxignore` entries are merged with these defaults and can add further exclusions.
258
+
259
+ **Security boundary**: Default secret exclusions are just patterns. Explicit `.ctxignore` negation patterns (e.g., `!.env`) can override these defaults. Users should treat custom re-inclusion rules as an explicit security opt-in — the tool cannot protect against intentional overrides in project configuration files.
260
+
261
+ To opt a specific secret file back in (e.g., a test fixture), add a negation pattern to `.ctxignore`:
262
+
263
+ ```text
264
+ !important/test-fixture.pem
265
+ ```
266
+
267
+ #### Strict mode
268
+
269
+ If you want the default secret exclusions to be non-overridable, pass `--strict-secrets`:
270
+
271
+ ```bash
272
+ python ctxpack.py pack --strict-secrets
273
+ ```
274
+
275
+ In strict mode, `.ctxignore` and CLI negation patterns **cannot** re-include any secret file (`.env`, `*.pem`, `*.key`, `*.gpg`, credential directories, etc.). The built-in `.env.example` template carve-out still applies, so conventional example files remain packable.
276
+
277
+ ### Path privacy
278
+
279
+ Generated packs use a privacy-preserving `.` representation for the project root by default. To include the absolute path (e.g., for debugging), use:
280
+
281
+ ```bash
282
+ python ctxpack.py pack --show-absolute-paths
283
+ ```
284
+
285
+ This prevents accidentally leaking local usernames, directory structures, or project locations when sharing context packs.
286
+
287
+ ## License
288
+
289
+ MIT License. See [LICENSE](LICENSE) for details.
290
+
291
+ ## Release Process
292
+
293
+ ### Versioning
294
+
295
+ `ctxpack` uses [Semantic Versioning](https://semver.org/):
296
+ - **MAJOR**: incompatible API changes
297
+ - **MINOR**: backwards-compatible functionality additions
298
+ - **PATCH**: backwards-compatible bug fixes
299
+
300
+ ### Building a Release
301
+
302
+ ```bash
303
+ # Clean build
304
+ rm -rf dist/
305
+ python -m build
306
+
307
+ # Verify artifacts
308
+ twine check dist/*
309
+ ```
310
+
311
+ ### Release Checklist
312
+
313
+ - [ ] All tests pass (`pytest -v`)
314
+ - [ ] Lint checks pass (`ruff check .`)
315
+ - [ ] Version bumped in `pyproject.toml`
316
+ - [ ] CHANGELOG updated (if applicable)
317
+ - [ ] Git tag created (`git tag v0.2.0`)
318
+ - [ ] CI green on all jobs (test, lint, security, packaging)
319
+
320
+ ### CI Release Validation
321
+
322
+ The `packaging` CI job validates:
323
+ - sdist and wheel build successfully
324
+ - CLI works after installation from wheel
325
+ - All tests pass against installed package