split-python4gpt 1.0.3__tar.gz → 1.0.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {split_python4gpt-1.0.3 → split_python4gpt-1.0.5}/.gitignore +1 -0
- split_python4gpt-1.0.5/AUTHORS.md +4 -0
- split_python4gpt-1.0.5/CHANGELOG.md +83 -0
- split_python4gpt-1.0.5/PKG-INFO +330 -0
- split_python4gpt-1.0.5/README.md +285 -0
- split_python4gpt-1.0.5/pyproject.toml +114 -0
- split_python4gpt-1.0.5/src/split_python4gpt/__init__.py +15 -0
- {split_python4gpt-1.0.3 → split_python4gpt-1.0.5}/src/split_python4gpt/__main__.py +17 -6
- split_python4gpt-1.0.5/src/split_python4gpt/_version.py +24 -0
- split_python4gpt-1.0.5/src/split_python4gpt/minifier.py +510 -0
- split_python4gpt-1.0.5/tests/conftest.py +18 -0
- split_python4gpt-1.0.5/tests/data/empty_in.py +0 -0
- split_python4gpt-1.0.5/tests/data/folder_in/file1.py +11 -0
- split_python4gpt-1.0.5/tests/data/folder_in/subdir/file2.py +18 -0
- {split_python4gpt-1.0.3 → split_python4gpt-1.0.5}/tests/data/in_test.py +0 -0
- split_python4gpt-1.0.5/tests/data/out_test.py +2 -0
- split_python4gpt-1.0.5/tests/data/syntax_error_in.py +6 -0
- split_python4gpt-1.0.5/tests/data/type_in.py +11 -0
- split_python4gpt-1.0.5/tests/test_cli.py +144 -0
- split_python4gpt-1.0.5/tests/test_minifier.py +334 -0
- split_python4gpt-1.0.5/tests/test_performance.py +211 -0
- split_python4gpt-1.0.5/tests/test_version.py +91 -0
- split_python4gpt-1.0.3/.coveragerc +0 -28
- split_python4gpt-1.0.3/.github/workflows/ci.yml +0 -80
- split_python4gpt-1.0.3/.isort.cfg +0 -3
- split_python4gpt-1.0.3/.pre-commit-config.yaml +0 -67
- split_python4gpt-1.0.3/AUTHORS.md +0 -3
- split_python4gpt-1.0.3/PKG-INFO +0 -145
- split_python4gpt-1.0.3/README.md +0 -119
- split_python4gpt-1.0.3/pydoc-markdown.yml +0 -6
- split_python4gpt-1.0.3/pyproject.toml +0 -9
- split_python4gpt-1.0.3/setup.cfg +0 -89
- split_python4gpt-1.0.3/setup.py +0 -21
- split_python4gpt-1.0.3/src/split_python4gpt/__init__.py +0 -17
- split_python4gpt-1.0.3/src/split_python4gpt/minifier.py +0 -146
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/PKG-INFO +0 -145
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/SOURCES.txt +0 -27
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/dependency_links.txt +0 -1
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/entry_points.txt +0 -2
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/not-zip-safe +0 -1
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/requires.txt +0 -9
- split_python4gpt-1.0.3/src/split_python4gpt.egg-info/top_level.txt +0 -1
- split_python4gpt-1.0.3/tests/conftest.py +0 -10
- split_python4gpt-1.0.3/tests/data/out_test.py +0 -1
- split_python4gpt-1.0.3/tests/test_minifier.py +0 -18
- split_python4gpt-1.0.3/tox.ini +0 -93
- {split_python4gpt-1.0.3 → split_python4gpt-1.0.5}/LICENSE.txt +0 -0
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- **MkDocs Material docs site** (`mkdocs.yml`, `docs/`) with pages for home,
|
|
12
|
+
installation, usage, API reference, and changelog.
|
|
13
|
+
- **GitHub Actions CI** (`.github/workflows/ci.yml`) covering Python 3.10–3.13
|
|
14
|
+
on Linux, macOS, and Windows; separate lint job with ruff; PyPI publish on tag.
|
|
15
|
+
- Full type hints and docstrings throughout `minifier.py` and `__main__.py`.
|
|
16
|
+
- `ruff` and `mypy` configuration in `pyproject.toml`.
|
|
17
|
+
- `astor` dependency eliminated — replaced with stdlib `ast.unparse` (Python 3.9+).
|
|
18
|
+
|
|
19
|
+
### Changed
|
|
20
|
+
- **Build system migrated** from PyScaffold/setuptools-scm to
|
|
21
|
+
`hatchling` + `hatch-vcs`; `setup.cfg`, `setup.py`, and `tox.ini` removed.
|
|
22
|
+
- **Python constraint broadened** to `>=3.10` (was `>=3.10,<3.11`), supporting
|
|
23
|
+
Python 3.10, 3.11, 3.12, and 3.13.
|
|
24
|
+
- `pytype` and LLM extras (`tiktoken`, `simpleaichat`) moved to optional
|
|
25
|
+
dependency groups `[types]` and `[llm]`; only `fire` and `python-minifier`
|
|
26
|
+
are required for the core CLI.
|
|
27
|
+
- `PyLLMSplitter` now imports `tiktoken` and `simpleaichat` lazily in
|
|
28
|
+
`__init__`; gracefully degrades when they are unavailable (character-count
|
|
29
|
+
estimate replaces token count; LLM summarisation silently disabled).
|
|
30
|
+
- `PyTypingMinifier.infer_types` now logs "Pytype failed for …" as `WARNING`
|
|
31
|
+
on subprocess failure or missing executable (previously silently swallowed
|
|
32
|
+
via `contextlib.suppress`).
|
|
33
|
+
- `PyTypingMinifier.process_py` now catches minification errors and logs
|
|
34
|
+
"Minification failed for …" as `ERROR`, writing the original source instead.
|
|
35
|
+
- `write_splits()` now guards against `None` `out_py_folder` and exits with a
|
|
36
|
+
warning rather than raising `TypeError`.
|
|
37
|
+
- CLI (`mdsplit4gpt --help`) now outputs help to **stdout** via
|
|
38
|
+
`fire.core.Display` override; program name shown as `mdsplit4gpt`.
|
|
39
|
+
- `tests/data/out_test.py` corrected to match actual `python-minifier` output.
|
|
40
|
+
- `test_process_folder_types_and_minify` xfail marker removed — the test now
|
|
41
|
+
passes with graceful pytype fallback.
|
|
42
|
+
|
|
43
|
+
### Removed
|
|
44
|
+
- `setup.cfg`, `setup.py`, `tox.ini`, `.isort.cfg` — superseded by
|
|
45
|
+
`pyproject.toml` with hatchling.
|
|
46
|
+
- `astor` as a runtime dependency (stdlib `ast.unparse` used instead).
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
- `contextlib` was referenced in `infer_types` and `visit_FunctionDef` but
|
|
50
|
+
not imported; import is now present.
|
|
51
|
+
- `PLAN.md` for outlining development steps and `TODO.md` for tracking task completion.
|
|
52
|
+
- Comprehensive test suite for `PyTypingMinifier`, including:
|
|
53
|
+
- Minification of single files and folders.
|
|
54
|
+
- Type inference testing (currently marked `xfail` due to issues with simple cases).
|
|
55
|
+
- Identity transformation checks (`types=False, mini=False`).
|
|
56
|
+
- Combined type inference and minification tests (marked `xfail` accordingly).
|
|
57
|
+
- Edge case handling for empty files and files with syntax errors.
|
|
58
|
+
- Configuration for `tox` to use Python 3.10 and `pyenv` for consistent test environments.
|
|
59
|
+
- Basic logging configuration in `tests/conftest.py` for improved test debugging.
|
|
60
|
+
- `__all__` to `src/split_python4gpt/__init__.py`.
|
|
61
|
+
|
|
62
|
+
### Changed
|
|
63
|
+
- **Core Logic & Robustness:**
|
|
64
|
+
- Clarified comments in `src/split_python4gpt/minifier.py`.
|
|
65
|
+
- Improved error handling and logging in `PyTypingMinifier.infer_types`:
|
|
66
|
+
- Pytype errors are now caught and logged as warnings, allowing processing to continue.
|
|
67
|
+
- Added a warning if the `pytype` executable is not found during `PyTypingMinifier` initialization.
|
|
68
|
+
- Improved error handling in `PyTypingMinifier.process_py` to catch and log errors during minification, allowing processing of other files to continue.
|
|
69
|
+
- Fixed `UnboundLocalError` in `PyTypingMinifier.process_py` that occurred when `types=False` and `mini=True`.
|
|
70
|
+
- **Documentation:**
|
|
71
|
+
- Updated `README.md` to accurately reflect current MVP functionality (focus on minification and type inference), corrected the CLI command, and linked to this `CHANGELOG.md`.
|
|
72
|
+
- **Development & CI:**
|
|
73
|
+
- Enabled and ran `pre-commit` hooks, applying formatting changes (isort, black, etc.).
|
|
74
|
+
- Minor `flake8` issues (E501, E231) were ignored as per user instruction after `black` formatting.
|
|
75
|
+
|
|
76
|
+
### Removed
|
|
77
|
+
- Unused instance variables (`py_path`, `out_py_path`, `pyi_path`, `py_code`) from `PyTypingMinifier`.
|
|
78
|
+
- Unused import of `contextlib` from `src/split_python4gpt/minifier.py`.
|
|
79
|
+
|
|
80
|
+
### Fixed
|
|
81
|
+
- Test failures related to minifier output expectations (e.g., quote style, trailing newlines).
|
|
82
|
+
- Indentation errors in test files introduced during development.
|
|
83
|
+
- Corrected `tests/data/out_test.py` to match actual minifier output.
|
|
@@ -0,0 +1,330 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: split-python4gpt
|
|
3
|
+
Version: 1.0.5
|
|
4
|
+
Summary: Reorganise large Python projects into minified, token-bounded files for LLM consumption.
|
|
5
|
+
Project-URL: Homepage, https://github.com/twardoch/split-python4gpt
|
|
6
|
+
Project-URL: Documentation, https://twardoch.github.io/split-python4gpt/
|
|
7
|
+
Project-URL: Source, https://github.com/twardoch/split-python4gpt
|
|
8
|
+
Project-URL: Changelog, https://github.com/twardoch/split-python4gpt/blob/main/CHANGELOG.md
|
|
9
|
+
Author-email: Adam Twardoch <adam+github@twardoch.com>
|
|
10
|
+
License: Apache-2.0
|
|
11
|
+
License-File: AUTHORS.md
|
|
12
|
+
License-File: LICENSE.txt
|
|
13
|
+
Keywords: code-summarization,data-preprocessing,gpt,gpt-4,openai,openai-gpt,python,text-summarization
|
|
14
|
+
Classifier: Development Status :: 4 - Beta
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Intended Audience :: Science/Research
|
|
17
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Classifier: Programming Language :: Python
|
|
20
|
+
Classifier: Programming Language :: Python :: 3
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
24
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
25
|
+
Classifier: Topic :: Text Processing
|
|
26
|
+
Requires-Python: >=3.10
|
|
27
|
+
Requires-Dist: fire>=0.5.0
|
|
28
|
+
Requires-Dist: python-minifier>=2.9.0
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: mypy>=1.0; extra == 'dev'
|
|
31
|
+
Requires-Dist: psutil>=5.9; extra == 'dev'
|
|
32
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
|
|
33
|
+
Requires-Dist: pytest>=7.0; extra == 'dev'
|
|
34
|
+
Requires-Dist: ruff>=0.4; extra == 'dev'
|
|
35
|
+
Provides-Extra: llm
|
|
36
|
+
Requires-Dist: simpleaichat>=0.2.0; extra == 'llm'
|
|
37
|
+
Requires-Dist: tiktoken>=0.4.0; extra == 'llm'
|
|
38
|
+
Provides-Extra: testing
|
|
39
|
+
Requires-Dist: psutil>=5.9; extra == 'testing'
|
|
40
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'testing'
|
|
41
|
+
Requires-Dist: pytest>=7.0; extra == 'testing'
|
|
42
|
+
Provides-Extra: types
|
|
43
|
+
Requires-Dist: pytype>=2023.6.16; extra == 'types'
|
|
44
|
+
Description-Content-Type: text/markdown
|
|
45
|
+
|
|
46
|
+
# split-python4gpt
|
|
47
|
+
|
|
48
|
+
[](https://pypi.org/project/split-python4gpt/)
|
|
49
|
+
[](https://github.com/twardoch/split-python4gpt/blob/main/LICENSE.txt)
|
|
50
|
+
[](https://twardoch.github.io/split-python4gpt/)
|
|
51
|
+
[](https://github.com/twardoch/split-python4gpt/actions/workflows/ci.yml)
|
|
52
|
+
|
|
53
|
+
`split-python4gpt` processes and reorganizes large Python projects into minified, type-annotated, and token-limited files. This prepares codebases for analysis by Large Language Models (LLMs) like OpenAI's GPT series, allowing them to handle data in manageable chunks.
|
|
54
|
+
|
|
55
|
+
## Overview
|
|
56
|
+
|
|
57
|
+
**What is `split-python4gpt`?**
|
|
58
|
+
|
|
59
|
+
A command-line and programmatic tool that takes Python files or project directories as input and performs:
|
|
60
|
+
1. **Type Inference:** Integrates with `pytype` to add type hints
|
|
61
|
+
2. **Minification:** Uses `python-minifier` with configurable options
|
|
62
|
+
3. **Code Summarization:** Replaces large functions/classes with `...` and AI-generated summaries (requires OpenAI API key)
|
|
63
|
+
4. **Splitting for LLMs:** Breaks processed code into smaller files respecting token limits
|
|
64
|
+
|
|
65
|
+
**Who is it for?**
|
|
66
|
+
|
|
67
|
+
* **Developers** working with LLMs who need to analyze large Python codebases
|
|
68
|
+
* **Researchers** in software engineering or NLP preprocessing Python code for studies
|
|
69
|
+
* Anyone reducing Python code size while preserving structure before passing to token-sensitive systems
|
|
70
|
+
|
|
71
|
+
**Why is it useful?**
|
|
72
|
+
|
|
73
|
+
* **Manages LLM Context Limits:** Breaks code into chunks that fit finite context windows
|
|
74
|
+
* **Reduces Token Count:** Minification and summarization significantly cut tokens, reducing processing time and API costs
|
|
75
|
+
* **Improves Code Clarity:** Type hints help LLMs understand code; summaries provide context for complex blocks
|
|
76
|
+
* **Automates Preprocessing:** Handles tedious parts of the MLOps pipeline for code-based LLM tasks
|
|
77
|
+
|
|
78
|
+
## Features
|
|
79
|
+
|
|
80
|
+
* **Process Single Files or Entire Directories:** Handles individual scripts or recursively processes all `.py` files
|
|
81
|
+
* **Optional Type Inference:** Uses `pytype` to add type annotations
|
|
82
|
+
* **Comprehensive Minification:** Leverages `python-minifier` with configurable options:
|
|
83
|
+
* Remove docstrings (`mini_docs`)
|
|
84
|
+
* Rename global/local variables (`mini_globs`, `mini_locs`)
|
|
85
|
+
* Hoist literal statements (`mini_lits`)
|
|
86
|
+
* Remove type annotations (`mini_annotations`)
|
|
87
|
+
* Remove `assert` and debugging statements (`mini_asserts`, `mini_debug`)
|
|
88
|
+
* Combine imports (`mini_imports`)
|
|
89
|
+
* Remove `object` base from classes (`mini_obj`)
|
|
90
|
+
* Remove `pass` statements (`mini_pass`)
|
|
91
|
+
* Convert positional to keyword arguments (`mini_posargs`)
|
|
92
|
+
* Remove explicit `return None` (`mini_retnone`)
|
|
93
|
+
* Remove shebang (`mini_shebang`)
|
|
94
|
+
* **AI-Powered Code Summarization:** Replaces large functions/classes with `...` and summaries via OpenAI models
|
|
95
|
+
* **Token-Based Splitting:** Uses `tiktoken` to count tokens and split code into multiple files respecting limits
|
|
96
|
+
* **Configurable Output:** Specify output directories for processed files and type stubs
|
|
97
|
+
* **Preserves Relative Paths:** Maintains original project structure in output
|
|
98
|
+
|
|
99
|
+
## Installation
|
|
100
|
+
|
|
101
|
+
**Prerequisites:**
|
|
102
|
+
|
|
103
|
+
* Python 3.10 or later (3.11, 3.12, 3.13 are all supported)
|
|
104
|
+
* An OpenAI API key is required **only** for LLM summarisation (`export OPENAI_API_KEY="your_api_key_here"`)
|
|
105
|
+
* `pytype` is optional — install the `[types]` extra for type inference
|
|
106
|
+
|
|
107
|
+
### Installation
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
pip install split-python4gpt
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Optional extras:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
# Type inference via pytype
|
|
117
|
+
pip install "split-python4gpt[types]"
|
|
118
|
+
|
|
119
|
+
# LLM summarisation (tiktoken + simpleaichat)
|
|
120
|
+
pip install "split-python4gpt[llm]"
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
#### From source
|
|
124
|
+
|
|
125
|
+
1. Install in virtual environment:
|
|
126
|
+
```bash
|
|
127
|
+
python -m venv .venv
|
|
128
|
+
source .venv/bin/activate
|
|
129
|
+
pip install -e ".[dev]"
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Installs dependencies: `fire`, `tiktoken`, `python-minifier`, `pytype`, and `simpleaichat`.
|
|
133
|
+
|
|
134
|
+
#### Option 2: Pre-built Binary
|
|
135
|
+
|
|
136
|
+
Download from [releases page](https://github.com/twardoch/split-python4gpt/releases):
|
|
137
|
+
|
|
138
|
+
- **Linux**: `mdsplit4gpt-linux-x86_64`
|
|
139
|
+
- **macOS**: `mdsplit4gpt-macos-x86_64`
|
|
140
|
+
- **Windows**: `mdsplit4gpt-windows-x86_64.exe`
|
|
141
|
+
|
|
142
|
+
Make executable and add to PATH:
|
|
143
|
+
```bash
|
|
144
|
+
chmod +x mdsplit4gpt-linux-x86_64
|
|
145
|
+
mv mdsplit4gpt-linux-x86_64 ~/.local/bin/mdsplit4gpt
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
#### Option 3: Source Installation
|
|
149
|
+
|
|
150
|
+
```bash
|
|
151
|
+
git clone https://github.com/twardoch/split-python4gpt.git
|
|
152
|
+
cd split-python4gpt
|
|
153
|
+
./scripts/install-dev.sh
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
## Usage
|
|
157
|
+
|
|
158
|
+
### Command-Line Interface
|
|
159
|
+
|
|
160
|
+
Primary command: `mdsplit4gpt`
|
|
161
|
+
|
|
162
|
+
```bash
|
|
163
|
+
mdsplit4gpt [PATH_OR_FOLDER] [OPTIONS]
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
**Key Arguments:**
|
|
167
|
+
|
|
168
|
+
* `path_or_folder`: Input Python file or folder
|
|
169
|
+
* `--out`: Output folder for processed files (defaults to input folder)
|
|
170
|
+
* `--pyis`: Directory for `.pyi` files (defaults to output folder)
|
|
171
|
+
* `--types`: Infer types using PyType (default: True)
|
|
172
|
+
* `--mini`: Minify scripts (default: True)
|
|
173
|
+
|
|
174
|
+
**Minification Options:**
|
|
175
|
+
* `--mini_docs`: Remove docstrings
|
|
176
|
+
* `--mini_globs`: Rename global names (default: False)
|
|
177
|
+
* `--mini_locs`: Rename local names (default: False)
|
|
178
|
+
* `--mini_lits`: Hoist literal statements
|
|
179
|
+
* `--mini_annotations`: Remove annotations
|
|
180
|
+
* `--mini_asserts`: Remove asserts
|
|
181
|
+
* `--mini_debug`: Remove debugging statements
|
|
182
|
+
* `--mini_imports`: Combine imports
|
|
183
|
+
* `--mini_obj`: Remove object base
|
|
184
|
+
* `--mini_pass`: Remove pass statements
|
|
185
|
+
* `--mini_posargs`: Convert positional to keyword args
|
|
186
|
+
* `--mini_retnone`: Remove explicit return None
|
|
187
|
+
* `--mini_shebang`: Remove shebang (set to False to preserve)
|
|
188
|
+
|
|
189
|
+
**Examples:**
|
|
190
|
+
|
|
191
|
+
1. Process single file with minification and type inference:
|
|
192
|
+
```bash
|
|
193
|
+
mdsplit4gpt my_script.py --out output_dir
|
|
194
|
+
```
|
|
195
|
+
Creates `output_dir/my_script.py` and `output_dir/split4gpt/split1.py`.
|
|
196
|
+
|
|
197
|
+
2. Process project without type inference, preserving docstrings:
|
|
198
|
+
```bash
|
|
199
|
+
mdsplit4gpt my_project/ --out processed_project/ --types=False --mini_docs=False
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
### Programmatic Usage
|
|
203
|
+
|
|
204
|
+
```python
|
|
205
|
+
from pathlib import Path
|
|
206
|
+
from split_python4gpt import PyLLMSplitter
|
|
207
|
+
|
|
208
|
+
# Set OPENAI_API_KEY environment variable for summarization
|
|
209
|
+
# os.environ["OPENAI_API_KEY"] = "your_api_key"
|
|
210
|
+
|
|
211
|
+
splitter = PyLLMSplitter(
|
|
212
|
+
gptok_model="gpt-3.5-turbo", gptok_limit=4000, gptok_threshold=200
|
|
213
|
+
)
|
|
214
|
+
|
|
215
|
+
input_path = "path/to/your/python_project_or_file"
|
|
216
|
+
output_dir = "path/to/output_directory"
|
|
217
|
+
pyi_dir = "path/to/pyi_files_directory"
|
|
218
|
+
|
|
219
|
+
processed_file_paths = splitter.process_py(
|
|
220
|
+
py_path_or_folder=input_path,
|
|
221
|
+
out_py_folder=output_dir,
|
|
222
|
+
pyi_folder=pyi_dir,
|
|
223
|
+
types=True,
|
|
224
|
+
mini=True,
|
|
225
|
+
remove_literal_statements=True,
|
|
226
|
+
rename_globals=False,
|
|
227
|
+
)
|
|
228
|
+
|
|
229
|
+
splitter.write_splits()
|
|
230
|
+
|
|
231
|
+
print(f"Processed files: {processed_file_paths}")
|
|
232
|
+
print(f"LLM splits: {Path(output_dir) / 'split4gpt'}")
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
## Technical Details
|
|
236
|
+
|
|
237
|
+
### Workflow
|
|
238
|
+
|
|
239
|
+
1. **File Discovery:** Processes single files or recursively finds all `*.py` files in folders
|
|
240
|
+
|
|
241
|
+
2. **Initialization:** Resolves and creates input/output directories; copies files if needed
|
|
242
|
+
|
|
243
|
+
3. **Per-file Processing:**
|
|
244
|
+
* **Type Inference:** Runs `pytype` subprocess, merges generated stubs back into source
|
|
245
|
+
* **Minification:** Passes code through `python-minifier` with specified options
|
|
246
|
+
|
|
247
|
+
4. **LLM Preparation:**
|
|
248
|
+
* Parses code into AST
|
|
249
|
+
* Processes top-level nodes (imports, assignments, functions, classes)
|
|
250
|
+
* For each function/class:
|
|
251
|
+
* Calculates token count with `tiktoken`
|
|
252
|
+
* If count exceeds threshold, replaces body with `...` and AI summary
|
|
253
|
+
|
|
254
|
+
5. **Splitting:**
|
|
255
|
+
* Collects all processed sections
|
|
256
|
+
* Accumulates sections into portions respecting token limits
|
|
257
|
+
* Writes `splitN.py` files in `split4gpt` subdirectory
|
|
258
|
+
|
|
259
|
+
### Output Structure
|
|
260
|
+
|
|
261
|
+
* **Processed Files:** Minified/type-annotated versions in output directory (modifies in-place if no output specified)
|
|
262
|
+
* **`.pyi` Files:** Type stubs in specified directory (defaults to output)
|
|
263
|
+
* **`split4gpt` Directory:** Contains final chunks for LLM consumption
|
|
264
|
+
|
|
265
|
+
### Core Components
|
|
266
|
+
|
|
267
|
+
* **`PyTypingMinifier`:** Handles file management, type inference, and minification
|
|
268
|
+
* **`PyBodySummarizer`:** AST transformer that summarizes large functions/classes
|
|
269
|
+
* **`PyLLMSplitter`:** Inherits from `PyTypingMinifier`, adds token counting and splitting
|
|
270
|
+
|
|
271
|
+
## Contributing
|
|
272
|
+
|
|
273
|
+
### Development Setup
|
|
274
|
+
|
|
275
|
+
1. Fork repository
|
|
276
|
+
2. Create branch: `git checkout -b feature/your-feature-name`
|
|
277
|
+
3. Set up environment:
|
|
278
|
+
```bash
|
|
279
|
+
./scripts/install-dev.sh
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
### Development Workflow
|
|
283
|
+
|
|
284
|
+
4. Make changes
|
|
285
|
+
5. Run tests:
|
|
286
|
+
```bash
|
|
287
|
+
./scripts/build-and-test.sh
|
|
288
|
+
./scripts/build-and-test.sh --with-coverage
|
|
289
|
+
./scripts/build-and-test.sh --with-performance
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
6. Code standards:
|
|
293
|
+
* Formatted with `black`
|
|
294
|
+
* Imports sorted with `isort`
|
|
295
|
+
* PEP 8 compliant
|
|
296
|
+
* Linted with `flake8`
|
|
297
|
+
* Pre-commit hooks run automatically
|
|
298
|
+
|
|
299
|
+
7. Add tests in `tests/` directory
|
|
300
|
+
8. Commit with descriptive message
|
|
301
|
+
9. Push branch: `git push origin feature/your-feature-name`
|
|
302
|
+
10. Create Pull Request
|
|
303
|
+
|
|
304
|
+
### Release Process
|
|
305
|
+
|
|
306
|
+
1. **Create release:**
|
|
307
|
+
```bash
|
|
308
|
+
./scripts/release.sh 1.2.3
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
2. **Automated CI/CD:**
|
|
312
|
+
* Tests run on push/PR for Linux, macOS, Windows
|
|
313
|
+
* Git tags trigger: PyPI publication, binary builds, GitHub releases
|
|
314
|
+
|
|
315
|
+
3. **Scripts:**
|
|
316
|
+
* `install-dev.sh`: Development setup
|
|
317
|
+
* `build-and-test.sh`: Testing
|
|
318
|
+
* `release.sh`: Create releases
|
|
319
|
+
* `get_version.py`: Check current version
|
|
320
|
+
* `validate_tag.py`: Validate version format
|
|
321
|
+
|
|
322
|
+
## License
|
|
323
|
+
|
|
324
|
+
Apache License 2.0. See [LICENSE.txt](LICENSE.txt).
|
|
325
|
+
|
|
326
|
+
## Authors
|
|
327
|
+
|
|
328
|
+
* Adam Twardoch ([@twardoch](https://github.com/twardoch))
|
|
329
|
+
|
|
330
|
+
Scaffolded with [PyScaffold](https://pyscaffold.org/).
|