live-vector-space 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- live_vector_space-0.1.0.dist-info/METADATA +171 -0
- live_vector_space-0.1.0.dist-info/RECORD +13 -0
- live_vector_space-0.1.0.dist-info/WHEEL +5 -0
- live_vector_space-0.1.0.dist-info/entry_points.txt +2 -0
- live_vector_space-0.1.0.dist-info/licenses/LICENSE +662 -0
- live_vector_space-0.1.0.dist-info/top_level.txt +1 -0
- lvs/__init__.py +8 -0
- lvs/__main__.py +5 -0
- lvs/cli.py +20 -0
- lvs/codemap.py +656 -0
- lvs/live.py +1346 -0
- lvs/mcp_server.py +226 -0
- lvs/skeleton.py +150 -0
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: live-vector-space
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: A whole codebase in a few KB, every byte one lookup away: a small map plus exact, numbered pieces for LLMs.
|
|
5
|
+
License-Expression: AGPL-3.0-or-later
|
|
6
|
+
Project-URL: Homepage, https://github.com/devkancheti4-design/live-vector-space
|
|
7
|
+
Keywords: llm,codebase map,context window,compression,mcp,code search
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Programming Language :: Python :: 3
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Topic :: Software Development
|
|
14
|
+
Requires-Python: >=3.14
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
License-File: LICENSE
|
|
17
|
+
Requires-Dist: numpy>=1.24
|
|
18
|
+
Dynamic: license-file
|
|
19
|
+
|
|
20
|
+
# lvs: a whole codebase in a few KB, every byte one lookup away
|
|
21
|
+
|
|
22
|
+
`lvs` gives an LLM a small map of a whole codebase (a few KB, however big the code) plus exact,
|
|
23
|
+
numbered pieces of the code to look up. The store keeps every byte, compressed about as small as zip,
|
|
24
|
+
and hands over any single piece on its own. So a model can know a codebase many times bigger than its
|
|
25
|
+
context window without reading it whole, and without ever decompressing anything itself.
|
|
26
|
+
|
|
27
|
+
Measured on two big Python codebases that no model had seen (66 blind runs, details in
|
|
28
|
+
[MILESTONE.md](https://github.com/devkancheti4-design/live-vector-space/blob/main/MILESTONE.md)):
|
|
29
|
+
|
|
30
|
+
| What was measured | Result |
|
|
31
|
+
|---|---|
|
|
32
|
+
| The map view alone (`map` plus a zoom into the main package, about 4.6 KB), 10 architecture questions per run | **100/100** right (raw code of the same size: 34/100) |
|
|
33
|
+
| Map + lookups, exact questions about codebases of 5-6 million tokens | **97.5%** right (468/480), for **2-4%** of the tokens a full read would take |
|
|
34
|
+
| A 6.2-million-token codebase, one session in a 1M-token window | **46/48** right in 111,501 tokens; **88.7%** of the code stayed compressed |
|
|
35
|
+
| Store size (reef, 14.5 MB of Python) | **30.7%** of the code (zip -9: 30.3%), any of 17,660 pieces readable alone, byte-exact |
|
|
36
|
+
|
|
37
|
+
## Install
|
|
38
|
+
|
|
39
|
+
Needs Python 3.14+ (for the built-in `compression.zstd`) and numpy.
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
pip install live-vector-space
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
The command it installs is `lvs`. For the latest code from GitHub instead:
|
|
46
|
+
`pip install git+https://github.com/devkancheti4-design/live-vector-space`
|
|
47
|
+
|
|
48
|
+
## Use
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
cd your-project
|
|
52
|
+
lvs build # the map and the compressed store, in .lvs/
|
|
53
|
+
lvs map # the whole codebase in a few KB: read this first
|
|
54
|
+
lvs files src/server # zoom into a folder: its files, intents and piece numbers
|
|
55
|
+
lvs find timeout retry # the numbered pieces that mention each word, with the line that matches
|
|
56
|
+
lvs get 812 813 # the exact code of those pieces
|
|
57
|
+
lvs rebuild /tmp/copy # every file back, byte-exact
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Lookups work from any subfolder (`lvs` finds `.lvs/` in a parent folder, the way git finds `.git`).
|
|
61
|
+
Run `lvs build` again after the code changes. `--store PREFIX` keeps several maps apart. Folders that
|
|
62
|
+
are never the project's own code (hidden folders, `venv`, `site-packages`, `node_modules`, `build`,
|
|
63
|
+
`dist`) are skipped; `--exclude NAME ...` skips more.
|
|
64
|
+
|
|
65
|
+
## Give it to an AI agent (MCP)
|
|
66
|
+
|
|
67
|
+
`lvs mcp` is a Model Context Protocol server over stdio, with five tools: `map`, `files`, `find`,
|
|
68
|
+
`get` (read-only lookups in the compressed store) and `build` (make or refresh the store). It tells
|
|
69
|
+
the agent to read the map first, then find and get only the pieces it needs.
|
|
70
|
+
|
|
71
|
+
With Claude Code, from your project folder:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
claude mcp add lvs -- lvs mcp --root "$PWD"
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Any other MCP client takes the same command in its config:
|
|
78
|
+
|
|
79
|
+
```json
|
|
80
|
+
{"mcpServers": {"lvs": {"command": "lvs", "args": ["mcp", "--root", "/path/to/your-project"]}}}
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Watch code live
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
lvs live path/to/file.py # every statement as a vector in 3-D space, redrawn on each save
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## How it works
|
|
90
|
+
|
|
91
|
+
- **Numbered exact pieces.** Each file is split at its natural joints (functions, classes, sections)
|
|
92
|
+
into pieces numbered across the whole codebase. Joined in order, they rebuild every file byte for byte.
|
|
93
|
+
- **The store.** Every piece is compressed with zstd and a dictionary shared by all pieces, so the whole
|
|
94
|
+
store is about as small as zip, yet any one piece unpacks on its own in an instant.
|
|
95
|
+
- **The map.** Folders and files with sizes and one-line intents, plus the code's shape: every statement
|
|
96
|
+
becomes a point in a 3-D cube, and the points gather into cores of similar code, each with its words
|
|
97
|
+
and its pieces. The map stays a few KB at any size (2.1 KB for reef's 14.5 MB of code), and `files`
|
|
98
|
+
zooms in, page by page.
|
|
99
|
+
- **The word index.** `find` goes from words to piece numbers and shows the line that matched, so the
|
|
100
|
+
model reads only the pieces it needs.
|
|
101
|
+
|
|
102
|
+
## Tests
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
python3 -m unittest discover -s tests
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## License
|
|
109
|
+
|
|
110
|
+
GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later); see [LICENSE](https://github.com/devkancheti4-design/live-vector-space/blob/main/LICENSE).
|
|
111
|
+
|
|
112
|
+
# The research behind it
|
|
113
|
+
|
|
114
|
+
Watch a Python file live, as vectors in 3-D space, plus a set of experiments on
|
|
115
|
+
squeezing code into shapes and maps that an LLM can read cheaply. The scripts below are the
|
|
116
|
+
research versions; the `lvs` package in `src/lvs/` is the product built from them.
|
|
117
|
+
|
|
118
|
+

|
|
119
|
+
|
|
120
|
+
## What's here
|
|
121
|
+
|
|
122
|
+
| File | What it does |
|
|
123
|
+
|---|---|
|
|
124
|
+
| `live_vector_space.py` | Turns every statement of a Python file into a 1024-d token vector, projects them to 3-D with PCA, and serves a live browser view: a cube filled with 14,000 particles of random "fluid" that settle into the code's shape. It updates every time the file is saved. |
|
|
125
|
+
| `skeleton.py` | Squeezes a file into 8 labeled cores of similar statements (a position in the cube plus a few words each). |
|
|
126
|
+
| `shape_codec.py` | Stores the 3-D points on a 12-bit grid in the cube, tied to anchor shapes, and decodes a perfect copy. |
|
|
127
|
+
| `optimal_shapes.py` | Lets the points choose their own shapes (blob, line, curve, sheet) by bit count (minimum description length). |
|
|
128
|
+
| `code_map.py` | A cheap map of a file plus exact, numbered pieces an LLM can look up; rebuilds the file byte-exact. |
|
|
129
|
+
|
|
130
|
+
Generated outputs: `code_square.svg`, `square_vs_scrambled.svg`, `shape_map.svg`,
|
|
131
|
+
`optimal_shapes.html`, `optimal_shapes_control.html` (open locally in a browser),
|
|
132
|
+
`code_map.json` and `code_store.bin`.
|
|
133
|
+
|
|
134
|
+
## Run it
|
|
135
|
+
|
|
136
|
+
Needs Python 3 with numpy; `code_map.py` needs Python 3.14+ for the built-in `compression.zstd`.
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
python3 live_vector_space.py # live 3-D view of itself at http://localhost:8765
|
|
140
|
+
python3 live_vector_space.py path/to/other.py # any other Python file
|
|
141
|
+
python3 live_vector_space.py --square square.svg # save the square-minus-the-path picture
|
|
142
|
+
python3 skeleton.py --stats
|
|
143
|
+
python3 shape_codec.py --svg shape_map.svg
|
|
144
|
+
python3 optimal_shapes.py --html optimal_shapes.html
|
|
145
|
+
python3 code_map.py build
|
|
146
|
+
python3 code_map.py find fluid svg
|
|
147
|
+
python3 code_map.py get 50
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Viewer controls: drag to rotate, scroll to zoom, hover or click a point. Keys: `H` shake the fluid,
|
|
151
|
+
`F` fluid on/off, `V` vectors, `P` code path, `L` labels, `T` table, `S` square, `Space` spin.
|
|
152
|
+
|
|
153
|
+
## What the experiments found
|
|
154
|
+
|
|
155
|
+
Measured on `live_vector_space.py` itself (52,552 bytes, 357 statements).
|
|
156
|
+
|
|
157
|
+
- **The shape alone doesn't carry meaning.** Swapping all 964 names and words for gibberish leaves the
|
|
158
|
+
fingerprint identical (exactly, with unhashed word counts). Fresh models given only the shape's numbers
|
|
159
|
+
scored 7/14 on a quiz about the program, the same as answering "no" to everything.
|
|
160
|
+
- **Words carry the gist cheaply.** The file's one-line intent plus the skeleton (about 750 bytes) scored
|
|
161
|
+
12/14 on what the program does, but 0/4 on details.
|
|
162
|
+
- **A map plus numbered lookups is the sweet spot.** With the map and exact lookups by number
|
|
163
|
+
(`code_map.py`), a fresh model scored 14/14 and 4/4, the same as reading the whole file, using about
|
|
164
|
+
9,200 extra tokens instead of about 30,400, and reading 8 of the 80 pieces.
|
|
165
|
+
- **Compression.** The cube grid stores the 357 points in 1,609 bytes; shapes the points chose themselves
|
|
166
|
+
(8 blobs) take 1,471. The cloud is a nearly round ball, so lines and sheets don't help here (on control
|
|
167
|
+
data that really forms lines and a sheet they save 32%). An exact copy of the code needs about 16 KB
|
|
168
|
+
(brotli); numbered pieces with a shared zstd dictionary take 24 KB, the price of reading any piece on
|
|
169
|
+
its own.
|
|
170
|
+
|
|
171
|
+
Each LLM test was a single run on one file, so small score differences may be chance.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
live_vector_space-0.1.0.dist-info/licenses/LICENSE,sha256=jVa0BUaKrRH4erV2P5AeJ24I2WRv9chIGxditreJ6e0,34524
|
|
2
|
+
lvs/__init__.py,sha256=CLykR8n8YFE55lDDfaT_5lJ0qtlJtQWdFfNdsCRmXl4,411
|
|
3
|
+
lvs/__main__.py,sha256=E6Gls0DNz8GQK2K-kOUIx8cYhgANW_CH54VKrfCfs14,52
|
|
4
|
+
lvs/cli.py,sha256=W0CchnUJj0p5H_BpnPBXm9XjbUNwBxKfiltK6WgvCgs,606
|
|
5
|
+
lvs/codemap.py,sha256=O9qCInQUYUorX5eO3ZH35tafwoeXeZSBs6ym3F-s-U8,29307
|
|
6
|
+
lvs/live.py,sha256=0L79jKtNblBYlbgXyUV4GneTkuWuphpHT1Wh4s_8jIs,52497
|
|
7
|
+
lvs/mcp_server.py,sha256=AbVSmEmO2LiMj9mpRjW2EjHAj4kvVWfPebiULv9hArY,11268
|
|
8
|
+
lvs/skeleton.py,sha256=NnIFtQp9DWfkHP1tgEQJJsAY1OVoDiF8nQerL2IqL2Q,6596
|
|
9
|
+
live_vector_space-0.1.0.dist-info/METADATA,sha256=3-LVsTxDMj0icDDXtM3pRyt6jsckSXJJCbfg9XUfeF0,8610
|
|
10
|
+
live_vector_space-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
|
|
11
|
+
live_vector_space-0.1.0.dist-info/entry_points.txt,sha256=vT_86H3DkCY3pca5jHqsBY4LI6hiaMMSC3-qW-HO_Y4,37
|
|
12
|
+
live_vector_space-0.1.0.dist-info/top_level.txt,sha256=K_kQa17eb9jRWXKIqY2S_8TPrD9ubHp1gDq7OFvIhiY,4
|
|
13
|
+
live_vector_space-0.1.0.dist-info/RECORD,,
|