skeletonpy 1.0.0__tar.gz

skeletonpy-1.0.0/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: skeletonpy
+Version: 1.0.0
+Project-URL: Homepage, https://github.com/Premik/skeletonpy
+Project-URL: Repository, https://github.com/Premik/skeletonpy
+Project-URL: Issues, https://github.com/Premik/skeletonpy/issues
+Description-Content-Type: text/markdown
+Requires-Dist: jedi
+Requires-Dist: pathspec
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+
+# SkeletonPy
+
+SkeletonPy is a Python utility for code analysis and summarization. It parses Python source code to generate a compact overview, which is particularly useful for reducing the context size when working with Large Language Models (LLMs). By providing a summarized version of the code, it helps improve the performance of AI-assisted coding and reduces token usage.
+
+## Motivation
+
+SkeletonPy is designed as a fast, pure code-driven alternative to complex local indexers (like those used in Continue or Cursor) for developers who want a lightweight, zero-overhead solution. It serves as an companion for Agentic Frameworks (by providing them with a highly accurate map of your Python repository.
+
+Why use SkeletonPy over full-context stuffing or maintaining local indexes?
+
+* **Zero Overhead Code Mapping:** Code changes frequently during development. Instead of maintaining complex embeddings, local vector databases, or dealing with expensive re-indexing processes, SkeletonPy runs instantly and entirely locally without LLMs.
+* **Focused Context:** Pumping entire repositories into the prompt window often leads to the "lost in the middle" phenomenon, where models overlook pieces of the context. A concise skeleton limits irrelevant information, which helps smaller local models and large models alike focus on what actually matters.
+* **Cost and Speed:** Passing a compact skeleton instead of full source files means significantly fewer input tokens. This directly translates to lower API costs and faster responses.
+* **Perfect for Agentic Workflows:** The generated summary contains original file names and precise line numbers down to class-level resolution.
+
+![side_by_side_example](doc/example-sbs.png)
+
+## Quick Start
+
+From your project's root directory, run `skeletonpy` with the path to your source code (`src`):
+
+```bash
+uvx skeletonpy src
+```
+
+This will scan all Python files in the `src` directory and create a `skeleton.txt` file inside it. You can then append the content of this file to your LLM prompt.
+
+## Installation
+
+You can install `skeletonpy` from PyPI using your favorite package manager like `pip` or `uv`.
+
+```bash
+pip install skeletonpy
+
+uv pip install skeletonpy
+```
+
+Alternatively, you can run it directly without a permanent installation:
+
+```bash
+pipx run skeletonpy -- --help
+
+uvx skeletonpy --help
+```
+
+Once installed, you can invoke the script:
+```bash
+skeletonpy --help
+```
+
+## Usage
+
+Run `skeletonpy` with the path to your source directory/directories. You can use include and exclude patterns to filter the files. The patterns are regular expressions.
+
+For example, to process the `src` directory, including all Python files but excluding test files, and save the output to `skeleton.txt`:
+
+```bash
+skeletonpy src --exclude "_test\.py" -o main_src.txt
+```
+
+This will generate a `main_src.txt` file. If you provide an absolute or relative path as output, it will be respected.
+See the [examples folder](examples/README.md) for more.

skeletonpy-1.0.0/README.md

@@ -0,0 +1,62 @@
+# SkeletonPy
+
+SkeletonPy is a Python utility for code analysis and summarization. It parses Python source code to generate a compact overview, which is particularly useful for reducing the context size when working with Large Language Models (LLMs). By providing a summarized version of the code, it helps improve the performance of AI-assisted coding and reduces token usage.
+
+## Motivation
+
+SkeletonPy is designed as a fast, pure code-driven alternative to complex local indexers (like those used in Continue or Cursor) for developers who want a lightweight, zero-overhead solution. It serves as an companion for Agentic Frameworks (by providing them with a highly accurate map of your Python repository.
+
+Why use SkeletonPy over full-context stuffing or maintaining local indexes?
+
+* **Zero Overhead Code Mapping:** Code changes frequently during development. Instead of maintaining complex embeddings, local vector databases, or dealing with expensive re-indexing processes, SkeletonPy runs instantly and entirely locally without LLMs.
+* **Focused Context:** Pumping entire repositories into the prompt window often leads to the "lost in the middle" phenomenon, where models overlook pieces of the context. A concise skeleton limits irrelevant information, which helps smaller local models and large models alike focus on what actually matters.
+* **Cost and Speed:** Passing a compact skeleton instead of full source files means significantly fewer input tokens. This directly translates to lower API costs and faster responses.
+* **Perfect for Agentic Workflows:** The generated summary contains original file names and precise line numbers down to class-level resolution.
+
+![side_by_side_example](doc/example-sbs.png)
+
+## Quick Start
+
+From your project's root directory, run `skeletonpy` with the path to your source code (`src`):
+
+```bash
+uvx skeletonpy src
+```
+
+This will scan all Python files in the `src` directory and create a `skeleton.txt` file inside it. You can then append the content of this file to your LLM prompt.
+
+## Installation
+
+You can install `skeletonpy` from PyPI using your favorite package manager like `pip` or `uv`.
+
+```bash
+pip install skeletonpy
+
+uv pip install skeletonpy
+```
+
+Alternatively, you can run it directly without a permanent installation:
+
+```bash
+pipx run skeletonpy -- --help
+
+uvx skeletonpy --help
+```
+
+Once installed, you can invoke the script:
+```bash
+skeletonpy --help
+```
+
+## Usage
+
+Run `skeletonpy` with the path to your source directory/directories. You can use include and exclude patterns to filter the files. The patterns are regular expressions.
+
+For example, to process the `src` directory, including all Python files but excluding test files, and save the output to `skeleton.txt`:
+
+```bash
+skeletonpy src --exclude "_test\.py" -o main_src.txt
+```
+
+This will generate a `main_src.txt` file. If you provide an absolute or relative path as output, it will be respected.
+See the [examples folder](examples/README.md) for more.

skeletonpy-1.0.0/pyproject.toml

@@ -0,0 +1,60 @@
+[project]
+name = "skeletonpy"
+version = "1.0.0"
+readme = "README.md"
+
+dependencies = [
+    "jedi",
+    "pathspec",
+]
+
+[project.urls]
+Homepage = "https://github.com/Premik/skeletonpy"
+Repository = "https://github.com/Premik/skeletonpy"
+Issues = "https://github.com/Premik/skeletonpy/issues"
+
+
+[project.scripts]
+skeletonpy = "app:main"
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "black",
+    "isort",
+    "ruff",
+    "mypy",    
+]
+
+
+[tool.pytest.ini_options]
+pythonpath = ["src", "tests"] 
+log_cli = true
+log_cli_level = "DEBUG"
+#log_cli_format = "%(asctime)s [%(levelname)8xs] %(message)s (%(filename)s:%(lineno)s)"
+#log_cli_date_format = "%Y-%m-%d %H:%M:%S"
+
+[tool.black]
+line-length = 160
+skip-string-normalization = true
+
+[tool.isort]
+profile = "black"
+src_paths = ["src", "tests"]
+line_length = 160
+
+[tool.ruff]
+line-length = 160
+ignore = ["E741", "E722", "E731"]
+
+[tool.mypy]
+strict_optional = false
+disallow_untyped_calls = true
+warn_unused_configs = true
+# disallow_untyped_defs = true 
+check_untyped_defs = true
+pretty = true
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+py-modules = ["app", "parsing", "overview", "sources"]

skeletonpy-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

skeletonpy-1.0.0/src/app.py

@@ -0,0 +1,122 @@
+import argparse
+import hashlib
+import json
+import os
+import shutil
+from pathlib import Path
+
+from parsing import ProjectModel
+from sources import FileMan
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Process project files.')
+    parser.add_argument('dirs', nargs='*', default=[os.getcwd()], help='Directories to process (default: current directory)')
+    parser.add_argument(
+        '-i',
+        '--include',
+        nargs='+',
+        default=[],
+        help="Include files if their path contains any of the given strings (substring match).",
+    )
+    parser.add_argument(
+        '-I',
+        '--include-exact',
+        nargs='+',
+        default=[],
+        help="Include files if their full path matches one of the given regular expressions. This performs a full regexp match, as opposed to the simple substring match of '--include'.",
+    )
+    parser.add_argument('-e', '--exclude', nargs='+', default=[], help="Exclude files if their full path matches one of the given regular expressions.")
+    parser.add_argument('-o', '--output', help='Output file name or path (default: same as summary file)')
+
+    args = parser.parse_args()
+    return args
+
+
+def get_args_hash(args: argparse.Namespace) -> str:
+    """Create a hash from the arguments."""
+    args_dict = vars(args)
+    # Create a sorted string representation of the dictionary
+    # to handle cases where arguments are not always in the same order.
+    sorted_args_str = json.dumps(args_dict, sort_keys=True)
+    return hashlib.sha256(sorted_args_str.encode('utf-8')).hexdigest()
+
+
+def get_output_path(src_path: Path, output_spec: str | None) -> Path:
+    """
+    Determine the output path based on source path and output specification.
+
+    Args:
+        src_path: The source directory path
+        output_spec: Output file specification - can be None, filename, or path
+
+    Returns:
+        Path object for the target output location
+    """
+    if not output_spec:
+        return src_path
+
+    output_path = Path(output_spec)
+
+    # If just a filename without path components, put in source dir
+    if len(output_path.parts) == 1:
+        return src_path / output_path
+
+    # If relative path, make it relative to source dir
+    if not output_path.is_absolute():
+        return src_path / output_path
+
+    # Otherwise use absolute path as-is
+    return output_path
+
+
+def debug() -> None:
+    import debugpy
+
+    # Allow other processes to connect to debugpy
+    debugpy.listen(("0.0.0.0", 5678))  # Use any available port, here 5678
+
+    # Pause execution until a debugger is attached
+    print("Waiting for debugger to attach...")
+    debugpy.wait_for_client()
+    print("Debugger is attached. Starting execution...")
+
+
+def main() -> None:
+    args = parse_args()
+
+    args_hash = get_args_hash(args)
+    trg = Path("/tmp/skels") / args_hash
+
+    if trg.exists() and trg.is_dir():
+        shutil.rmtree(trg)
+    trg.mkdir(parents=True, exist_ok=True)
+
+    pm = ProjectModel(proj_root_path=trg)
+    pm.file_man.src_paths = [Path(d) for d in args.dirs]
+
+    includes = []
+    if args.include:
+        includes.extend(FileMan.make_substring_match(p) for p in args.include)
+    if args.include_exact:
+        includes.extend(args.include_exact)
+
+    if not includes:
+        includes = ['.*']
+
+    pm.file_man.includes = includes
+    pm.file_man.excludes = args.exclude
+    # debug()
+
+    pm.parse_all()
+
+    for src_path in pm.file_man.src_paths:
+        output_path = get_output_path(src_path, args.output)
+        # Create parent directories if needed
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        print(f"{pm.file_man.summary_file_path}->{output_path}")
+        shutil.copy2(pm.file_man.summary_file_path, output_path)
+
+
+if __name__ == "__main__":
+    main()

skeletonpy-1.0.0/src/overview.py

@@ -0,0 +1,186 @@
+import re
+from dataclasses import dataclass, field
+from functools import cached_property
+from pathlib import Path
+from typing import Iterable, Iterator
+
+from sources import SrcFile, SrcFragment
+
+
+def trim_string(value: str, max_length: int = 200) -> str:
+    """Trims a string from the middle if it exceeds the max length."""
+    if len(value) <= max_length * 1.2:
+        return value
+    prefix_length = int(max_length * 0.7)  # 70% prefix
+    suffix_length = max_length - prefix_length - 3  # Account for '…'
+    return f"{value[:prefix_length]}…{value[-suffix_length:]}"
+
+
+def trim_collection(value: dict | list | set, max_items: int = 4) -> str:
+    """Trims collections like dicts, lists, or sets to max_items."""
+    if len(value) <= max_items:
+        return str(value)  # Return unmodified if within limit
+
+    if isinstance(value, dict):
+        header = list(value.items())[: max_items // 2]
+        footer = list(value.items())[-max_items // 2 :]
+        return f"{{{', '.join(map(str, header))}, …, {', '.join(map(str, footer))}}}"
+
+    if isinstance(value, (list, set)):
+        header = list(value)[: max_items // 2]
+        footer = list(value)[-max_items // 2 :]
+        return f"[{', '.join(map(str, header))}, …, {', '.join(map(str, footer))}]"
+
+    return str(value)
+
+
+@dataclass
+class OverviewSection:
+    frag: SrcFragment = field(repr=False)
+    name: str = "test"
+    parent_name: str = ""
+
+    out_lines_raw: list[str] = field(default_factory=list, repr=False)
+
+    def _ensure_index(self, index: int) -> None:
+        if index < len(self.out_lines_raw):
+            return
+        padding = [''] * (index + 1 - len(self.out_lines_raw))
+        self.out_lines_raw.extend(padding)
+
+    def out_lines(self) -> Iterator[str]:
+        started = True
+        for l in self.out_lines_raw:
+            if l is None:
+                continue
+            if started:  # Skip leading blank lines
+                if l.strip() == "":
+                    continue
+                started = False
+            yield l
+
+    @property
+    def content(self) -> str:
+        return "\n".join(self.out_lines())
+
+    def remove_whitespaces(self) -> None:
+        self.out_lines_raw = [re.sub(r'\s+', '', line) if line is not None else None for line in self.out_lines_raw]
+
+    def remove_blank_lines(self) -> None:
+        self.out_lines_raw = [line if line is not None and line.strip() != "" else None for line in self.out_lines_raw]
+
+    def remove_comments(self) -> None:
+        self.out_lines_raw = [re.sub(r'#.*', '', line) if line is not None else None for line in self.out_lines_raw]
+
+    def remove_single_line_docstrings(self) -> None:
+        pattern = r'(\'\'\'.*?\'\'\')|(\"\"\".*?\"\"\")'
+        self.out_lines_raw = [re.sub(pattern, '', line) if line is not None else None for line in self.out_lines_raw]
+
+    def remove_multi_line_docstrings(self, sepa='"""') -> None:
+        in_docstring = False
+
+        for i, line in enumerate(self.out_lines_raw):
+            if line is None:
+                continue
+
+            # if line.find("'''") > -1 or line.find('"""') > -1:
+            if line.find(sepa) > -1:
+                in_docstring = not in_docstring
+                if not in_docstring:
+                    self.out_lines_raw[i] = None
+            if in_docstring:
+                self.out_lines_raw[i] = None
+
+    def type_from_name(self, name: str) -> str:
+        if not name:
+            return ""
+        if '-' in name:
+            return name.split('-', 1)[0]
+        else:
+            return ""
+
+    @property
+    def type(self) -> str:
+        return self.type_from_name(self.name)
+
+    @property
+    def parent_type(self) -> str:
+        return self.type_from_name(self.parent_name)
+
+    @property
+    def src(self) -> SrcFile:
+        return self.frag.src
+
+    def __getitem__(self, index: int | slice) -> str | list[str]:
+        if isinstance(index, slice):
+            return self.out_lines_raw[index]
+        if index < 0:
+            return None
+        self._ensure_index(index)
+        return self.out_lines_raw[index]
+
+    def __setitem__(self, index: int, value: str) -> None:
+        self._ensure_index(index)
+        self.out_lines_raw[index] = value
+
+    def cut_lines(self, start: int, end: int) -> list[str]:
+        if start < 0 or start >= len(self.out_lines_raw):
+            return []
+
+        end = min(end, len(self.out_lines_raw))
+        if end < start:
+            return []
+
+        cut_lines = self.out_lines_raw[start : end + 1]
+        self.drop_lines(start, end)
+        self.frag.exclude_range_indices(start, end)
+        return cut_lines
+
+    def paste_lines(self, lines: Iterable[str], start: int = 0) -> None:
+        for i, line in enumerate(lines, start=start):
+            self[i] = line
+
+    def drop_lines(self, start: int = 0, end: int | None = None, val: str = None) -> None:
+        if end is None or start >= len(self.out_lines_raw):
+            end = len(self.out_lines_raw) - 1
+        if start < 0:
+            start = 0
+
+        if end < start:
+            return
+        self.out_lines_raw[start : end + 1] = [val] * (end - start + 1)
+
+    def set_lines(self, lines: list[str], start: int = 0) -> None:
+        self.drop_lines()
+        self.paste_lines(lines, start)
+
+    @property
+    def out_file_path(self) -> Path:
+        p = self.src.skel_path
+        if self.parent_name:
+            p = p / self.parent_name
+        return p / f"{self.name}.py.txt"
+
+    @cached_property
+    def summary_file_path(self) -> Path:
+        return self.src.file_man.summary_file_path
+
+    @cached_property
+    def previous_content(self) -> str:
+        try:
+            with open(self.out_file_path, 'r', encoding='utf-8') as f:
+                return f.read()
+        except FileNotFoundError:
+            return ''
+
+    def append_to_summary(self, text: str) -> None:
+        with open(self.summary_file_path, 'a', encoding='utf-8') as f:
+            f.write(text)
+
+    def append_content_to_summary(self) -> None:
+        self.append_to_summary(self.content)
+
+    def save(self) -> None:
+        self.out_file_path.parent.mkdir(parents=True, exist_ok=True)
+        with open(self.out_file_path, 'w', encoding='utf-8') as f:
+            f.write(self.content)

skeletonpy-1.0.0/src/parsing.py

@@ -0,0 +1,342 @@
+import re
+import shutil
+from dataclasses import dataclass, field
+from functools import cached_property
+from pathlib import Path
+from typing import Callable, Iterator
+
+import jedi
+from jedi.api.classes import Name
+
+from overview import OverviewSection, trim_string
+from sources import FileMan, SrcFile, SrcFragment
+
+
+@dataclass
+class PythonParser:
+    src: SrcFile
+    current_name: Name = None
+    extra_lines_before_current: int = 0
+    tags: list[str] = field(default_factory=list)
+    sections: list[OverviewSection] = field(default_factory=list)
+    decorator_re = re.compile(r"@(?P<name>\w+)")
+    dispatch_methods: dict[str, Callable] = field(default_factory=dict, repr=False)
+    max_string_length: int = 200
+    max_items: int = 4
+
+    def __post_init__(self) -> None:
+        self.dispatch_methods.update(
+            {
+                'module-import': self.handle_import,
+                'class-import': self.handle_import,
+                'function-import': self.handle_import,
+                'function-@property': self.handle_property,
+                'function-@cached_property': self.handle_property,
+                'class': self.handle_class,
+                'class.statement': self.handle_class_statement,
+                'module.statement': self.handle_module_statement,
+                'function': self.handle_function,
+                '': self.handle_skip,
+            }
+        )
+
+    @property
+    def module_section(self) -> OverviewSection:
+        if not self.sections:
+            return None
+        return self.sections[0]
+
+    @property
+    def current_section(self) -> OverviewSection:
+        if not self.sections:
+            return None
+        return self.sections[-1]
+
+    @property
+    def parent_section(self) -> OverviewSection:
+        if len(self.sections) < 2:
+            return None
+        return self.sections[-2]
+
+    def push_new_section(self, name: str) -> OverviewSection:
+        frg = SrcFragment(self.src)
+        ret = OverviewSection(frg, name)
+        self.sections.append(ret)
+        self.claim_current_lines()
+        return ret
+
+    def push_module(self, name: str) -> OverviewSection:
+        ret = self.push_new_section(f"module-{name}")
+        return ret
+
+    def push_class(self, name: str) -> OverviewSection:
+        ret = self.push_new_section(f"class-{name}")
+        ret.parent_name = ret.name
+        return ret
+
+    def push_function(self, name: str) -> OverviewSection:
+        pn = self.current_section.name
+        ret = self.push_new_section(f"function-{name}")
+        ret.parent_name = pn
+
+        return ret
+
+    def pop(self) -> OverviewSection:
+        cs = self.current_section
+        assert cs, "Nothing to pop"
+        self.sections.pop()
+        parent_cs = self.current_section
+
+        if cs.type in ["module", "class"]:
+            cs.remove_comments()
+            cs.remove_single_line_docstrings()
+            cs.remove_multi_line_docstrings('"""')
+            cs.remove_multi_line_docstrings("'''")
+            cs.remove_blank_lines()
+            cs.save()
+            cs.append_to_summary(f"\n\n# {self.src.module_path} {cs.frag.range_str}\n")
+            cs.append_content_to_summary()
+            return cs
+
+        parent_cs.paste_lines(cs.out_lines(), cs.frag.start)
+        return cs
+
+    def claim_current_lines(self) -> None:
+        start = self.current_range[0]
+        end = self.current_range[1]
+        for line_loc, st in enumerate(self.current_lines):
+            line = start + line_loc
+            self.current_section[line] = st
+
+        self.current_section.frag.include_range_indices(start, end)
+
+        if self.parent_section:
+            self.parent_section.cut_lines(start, end)
+
+    def handle_import(self) -> None:
+        cs = self.current_section
+        cs.drop_lines(self.current_range[0], self.current_range[1])
+
+    def handle_function(self) -> None:
+        cn = self.current_name
+        self.push_function(cn.name)
+        cs = self.current_section
+
+        try:
+            sum = f"{cn.get_type_hint()}"
+        except Exception as e:
+            # Probaby bug in Jedi, workaround by only using basic desc without types
+            print(f"Exception: {str(e)}")
+            # traceback.print_exc()
+            sum = cn.description
+
+        sum = re.sub(r'self\s*,?', '', sum)
+        q = "[\"']?"
+        sum = re.sub(f'{q}', '', sum)
+        sum = re.sub(r'\s+', '', sum)
+        sum = re.sub(rf'->\s*{q}None{q}', '', sum)
+        tags = ''.join(self.tags)
+        if tags:
+            tags += ' '
+        sum = f"{tags}{sum}"
+        cs.set_lines([sum])
+
+    def handle_class_statement(self) -> None:
+        cs = self.current_section
+        cn = self.current_name
+        code = cn.get_line_code()
+        code = re.sub(r'\s+', '', code)
+        # Apply trimming to possibly long values in the code
+        code = trim_string(code, self.max_string_length)
+        cs.paste_lines([code], self.current_range[0])
+
+    def handle_module_statement(self) -> None:
+        cs = self.current_section
+        assert cs == self.module_section
+        cn = self.current_name
+
+    def handle_property(self) -> None:
+        cn = self.current_name
+        self.push_function(cn.name)
+        cs = self.current_section
+        function_signature = cn.get_line_code()
+        # Regular expression to find return type hint
+        match = re.search(r'(->\s*[a-zA-Z_][a-zA-Z0-9_]*)', function_signature)
+        return_type = ""
+        if match:
+            return_type = match.group(1)
+        sum = f"{cn.name}{return_type}"
+        cs.set_lines([sum])
+        cs.remove_whitespaces()
+
+    def handle_class(self) -> None:
+        self.push_class(self.current_name.name)
+
+    def handle_fail(self) -> None:
+        raise NotImplementedError(f"Handler not implemented for {self.current_name}")
+
+    def handle_skip(self) -> None:
+        s = self.current_name
+        cr = self.current_range
+
+    @cached_property
+    def script(self) -> jedi.Script:
+        return jedi.Script(path=self.src.path)
+
+    @property
+    def current_range(self) -> tuple[int, int]:
+        if not self.current_name:
+            return (0, len(self.src.lines) - 1)
+        start_line, _ = self.current_name.get_definition_start_position()
+        end_line, _ = self.current_name.get_definition_end_position()
+        # Convert lines to start from 0
+        return (start_line - self.extra_lines_before_current - 1, end_line - 1)
+
+    @property
+    def current_range_str(self) -> str:
+        start = self.current_range[0]
+        end = self.current_range[1]
+        if end - start == 0:
+            return f"({start+1})"
+        return f"({start+1}-{end+1})"
+
+    @property
+    def current_lines(self) -> list[str]:
+        start, end = self.current_range
+        return self.src.lines[start : end + 1]
+
+    def get_prev_lines(self) -> Iterator[str]:
+        start, _ = self.current_range
+        for i in range(start - 1, 0, -1):
+            yield self.src.lines[i]
+
+    @property
+    def current_line(self) -> list[str]:
+        return self.current_name.get_line_code()
+
+    def find_tags(self) -> None:
+        self.tags.clear()
+        if "import" in self.current_line:
+            self.tags.append("import")
+
+        self.extra_lines_before_current = 0
+        for pl in self.get_prev_lines():
+            # Walk back through the lines to find any decorators.
+            decors = PythonParser.decorator_re.findall(pl)
+            if not decors:
+                break  # No more (or any) are found
+            self.tags += [f"@{d}" for d in decors]
+            self.extra_lines_before_current += 1
+
+    def pop_if_neded(self, line_end: int) -> None:
+        while len(self.sections) > 1 and line_end > self.current_section.frag.end:
+            self.pop()
+
+    def parse_src(self) -> None:
+        self.push_module(self.src.base_name)
+
+        for name in self.script.get_names(all_scopes=True, definitions=True):
+            self.current_name = name
+            self.find_tags()
+            line_code = name.get_line_code()
+            if "TYPE_CHECKING" in line_code or line_code.strip() == "pass":
+                continue
+
+            self.pop_if_neded(self.current_range[0])
+            if not self.sections:
+                break
+
+            self.dispatch_handler()
+        self.pop_if_neded(len(self.src.lines))  # After file end, to pop all section
+
+    def parse_debug(self) -> None:
+        self.push_new_section("module-debug")
+        self.module_section.drop_lines(val="")
+        for name in self.script.get_names(all_scopes=True):
+            self.current_name = name
+            self.find_tags()
+            self.debug_output()
+        self.current_section.save()
+
+    def dispatch_handler(self) -> None:
+        parent_type = self.current_section.type
+        if parent_type:
+            parent_type += "."
+
+        dispatch_keys = [self.current_name.type, f"{parent_type}{self.current_name.type}", '']
+
+        for tag in self.tags:
+            dispatch_keys.append(f"{self.current_name.type}-{tag}")
+            dispatch_keys.append(f"{parent_type}{self.current_name.type}-{tag}")
+
+        dispatch_keys = list(dict.fromkeys(dispatch_keys))
+        # Try dispatch keys from most specific to least
+        dispatch_keys.sort(key=len, reverse=True)
+
+        for key in dispatch_keys:
+            if key in self.dispatch_methods:
+                self.dispatch_methods[key]()
+                return
+        raise NotImplementedError(f"Handler not found for:{dispatch_keys}")
+
+    def debug_output(self) -> None:
+        s = self.current_name
+        ls = self.current_lines
+
+        for line_loc, st in enumerate(self.current_lines):
+            line = self.current_range[0] + line_loc
+            if line_loc == 0:
+                # Create tag string separately to avoid nested f-string issues
+                tag_str = ''.join([f"-{t}" for t in self.tags])
+                tp = f"{s.name[:10]}:{s.type}{tag_str}"
+            else:
+                tp = "-" * 40
+
+            formatted_line = f"{st+ '#':<82} {line+1:<3}{tp[:25]:<25}"
+            prev_line = self.module_section[line]
+            if prev_line:
+                formatted_line = f"{prev_line} {tp[:25]:<25}"
+
+            self.module_section[line] = formatted_line
+
+
+@dataclass
+class ProjectModel:
+    proj_root_path: Path
+
+    file_man: FileMan = field(init=False)
+
+    def __post_init__(self) -> None:
+        self.file_man = FileMan(proj_root_path=self.proj_root_path)
+
+    def parse_all(self) -> None:
+        self.file_man.detect_changes()
+        vals = self.file_man.src_files.values()
+        if not vals:
+            raise RuntimeError(f"No src file matched in {self.file_man.src_paths}")
+        for src in vals:
+            print(src)
+            parser = PythonParser(src)
+            parser.script
+            parser.parse_src()
+
+            parser = PythonParser(src)
+            parser.script
+            parser.parse_debug()
+
+
+if __name__ == "__main__":
+    root = Path("/home/premik/.conda/envs/unsloth/lib/python3.12/site-packages/transformers/models/gemma3")
+
+    trg = Path("/tmp/skels")
+    if trg.exists() and trg.is_dir():
+        shutil.rmtree(trg)
+    trg.mkdir(exist_ok=True)
+
+    sk_path = root / "skels"
+    if not sk_path.exists():
+        sk_path.symlink_to(trg, target_is_directory=True)
+
+    pm = ProjectModel(root)
+    pm.file_man.includes = ["modeling_gemma3"]
+    pm.parse_all()

skeletonpy-1.0.0/src/skeletonpy.egg-info/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: skeletonpy
+Version: 1.0.0
+Project-URL: Homepage, https://github.com/Premik/skeletonpy
+Project-URL: Repository, https://github.com/Premik/skeletonpy
+Project-URL: Issues, https://github.com/Premik/skeletonpy/issues
+Description-Content-Type: text/markdown
+Requires-Dist: jedi
+Requires-Dist: pathspec
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+
+# SkeletonPy
+
+SkeletonPy is a Python utility for code analysis and summarization. It parses Python source code to generate a compact overview, which is particularly useful for reducing the context size when working with Large Language Models (LLMs). By providing a summarized version of the code, it helps improve the performance of AI-assisted coding and reduces token usage.
+
+## Motivation
+
+SkeletonPy is designed as a fast, pure code-driven alternative to complex local indexers (like those used in Continue or Cursor) for developers who want a lightweight, zero-overhead solution. It serves as an companion for Agentic Frameworks (by providing them with a highly accurate map of your Python repository.
+
+Why use SkeletonPy over full-context stuffing or maintaining local indexes?
+
+* **Zero Overhead Code Mapping:** Code changes frequently during development. Instead of maintaining complex embeddings, local vector databases, or dealing with expensive re-indexing processes, SkeletonPy runs instantly and entirely locally without LLMs.
+* **Focused Context:** Pumping entire repositories into the prompt window often leads to the "lost in the middle" phenomenon, where models overlook pieces of the context. A concise skeleton limits irrelevant information, which helps smaller local models and large models alike focus on what actually matters.
+* **Cost and Speed:** Passing a compact skeleton instead of full source files means significantly fewer input tokens. This directly translates to lower API costs and faster responses.
+* **Perfect for Agentic Workflows:** The generated summary contains original file names and precise line numbers down to class-level resolution.
+
+![side_by_side_example](doc/example-sbs.png)
+
+## Quick Start
+
+From your project's root directory, run `skeletonpy` with the path to your source code (`src`):
+
+```bash
+uvx skeletonpy src
+```
+
+This will scan all Python files in the `src` directory and create a `skeleton.txt` file inside it. You can then append the content of this file to your LLM prompt.
+
+## Installation
+
+You can install `skeletonpy` from PyPI using your favorite package manager like `pip` or `uv`.
+
+```bash
+pip install skeletonpy
+
+uv pip install skeletonpy
+```
+
+Alternatively, you can run it directly without a permanent installation:
+
+```bash
+pipx run skeletonpy -- --help
+
+uvx skeletonpy --help
+```
+
+Once installed, you can invoke the script:
+```bash
+skeletonpy --help
+```
+
+## Usage
+
+Run `skeletonpy` with the path to your source directory/directories. You can use include and exclude patterns to filter the files. The patterns are regular expressions.
+
+For example, to process the `src` directory, including all Python files but excluding test files, and save the output to `skeleton.txt`:
+
+```bash
+skeletonpy src --exclude "_test\.py" -o main_src.txt
+```
+
+This will generate a `main_src.txt` file. If you provide an absolute or relative path as output, it will be respected.
+See the [examples folder](examples/README.md) for more.

skeletonpy-1.0.0/src/skeletonpy.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+README.md
+pyproject.toml
+src/app.py
+src/overview.py
+src/parsing.py
+src/sources.py
+src/skeletonpy.egg-info/PKG-INFO
+src/skeletonpy.egg-info/SOURCES.txt
+src/skeletonpy.egg-info/dependency_links.txt
+src/skeletonpy.egg-info/entry_points.txt
+src/skeletonpy.egg-info/requires.txt
+src/skeletonpy.egg-info/top_level.txt

skeletonpy-1.0.0/src/skeletonpy.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

skeletonpy-1.0.0/src/skeletonpy.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+skeletonpy = app:main

skeletonpy-1.0.0/src/skeletonpy.egg-info/requires.txt

@@ -0,0 +1,9 @@
+jedi
+pathspec
+
+[dev]
+pytest
+black
+isort
+ruff
+mypy

skeletonpy-1.0.0/src/skeletonpy.egg-info/top_level.txt

@@ -0,0 +1,4 @@
+app
+overview
+parsing
+sources

skeletonpy-1.0.0/src/sources.py

@@ -0,0 +1,365 @@
+import base64
+import hashlib
+import json
+import os
+import re
+from dataclasses import dataclass, field
+from enum import Enum, auto
+from functools import cached_property
+from pathlib import Path
+from typing import Any, Generator, Iterator, Optional
+
+from pathspec import PathSpec
+
+
+def hash(s: str, digest=9) -> str:
+    hash_bytes = hashlib.blake2s(s.encode('utf-8'), digest_size=digest).digest()
+    return base64.b64encode(hash_bytes).decode('utf-8')
+
+
+def hash_short(s: str) -> str:
+    return hash(s, 3)
+
+
+class ContentState(Enum):
+    UNKNOWN = auto()
+    NEW = auto()
+    CHANGED = auto()
+    UNCHANGED = auto()
+    REMOVED = auto()
+
+
+@dataclass
+class SrcFragment:
+    src: 'SrcFile' = field(compare=False, repr=False)
+    line_indexes: list[int] = field(default_factory=list)
+    state: ContentState = field(default=ContentState.UNKNOWN, repr=False)
+
+    def __getitem__(self, index: int | slice) -> str | list[str]:
+        if isinstance(index, slice):
+            return [self.src.lines[i] for i in self.line_indexes[index]]
+        return self.src.lines[self.line_indexes[index]]
+
+    def __iter__(self) -> Generator[str, Any, None]:
+        for i in self.line_indexes:
+            yield self.src.lines[i]
+
+    @property
+    def content(self) -> str:
+        return "\n".join(self)
+
+    def to_json(self) -> dict[int, str]:
+        if self.src.path.exists():
+            return {i: hash_short(self.src.lines[i]) for i in self.line_indexes}
+        else:
+            return {}
+
+    def merge_with(self, other: 'SrcFragment') -> 'None':
+        self.line_indexes = sorted(set(self.line_indexes + other.line_indexes))
+
+    def include_range_indices(self, start: int, end: int) -> None:
+        new_indices = list(range(start, end + 1))
+        self.line_indexes = sorted(set(self.line_indexes + new_indices))
+
+    def exclude_range_indices(self, start: int, end: int) -> None:
+        exclude_set = set(range(start, end + 1))
+        self.line_indexes = sorted(i for i in self.line_indexes if i not in exclude_set)
+
+    @property
+    def start(self) -> int:
+        if not self.line_indexes:
+            return -1
+        return self.line_indexes[0]
+
+    @property
+    def end(self) -> int:
+        if not self.line_indexes:
+            return -1
+        return self.line_indexes[-1]
+
+    @property
+    def range_str(self) -> str:
+        if self.end - self.start == 0:
+            return f"({self.start+1})"
+        return f"({self.start+1}-{self.end+1})"
+
+    @staticmethod
+    def of_json(src: 'SrcFile', json_dict: dict[int, str]) -> "SrcFragment":
+        line_indexes = []
+        state = ContentState.UNCHANGED
+
+        for index_str, hashed in json_dict.items():
+            index = int(index_str)
+            line_indexes.append(index)
+            # Check if the hash corresponds to the source line
+            # if hash_short(src.lines[index]) != hashed:
+            #     state = ContentState.CHANGED
+
+        return SrcFragment(src=src, line_indexes=line_indexes, state=state)
+
+
+@dataclass
+class SrcFile:
+    path: Path
+    file_man: 'FileMan' = field(compare=False, repr=False)
+    state: ContentState = ContentState.UNKNOWN
+    fragments: list[SrcFragment] = field(default_factory=list, repr=False)
+    _hash: str = None
+
+    @cached_property
+    def content(self) -> str:
+        if self.path.exists():
+            return self.path.read_text(encoding="utf8")
+        else:
+            return ""
+
+    @cached_property
+    def lines(self) -> list[str]:
+        return self.content.splitlines()
+
+    @cached_property
+    def base_name(self) -> str:
+        return self.path.stem
+
+    @cached_property
+    def module_path(self) -> str:
+        rel_path = self.file_man.safe_relative_path(self.path)
+        return str(rel_path)
+
+    @cached_property
+    def packages(self) -> list[str]:
+        rel_path = self.file_man.safe_relative_path(self.path)
+        return list(rel_path.parent.parts)
+
+    @cached_property
+    def skel_path(self) -> Path:
+        if not self.file_man:
+            raise ValueError("FileMan ref is null")
+        ret = self.file_man.skel_path
+        for pkg in self.packages:
+            ret = ret / pkg
+        ret = ret / self.base_name
+        ret.mkdir(parents=True, exist_ok=True)
+        return ret
+
+    @property
+    def hash(self) -> str:
+        if self._hash is None:
+            self.rehash()
+        return self._hash
+
+    def rehash(self) -> None:
+        if self.path.exists():
+            self._hash = hash(self.content)
+        else:
+            self._hash = None
+
+    def set_fragments_state(self, state: ContentState) -> None:
+        """Set the state on this file and all its fragments"""
+        self.state = state
+        for fragment in self.fragments:
+            fragment.state = state
+
+    def compare_content(self) -> None:
+        if not self.path.exists():  # Set state as REMOVED if file no longer exists
+            self.set_fragments_state(ContentState.REMOVED)
+            return
+
+        old_hash = self._hash
+        self.rehash()
+
+        if old_hash is None:  # New file
+            self.set_fragments_state(ContentState.NEW)
+            return
+        if old_hash == self._hash:  # Content unchanged
+            self.set_fragments_state(ContentState.UNCHANGED)
+            return
+
+        # Content changed
+        self.set_fragments_state(ContentState.CHANGED)
+
+    def to_json(self) -> dict:
+        return {
+            "path": str(self.path),
+            "hash": self.hash,
+            "fragments": [f.to_json() for f in self.fragments],
+        }
+
+    @staticmethod
+    def of_json(json_dict: dict, file_man: 'FileMan') -> "SrcFile":
+        src_file = SrcFile(path=Path(json_dict["path"]), file_man=file_man, _hash=json_dict["hash"])
+        if "fragments" in json_dict:
+            src_file.fragments = [SrcFragment.of_json(src_file, f) for f in json_dict["fragments"]]
+        return src_file
+
+
+@dataclass
+class FileMan:
+    proj_root_path: Path
+    src_paths: list[Path] = field(default_factory=list)
+    src_files: dict[Path, SrcFile] = field(default_factory=dict)
+    skel_path: Path = field(init=False)
+    cache_path: Optional[Path] = None
+    includes: list[str] = field(default_factory=lambda: [".*"])
+    excludes: list[str] = field(default_factory=list)
+
+    def __post_init__(self) -> None:
+        if not self.proj_root_path.is_dir() or not self.proj_root_path.exists():
+            raise ValueError(f"Project root path {self.proj_root_path} must be an existing directory")
+
+        self.skel_path = self.proj_root_path / "skels"
+        self.skel_path.mkdir(exist_ok=True)
+
+        if not self.cache_path:
+            self.cache_path = Path(self.skel_path, "cache.json")
+
+        if not self.src_paths:
+            self.src_paths = [self.proj_root_path]
+
+    @cached_property
+    def gitignore(self) -> PathSpec:
+        patterns = []
+        # Collect patterns from all source paths
+        for src_path in self.src_paths:
+            gitignore_path = src_path / '.gitignore'
+            if gitignore_path.exists():
+                with open(gitignore_path) as f:
+                    patterns.extend(f.readlines())
+
+        # Also check project root if different from source paths
+        if self.proj_root_path not in self.src_paths:
+            gitignore_path = self.proj_root_path / '.gitignore'
+            if gitignore_path.exists():
+                with open(gitignore_path) as f:
+                    patterns.extend(f.readlines())
+
+        # Create a single PathSpec from all collected patterns
+        return PathSpec.from_lines('gitwildmatch', patterns)
+
+    def clear(self) -> None:
+        self.src_files.clear()
+
+    def load(self) -> None:
+        self.clear()
+        if not self.cache_path.exists():
+            return
+        with open(self.cache_path, encoding='utf-8') as f:
+            json_data = json.loads(f.read())
+        self.src_files = {Path(key): SrcFile.of_json(value, self) for key, value in json_data.items()}
+
+    def save(self) -> None:
+        json_data = {str(key): value.to_json() for key, value in self.src_files.items()}
+        with open(self.cache_path, 'w', encoding='utf-8') as f:
+            json.dump(json_data, f, indent=2)
+
+    @cached_property
+    def summary_file_path(self) -> Path:
+        return self.skel_path / "summary.py.txt"
+
+    def detect_changes(self) -> None:
+        self.load()
+
+        for src_file_path in self.source_files():
+            src_file = self.src_files.get(src_file_path, None)
+            if not src_file:  # new file
+                self.src_files[src_file_path] = SrcFile(src_file_path, self, ContentState.NEW)
+                continue
+            src_file.compare_content()
+        for src_file in self.src_files.values():
+            if src_file.state == ContentState.UNKNOWN:
+                # Unknown -> has not been visited since load, means the file has been removed
+                src_file.state = ContentState.REMOVED
+
+    @cached_property
+    def src_root(self) -> Path:
+        if not self.src_paths:
+            return self.proj_root_path
+
+        # Convert paths to absolute and get parts
+        abs_paths = [p.absolute() for p in self.src_paths]
+        path_parts = [p.parts for p in abs_paths]
+
+        common = []
+        for parts in zip(*path_parts):  # Find common prefix among all paths
+            if len(set(parts)) != 1:
+                break
+            common.append(parts[0])
+
+        if not common:
+            return self.proj_root_path
+
+        return Path(*common)
+
+    @cached_property
+    def includes_rx(self) -> list[re.Pattern]:
+        return [re.compile(p) for p in self.includes]
+
+    @cached_property
+    def excludes_rx(self) -> list[re.Pattern]:
+        return [re.compile(p) for p in self.excludes]
+
+    def rx_list_match(self, text: str, rx_list: list[re.Pattern]) -> bool:
+        return any(rx.fullmatch(text) for rx in rx_list)
+
+    def includes_match(self, text: str) -> bool:
+        return self.rx_list_match(text, self.includes_rx)
+
+    def excludes_match(self, text: str) -> bool:
+        return self.rx_list_match(text, self.excludes_rx)
+
+    @staticmethod
+    def make_substring_match(pattern: str) -> str:
+        return f".*{pattern}.*"
+
+    def safe_relative_path(self, path: Path | str) -> Path:
+        """
+        Safely get a path relative to src_root, handling cases where the path is not within src_root.
+        If the path is not within src_root, returns the path as is.
+        """
+        path = Path(path) if isinstance(path, str) else path
+        try:
+            return path.relative_to(self.src_root)
+        except ValueError:
+            # Path is not within src_root, return the path as is
+            # This handles relative imports and files outside the source root
+            return path
+
+    def fn_match(self, filename: str) -> bool:
+        # Early returns for non-Python files and gitignore matches
+        if not filename.endswith('.py'):
+            return False
+
+        # Get relative path safely
+        rel_path = self.safe_relative_path(filename)
+
+        # Only check gitignore if the path is actually relative
+        # (i.e., it was successfully made relative to src_root)
+        if str(rel_path) != filename:
+            if self.gitignore.match_file(str(rel_path)):
+                return False
+
+        # Check include/exclude patterns
+        # Use relative path for matching to allow cleaner patterns
+        match_target = str(rel_path)
+        if not self.includes_match(match_target):
+            return False
+
+        if self.excludes_match(match_target):
+            return False
+
+        return True
+
+    def source_files(self) -> Iterator[Path]:
+        for src_path in self.src_paths:
+            for dir, _, filenames in os.walk(src_path):
+                dirpath = Path(dir)
+                for filename in filenames:
+                    filepath = dirpath / filename
+                    if self.fn_match(str(filepath)):
+                        yield filepath
+
+
+if __name__ == "__main__":
+    fm = FileMan(proj_root_path=Path("/wrk/dev/Skeleton"))
+    # for src in fm.sources.values():
+    #     print(src.lines)