skylos 2.0.0__tar.gz → 2.1.0__tar.gz

{skylos-2.0.0 → skylos-2.1.0}/PKG-INFO

@@ -1,8 +1,11 @@
 Metadata-Version: 2.4
 Name: skylos
-Version: 2.0.0
+Version: 2.1.0
 Summary: A static analysis tool for Python codebases
 Author-email: oha <aaronoh2015@gmail.com>
 Requires-Python: >=3.9
 Requires-Dist: inquirer>=3.0.0
+Requires-Dist: flask>=2.1.0
+Requires-Dist: flask-cors>=3.0.0
+Requires-Dist: libcst>=1.8.2
 Dynamic: requires-python

{skylos-2.0.0 → skylos-2.1.0}/README.md

@@ -9,7 +9,7 @@
    <img src="assets/SKYLOS.png" alt="Skylos Logo" width="200">
 </div>
 
-> A static analysis tool for Python codebases written in Python (formerly was written in Rust but we ditched that) that detects unreachable functions and unused imports, aka dead code. Faster and better results than many alternatives like Flake8 and Pylint, and finding more dead code than Vulture in our tests with comparable speed.
+> A static analysis tool for Python codebases written in Python that detects unreachable functions and unused imports, aka dead code. Faster and better results than many alternatives like Flake8 and Pylint, and finding more dead code than Vulture in our tests with comparable speed.
 
 <div align="center">
    <img src="assets/FE_SS.png" alt="FE" width="800">
@@ -23,7 +23,7 @@
 - [Installation](#installation)
 - [Quick Start](#quick-start)
 - [Web Interface](#web-interface)
-- [Understanding Confidence Levels](#understanding-confidence-levels)
+- [Design](#design)
 - [Test File Detection](#test-file-detection)
 - [Folder Management](#folder-management)
 - [Ignoring Pragmas](#ignoring-pragmas)
@@ -42,6 +42,7 @@
 
 ## Features
 
+* **CST-safe removals:** Uses LibCST to remove selected imports or functions (handles multiline imports, aliases, decorators, async etc..)
 * **Framework-Aware Detection**: Attempt at handling Flask, Django, FastAPI routes and decorators  
 * **Test File Exclusion**: Auto excludes test files (you can include it back if you want)
 * **Interactive Cleanup**: Select specific items to remove from CLI
@@ -82,18 +83,15 @@ pip install skylos
 ### From Source
 
 ```bash
-# Clone the repository
 git clone https://github.com/duriantaco/skylos.git
 cd skylos
 
-## Install your dependencies 
 pip install .
 ```
 
 ## Quick Start
 
 ```bash
-# Analyze a project
 skylos /path/to/your/project
 
 # To launch the front end
@@ -105,7 +103,6 @@ skylos --interactive /path/to/your/project
 # Dry run - see what would be removed
 skylos --interactive --dry-run /path/to/your/project 
 
-# Output to JSON
 skylos --json /path/to/your/project 
 
 # With confidence
@@ -117,13 +114,22 @@ skylos path/to/your/file --confidence 20 ## or whatever value u wanna set
 Skylos includes a modern web dashboard for interactive analysis:
 
 ```bash
-# Start web interface
 skylos run
 
 # Opens browser at http://localhost:5090
 ```
 
-## Understanding Confidence Levels
+## Design
+
+### Summary 
+
+Framework endpoints are often invoked externally (eg, via HTTP, signals), so we use framework aware signals + confidence scores to try avoid false positives while still catching dead codes
+
+Name resolution handles aliases and modules, but when things get ambiguous, we rather miss some dead code than accidentally mark live code as dead lmao
+
+Tests are excluded by default because their call patterns are noisy. You can opt in when you really need to audit it
+
+### Understanding Confidence Levels
 
 Skylos uses a confidence-based system to try to handle Python's dynamic nature and web frameworks.
 
@@ -204,7 +210,6 @@ By default, Skylos excludes common folders: `__pycache__`, `.git`, `.pytest_cach
 
 ### Folder Options
 ```bash
-# List default excluded folders
 skylos --list-default-excludes
 
 # Exclude single folder (The example here will be venv)
@@ -228,8 +233,8 @@ Arguments:
   PATH  Path to the Python project to analyze
 
 Options:
-  -h, --help                    Show this help message and exit
-  -j, --json                   Output raw JSON instead of formatted text  
+  -h, --help                   Show this help message and exit
+  --json                       Output raw JSON instead of formatted text  
   -o, --output FILE            Write output to file instead of stdout
   -v, --verbose                Enable verbose output
   -i, --interactive            Interactively select items to remove
@@ -241,48 +246,11 @@ Options:
   -c, --confidence LEVEL       Confidence threshold (0-100). Lower values will show more items.
 ```
 
-## Example Output
-
-```
-Python Static Analysis Results
-===================================
-
-Summary:
-  • Unreachable functions: 48
-  • Unused imports: 8
-
-Unreachable Functions
-========================
-
- 1. module_13.test_function
-    └─ /Users/oha/project/module_13.py:5
- 2. module_13.unused_function
-    └─ /Users/oha/project/module_13.py:13
-...
-
-
-Unused Imports
-=================
-
- 1. os
-    └─ /Users/oha/project/module_13.py:1
- 2. json
-    └─ /Users/oha/project/module_13.py:3
-...
-```
-
-Next steps:
-
-  • Use `--interactive` to select specific items to remove
-
-  • Use `--dry-run` to preview changes before applying them
-
-
 ## Interactive Mode
 
 The interactive mode lets you select specific functions and imports to remove:
 
-1. **Select items**: Use arrow keys and space to select/deselect
+1. **Select items**: Use arrow keys and `spacebar` to select/unselect
 2. **Confirm changes**: Review selected items before applying
 3. **Auto-cleanup**: Files are automatically updated
 
@@ -297,11 +265,9 @@ The interactive mode lets you select specific functions and imports to remove:
 ### Setup
 
 ```bash
-# Clone the repository
 git clone https://github.com/duriantaco/skylos.git
 cd skylos
 
-# Create a virtual environment
 python -m venv venv
 source venv/bin/activate  # On Windows: venv\Scripts\activate
 ```
@@ -309,7 +275,6 @@ source venv/bin/activate  # On Windows: venv\Scripts\activate
 ### Running Tests
 
 ```bash
-# Run Python tests
 python -m pytest tests/
 ```
 
@@ -343,8 +308,6 @@ A: Start with 60 (default) for safe cleanup. Use 30 for framework applications.
 - **Test data**: Limited scenarios, your mileage may vary
 - **False positives**: Always manually review before deleting code
 
-This is a STATIC code analyzer. If we can detect 100% of all dead code in any structure and framework, we wouldn't be sitting here. But we definitely tried our best.
-
 ## Troubleshooting
 
 ### Common Issues
@@ -378,7 +341,6 @@ We welcome contributions! Please read our [Contributing Guidelines](CONTRIBUTING
 - [ ] Configuration file support 
 - [ ] Git hooks integration
 - [ ] CI/CD integration examples
-~~- [] Support for other languages (unless I have contributors, this ain't possible)~~
 - [ ] Further optimization
 
 ## License

{skylos-2.0.0 → skylos-2.1.0}/pyproject.toml

@@ -4,11 +4,11 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "skylos"
-version = "2.0.0"
+version = "2.1.0"
 requires-python = ">=3.9"
 description = "A static analysis tool for Python codebases"
 authors = [{name = "oha", email = "aaronoh2015@gmail.com"}]
-dependencies = ["inquirer>=3.0.0"]
+dependencies = ["inquirer>=3.0.0", "flask>=2.1.0", "flask-cors>=3.0.0", "libcst>=1.8.2"]
 
 [project.scripts]
 skylos = "skylos.cli:main"

{skylos-2.0.0 → skylos-2.1.0}/setup.py

@@ -2,13 +2,15 @@ from setuptools import setup, find_packages
 
 setup(
     name="skylos",
-    version="2.0.0",
+    version="2.1.0",
     packages=find_packages(),
     python_requires=">=3.9",
     install_requires=[
         "inquirer>=3.0.0",
-        "flask>=2.0.0",
-        "flask-cors>=3.0.0"],
+        "flask>=2.1.0",
+        "flask-cors>=3.0.0",
+        "libcst>=1.8.2"],
+
     classifiers=[
         "Development Status :: 4 - Beta",
         "Intended Audience :: Developers",

{skylos-2.0.0 → skylos-2.1.0}/skylos/__init__.py

@@ -1,6 +1,6 @@
 from skylos.analyzer import analyze
 
-__version__ = "2.0.0"
+__version__ = "2.1.0"
 
 def debug_test():
     return "debug-ok"

{skylos-2.0.0 → skylos-2.1.0}/skylos/analyzer.py

@@ -6,7 +6,7 @@ import logging
 from pathlib import Path
 from collections import defaultdict
 from skylos.visitor import Visitor
-from skylos.constants import ( DEFAULT_EXCLUDE_FOLDERS, PENALTIES, AUTO_CALLED )
+from skylos.constants import ( PENALTIES, AUTO_CALLED )
 from skylos.test_aware import TestAwareVisitor
 import os
 import traceback
@@ -15,21 +15,6 @@ from skylos.framework_aware import FrameworkAwareVisitor, detect_framework_usage
 logging.basicConfig(level=logging.INFO,format='%(asctime)s - %(levelname)s - %(message)s')
 logger=logging.getLogger('Skylos')
 
-def parse_exclude_folders(user_exclude_folders, use_defaults=True, include_folders=None):
-    exclude_set = set()
-    
-    if use_defaults:
-        exclude_set.update(DEFAULT_EXCLUDE_FOLDERS)
-        
-    if user_exclude_folders:
-        exclude_set.update(user_exclude_folders)
-    
-    if include_folders:
-        for folder in include_folders:
-            exclude_set.discard(folder)
-    
-    return exclude_set
-
 class Skylos:
     def __init__(self):
         self.defs={}
@@ -126,17 +111,31 @@ class Skylos:
         for ref, _ in self.refs:
             if ref in self.defs:
                 self.defs[ref].references += 1
-                
                 if ref in import_to_original:
                     original = import_to_original[ref]
                     self.defs[original].references += 1
                 continue
-            
+
             simple = ref.split('.')[-1]
-            matches = simple_name_lookup.get(simple, [])
-            for definition in matches:
-                definition.references += 1
- 
+            ref_mod = ref.rsplit(".", 1)[0]
+            candidates = simple_name_lookup.get(simple, [])
+
+            if ref_mod:
+                filtered = []
+                for d in candidates:
+                    if d.name.startswith(ref_mod + ".") and d.type != "import":
+                        filtered.append(d)
+                candidates = filtered
+            else:
+                filtered = []
+                for d in candidates:
+                    if d.type != "import":
+                        filtered.append(d)
+                candidates = filtered
+
+            if len(candidates) == 1:
+                candidates[0].references += 1
+
         for module_name in self.dynamic:
             for def_name, def_obj in self.defs.items():
                 if def_obj.name.startswith(f"{module_name}."):
@@ -160,8 +159,8 @@ class Skylos:
             confidence -= PENALTIES["private_name"] 
         if def_obj.simple_name.startswith("__") and def_obj.simple_name.endswith("__"):
             confidence -= PENALTIES["dunder_or_magic"]
-        if def_obj.type == "variable" and def_obj.simple_name == "_":
-            confidence -= PENALTIES["underscored_var"]
+        if def_obj.type == "variable" and def_obj.simple_name.isupper():
+            confidence = 0
         if def_obj.in_init and def_obj.type in ("function", "class"):
             confidence -= PENALTIES["in_init_file"]
         if def_obj.name.split(".")[0] in self.dynamic:
@@ -207,6 +206,14 @@ class Skylos:
                 for method in methods:
                     if method.simple_name in AUTO_CALLED:
                         method.references += 1
+                    
+                    if (method.simple_name.startswith("visit_") or
+                        method.simple_name.startswith("leave_") or
+                        method.simple_name.startswith("transform_")):
+                        method.references += 1
+
+                    if method.simple_name == "format" and cls.endswith("Formatter"):
+                        method.references += 1
 
     def analyze(self, path, thr=60, exclude_folders=None):
         files, root = self._get_python_files(path, exclude_folders)
@@ -246,8 +253,17 @@ class Skylos:
         self._mark_refs()
         self._apply_heuristics() 
         self._mark_exports()
-        
-        thr = max(0, thr)
+
+        shown = 0
+
+        def def_sort_key(d):
+            return (d.type, d.name)
+
+        for d in sorted(self.defs.values(), key=def_sort_key):
+            if shown >= 50:
+                break
+            print(f" {d.type:<8} refs={d.references:<2} conf={d.confidence:<3} exported={d.is_exported} line={d.line:<4} {d.name}")
+            shown += 1
 
         unused = []
         for definition in self.defs.values():
@@ -262,7 +278,7 @@ class Skylos:
             "unused_parameters": [],
             "analysis_summary": {
                 "total_files": len(files),
-                "excluded_folders": exclude_folders if exclude_folders else [],
+                "excluded_folders": exclude_folders or [],
             }
         }
         
@@ -295,7 +311,7 @@ def proc_file(file_or_args, mod=None):
 
         fv = FrameworkAwareVisitor(filename=file)
         fv.visit(tree)
-
+        fv.finalize()
         v  = Visitor(mod, file)
         v.visit(tree)
 
@@ -322,7 +338,10 @@ if __name__ == "__main__":
         print("\n Python Static Analysis Results")
         print("===================================\n")
         
-        total_items = sum(len(items) for items in data.values())
+        total_items = 0
+        for key, items in data.items():
+            if key.startswith("unused_") and isinstance(items, list):
+                total_items += len(items)
         
         print("Summary:")
         if data["unused_functions"]:

{skylos-2.0.0 → skylos-2.1.0}/skylos/cli.py

@@ -2,10 +2,14 @@ import argparse
 import json
 import sys
 import logging
-import ast
-import skylos
 from skylos.constants import parse_exclude_folders, DEFAULT_EXCLUDE_FOLDERS
 from skylos.server import start_server
+from skylos.analyzer import analyze as run_analyze
+from skylos.codemods import (
+    remove_unused_import_cst,
+    remove_unused_function_cst,
+)
+import pathlib
 
 try:
     import inquirer
@@ -50,87 +54,30 @@ def setup_logger(output_file=None):
     
     return logger
 
-def remove_unused_import(file_path: str, import_name: str, line_number: int) -> bool:
+def remove_unused_import(file_path, import_name, line_number):
+    path = pathlib.Path(file_path)
     try:
-        with open(file_path, 'r') as f:
-            lines = f.readlines()
-        
-        line_idx = line_number - 1
-        original_line = lines[line_idx].strip()
-        
-        if original_line.startswith(f'import {import_name}'):
-            lines[line_idx] = ''
-            
-        elif original_line.startswith('import ') and f' {import_name}' in original_line:
-            parts = original_line.split(' ', 1)[1].split(',')
-            new_parts = [p.strip() for p in parts if p.strip() != import_name]
-            if new_parts:
-                lines[line_idx] = f'import {", ".join(new_parts)}\n'
-            else:
-                lines[line_idx] = ''
-
-        elif original_line.startswith('from ') and import_name in original_line:
-            if f'import {import_name}' in original_line and ',' not in original_line:
-                lines[line_idx] = ''
-            else:
-                parts = original_line.split('import ', 1)[1].split(',')
-                new_parts = [p.strip() for p in parts if p.strip() != import_name]
-                if new_parts:
-                    prefix = original_line.split(' import ')[0]
-                    lines[line_idx] = f'{prefix} import {", ".join(new_parts)}\n'
-                else:
-                    lines[line_idx] = ''
-        
-        with open(file_path, 'w') as f:
-            f.writelines(lines)
-        
+        src = path.read_text(encoding="utf-8")
+        new_code, changed = remove_unused_import_cst(src, import_name, line_number)
+        if not changed:
+            return False
+        path.write_text(new_code, encoding="utf-8")
         return True
     except Exception as e:
         logging.error(f"Failed to remove import {import_name} from {file_path}: {e}")
         return False
 
-def remove_unused_function(file_path: str, function_name: str, line_number: int) -> bool:
+def remove_unused_function(file_path, function_name, line_number):
+    path = pathlib.Path(file_path)
     try:
-        with open(file_path, 'r') as f:
-            content = f.read()
-        
-        tree = ast.parse(content)
-        
-        lines = content.splitlines()
-        for node in ast.walk(tree):
-            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
-                if (node.name in function_name and 
-                    node.lineno == line_number):
-                    
-                    start_line = node.lineno - 1
-                    
-                    if node.decorator_list:
-                        start_line = node.decorator_list[0].lineno - 1
-                    
-                    end_line = len(lines)
-                    base_indent = len(lines[start_line]) - len(lines[start_line].lstrip())
-                    
-                    for i in range(node.end_lineno, len(lines)):
-                        if lines[i].strip() == '':
-                            continue
-                        current_indent = len(lines[i]) - len(lines[i].lstrip())
-                        if current_indent <= base_indent and lines[i].strip():
-                            end_line = i
-                            break
-                    
-                    while end_line < len(lines) and lines[end_line].strip() == '':
-                        end_line += 1
-                    
-                    new_lines = lines[:start_line] + lines[end_line:]
-                    
-                    with open(file_path, 'w') as f:
-                        f.write('\n'.join(new_lines) + '\n')
-                    
-                    return True
-        
-        return False
-    except:
-        logging.error(f"Failed to remove function {function_name}")
+        src = path.read_text(encoding="utf-8")
+        new_code, changed = remove_unused_function_cst(src, function_name, line_number)
+        if not changed:
+            return False
+        path.write_text(new_code, encoding="utf-8")
+        return True
+    except Exception as e:
+        logging.error(f"Failed to remove function {function_name} from {file_path}: {e}")
         return False
 
 def interactive_selection(logger, unused_functions, unused_imports):
@@ -142,7 +89,7 @@ def interactive_selection(logger, unused_functions, unused_imports):
     selected_imports = []
     
     if unused_functions:
-        logger.info(f"\n{Colors.CYAN}{Colors.BOLD}Select unused functions to remove:{Colors.RESET}")
+        logger.info(f"\n{Colors.CYAN}{Colors.BOLD}Select unused functions to remove (hit spacebar to select):{Colors.RESET}")
         
         function_choices = []
 
@@ -162,7 +109,7 @@ def interactive_selection(logger, unused_functions, unused_imports):
             selected_functions = answers['functions']
     
     if unused_imports:
-        logger.info(f"\n{Colors.MAGENTA}{Colors.BOLD}Select unused imports to remove:{Colors.RESET}")
+        logger.info(f"\n{Colors.MAGENTA}{Colors.BOLD}Select unused imports to remove (hit spacebar to select):{Colors.RESET}")
         
         import_choices = []
 
@@ -310,7 +257,7 @@ def main() -> None:
             logger.info(f"{Colors.GREEN}📁 No folders excluded{Colors.RESET}")
 
     try:
-        result_json = skylos.analyze(args.path, conf=args.confidence, exclude_folders=list(final_exclude_folders))
+        result_json = run_analyze(args.path, conf=args.confidence, exclude_folders=list(final_exclude_folders))
         result = json.loads(result_json)
 
     except Exception as e:

skylos-2.1.0/skylos/codemods.py

@@ -0,0 +1,89 @@
+from __future__ import annotations
+import libcst as cst
+from libcst.metadata import PositionProvider
+
+def _bound_name_for_import_alias(alias: cst.ImportAlias):
+    if alias.asname:
+        return alias.asname.name.value
+    node = alias.name
+    while isinstance(node, cst.Attribute):
+        node = node.value
+    return node.value 
+
+class _RemoveImportAtLine(cst.CSTTransformer):
+    METADATA_DEPENDENCIES = (PositionProvider,)
+
+    def __init__(self, target_name, target_line):
+        self.target_name = target_name
+        self.target_line = target_line
+        self.changed = False
+
+    def _is_target_line(self, node: cst.CSTNode):
+        pos = self.get_metadata(PositionProvider, node, None)
+        return bool(pos and pos.start.line == self.target_line)
+
+    def _filter_aliases(self, aliases):
+        kept = []
+        for alias in aliases:
+            bound = _bound_name_for_import_alias(alias)
+            if bound == self.target_name:
+                self.changed = True
+                continue
+            kept.append(alias)
+        return kept
+
+    def leave_Import(self, orig: cst.Import, updated: cst.Import):
+        if not self._is_target_line(orig):
+            return updated
+        kept = self._filter_aliases(updated.names)
+        if not kept:
+            return cst.RemoveFromParent()
+        return updated.with_changes(names=tuple(kept))
+
+    def leave_ImportFrom(self, orig: cst.ImportFrom, updated: cst.ImportFrom):
+        if not self._is_target_line(orig):
+            return updated
+        if isinstance(updated.names, cst.ImportStar):
+            return updated 
+        kept = self._filter_aliases(list(updated.names))
+        if not kept:
+            return cst.RemoveFromParent()
+
+        return updated.with_changes(names=tuple(kept))
+
+class _RemoveFunctionAtLine(cst.CSTTransformer):
+    METADATA_DEPENDENCIES = (PositionProvider,)
+
+    def __init__(self, func_name, target_line):
+        self.func_name = func_name
+        self.target_line = target_line
+        self.changed = False
+
+    def _is_target(self, node: cst.CSTNode):
+        pos = self.get_metadata(PositionProvider, node, None)
+        return bool(pos and pos.start.line == self.target_line)
+
+    def leave_FunctionDef(self, orig: cst.FunctionDef, updated: cst.FunctionDef):
+        if self._is_target(orig) and (orig.name.value in self.func_name):
+            self.changed = True
+            return cst.RemoveFromParent()
+        return updated
+
+    def leave_AsyncFunctionDef(self, orig: cst.AsyncFunctionDef, updated: cst.AsyncFunctionDef):
+        if self._is_target(orig) and (orig.name.value in self.func_name):
+            self.changed = True
+            return cst.RemoveFromParent()
+
+        return updated
+
+def remove_unused_import_cst(code, import_name, line_number):
+    wrapper = cst.MetadataWrapper(cst.parse_module(code))
+    tx = _RemoveImportAtLine(import_name, line_number)
+    new_mod = wrapper.visit(tx)
+    return new_mod.code, tx.changed
+
+def remove_unused_function_cst(code, func_name, line_number):
+    wrapper = cst.MetadataWrapper(cst.parse_module(code))
+    tx = _RemoveFunctionAtLine(func_name, line_number)
+    new_mod = wrapper.visit(tx)
+    return new_mod.code, tx.changed