skylos 1.2.2__tar.gz → 2.1.0__tar.gz

{skylos-1.2.2 → skylos-2.1.0}/PKG-INFO

@@ -1,8 +1,11 @@
 Metadata-Version: 2.4
 Name: skylos
-Version: 1.2.2
+Version: 2.1.0
 Summary: A static analysis tool for Python codebases
 Author-email: oha <aaronoh2015@gmail.com>
 Requires-Python: >=3.9
 Requires-Dist: inquirer>=3.0.0
+Requires-Dist: flask>=2.1.0
+Requires-Dist: flask-cors>=3.0.0
+Requires-Dist: libcst>=1.8.2
 Dynamic: requires-python

{skylos-1.2.2 → skylos-2.1.0}/README.md

@@ -9,7 +9,12 @@
    <img src="assets/SKYLOS.png" alt="Skylos Logo" width="200">
 </div>
 
-> A static analysis tool for Python codebases written in Python (formerly was written in Rust but we ditched that) that detects unreachable functions and unused imports, aka dead code. Faster and better results than many alternatives like Flake8 and Pylint, and finding more dead code than Vulture in our tests with comparable speed.
+> A static analysis tool for Python codebases written in Python that detects unreachable functions and unused imports, aka dead code. Faster and better results than many alternatives like Flake8 and Pylint, and finding more dead code than Vulture in our tests with comparable speed.
+
+<div align="center">
+   <img src="assets/FE_SS.png" alt="FE" width="800">
+</div>
+
 
 ## Table of Contents
 
@@ -17,9 +22,12 @@
 - [Benchmark](#benchmark-you-can-find-this-benchmark-test-in-test-folder)
 - [Installation](#installation)
 - [Quick Start](#quick-start)
-- [Understanding Confidence Levels](#understanding-confidence-levels)
+- [Web Interface](#web-interface)
+- [Design](#design)
 - [Test File Detection](#test-file-detection)
 - [Folder Management](#folder-management)
+- [Ignoring Pragmas](#ignoring-pragmas)
+- [Including & Excluding Files](#including--excluding-files)
 - [CLI Options](#cli-options)
 - [Example Output](#example-output)
 - [Interactive Mode](#interactive-mode)
@@ -34,6 +42,7 @@
 
 ## Features
 
+* **CST-safe removals:** Uses LibCST to remove selected imports or functions (handles multiline imports, aliases, decorators, async etc..)
 * **Framework-Aware Detection**: Attempt at handling Flask, Django, FastAPI routes and decorators  
 * **Test File Exclusion**: Auto excludes test files (you can include it back if you want)
 * **Interactive Cleanup**: Select specific items to remove from CLI
@@ -74,34 +83,53 @@ pip install skylos
 ### From Source
 
 ```bash
-# Clone the repository
 git clone https://github.com/duriantaco/skylos.git
 cd skylos
 
-## Install your dependencies 
 pip install .
 ```
 
 ## Quick Start
 
 ```bash
-# Analyze a project
 skylos /path/to/your/project
 
+# To launch the front end
+skylos run
+
 # Interactive mode - select items to remove
 skylos --interactive /path/to/your/project 
 
 # Dry run - see what would be removed
 skylos --interactive --dry-run /path/to/your/project 
 
-# Output to JSON
 skylos --json /path/to/your/project 
 
 # With confidence
 skylos path/to/your/file --confidence 20 ## or whatever value u wanna set
 ```
 
-## Understanding Confidence Levels
+## Web Interface
+
+Skylos includes a modern web dashboard for interactive analysis:
+
+```bash
+skylos run
+
+# Opens browser at http://localhost:5090
+```
+
+## Design
+
+### Summary 
+
+Framework endpoints are often invoked externally (eg, via HTTP, signals), so we use framework aware signals + confidence scores to try avoid false positives while still catching dead codes
+
+Name resolution handles aliases and modules, but when things get ambiguous, we rather miss some dead code than accidentally mark live code as dead lmao
+
+Tests are excluded by default because their call patterns are noisy. You can opt in when you really need to audit it
+
+### Understanding Confidence Levels
 
 Skylos uses a confidence-based system to try to handle Python's dynamic nature and web frameworks.
 
@@ -163,6 +191,18 @@ When Skylos detects a test file, it by default, will apply a confidence penalty
 /project/test_data.py 
 ```
 
+## Ignoring Pragmas
+
+To ignore any warning, indicate `# pragma: no skylos` **ON THE SAME LINE** as the function/class you want to ignore
+
+Example
+
+```python
+    def with_logging(self, enabled: bool = True) -> "WebPath":     # pragma: no skylos
+        new_path = WebPath(self._url)
+        return new_path
+```
+
 ## Including & Excluding Files
 
 ### Default Exclusions
@@ -170,7 +210,6 @@ By default, Skylos excludes common folders: `__pycache__`, `.git`, `.pytest_cach
 
 ### Folder Options
 ```bash
-# List default excluded folders
 skylos --list-default-excludes
 
 # Exclude single folder (The example here will be venv)
@@ -194,8 +233,8 @@ Arguments:
   PATH  Path to the Python project to analyze
 
 Options:
-  -h, --help                    Show this help message and exit
-  -j, --json                   Output raw JSON instead of formatted text  
+  -h, --help                   Show this help message and exit
+  --json                       Output raw JSON instead of formatted text  
   -o, --output FILE            Write output to file instead of stdout
   -v, --verbose                Enable verbose output
   -i, --interactive            Interactively select items to remove
@@ -207,48 +246,11 @@ Options:
   -c, --confidence LEVEL       Confidence threshold (0-100). Lower values will show more items.
 ```
 
-## Example Output
-
-```
-Python Static Analysis Results
-===================================
-
-Summary:
-  • Unreachable functions: 48
-  • Unused imports: 8
-
-Unreachable Functions
-========================
-
- 1. module_13.test_function
-    └─ /Users/oha/project/module_13.py:5
- 2. module_13.unused_function
-    └─ /Users/oha/project/module_13.py:13
-...
-
-
-Unused Imports
-=================
-
- 1. os
-    └─ /Users/oha/project/module_13.py:1
- 2. json
-    └─ /Users/oha/project/module_13.py:3
-...
-```
-
-Next steps:
-
-  • Use `--interactive` to select specific items to remove
-
-  • Use `--dry-run` to preview changes before applying them
-
-
 ## Interactive Mode
 
 The interactive mode lets you select specific functions and imports to remove:
 
-1. **Select items**: Use arrow keys and space to select/deselect
+1. **Select items**: Use arrow keys and `spacebar` to select/unselect
 2. **Confirm changes**: Review selected items before applying
 3. **Auto-cleanup**: Files are automatically updated
 
@@ -263,11 +265,9 @@ The interactive mode lets you select specific functions and imports to remove:
 ### Setup
 
 ```bash
-# Clone the repository
 git clone https://github.com/duriantaco/skylos.git
 cd skylos
 
-# Create a virtual environment
 python -m venv venv
 source venv/bin/activate  # On Windows: venv\Scripts\activate
 ```
@@ -275,7 +275,6 @@ source venv/bin/activate  # On Windows: venv\Scripts\activate
 ### Running Tests
 
 ```bash
-# Run Python tests
 python -m pytest tests/
 ```
 
@@ -309,8 +308,6 @@ A: Start with 60 (default) for safe cleanup. Use 30 for framework applications.
 - **Test data**: Limited scenarios, your mileage may vary
 - **False positives**: Always manually review before deleting code
 
-This is a STATIC code analyzer. If we can detect 100% of all dead code in any structure and framework, we wouldn't be sitting here. But we definitely tried our best.
-
 ## Troubleshooting
 
 ### Common Issues
@@ -344,7 +341,6 @@ We welcome contributions! Please read our [Contributing Guidelines](CONTRIBUTING
 - [ ] Configuration file support 
 - [ ] Git hooks integration
 - [ ] CI/CD integration examples
-~~- [] Support for other languages (unless I have contributors, this ain't possible)~~
 - [ ] Further optimization
 
 ## License

{skylos-1.2.2 → skylos-2.1.0}/pyproject.toml

@@ -4,11 +4,11 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "skylos"
-version = "1.2.2"
+version = "2.1.0"
 requires-python = ">=3.9"
 description = "A static analysis tool for Python codebases"
 authors = [{name = "oha", email = "aaronoh2015@gmail.com"}]
-dependencies = ["inquirer>=3.0.0"]
+dependencies = ["inquirer>=3.0.0", "flask>=2.1.0", "flask-cors>=3.0.0", "libcst>=1.8.2"]
 
 [project.scripts]
 skylos = "skylos.cli:main"

{skylos-1.2.2 → skylos-2.1.0}/setup.py

@@ -2,10 +2,15 @@ from setuptools import setup, find_packages
 
 setup(
     name="skylos",
-    version="1.2.2",
+    version="2.1.0",
     packages=find_packages(),
     python_requires=">=3.9",
-    install_requires=["inquirer>=3.0.0"],
+    install_requires=[
+        "inquirer>=3.0.0",
+        "flask>=2.1.0",
+        "flask-cors>=3.0.0",
+        "libcst>=1.8.2"],
+
     classifiers=[
         "Development Status :: 4 - Beta",
         "Intended Audience :: Developers",

{skylos-1.2.2 → skylos-2.1.0}/skylos/__init__.py

@@ -1,6 +1,6 @@
 from skylos.analyzer import analyze
 
-__version__ = "1.0.22"
+__version__ = "2.1.0"
 
 def debug_test():
     return "debug-ok"

{skylos-1.2.2 → skylos-2.1.0}/skylos/analyzer.py

@@ -6,61 +6,29 @@ import logging
 from pathlib import Path
 from collections import defaultdict
 from skylos.visitor import Visitor
-from skylos.constants import ( DEFAULT_EXCLUDE_FOLDERS, PENALTIES, AUTO_CALLED )
+from skylos.constants import ( PENALTIES, AUTO_CALLED )
 from skylos.test_aware import TestAwareVisitor
 import os
 import traceback
 from skylos.framework_aware import FrameworkAwareVisitor, detect_framework_usage
-import io
-import tokenize
-import re
-import warnings
 
 logging.basicConfig(level=logging.INFO,format='%(asctime)s - %(levelname)s - %(message)s')
 logger=logging.getLogger('Skylos')
 
-def parse_exclude_folders(user_exclude_folders, use_defaults=True, include_folders=None):
-    exclude_set = set()
-    
-    if use_defaults:
-        exclude_set.update(DEFAULT_EXCLUDE_FOLDERS)
-        
-    if user_exclude_folders:
-        exclude_set.update(user_exclude_folders)
-    
-    if include_folders:
-        for folder in include_folders:
-            exclude_set.discard(folder)
-    
-    return exclude_set
-
-IGNORE_PATTERNS = (
-    r"#\s*pragma:\s*no\s+skylos", ## our own pragma
-    r"#\s*pragma:\s*no\s+cover",
-    r"#\s*noqa(?:\b|:)", # flake8 style
-)
-
-def _collect_ignored_lines(source: str) -> set[int]:
-    ignores = set()
-    for tok in tokenize.generate_tokens(io.StringIO(source).readline):
-        if tok.type == tokenize.COMMENT:
-            if any(re.search(pat, tok.string, flags=re.I) for pat in IGNORE_PATTERNS):
-                ignores.add(tok.start[0])
-    return ignores
-
 class Skylos:
     def __init__(self):
         self.defs={}
         self.refs=[]
         self.dynamic=set()
         self.exports=defaultdict(set)
-        self.ignored_lines:set[int]=set()
 
     def _module(self,root,f):
-        p=list(f.relative_to(root).parts)
-        if p[-1].endswith(".py"):p[-1]=p[-1][:-3]
-        if p[-1]=="__init__":p.pop()
-        return".".join(p)
+        p = list(f.relative_to(root).parts)
+        if p[-1].endswith(".py"):
+            p[-1] = p[-1][:-3]
+        if p[-1] == "__init__":
+            p.pop()
+        return ".".join(p)
     
     def _should_exclude_file(self, file_path, root_path, exclude_folders):
         if not exclude_folders:
@@ -111,9 +79,9 @@ class Skylos:
         return all_files, root
     
     def _mark_exports(self):
-        for name, d in self.defs.items():
-            if d.in_init and not d.simple_name.startswith('_'):
-                d.is_exported = True
+        for name, definition in self.defs.items():
+            if definition.in_init and not definition.simple_name.startswith('_'):
+                definition.is_exported = True
         
         for mod, export_names in self.exports.items():
             for name in export_names:
@@ -137,23 +105,37 @@ class Skylos:
                         break
 
         simple_name_lookup = defaultdict(list)
-        for d in self.defs.values():
-            simple_name_lookup[d.simple_name].append(d)
+        for definition in self.defs.values():
+            simple_name_lookup[definition.simple_name].append(definition)
         
         for ref, _ in self.refs:
             if ref in self.defs:
                 self.defs[ref].references += 1
-                
                 if ref in import_to_original:
                     original = import_to_original[ref]
                     self.defs[original].references += 1
                 continue
-            
+
             simple = ref.split('.')[-1]
-            matches = simple_name_lookup.get(simple, [])
-            for d in matches:
-                d.references += 1
- 
+            ref_mod = ref.rsplit(".", 1)[0]
+            candidates = simple_name_lookup.get(simple, [])
+
+            if ref_mod:
+                filtered = []
+                for d in candidates:
+                    if d.name.startswith(ref_mod + ".") and d.type != "import":
+                        filtered.append(d)
+                candidates = filtered
+            else:
+                filtered = []
+                for d in candidates:
+                    if d.type != "import":
+                        filtered.append(d)
+                candidates = filtered
+
+            if len(candidates) == 1:
+                candidates[0].references += 1
+
         for module_name in self.dynamic:
             for def_name, def_obj in self.defs.items():
                 if def_obj.name.startswith(f"{module_name}."):
@@ -172,59 +154,66 @@ class Skylos:
         return []
     
     def _apply_penalties(self, def_obj, visitor, framework):
-        c = 100
-        
+        confidence=100
         if def_obj.simple_name.startswith("_") and not def_obj.simple_name.startswith("__"):
-            c -= PENALTIES["private_name"] 
+            confidence -= PENALTIES["private_name"] 
         if def_obj.simple_name.startswith("__") and def_obj.simple_name.endswith("__"):
-            c -= PENALTIES["dunder_or_magic"]
-        if def_obj.type == "variable" and def_obj.simple_name == "_":
-            c -= PENALTIES["underscored_var"]
+            confidence -= PENALTIES["dunder_or_magic"]
+        if def_obj.type == "variable" and def_obj.simple_name.isupper():
+            confidence = 0
         if def_obj.in_init and def_obj.type in ("function", "class"):
-            c -= PENALTIES["in_init_file"]
+            confidence -= PENALTIES["in_init_file"]
         if def_obj.name.split(".")[0] in self.dynamic:
-            c -= PENALTIES["dynamic_module"]
+            confidence -= PENALTIES["dynamic_module"]
         if visitor.is_test_file or def_obj.line in visitor.test_decorated_lines:
-            c -= PENALTIES["test_related"]
+            confidence -= PENALTIES["test_related"]
         
         framework_confidence = detect_framework_usage(def_obj, visitor=framework)
         if framework_confidence is not None:
-            c = min(c, framework_confidence)
+            confidence = min(confidence, framework_confidence)
 
         if (def_obj.simple_name.startswith("__") and def_obj.simple_name.endswith("__")):
-            c = 0
+            confidence = 0
 
         if def_obj.type == "parameter":
             if def_obj.simple_name in ("self", "cls"):
-                c = 0
+                confidence = 0
             elif "." in def_obj.name:
                 method_name = def_obj.name.split(".")[-2]
                 if method_name.startswith("__") and method_name.endswith("__"):
-                    c = 0
+                    confidence = 0
         
         if visitor.is_test_file or def_obj.line in visitor.test_decorated_lines:
-            c = 0
+            confidence = 0
         
         if (def_obj.type == "import" and def_obj.name.startswith("__future__.") and
             def_obj.simple_name in ("annotations", "absolute_import", "division", 
                                 "print_function", "unicode_literals", "generator_stop")):
-            c = 0
+            confidence = 0
         
-        def_obj.confidence = max(c, 0)
+        def_obj.confidence = max(confidence, 0)
 
     def _apply_heuristics(self):
         class_methods = defaultdict(list)
-        for d in self.defs.values():
-            if d.type in ("method", "function") and "." in d.name:
-                cls = d.name.rsplit(".", 1)[0]
+        for definition in self.defs.values():
+            if definition.type in ("method", "function") and "." in definition.name:
+                cls = definition.name.rsplit(".", 1)[0]
                 if cls in self.defs and self.defs[cls].type == "class":
-                    class_methods[cls].append(d)
+                    class_methods[cls].append(definition)
         
         for cls, methods in class_methods.items():
             if self.defs[cls].references > 0:
-                for m in methods:
-                    if m.simple_name in AUTO_CALLED:  # __init__, __enter__, __exit__
-                        m.references += 1
+                for method in methods:
+                    if method.simple_name in AUTO_CALLED:
+                        method.references += 1
+                    
+                    if (method.simple_name.startswith("visit_") or
+                        method.simple_name.startswith("leave_") or
+                        method.simple_name.startswith("transform_")):
+                        method.references += 1
+
+                    if method.simple_name == "format" and cls.endswith("Formatter"):
+                        method.references += 1
 
     def analyze(self, path, thr=60, exclude_folders=None):
         files, root = self._get_python_files(path, exclude_folders)
@@ -251,25 +240,11 @@ class Skylos:
         
         for file in files:
             mod = modmap[file]
+            defs, refs, dyn, exports, test_flags, framework_flags = proc_file(file, mod)
 
-            result = proc_file(file, mod)
-
-            if len(result) == 7: ##new
-                defs, refs, dyn, exports, test_flags, framework_flags, ignored = result
-                self.ignored_lines.update(ignored)
-            else: ##legacy
-                warnings.warn(
-                    "proc_file() now returns 7 values (added ignored_lines). "
-                    "The 6-value form is deprecated and will disappear.",
-                    DeprecationWarning,
-                    stacklevel=2,
-                )
-                defs, refs, dyn, exports, test_flags, framework_flags = result
-
-            # apply penalties while we still have the file-specific flags
-            for d in defs:
-                self._apply_penalties(d, test_flags, framework_flags) 
-                self.defs[d.name] = d
+            for definition in defs:
+                self._apply_penalties(definition, test_flags, framework_flags) 
+                self.defs[definition.name] = definition
 
             self.refs.extend(refs)
             self.dynamic.update(dyn)
@@ -278,18 +253,22 @@ class Skylos:
         self._mark_refs()
         self._apply_heuristics() 
         self._mark_exports()
-        
-        thr = max(0, thr)
 
-        unused = []
-        for d in self.defs.values():
-            # skip anything on line that carries ignore pragma
-            if d.line in self.ignored_lines:
-                continue
+        shown = 0
+
+        def def_sort_key(d):
+            return (d.type, d.name)
 
-            if (d.references == 0 and not d.is_exported
-                and d.confidence >= thr):
-                    unused.append(d.to_dict())
+        for d in sorted(self.defs.values(), key=def_sort_key):
+            if shown >= 50:
+                break
+            print(f" {d.type:<8} refs={d.references:<2} conf={d.confidence:<3} exported={d.is_exported} line={d.line:<4} {d.name}")
+            shown += 1
+
+        unused = []
+        for definition in self.defs.values():
+                if definition.references == 0 and not definition.is_exported and definition.confidence > 0 and definition.confidence >= thr:
+                    unused.append(definition.to_dict())
 
         result = {
             "unused_functions": [], 
@@ -299,7 +278,7 @@ class Skylos:
             "unused_parameters": [],
             "analysis_summary": {
                 "total_files": len(files),
-                "excluded_folders": exclude_folders if exclude_folders else [],
+                "excluded_folders": exclude_folders or [],
             }
         }
         
@@ -325,7 +304,6 @@ def proc_file(file_or_args, mod=None):
 
     try:
         source = Path(file).read_text(encoding="utf-8")
-        ignored = _collect_ignored_lines(source)   
         tree   = ast.parse(source)
 
         tv = TestAwareVisitor(filename=file)
@@ -333,11 +311,11 @@ def proc_file(file_or_args, mod=None):
 
         fv = FrameworkAwareVisitor(filename=file)
         fv.visit(tree)
-
+        fv.finalize()
         v  = Visitor(mod, file)
         v.visit(tree)
 
-        return v.defs, v.refs, v.dyn, v.exports, tv, fv, ignored
+        return v.defs, v.refs, v.dyn, v.exports, tv, fv
     except Exception as e:
         logger.error(f"{file}: {e}")
         if os.getenv("SKYLOS_DEBUG"):
@@ -345,55 +323,59 @@ def proc_file(file_or_args, mod=None):
         dummy_visitor = TestAwareVisitor(filename=file)
         dummy_framework_visitor = FrameworkAwareVisitor(filename=file)
 
-        return [], [], set(), set(), dummy_visitor, dummy_framework_visitor, set()
+        return [], [], set(), set(), dummy_visitor, dummy_framework_visitor
 
 def analyze(path,conf=60, exclude_folders=None):
     return Skylos().analyze(path,conf, exclude_folders)
 
-if __name__=="__main__":
+if __name__ == "__main__":
     if len(sys.argv)>1:
-        p=sys.argv[1];c=int(sys.argv[2])if len(sys.argv)>2 else 60
-        result = analyze(p,c)
+        p = sys.argv[1]
+        confidence = int(sys.argv[2]) if len(sys.argv) >2 else 60
+        result = analyze(p,confidence)
         
         data = json.loads(result)
-        print("\n🔍 Python Static Analysis Results")
+        print("\n Python Static Analysis Results")
         print("===================================\n")
         
-        total_items = sum(len(items) for items in data.values())
+        total_items = 0
+        for key, items in data.items():
+            if key.startswith("unused_") and isinstance(items, list):
+                total_items += len(items)
         
         print("Summary:")
         if data["unused_functions"]:
-            print(f"  • Unreachable functions: {len(data['unused_functions'])}")
+            print(f" * Unreachable functions: {len(data['unused_functions'])}")
         if data["unused_imports"]:
-            print(f"  • Unused imports: {len(data['unused_imports'])}")
+            print(f" * Unused imports: {len(data['unused_imports'])}")
         if data["unused_classes"]:
-            print(f"  • Unused classes: {len(data['unused_classes'])}")
+            print(f" * Unused classes: {len(data['unused_classes'])}")
         if data["unused_variables"]:
-            print(f"  • Unused variables: {len(data['unused_variables'])}")
+            print(f" * Unused variables: {len(data['unused_variables'])}")
         
         if data["unused_functions"]:
-            print("\n📦 Unreachable Functions")
+            print("\n - Unreachable Functions")
             print("=======================")
             for i, func in enumerate(data["unused_functions"], 1):
                 print(f" {i}. {func['name']}")
                 print(f"    └─ {func['file']}:{func['line']}")
         
         if data["unused_imports"]:
-            print("\n📥 Unused Imports")
+            print("\n - Unused Imports")
             print("================")
             for i, imp in enumerate(data["unused_imports"], 1):
                 print(f" {i}. {imp['simple_name']}")
                 print(f"    └─ {imp['file']}:{imp['line']}")
         
         if data["unused_classes"]:
-            print("\n📋 Unused Classes")
+            print("\n - Unused Classes")
             print("=================")
             for i, cls in enumerate(data["unused_classes"], 1):
                 print(f" {i}. {cls['name']}")
                 print(f"    └─ {cls['file']}:{cls['line']}")
                 
         if data["unused_variables"]:
-            print("\n📊 Unused Variables")
+            print("\n - Unused Variables")
             print("==================")
             for i, var in enumerate(data["unused_variables"], 1):
                 print(f" {i}. {var['name']}")
@@ -406,7 +388,7 @@ if __name__=="__main__":
         print(f"```")
         
         print("\nNext steps:")
-        print("  • Use --interactive to select specific items to remove")
-        print("  • Use --dry-run to preview changes before applying them")
+        print("  * Use --interactive to select specific items to remove")
+        print("  * Use --dry-run to preview changes before applying them")
     else:
         print("Usage: python Skylos.py <path> [confidence_threshold]")