skylos 0.0.1__tar.gz → 1.0.8__tar.gz

skylos-1.0.8/PKG-INFO

@@ -0,0 +1,8 @@
+Metadata-Version: 2.4
+Name: skylos
+Version: 1.0.8
+Summary: A static analysis tool for Python codebases
+Author-email: oha <aaronoh2015@gmail.com>
+Requires-Python: >=3.9
+Requires-Dist: inquirer>=3.0.0
+Dynamic: requires-python

skylos-1.0.8/README.md

@@ -0,0 +1,233 @@
+# Skylos 🔍
+
+![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)
+![100% Local](https://img.shields.io/badge/privacy-100%25%20local-brightgreen)
+![PyPI version](https://img.shields.io/pypi/v/skylos)
+
+<div align="center">
+   <img src="assets/SKYLOS.png" alt="Skylos Logo" width="200">
+</div>
+
+> A static analysis tool for Python codebases written in Python (formerly was written in Rust but we ditched that) that detects unreachable functions and unused imports, aka dead code. Faster and better results than many alternatives like Flake8 and Pylint, and finding more dead code than Vulture in our tests with comparable speed.
+
+## Features
+
+* Unused Functions & Methods: Finds functions and methods that are never called
+* Unused Classes: Detects classes that are never instantiated or inherited
+* Unused Imports: Identifies imports that serve no purpose
+* Cross-Module Tracking: Analyzes usage patterns across your entire codebase
+
+## Benchmark (You can find this benchmark test in `test` folder)
+
+The benchmark checks how well static analysis tools spot dead code in Python. Things such as unused functions, classes, imports, variables, that kinda stuff. To read more refer down below.
+
+**The methodology and process for benchmarking can be found in `BENCHMARK.md`** 
+
+| Tool | Time (s) | Items | TP | FP | FN | Precision | Recall | F1 Score |
+|------|----------|-------|----|----|----|-----------|---------|---------| 
+| **Skylos (Local Dev)** | **0.013** | **34** | **22** | **12** | **7** | **0.6471** | **0.7586** | **0.6984** |
+| Vulture (0%) | 0.054 | 32 | 11 | 20 | 18 | 0.3548 | 0.3793 | 0.3667 |
+| Vulture (60%) | 0.044 | 32 | 11 | 20 | 18 | 0.3548 | 0.3793 | 0.3667 |
+| Flake8 | 0.371 | 16 | 5 | 7 | 24 | 0.4167 | 0.1724 | 0.2439 |
+| Pylint | 0.705 | 11 | 0 | 8 | 29 | 0.0000 | 0.0000 | 0.0000 |
+| Ruff | 0.140 | 16 | 5 | 7 | 24 | 0.4167 | 0.1724 | 0.2439 |
+
+To run the benchmark:
+`python compare_tools.py /path/to/sample_repo`
+
+**Note: More can be found in `BENCHMARK.md`**
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install skylos
+```
+
+### From Source
+
+```bash
+# Clone the repository
+git clone https://github.com/duriantaco/skylos.git
+cd skylos
+
+## Install your dependencies 
+pip install .
+```
+
+## Quick Start
+
+```bash
+# Analyze a project
+skylos /path/to/your/project
+
+# Interactive mode - select items to remove
+skylos --interactive /path/to/your/project 
+
+# Dry run - see what would be removed
+skylos --interactive --dry-run /path/to/your/project 
+
+# Output to JSON
+skylos --json /path/to/your/project 
+```
+
+## CLI Options
+
+```
+Usage: skylos [OPTIONS] PATH
+
+Arguments:
+  PATH  Path to the Python project to analyze
+
+Options:
+  -h, --help              Show this help message and exit
+  -j, --json             Output raw JSON instead of formatted text
+  -o, --output FILE      Write output to file instead of stdout
+  -v, --verbose          Enable verbose output
+  -i, --interactive      Interactively select items to remove
+  --dry-run              Show what would be removed without modifying files
+```
+
+## Example Output
+
+```
+🔍 Python Static Analysis Results
+===================================
+
+Summary:
+  • Unreachable functions: 48
+  • Unused imports: 8
+
+📦 Unreachable Functions
+========================
+ 1. module_13.test_function
+    └─ /Users/oha/project/module_13.py:5
+ 2. module_13.unused_function
+    └─ /Users/oha/project/module_13.py:13
+...
+
+📥 Unused Imports
+=================
+ 1. os
+    └─ /Users/oha/project/module_13.py:1
+ 2. json
+    └─ /Users/oha/project/module_13.py:3
+...
+
+Next steps:
+  • Use --interactive to select specific items to remove
+  • Use --dry-run to preview changes before applying them
+```
+
+## Interactive Mode
+
+The interactive mode lets you select specific functions and imports to remove:
+
+![Interactive Demo](docs/interactive-demo.gif)
+
+1. **Select items**: Use arrow keys and space to select/deselect
+2. **Confirm changes**: Review selected items before applying
+3. **Auto-cleanup**: Files are automatically updated
+
+## Development
+
+### Prerequisites
+
+- `Python ≥3.9`
+- `pytest`
+- `inquirer`
+
+### Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/duriantaco/skylos.git
+cd skylos
+
+# Create a virtual environment
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+### Running Tests
+
+```bash
+# Run Python tests
+python -m pytest tests/
+```
+
+## FAQ 
+
+**Q: Why doesn't Skylos find 100% of dead code?**
+A: Python's dynamic features (getattr, globals, etc.) can't be perfectly analyzed statically. No tool can achieve 100% accuracy. If they say they can, they're lying.
+
+**Q: Why are the results different on my codebase?**
+A: These benchmarks use specific test cases. Your code patterns (frameworks, legacy code, etc.) will give different results.
+
+**Q: Are these benchmarks realistic?**
+A: They test common scenarios but can't cover every edge case. Use them as a guide, not gospel.
+
+**Q: Should I automatically delete everything flagged as unused?**
+A: No. Always review results manually, especially for framework code, APIs, and test utilities.
+
+**Q: Why did Ruff underperform?**
+A: Like all other tools, Ruff is focused on detecting specific, surface-level issues. Tools like Vulture and Skylos are built SPECIFICALLY for dead code detection. It is NOT a specialized dead code detector. If your goal is dead code, then ruff is the wrong tool. It is a good tool but it's like using a wrench to hammer a nail. Good tool, wrong purpose. 
+
+## Limitations
+
+- **Dynamic code**: `getattr()`, `globals()`, runtime imports are hard to detect
+- **Frameworks**: Django models, Flask routes may appear unused but aren't
+- **Test data**: Limited scenarios, your mileage may vary
+- **False positives**: Always manually review before deleting code
+
+If we can detect 100% of all dead code in any structure, we wouldn't be sitting here. But we definitely tried our best
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Permission Errors**
+   ```
+   Error: Permission denied when removing function
+   ```
+   Check file permissions before running in interactive mode.
+
+2. **Missing Dependencies**
+   ```
+   Interactive mode requires 'inquirer' package
+   ```
+   Install with: `pip install skylos[interactive]`
+
+## Contributing
+
+We welcome contributions! Please read our [Contributing Guidelines](CONTRIBUTING.md) before submitting pull requests.
+
+### Quick Contribution Guide
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## Roadmap
+
+- [ ] Expand our test cases
+- [ ] Configuration file support 
+- [ ] Custom analysis rules
+- [ ] Git hooks integration
+- [ ] CI/CD integration examples
+- [ ] Web interface
+- [ ] Support for other languages
+- [ ] Further optimization
+
+## License
+
+This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
+
+## Contact
+
+- **Author**: oha
+- **Email**: aaronoh2015@gmail.com
+- **GitHub**: [@duriantaco](https://github.com/duriantaco)

skylos-1.0.8/pyproject.toml

@@ -0,0 +1,14 @@
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "skylos"
+version = "1.0.8"
+requires-python = ">=3.9"
+description = "A static analysis tool for Python codebases"
+authors = [{name = "oha", email = "aaronoh2015@gmail.com"}]
+dependencies = ["inquirer>=3.0.0"]
+
+[project.scripts]
+skylos = "skylos.cli:main"

skylos-1.0.8/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

skylos-1.0.8/setup.py

@@ -0,0 +1,26 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="skylos",
+    version="1.0.8",
+    packages=find_packages(),
+    python_requires=">=3.9",
+    install_requires=["inquirer>=3.0.0"],
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: Apache 2.0 License",
+        "Operating System :: OS Independent",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
+        "Topic :: Software Development :: Quality Assurance",
+        "Topic :: Software Development :: Testing",
+    ],
+    entry_points={
+        "console_scripts": [
+            "skylos=skylos.cli:main",
+        ],
+    },
+)

skylos-1.0.8/skylos/__init__.py

@@ -0,0 +1,8 @@
+from skylos.analyzer import analyze
+
+__version__ = "1.0.8"
+
+def debug_test():
+    return "debug-ok"
+
+__all__ = ["analyze", "debug_test", "__version__"]

skylos-1.0.8/skylos/analyzer.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+import ast,sys,json,logging,re
+from pathlib import Path
+from collections import defaultdict
+from skylos.visitor import Visitor
+
+logging.basicConfig(level=logging.INFO,format='%(asctime)s - %(levelname)s - %(message)s')
+logger=logging.getLogger('Skylos')
+
+AUTO_CALLED={"__init__","__enter__","__exit__"}
+TEST_BASE_CLASSES = {"TestCase", "AsyncioTestCase", "unittest.TestCase", "unittest.AsyncioTestCase"}
+TEST_METHOD_PATTERN = re.compile(r"^test_\w+$")
+MAGIC_METHODS={f"__{n}__"for n in["init","new","call","getattr","getattribute","enter","exit","str","repr","hash","eq","ne","lt","gt","le","ge","iter","next","contains","len","getitem","setitem","delitem","iadd","isub","imul","itruediv","ifloordiv","imod","ipow","ilshift","irshift","iand","ixor","ior","round","format","dir","abs","complex","int","float","bool","bytes","reduce","await","aiter","anext","add","sub","mul","truediv","floordiv","mod","divmod","pow","lshift","rshift","and","or","xor","radd","rsub","rmul","rtruediv","rfloordiv","rmod","rdivmod","rpow","rlshift","rrshift","rand","ror","rxor"]}
+
+class Skylos:
+    def __init__(self):
+        self.defs={}
+        self.refs=[]
+        self.dynamic=set()
+        self.exports=defaultdict(set)
+
+    def _module(self,root,f):
+        p=list(f.relative_to(root).parts)
+        if p[-1].endswith(".py"):p[-1]=p[-1][:-3]
+        if p[-1]=="__init__":p.pop()
+        return".".join(p)
+    
+    def _mark_exports(self):
+        
+        for name, d in self.defs.items():
+            if d.in_init and not d.simple_name.startswith('_'):
+                d.is_exported = True
+        
+        for mod, export_names in self.exports.items():
+            for name in export_names:
+                for def_name, def_obj in self.defs.items():
+                    if (def_name.startswith(f"{mod}.") and 
+                        def_obj.simple_name == name and
+                        def_obj.type != "import"):
+                        def_obj.is_exported = True
+
+    def _mark_refs(self):
+        import_to_original = {}
+        for name, def_obj in self.defs.items():
+            if def_obj.type == "import":
+                import_name = name.split('.')[-1]
+                
+                for def_name, orig_def in self.defs.items():
+                    if (orig_def.type != "import" and 
+                        orig_def.simple_name == import_name and
+                        def_name != name):
+                        import_to_original[name] = def_name
+                        break
+
+        simple_name_lookup = defaultdict(list)
+        for d in self.defs.values():
+            simple_name_lookup[d.simple_name].append(d)
+        
+        for ref, file in self.refs:
+            if ref in self.defs:
+                self.defs[ref].references += 1
+                
+                if ref in import_to_original:
+                    original = import_to_original[ref]
+                    self.defs[original].references += 1
+                continue
+            
+            simple = ref.split('.')[-1]
+            matches = simple_name_lookup.get(simple, [])
+            for d in matches:
+                d.references += 1
+    
+    def _get_base_classes(self, class_name):
+        """Get base classes for a given class name"""
+        if class_name not in self.defs:
+            return []
+        
+        class_def = self.defs[class_name]
+        
+        if hasattr(class_def, 'base_classes'):
+            return class_def.base_classes
+        
+        return []
+            
+    def _apply_heuristics(self):
+
+        class_methods=defaultdict(list)
+        for d in self.defs.values():
+            if d.type in("method","function") and"." in d.name:
+                cls=d.name.rsplit(".",1)[0]
+                if cls in self.defs and self.defs[cls].type=="class":
+                    class_methods[cls].append(d)
+
+        for cls,methods in class_methods.items():
+            if self.defs[cls].references>0:
+                for m in methods:
+                    if m.simple_name in AUTO_CALLED:m.references+=1
+                    
+        for d in self.defs.values():
+            if d.simple_name in MAGIC_METHODS or d.simple_name.startswith("__")and d.simple_name.endswith("__"):d.confidence=0
+            if not d.simple_name.startswith("_")and d.type in("function","method","class"):d.confidence=min(d.confidence,90)
+            if d.in_init and d.type in("function","class"):d.confidence=min(d.confidence,85)
+            if d.name.split(".")[0] in self.dynamic:d.confidence=min(d.confidence,50)
+        
+        for d in self.defs.values():
+            if d.type == "method" and TEST_METHOD_PATTERN.match(d.simple_name):
+                # check if its in a class that inherits from a test base class
+                class_name = d.name.rsplit(".", 1)[0]
+                class_simple_name = class_name.split(".")[-1]
+                # class name suggests it's a test class, ignore test methods
+                if "Test" in class_simple_name or class_simple_name.endswith("TestCase"):
+                    d.confidence = 0
+
+    def analyze(self, path, thr=60):
+        p = Path(path).resolve()
+        files = [p] if p.is_file() else list(p.glob("**/*.py"))
+        root = p.parent if p.is_file() else p
+        
+        modmap = {}
+        for f in files:
+            modmap[f] = self._module(root, f)
+        
+        for file in files:
+            mod = modmap[file]
+            defs, refs, dyn, exports = proc_file(file, mod)
+            
+            for d in defs: 
+                self.defs[d.name] = d
+            self.refs.extend(refs)
+            self.dynamic.update(dyn)
+            self.exports[mod].update(exports)
+        
+        self._mark_refs()
+        self._apply_heuristics()
+        self._mark_exports()
+        
+        # for name, d in self.defs.items():
+        #     print(f"  {d.type} '{name}': {d.references} refs, exported: {d.is_exported}, confidence: {d.confidence}")
+            
+        thr = max(0, thr)
+
+        unused = []
+        for d in self.defs.values():
+            if d.references == 0 and not d.is_exported and d.confidence >= thr:
+                unused.append(d.to_dict())
+        
+        result = {"unused_functions": [], "unused_imports": [], "unused_classes": []}
+        for u in unused:
+            if u["type"] in ("function", "method"):
+                result["unused_functions"].append(u)
+            elif u["type"] == "import":
+                result["unused_imports"].append(u)
+            elif u["type"] == "class": 
+                result["unused_classes"].append(u)
+                
+        return json.dumps(result, indent=2)
+
+def proc_file(file_or_args, mod=None):
+    if mod is None and isinstance(file_or_args, tuple):
+        file, mod = file_or_args 
+    else:
+        file = file_or_args 
+
+    try:
+        tree = ast.parse(Path(file).read_text(encoding="utf-8"))
+        v = Visitor(mod, file)
+        v.visit(tree)
+        return v.defs, v.refs, v.dyn, v.exports
+    except Exception as e:
+        logger.error(f"{file}: {e}")
+        return [], [], set(), set()
+
+def analyze(path,conf=60):return Skylos().analyze(path,conf)
+
+if __name__=="__main__":
+    if len(sys.argv)>1:
+        p=sys.argv[1];c=int(sys.argv[2])if len(sys.argv)>2 else 60
+        print(analyze(p,c))
+    else:
+        print("Usage: python Skylos.py <path> [confidence_threshold]")

skylos-1.0.8/skylos/visitor.py

@@ -0,0 +1,181 @@
+#!/usr/bin/env python3
+import ast,re
+from pathlib import Path
+
+PYTHON_BUILTINS={"print","len","str","int","float","list","dict","set","tuple","range","open","super","object","type","enumerate","zip","map","filter","sorted","reversed","sum","min","max","all","any","next","iter","repr","chr","ord","bytes","bytearray","memoryview","format","round","abs","pow","divmod","complex","hash","id","bool","callable","getattr","setattr","delattr","hasattr","isinstance","issubclass","globals","locals","vars","dir","property","classmethod","staticmethod"}
+DYNAMIC_PATTERNS={"getattr","globals","eval","exec"}
+
+class Definition:
+    __slots__ = ('name', 'type', 'filename', 'line', 'simple_name', 'confidence', 'references', 'is_exported', 'in_init')
+    
+    def __init__(self, n, t, f, l):
+        self.name = n
+        self.type = t
+        self.filename = f
+        self.line = l
+        self.simple_name = n.split('.')[-1]
+        self.confidence = 100
+        self.references = 0
+        self.is_exported = False
+        self.in_init = "__init__.py" in str(f)
+    
+    def to_dict(self):
+        if self.type == "method" and "." in self.name:
+            parts = self.name.split(".")
+            if len(parts) >= 3:
+                output_name = ".".join(parts[-2:])
+            else:
+                output_name = self.name
+        else:
+            output_name = self.simple_name
+            
+        return{
+            "name": output_name,
+            "full_name": self.name,
+            "simple_name": self.simple_name,
+            "type": self.type,
+            "file": str(self.filename),
+            "basename": Path(self.filename).name,
+            "line": self.line,
+            "confidence": self.confidence,
+            "references": self.references
+        }
+
+class Visitor(ast.NodeVisitor):
+    def __init__(self,mod,file):
+        self.mod=mod
+        self.file=file
+        self.defs=[]
+        self.refs=[]
+        self.cls=None
+        self.alias={}
+        self.dyn=set()
+        self.exports=set()
+        self.current_function_scope = []
+
+    def add_def(self,n,t,l):
+        if n not in{d.name for d in self.defs}:self.defs.append(Definition(n,t,self.file,l))
+
+    def add_ref(self,n):self.refs.append((n,self.file))
+
+    def qual(self,n):
+        if n in self.alias:return self.alias[n]
+        if n in PYTHON_BUILTINS:return n
+        return f"{self.mod}.{n}"if self.mod else n
+    
+    def visit_Import(self,node):
+        for a in node.names:
+            full=a.name
+            self.alias[a.asname or a.name.split(".")[-1]]=full
+            self.add_def(full,"import",node.lineno)
+
+    def visit_ImportFrom(self,node):
+        if node.module is None:return
+        for a in node.names:
+            if a.name=="*":continue
+            base=node.module
+            if node.level:
+                parts=self.mod.split(".")
+                base=".".join(parts[:-node.level])+(f".{node.module}"if node.module else"")
+            full=f"{base}.{a.name}"
+            self.alias[a.asname or a.name]=full
+            self.add_def(full,"import",node.lineno)
+
+    def visit_FunctionDef(self,node):
+        outer_scope_prefix = '.'.join(self.current_function_scope) + '.' if self.current_function_scope else ''
+        
+        if self.cls:
+            name_parts = [self.mod, self.cls, outer_scope_prefix + node.name]
+        else:
+            name_parts = [self.mod, outer_scope_prefix + node.name]
+        
+        qualified_name = ".".join(filter(None, name_parts))
+
+        self.add_def(qualified_name,"method"if self.cls else"function",node.lineno)
+        
+        self.current_function_scope.append(node.name)
+        for d_node in node.decorator_list:
+            self.visit(d_node)
+        for stmt in node.body:
+            self.visit(stmt)
+        self.current_function_scope.pop()
+        
+    visit_AsyncFunctionDef=visit_FunctionDef
+
+    def visit_ClassDef(self,node):
+        cname=f"{self.mod}.{node.name}"
+        self.add_def(cname,"class",node.lineno)
+        prev=self.cls;self.cls=node.name
+        for b in node.body:self.visit(b)
+        self.cls=prev
+
+    def visit_Assign(self, node):
+        for target in node.targets:
+            if isinstance(target, ast.Name) and target.id == "__all__":
+                if isinstance(node.value, (ast.List, ast.Tuple)):
+                    for elt in node.value.elts:
+                        value = None
+                        if isinstance(elt, ast.Constant) and isinstance(elt.value, str):
+                            value = elt.value
+                        elif hasattr(elt, 's') and isinstance(elt.s, str):
+                            value = elt.s
+                            
+                        if value is not None:
+                            full_name = f"{self.mod}.{value}"
+                            self.add_ref(full_name)
+                            self.add_ref(value)
+        self.generic_visit(node)
+
+    def visit_Call(self, node):
+        self.generic_visit(node)
+        
+        if isinstance(node.func, ast.Name) and node.func.id in ("getattr", "hasattr") and len(node.args) >= 2:
+            if isinstance(node.args[1], ast.Constant) and isinstance(node.args[1].value, str):
+                attr_name = node.args[1].value
+                self.add_ref(attr_name)
+                
+                if isinstance(node.args[0], ast.Name):
+                    module_name = node.args[0].id
+                    if module_name != "self": 
+                        qualified_name = f"{self.qual(module_name)}.{attr_name}"
+                        self.add_ref(qualified_name)
+        
+        elif isinstance(node.func, ast.Name) and node.func.id == "globals":
+            parent = getattr(node, 'parent', None)
+            if (isinstance(parent, ast.Subscript) and 
+                isinstance(parent.slice, ast.Constant) and 
+                isinstance(parent.slice.value, str)):
+                func_name = parent.slice.value
+                self.add_ref(func_name)
+                self.add_ref(f"{self.mod}.{func_name}")
+        
+        elif (isinstance(node.func, ast.Attribute) and 
+            node.func.attr == "format" and 
+            isinstance(node.func.value, ast.Constant) and 
+            isinstance(node.func.value.value, str)):
+            fmt = node.func.value.value
+            if any(isinstance(k.arg, str) and k.arg is None for k in node.keywords):
+                for _, n, _, _ in re.findall(r'\{([^}:!]+)', fmt):
+                    if n:
+                        self.add_ref(self.qual(n))
+
+    def visit_Name(self,node):
+        if isinstance(node.ctx,ast.Load):
+            self.add_ref(self.qual(node.id))
+            if node.id in DYNAMIC_PATTERNS:self.dyn.add(self.mod.split(".")[0])
+
+    def visit_Attribute(self,node):
+        self.generic_visit(node)
+        if isinstance(node.ctx,ast.Load)and isinstance(node.value,ast.Name):
+            self.add_ref(f"{self.qual(node.value.id)}.{node.attr}")
+
+    def generic_visit(self, node):
+        for field, value in ast.iter_fields(node):
+            if isinstance(value, list):
+                for item in value:
+                    if isinstance(item, ast.AST):
+                        item.parent = node
+                        self.visit(item)
+            elif isinstance(value, ast.AST):
+                value.parent = node
+                self.visit(value)

skylos-1.0.8/skylos.egg-info/PKG-INFO

@@ -0,0 +1,8 @@
+Metadata-Version: 2.4
+Name: skylos
+Version: 1.0.8
+Summary: A static analysis tool for Python codebases
+Author-email: oha <aaronoh2015@gmail.com>
+Requires-Python: >=3.9
+Requires-Dist: inquirer>=3.0.0
+Dynamic: requires-python