depyo 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2011-2024 Sergey Kuznetsov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,97 @@
+# depyo — Python bytecode decompiler in Node.js
+
+Depyo converts Python `.pyc` files (or archives of them) back to readable Python source. It aims for broad coverage (Python 1.0 through 3.14) and fast throughput, with fixtures for modern features (exception groups, pattern matching, walrus, f-strings, async, context managers).
+
+## Why depyo?
+- **Wide version coverage:** Opcode tables and expected outputs for Python 1.0–3.14, plus decompilation support for PyPy bytecode sets.
+- **Modern features:** WITH_EXCEPT_START/PREP_RERAISE_STAR, async/await, walrus, match/case, f-strings, type params, dict/set merges.
+- **Workflow friendly:** CLI options for asm dumps, raw spacing hints, raw `.pyc` preservation, and flattened output paths.
+- **Verification harness:** `run-fixtures.js` and `run-matrix.js` compare decompiled output against expected fixtures across versions.
+
+## Install
+- Global: `npm i -g depyo`
+- One-off: `npx depyo <file.pyc>`
+
+Node.js 20+ recommended (matches CI).
+
+## Quick start
+
+```bash
+# Decompile a single .pyc
+node depyo.js /path/to/file.pyc
+
+# Decompile a zip of .pyc files, emit asm and keep raw bytes
+node depyo.js --asm --raw my_archive.zip
+
+# Write sources next to inputs (skip directory mirroring)
+node depyo.js --skip-path /path/to/file.pyc
+
+# Dump to stdout instead of files
+node depyo.js --out /path/to/file.pyc
+```
+
+### CLI options
+- `--asm` emit `.pyasm` disassembly alongside source
+- `--raw` emit raw `.pyc` next to output
+- `--raw-spacing` preserve blank lines/comment gaps
+- `--dump` dump marshalled object tree
+- `--stats` print throughput stats
+- `--skip-source-gen` skip writing `.py` (use with `--asm/--dump`)
+- `--skip-path` flatten output paths (write next to input)
+- `--out` print source to stdout instead of files
+- `--basedir <dir>` override output root (default: alongside input)
+- `--file-ext <ext>` change emitted extension (default `py`)
+
+## Examples
+- Disassemble only (no source): `node depyo.js --skip-source-gen --asm file.pyc`
+- Keep raw + disassembly next to source: `node depyo.js --raw --asm path/to/file.pyc`
+- Flatten outputs (helpful for bulk zips): `node depyo.js --skip-path archive.zip`
+
+## Testing
+- Smoke per version:
+  ```bash
+  node scripts/run-fixtures.js --root test/bytecode_3.14 --pattern py314_with_except_star --fail-fast
+  node scripts/run-fixtures.js --root test/bytecode_3.6 --pattern py36_fstrings --fail-fast
+  ```
+- Matrix (all versions, optional PyPy):
+  ```bash
+  node scripts/run-matrix.js                  # full sweep
+  node scripts/run-matrix.js --pattern py311_exception_groups --fail-fast
+  ```
+- Modern fixtures are generated via `test/generate_modern_tests.py` (Python 3.8+ on PATH).
+
+## Support matrix
+- Python 1.0–3.14 opcode tables with expected fixtures.
+- Modern features: match/case, walrus, f-strings, exception groups, type params.
+- PyPy bytecode sets decompile; expected files are not yet part of CI.
+- Legacy CI smokes (1.x/2.7/3.0–3.6) are informational (`continue-on-error`); modern feature checks are blocking.
+
+## Known limitations
+- **Inline comprehensions (Python 3.12+):** PEP 709 inlines list/dict/set comprehensions into parent code objects. Depyo currently reconstructs these as for-loops rather than comprehension expressions. Functions, classes, match/case, exception handling, and other constructs work correctly.
+
+## Contributing / DX tips
+- Use `node scripts/run-fixtures.js --pattern <piece>` for fast repros.
+- For full coverage, `node scripts/run-matrix.js --fail-fast` (optionally add `--pattern`).
+- Enable `--raw-spacing` to inspect potential comment/blank-line gaps.
+- `--stats` helps when profiling throughput.
+
+Comments and docs are in English; output mirrors the target Python version syntax.
+
+## Comparison snapshot (at a glance)
+
+| Project            | Supported versions          | Modern features (match, walrus, f-strings, exc groups) | Delivery     | Expected fixtures | Notes                                     |
+| ------------------ | --------------------------- | ------------------------------------------------------ | ------------ | ----------------- | ----------------------------------------- |
+| depyo              | 1.0–3.14 (PyPy decompiles)  | Yes                                                    | npm/npx, CLI | Yes (1.0–3.14)    | Node.js CLI, asm/raw-spacing options      |
+| uncompyle6/decompyle3 | 2.x–3.12+ (lag on 3.13/3.14) | Partial (depends on branch)                           | pip          | Partial           | Python-based, slower adoption of new ops  |
+| pycdc (C++)        | Mostly 2.x–3.x (limited new) | Partial                                                | source build | No                | Fast, but modern coverage limited         |
+
+## Quick benchmark (informal)
+- Machine: local Node 25, single-thread.
+- Case: `py314_exception_groups.pyc` decompiled 50× in-process: ~5.3 ms total (≈0.1 ms per decompile).  
+Use `node depyo.js --stats <file.pyc>` for your environment.
+
+## Promotion ideas (OSS)
+- Announce on HN/Reddit (Show HN / r/Python) with npm/npx one-liners.
+- Add to awesome lists (`awesome-python`, `awesome-reverse-engineering`).
+- Provide asciinema/GIF of `npx depyo file.pyc` + `--asm`.
+- Encourage contributions via Issues/Discussions and `help wanted` labels.

package/depyo.js

@@ -0,0 +1,213 @@
+#!/usr/bin/env node
+
+const {PycReader} = require('./lib/PycReader')
+const PycDecompiler = require('./lib/PycDecompiler');
+const PycDisassembler = require('./lib/PycDisassembler');
+const ZipReader = require('./lib/zip_reader');
+const zlib = require('zlib');
+const fs = require('fs');
+const Path = require('path');
+const PycResult = require('./lib/PycResult');
+
+let g_baseDir = './decompiled/';
+global.g_cliArgs = {
+    debug: false,
+    raw: false,
+    rawSpacing: false,
+    dump: false,
+    asm: false,
+    stats: false,
+    skipSource: false,
+    skipPath: false,
+    sendToStdout: false,
+    fileExt: 'py',
+    baseDir: null,
+    filenames: []
+};
+
+let g_totalInThroughput = 0;
+let g_totalOutThroughput = 0;
+let g_totalExecTime = 0;
+let g_totalFiles = 0;
+
+function printUsage() {
+    console.log(`Usage: node depyo.js [options] <file.pyc|archive.zip> [...]
+
+Options:
+  --asm               Emit .pyasm disassembly alongside source
+  --debug             Verbose logging during decompilation
+  --raw               Save raw .pyc next to output
+  --raw-spacing       Preserve blank lines (show potential comment gaps)
+  --dump              Dump marshalled object tree (.dump)
+  --stats             Print throughput stats
+  --skip-source-gen   Do not emit .py source (useful with --asm/--dump)
+  --skip-path         Flatten output paths (write files next to inputs)
+  --out               Print decompiled source to stdout instead of files
+  --basedir <path>    Output base directory (default: alongside input)
+  --file-ext <ext>    Extension for generated source (default: py)
+`);
+}
+
+function parseCLIParams() {
+    for (let idx = 2; idx < process.argv.length; idx++ ) {
+        let cliParam = process.argv[idx];
+        if (cliParam.toLowerCase() == "--asm") {
+            g_cliArgs.asm = true;
+        } else if (cliParam.toLowerCase() == "--debug") {
+            g_cliArgs.debug = true;
+        } else if (cliParam.toLowerCase() == "--raw") {
+            g_cliArgs.raw = true;
+        } else if (cliParam.toLowerCase() == "--raw-spacing") {
+            g_cliArgs.rawSpacing = true;
+        } else if (cliParam.toLowerCase() == "--dump") {
+            g_cliArgs.dump = true;
+        } else if (cliParam.toLowerCase() == "--stats") {
+            g_cliArgs.stats = true;
+        } else if (cliParam.toLowerCase() == "--skip-source-gen") {
+            g_cliArgs.skipSource = true;
+        } else if (cliParam.toLowerCase() == "--skip-path") {
+            g_cliArgs.skipPath = true;
+        } else if (cliParam.toLowerCase() == "--out") {
+            g_cliArgs.sendToStdout = true;
+        } else if (cliParam.toLowerCase() == "--basedir") {
+            g_cliArgs.baseDir = process.argv[++idx];
+        } else if (cliParam.toLowerCase() == "--file-ext") {
+            g_cliArgs.fileExt = process.argv[++idx];
+        } else if (cliParam.toLowerCase() == "--help" || cliParam.toLowerCase() == "-h") {
+            printUsage();
+            process.exit(0);
+        } else {
+            if (cliParam && cliParam.trim().length > 0) {
+                g_cliArgs.filenames.push(cliParam);
+            }
+        }
+    }
+}
+
+function decompilePycObject(data) {
+    try
+    {
+        let filename = null, obj = null;
+        let startTS = process.hrtime.bigint();
+        let rdr = new PycReader(data);
+        try {
+            obj = rdr.ReadObject();
+            filename = g_baseDir + obj.FileName;
+        } catch (ex) {
+            if (ex instanceof PycReader.LoadError) {
+                // Save the binary file if it not already exists for future manual analysis.
+                if (!ex.FileName) {
+                    if (global.g_cliArgs?.debug) {
+                        console.log(`LoadError: ${ex.message} at position ${ex.position}`);
+                    }
+                    return;
+                }
+                filename = g_baseDir + ex.FileName;
+                let dirPath =  Path.dirname(filename);
+                if (!g_cliArgs.skipPath && !fs.existsSync(dirPath)) {
+                    fs.mkdirSync(dirPath, {recursive: true});
+                }
+                let filenamePyc = filename.substring(0, filename.lastIndexOf('.')) + ".pyc";
+                fs.writeFileSync(filenamePyc, rdr.Reader);
+                console.log(`Error: ${ex.message}\nFile: ${filenamePyc}\nPosition: ${ex.position}`);
+                return;
+            }
+            console.log(`Error: ${ex.message}\nStack:\n${ex.stacktrace}`);
+            return;
+        }
+
+        if (!g_cliArgs.sendToStdout) {
+            console.log(`Processing ${filename}...`);
+        }
+        let dirPath =  Path.dirname(filename);
+        let filenameBase = filename.substring(0, filename.lastIndexOf('.'));
+        if (g_cliArgs.skipPath) {
+            filenameBase = "./" + filenameBase.substring(dirPath.length + 1);
+        }
+
+        if (!g_cliArgs.sendToStdout) {
+            if (!g_cliArgs.skipPath && !fs.existsSync(dirPath)) {
+                fs.mkdirSync(dirPath, {recursive: true});
+            }
+            if (g_cliArgs.raw) {
+                fs.writeFileSync(filenameBase + ".pyc", rdr.Reader);
+            }
+            if (g_cliArgs.dump) {
+                fs.writeFileSync(filenameBase + ".dump", PycReader.DumpObject(obj));
+            }
+            if (g_cliArgs.asm) {
+                fs.writeFileSync(filenameBase + ".pyasm", PycDisassembler.Disassemble(rdr, obj));
+            }
+        }
+        let genStartTS = process.hrtime.bigint();
+        let decompiler = new PycDecompiler(obj);
+        let ast = decompiler.decompile();
+        let pycResult = ast.codeFragment();
+        let pySrc = pycResult.toString();
+        if (!pySrc.endsWith("\n")) {
+            pySrc += "\n";
+        }
+        let genSecs = parseInt(process.hrtime.bigint() - genStartTS) / 1000000000;
+        if (g_cliArgs.sendToStdout) {
+//            console.log(`\n\n${filenameBase}.${g_cliArgs.fileExt}\n-------\n${pySrc}`);
+            console.log(pySrc);
+        } else {
+            fs.writeFileSync(filenameBase + "." + g_cliArgs.fileExt, pySrc);
+        }
+        let secs = parseInt(process.hrtime.bigint() - startTS) / 1000000000;
+        g_totalExecTime += secs;
+        let inThroughput = data.length / genSecs;
+        let outThroughput = pySrc.length / genSecs;
+        g_totalInThroughput += data.length;
+        g_totalOutThroughput += pySrc.length;
+        g_totalFiles++;
+        if (g_cliArgs.stats) {
+            console.log(`Done in ${genSecs.toFixed(3)}s. In: ${data.length} bytes (${inThroughput.toFixed(2)} B/s). Out: ${pySrc.length} bytes (${outThroughput.toFixed(2)} B/s).`);
+        }
+    }
+    catch (ex)
+    {
+        console.log(`EXCEPTION: ${ex}`);
+        if (g_cliArgs.debug) {
+            console.log(ex.stack);
+        }
+    }
+}
+
+
+function DecompileModule(filenames)
+{
+    for (let filename of filenames) {
+        try {
+            let zipReader = new ZipReader(filename);
+            if (zipReader.isZipFile()) {
+                for (let entry of zipReader.entries()) {
+                    let uncompressedData = zlib.inflateSync(entry.data);
+
+                    decompilePycObject(uncompressedData);
+                }
+            } else {
+                decompilePycObject(filename);
+            }
+        } catch (ex) {
+            console.log(`Processing of file ${filename} was unsuccessful due to: ${ex.message}`);
+        }
+    }
+}
+
+parseCLIParams()
+if (g_cliArgs.filenames.length === 0) {
+    printUsage();
+    process.exit(1);
+}
+
+const baseInputDir = g_cliArgs.baseDir ? g_cliArgs.baseDir : Path.dirname(g_cliArgs.filenames[0] || '.');
+g_baseDir = Path.resolve(baseInputDir, 'decompiled') + '/';
+
+DecompileModule(g_cliArgs.filenames);
+
+if (!g_cliArgs.sendToStdout) {
+    const inRate = (g_totalInThroughput / g_totalExecTime).toFixed(2);
+    const outRate = (g_totalOutThroughput / g_totalExecTime).toFixed(2);
+    console.log(`Processed ${g_totalFiles} files in ${g_totalExecTime.toFixed(3)}s. In: ${g_totalInThroughput} bytes (${inRate} B/s). Out: ${g_totalOutThroughput} bytes (${outRate} B/s).`);
+}

package/lib/BinaryReader.js

@@ -0,0 +1,153 @@
+class BinaryReader {
+    constructor(code) {
+        if (!code) {
+            return;
+        }
+        
+        this._reader = Buffer.from(code);
+        this._pc = 0;
+    }
+
+    get pc() {
+        return this._pc;
+    }
+
+    set pc(pos) {
+        this._pc = pos;
+    }
+
+    get EOF() {
+        return this._pc >= this._reader.length;
+    }
+
+    get Reader() {
+        return this._reader;
+    }
+
+    readChar() {
+        return String.fromCharCode([this.readByte()]);
+    }
+
+    readByte() {
+        let value = this._reader.readUInt8(this.pc, true);
+        this.pc++;
+        return value;
+    }
+    
+    readUInt16() {
+        let value = this._reader.readUInt16LE(this.pc, true);
+        this.pc += 2;
+        return value;
+    }
+
+    readUInt16BE() {
+        let value = this._reader.readUInt16BE(this.pc, true);
+        this.pc += 2;
+        return value;
+    }
+
+    readInt16() {
+        let value = this._reader.readInt16LE(this.pc, true);
+        this.pc += 2;
+        return value;
+    }
+
+    readInt16BE() {
+        let value = this._reader.readInt16BE(this.pc, true);
+        this.pc += 2;
+        return value;
+    }
+
+    readInt32() {
+        let value = this._reader.readInt32LE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+
+    readInt32BE() {
+        let value = this._reader.readInt32BE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+
+    readUInt32() {
+        let value = this._reader.readUInt32LE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+
+    readUInt32BE() {
+        let value = this._reader.readUInt32BE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+
+    readLong() {
+        let high = this.readInt32()
+        let low = this.readInt32()
+        return {low, high};
+    }
+
+    readLongBE() {
+        let high = this.readInt32BE()
+        let low = this.readInt32BE()
+        return {low, high};
+    }
+
+    readULong() {
+        let high = this.readUInt()
+        let low = this.readUInt()
+        return {low, high};
+    }
+
+    readULongBE() {
+        let high = this.readUIntBE()
+        let low = this.readUIntBE()
+        return {low, high};
+    }
+
+    readFloat() {
+        let value = this._reader.readFloatLE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+
+    readFloatBE() {
+        let value = this._reader.readFloatBE(this.pc, true);
+        this.pc += 4;
+        return value;
+    }
+    
+    readDouble() {
+        let value = this._reader.readDoubleLE(this.pc, true);
+        this.pc += 8;
+        return value;
+    }
+
+    readDoubleBE() {
+        let value = this._reader.readDoubleBE(this.pc, true);
+        this.pc += 8;
+        return value;
+    }
+    
+    readBytes(length) {
+        if (this.pc + length > this._reader.length) {
+            throw Error("Requested more data than buffer holds");
+        }
+        let value = this._reader.slice(this.pc, this.pc + length);
+        this.pc += length;
+        return value;
+    }
+
+    readString(length) {
+        let value = this.readBytes(length).toString('utf8');
+        return value;
+    }
+}
+
+module.exports = {BinaryReader};
+
+// Registering classes in global scope for propoer class deserialization.
+for (let className of Object.keys(module.exports)) {
+    global[className] = new module.exports[className]();
+}

package/lib/OpCode.js

@@ -0,0 +1,90 @@
+class OpCode {
+    OpCodeID = 0;
+    InstructionName = null;
+    HasArgument = false;
+    Argument = 0;
+    HasName = false;
+    Name = null;
+    HasJumpRelative = false;
+    HasJumpAbsolute = false;
+    HasNegativeOffset = false;
+    HasConstant = false;
+    Constant = null;
+    ConstantObject = null;
+    HasCompare = false;
+    CompareOperator = null;
+    HasLocal = false;
+    LocalName = null;
+    HasBinaryOp = false;
+    BinaryOp = null;
+    HasIntrisic1 = false;
+    Intrisic1 = null;
+    HasIntrisic2 = false;
+    Intrisic2 = null;
+    HasFree = false;
+    FreeName = null;
+    Offset = 0;
+    LineNo = 0;
+    CodeBlock = null;
+    InstructionIndex = -1;
+    Size = 2;  // Default size for Python 3.6+ word-aligned instructions
+
+    constructor(opcode, name, opts) {
+        this.OpCodeID = opcode;
+        this.InstructionName = name;
+
+        if (opts && typeof(opts) == "object") {
+            for (let [key, value] of Object.entries(opts)) {
+                if (key in this) {
+                    this[key] = value;
+                }
+            }
+        }
+
+    }
+
+    get Prev() {
+        if (!this.CodeBlock || this.InstructionIndex < 1) {
+            return null;
+        }
+        return this.CodeBlock.Instructions[this.InstructionIndex - 1];
+    }
+
+    get Next() {
+        if (!this.CodeBlock || this.InstructionIndex < 0 || this.InstructionIndex >= this.CodeBlock.Instructions.length - 1) {
+            return null;
+        }
+        return this.CodeBlock.Instructions[this.InstructionIndex + 1];
+    }
+
+    get JumpTarget () {
+        if (this.HasJumpRelative) {
+            // Python < 3.6: 3-byte instructions (opcode + 2 arg bytes)
+            // Python 3.6+: 2-byte instructions (opcode + arg byte)
+            return this.Offset + this.Size + this.Argument;
+        } else if (this.HasJumpAbsolute) {
+            return this.Argument;
+        } else {
+            return 0;
+        }
+    }
+
+    get Label() {
+        return this.Name || this.LocalName || this.FreeName;
+    }
+
+    Clone() {
+        let clone = new OpCode();
+        for (let [key, value] of Object.entries(this)) {
+            clone[key] = value;
+        }
+        return clone;
+    }
+}
+
+module.exports = OpCode;
+
+// Registering classes in global scope for propoer class deserialization.
+for (let className of Object.keys(module.exports)) {
+    global[className] = new module.exports[className]();
+}