appbinary-inspector 0.1.0__tar.gz → 0.4.0__tar.gz

appbinary_inspector-0.4.0/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.5
+Name: appbinary-inspector
+Version: 0.4.0
+Summary: Open-source lazy-card character model plus static binary inspection tools.
+Author: William Laurento
+License: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Provides-Extra: torch
+Requires-Dist: torch>=2.1; extra == 'torch'
+Description-Content-Type: text/markdown
+
+# appbinary-inspector
+
+Static inspection tools plus an open-source, disk-paged character-model architecture.
+
+## Features
+
+- Detects PE, ELF, Mach-O, ZIP, 7z, and RAR headers
+- Calculates SHA-256 and SHA-1 hashes
+- Extracts readable strings and entropy
+- Reads basic PE header metadata
+- Extracts embedded ZIP archives and carves recognizable binary signatures
+- Exports PE headers and raw binary sections for authorized analysis
+- Builds a compact tokenizer for source code, logs, or extracted text
+- Provides editable PyTorch classifier, autoencoder, and transformer templates
+- Keeps CardStack model cards as separately stored 4-bit SSD files
+- Loads only router-selected cards into a small LRU cache during generation
+- Generates one next byte at a time for true character-level output
+
+## Install
+
+```bash
+pip install appbinary-inspector
+```
+
+## Use
+
+```bash
+appbin inspect program.exe
+appbin extract program.exe --output extracted
+appbin binary-extract program.exe --output pe_parts
+appbin tokenize source.py --vocab source-tokenizer.json
+appbin model-template classifier
+```
+
+## CardStack: low-VRAM model experiments
+
+CardStack is a small trainable byte-level language model. Its expert cards are
+stored as separate 4-bit files. At inference, inactive cards remain on disk and
+the cache retains only a limited number of recently selected cards.
+
+```python
+from appbinary_inspector import CardStackConfig, CardStackLM, train_next_character
+
+model = CardStackLM(CardStackConfig(width=128, layers=4, cards_per_layer=8, active_cards=2, cache_cards=4))
+losses = train_next_character(model, open("training.txt", encoding="utf-8").read(), steps=500)
+model.save("my-card-model")  # writes a base file plus separate 4-bit card files
+
+paged = CardStackLM.load("my-card-model")
+print(paged.generate("Hello", new_characters=80))
+```
+
+CLI version:
+
+```bash
+appbin card-create --output my-card-model --width 128 --layers 4 --cards 8 --active-cards 2
+appbin card-train my-card-model training.txt --steps 500
+appbin card-generate my-card-model "Hello" --characters 80
+```
+
+This reduces active model-card memory. It does not make a dense, unmodified
+70B model fit instantly on a low-VRAM GPU: disk bandwidth and CPU/GPU transfer
+time become the tradeoff, and models need to be trained for sparse routing.
+
+## PyTorch models
+
+Install optional PyTorch support:
+
+```bash
+pip install "appbinary-inspector[torch]"
+```
+
+```python
+from appbinary_inspector.torch_builder import create_model, preset
+
+config = preset("classifier")
+config["layers"][0]["out_features"] = 512
+model = create_model(config)
+```
+
+## Safety
+
+Targets are read as bytes only and never executed. The package does not defeat encryption, DRM, licensing controls, passwords, or code protections. It can analyze plaintext, export raw PE sections, and extract recognizable embedded content from files you are authorized to inspect.

appbinary_inspector-0.4.0/README.md

@@ -0,0 +1,82 @@
+# appbinary-inspector
+
+Static inspection tools plus an open-source, disk-paged character-model architecture.
+
+## Features
+
+- Detects PE, ELF, Mach-O, ZIP, 7z, and RAR headers
+- Calculates SHA-256 and SHA-1 hashes
+- Extracts readable strings and entropy
+- Reads basic PE header metadata
+- Extracts embedded ZIP archives and carves recognizable binary signatures
+- Exports PE headers and raw binary sections for authorized analysis
+- Builds a compact tokenizer for source code, logs, or extracted text
+- Provides editable PyTorch classifier, autoencoder, and transformer templates
+- Keeps CardStack model cards as separately stored 4-bit SSD files
+- Loads only router-selected cards into a small LRU cache during generation
+- Generates one next byte at a time for true character-level output
+
+## Install
+
+```bash
+pip install appbinary-inspector
+```
+
+## Use
+
+```bash
+appbin inspect program.exe
+appbin extract program.exe --output extracted
+appbin binary-extract program.exe --output pe_parts
+appbin tokenize source.py --vocab source-tokenizer.json
+appbin model-template classifier
+```
+
+## CardStack: low-VRAM model experiments
+
+CardStack is a small trainable byte-level language model. Its expert cards are
+stored as separate 4-bit files. At inference, inactive cards remain on disk and
+the cache retains only a limited number of recently selected cards.
+
+```python
+from appbinary_inspector import CardStackConfig, CardStackLM, train_next_character
+
+model = CardStackLM(CardStackConfig(width=128, layers=4, cards_per_layer=8, active_cards=2, cache_cards=4))
+losses = train_next_character(model, open("training.txt", encoding="utf-8").read(), steps=500)
+model.save("my-card-model")  # writes a base file plus separate 4-bit card files
+
+paged = CardStackLM.load("my-card-model")
+print(paged.generate("Hello", new_characters=80))
+```
+
+CLI version:
+
+```bash
+appbin card-create --output my-card-model --width 128 --layers 4 --cards 8 --active-cards 2
+appbin card-train my-card-model training.txt --steps 500
+appbin card-generate my-card-model "Hello" --characters 80
+```
+
+This reduces active model-card memory. It does not make a dense, unmodified
+70B model fit instantly on a low-VRAM GPU: disk bandwidth and CPU/GPU transfer
+time become the tradeoff, and models need to be trained for sparse routing.
+
+## PyTorch models
+
+Install optional PyTorch support:
+
+```bash
+pip install "appbinary-inspector[torch]"
+```
+
+```python
+from appbinary_inspector.torch_builder import create_model, preset
+
+config = preset("classifier")
+config["layers"][0]["out_features"] = 512
+model = create_model(config)
+```
+
+## Safety
+
+Targets are read as bytes only and never executed. The package does not defeat encryption, DRM, licensing controls, passwords, or code protections. It can analyze plaintext, export raw PE sections, and extract recognizable embedded content from files you are authorized to inspect.

appbinary_inspector-0.4.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[project]
+name = "appbinary-inspector"
+version = "0.4.0"
+description = "Open-source lazy-card character model plus static binary inspection tools."
+readme = "README.md"
+requires-python = ">=3.10"
+license = {text = "MIT"}
+authors = [{name = "William Laurento"}]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+]
+dependencies = []
+
+[project.optional-dependencies]
+torch = ["torch>=2.1"]
+
+[project.scripts]
+appbin = "appbinary_inspector.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/appbinary_inspector"]

appbinary_inspector-0.4.0/src/appbinary_inspector/__init__.py

@@ -0,0 +1,9 @@
+"""Static binary inspection without executing the analyzed file."""
+
+from .analyzer import inspect_file
+from .extract import extract_binary_parts, extract_embedded
+from .tokenizer import WordTokenizer
+from .cardstack import CardStackConfig, CardStackLM, train_next_character
+
+__all__ = ["inspect_file", "extract_embedded", "extract_binary_parts", "WordTokenizer", "CardStackConfig", "CardStackLM", "train_next_character"]
+__version__ = "0.4.0"

appbinary_inspector-0.4.0/src/appbinary_inspector/cardstack.py

@@ -0,0 +1,246 @@
+"""A compact, character-by-character language model with on-demand expert cards.
+
+This is a research-sized model architecture, not a pre-trained replacement for
+large commercial models. Each layer routes work to only ``active_cards`` expert
+cards. Serialized cards live in separate files and are loaded into an LRU cache
+only when the router selects them.
+"""
+from __future__ import annotations
+
+import json
+import math
+from collections import OrderedDict
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+
+def _torch():
+    try:
+        import torch
+        import torch.nn as nn
+        import torch.nn.functional as functional
+        return torch, nn, functional
+    except ImportError as error:
+        raise RuntimeError("Install PyTorch first: pip install appbinary-inspector[torch]") from error
+
+
+@dataclass
+class CardStackConfig:
+    width: int = 128
+    layers: int = 4
+    cards_per_layer: int = 8
+    active_cards: int = 2
+    context_size: int = 128
+    cache_cards: int = 4
+    card_bits: int = 4
+
+
+def _pack_int4(values):
+    """Pack signed 4-bit values two per byte for SSD storage."""
+    torch, _, _ = _torch()
+    unsigned = (values.to(torch.int16) + 8).to(torch.uint8).flatten()
+    if unsigned.numel() % 2:
+        unsigned = torch.cat([unsigned, torch.zeros(1, dtype=torch.uint8)])
+    return unsigned[0::2] | (unsigned[1::2] << 4)
+
+
+def _unpack_int4(packed, count: int):
+    torch, _, _ = _torch()
+    low = packed & 0x0F
+    high = packed >> 4
+    return (torch.stack((low, high), dim=1).flatten()[:count].to(torch.int16) - 8).to(torch.float32)
+
+
+def quantize_card(state: dict) -> dict:
+    """Convert a card state dict to independently scaled 4-bit tensors."""
+    torch, _, _ = _torch()
+    tensors = {}
+    for name, tensor in state.items():
+        source = tensor.detach().cpu().float()
+        scale = max(float(source.abs().max()) / 7.0, 1e-8)
+        quantized = torch.round(source / scale).clamp(-8, 7).to("int8")
+        tensors[name] = {"shape": list(source.shape), "scale": scale, "count": source.numel(), "packed": _pack_int4(quantized)}
+    return {"format": "cardstack-int4-v1", "tensors": tensors}
+
+
+def dequantize_card(payload: dict) -> dict:
+    """Restore a quantized card state dict just before the card is needed."""
+    if payload.get("format") != "cardstack-int4-v1":
+        return payload
+    torch, _, _ = _torch()
+    result = {}
+    for name, item in payload["tensors"].items():
+        values = _unpack_int4(item["packed"], item["count"]) * item["scale"]
+        result[name] = values.reshape(item["shape"])
+    return result
+
+
+class CardStackLM:
+    """Byte-level LM. A generation step emits exactly one next byte/character."""
+
+    def __init__(self, config: CardStackConfig, card_directory: str | Path | None = None) -> None:
+        torch, nn, _ = _torch()
+        if not 1 <= config.active_cards <= config.cards_per_layer:
+            raise ValueError("active_cards must be between 1 and cards_per_layer")
+        self.torch, self.nn, self.config = torch, nn, config
+        self.embedding = nn.Embedding(256, config.width)
+        self.position = nn.Embedding(config.context_size, config.width)
+        self.routers = nn.ModuleList([nn.Linear(config.width, config.cards_per_layer) for _ in range(config.layers)])
+        self.norms = nn.ModuleList([nn.LayerNorm(config.width) for _ in range(config.layers)])
+        self.head = nn.Linear(config.width, 256)
+        self.card_directory = Path(card_directory) if card_directory else None
+        self.experts = nn.ModuleList() if not card_directory else None
+        if self.experts is not None:
+            for _ in range(config.layers):
+                self.experts.append(nn.ModuleList([self._new_card() for _ in range(config.cards_per_layer)]))
+        self.cache: OrderedDict[tuple[int, int], object] = OrderedDict()
+
+    def _new_card(self):
+        nn = self.nn
+        return nn.Sequential(nn.Linear(self.config.width, self.config.width * 2), nn.GELU(), nn.Linear(self.config.width * 2, self.config.width))
+
+    def _card(self, layer: int, index: int):
+        if self.experts is not None:
+            return self.experts[layer][index]
+        key = (layer, index)
+        if key in self.cache:
+            self.cache.move_to_end(key)
+            return self.cache[key]
+        if self.card_directory is None:
+            raise RuntimeError("No card storage configured")
+        card = self._new_card()
+        state = self.torch.load(self.card_directory / f"layer_{layer:02d}_card_{index:02d}.pt", map_location="cpu", weights_only=True)
+        state = dequantize_card(state)
+        card.load_state_dict(state)
+        card.eval()
+        self.cache[key] = card
+        while len(self.cache) > self.config.cache_cards:
+            self.cache.popitem(last=False)
+        return card
+
+    def prefetch(self, layer: int, card_indices: list[int]) -> None:
+        """Load likely next cards before the caller reaches that layer."""
+        if self.experts is None:
+            for index in card_indices:
+                self._card(layer, int(index))
+
+    def parameters(self):
+        modules = [self.embedding, self.position, self.routers, self.norms, self.head]
+        if self.experts is not None:
+            modules.append(self.experts)
+        for module in modules:
+            yield from module.parameters()
+
+    def train(self) -> None:
+        for module in (self.embedding, self.position, self.routers, self.norms, self.head):
+            module.train()
+        if self.experts is not None:
+            self.experts.train()
+
+    def eval(self) -> None:
+        for module in (self.embedding, self.position, self.routers, self.norms, self.head):
+            module.eval()
+        if self.experts is not None:
+            self.experts.eval()
+
+    def forward(self, token_ids):
+        torch = self.torch
+        _, sequence = token_ids.shape
+        if sequence > self.config.context_size:
+            token_ids = token_ids[:, -self.config.context_size:]
+            sequence = token_ids.shape[1]
+        positions = torch.arange(sequence, device=token_ids.device)
+        hidden = self.embedding(token_ids) + self.position(positions)[None, :, :]
+        for layer, (router, norm) in enumerate(zip(self.routers, self.norms)):
+            normalized = norm(hidden)
+            scores = router(normalized)
+            values, indices = scores.topk(self.config.active_cards, dim=-1)
+            # After layer 0 the current representation is a useful cheap hint for
+            # the next router, so prefetch its likely cards from SSD into the LRU.
+            if self.experts is None and layer + 1 < self.config.layers:
+                next_scores = self.routers[layer + 1](normalized)
+                next_indices = next_scores.topk(self.config.active_cards, dim=-1).indices.unique().tolist()
+                self.prefetch(layer + 1, next_indices)
+            weights = values.softmax(dim=-1)
+            routed = torch.zeros_like(hidden)
+            # Run only cards selected anywhere in this batch, then apply their gate.
+            for card_index in indices.unique().tolist():
+                card_output = self._card(layer, card_index)(normalized)
+                gate = (weights * (indices == card_index)).sum(dim=-1, keepdim=True)
+                routed = routed + card_output * gate
+            hidden = hidden + routed
+        return self.head(hidden)
+
+    def save(self, directory: str | Path, quantize_cards: bool = True) -> None:
+        if self.experts is None:
+            raise RuntimeError("A paged inference model cannot be saved as a new training bundle")
+        destination = Path(directory)
+        destination.mkdir(parents=True, exist_ok=True)
+        (destination / "config.json").write_text(json.dumps(asdict(self.config), indent=2), encoding="utf-8")
+        self.torch.save({"embedding": self.embedding.state_dict(), "position": self.position.state_dict(), "routers": self.routers.state_dict(), "norms": self.norms.state_dict(), "head": self.head.state_dict()}, destination / "base.pt")
+        for layer, cards in enumerate(self.experts):
+            for index, card in enumerate(cards):
+                payload = quantize_card(card.state_dict()) if quantize_cards else card.state_dict()
+                self.torch.save(payload, destination / f"layer_{layer:02d}_card_{index:02d}.pt")
+
+    @classmethod
+    def load(cls, directory: str | Path) -> "CardStackLM":
+        folder = Path(directory)
+        config = CardStackConfig(**json.loads((folder / "config.json").read_text(encoding="utf-8")))
+        model = cls(config, card_directory=folder)
+        state = model.torch.load(folder / "base.pt", map_location="cpu", weights_only=True)
+        for name in ("embedding", "position", "routers", "norms", "head"):
+            getattr(model, name).load_state_dict(state[name])
+        model.eval()
+        return model
+
+    @classmethod
+    def load_for_training(cls, directory: str | Path) -> "CardStackLM":
+        """Load every card into memory for training. Use ``load`` for low-memory inference."""
+        folder = Path(directory)
+        config = CardStackConfig(**json.loads((folder / "config.json").read_text(encoding="utf-8")))
+        model = cls(config)
+        state = model.torch.load(folder / "base.pt", map_location="cpu", weights_only=True)
+        for name in ("embedding", "position", "routers", "norms", "head"):
+            getattr(model, name).load_state_dict(state[name])
+        for layer, cards in enumerate(model.experts):
+            for index, card in enumerate(cards):
+                payload = model.torch.load(folder / f"layer_{layer:02d}_card_{index:02d}.pt", map_location="cpu", weights_only=True)
+                card.load_state_dict(dequantize_card(payload))
+        return model
+
+    def generate(self, prompt: str, new_characters: int = 80, temperature: float = 0.8) -> str:
+        torch = self.torch
+        if temperature <= 0:
+            raise ValueError("temperature must be positive")
+        values = list(prompt.encode("utf-8", "replace")) or [32]
+        self.eval()
+        with torch.no_grad():
+            for _ in range(new_characters):
+                input_ids = torch.tensor([values[-self.config.context_size:]], dtype=torch.long)
+                logits = self.forward(input_ids)[0, -1] / temperature
+                next_value = torch.multinomial(torch.softmax(logits, dim=-1), 1).item()
+                values.append(next_value)
+        return bytes(values).decode("utf-8", "replace")
+
+
+def train_next_character(model: CardStackLM, text: str, steps: int = 500, learning_rate: float = 3e-4) -> list[float]:
+    """Train a small model from user-supplied text. Returns sampled loss values."""
+    torch, _, functional = _torch()
+    data = torch.tensor(list(text.encode("utf-8", "replace")), dtype=torch.long)
+    if len(data) <= model.config.context_size:
+        raise ValueError("Training text must exceed context_size bytes")
+    model.train()
+    optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
+    losses = []
+    for step in range(steps):
+        start = torch.randint(0, len(data) - model.config.context_size - 1, (1,)).item()
+        inputs = data[start:start + model.config.context_size].unsqueeze(0)
+        targets = data[start + 1:start + model.config.context_size + 1].unsqueeze(0)
+        optimizer.zero_grad()
+        loss = functional.cross_entropy(model.forward(inputs).reshape(-1, 256), targets.reshape(-1))
+        loss.backward()
+        optimizer.step()
+        if step % max(1, steps // 10) == 0 or step == steps - 1:
+            losses.append(round(float(loss.detach()), 4))
+    return losses

appbinary_inspector-0.4.0/src/appbinary_inspector/cli.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from .analyzer import inspect_file
+from .extract import extract_embedded
+from .extract import extract_binary_parts
+from .tokenizer import WordTokenizer
+from .torch_builder import PRESETS
+from .cardstack import CardStackConfig, CardStackLM, train_next_character
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Static application-binary inspector. Never executes targets.")
+    commands = parser.add_subparsers(dest="command", required=True)
+    inspect_cmd = commands.add_parser("inspect", help="inspect a binary")
+    inspect_cmd.add_argument("file")
+    inspect_cmd.add_argument("--strings", type=int, default=200)
+    extract_cmd = commands.add_parser("extract", help="extract recognizable embedded files")
+    extract_cmd.add_argument("file")
+    extract_cmd.add_argument("--output", required=True)
+    parts_cmd = commands.add_parser("binary-extract", help="export PE headers and raw sections")
+    parts_cmd.add_argument("file")
+    parts_cmd.add_argument("--output", required=True)
+    tokenize_cmd = commands.add_parser("tokenize", help="train a tokenizer and encode a text/source file")
+    tokenize_cmd.add_argument("file")
+    tokenize_cmd.add_argument("--vocab", default="tokenizer.json")
+    tokenize_cmd.add_argument("--max-vocab", type=int, default=20_000)
+    preset_cmd = commands.add_parser("model-template", help="print a PyTorch model configuration preset")
+    preset_cmd.add_argument("name", choices=sorted(PRESETS))
+    card_create = commands.add_parser("card-create", help="create a disk-paged character model")
+    card_create.add_argument("--output", required=True)
+    card_create.add_argument("--width", type=int, default=128)
+    card_create.add_argument("--layers", type=int, default=4)
+    card_create.add_argument("--cards", type=int, default=8)
+    card_create.add_argument("--active-cards", type=int, default=2)
+    card_create.add_argument("--cache-cards", type=int, default=4)
+    card_train = commands.add_parser("card-train", help="train a CardStack model from text")
+    card_train.add_argument("model")
+    card_train.add_argument("text_file")
+    card_train.add_argument("--steps", type=int, default=500)
+    card_generate = commands.add_parser("card-generate", help="generate one character at a time from disk-paged cards")
+    card_generate.add_argument("model")
+    card_generate.add_argument("prompt")
+    card_generate.add_argument("--characters", type=int, default=80)
+    args = parser.parse_args()
+    if args.command == "inspect":
+        result = inspect_file(args.file, args.strings)
+    elif args.command == "extract":
+        result = extract_embedded(args.file, args.output)
+    elif args.command == "binary-extract":
+        result = extract_binary_parts(args.file, args.output)
+    elif args.command == "tokenize":
+        text = Path(args.file).read_text(encoding="utf-8", errors="replace")
+        tokenizer = WordTokenizer().fit([text], max_vocab=args.max_vocab)
+        tokenizer.save(args.vocab)
+        result = {"vocabulary_path": args.vocab, "vocabulary_size": len(tokenizer.vocabulary), "tokens": tokenizer.tokenize(text), "ids": tokenizer.encode(text)}
+    elif args.command == "card-create":
+        config = CardStackConfig(width=args.width, layers=args.layers, cards_per_layer=args.cards, active_cards=args.active_cards, cache_cards=args.cache_cards)
+        CardStackLM(config).save(args.output)
+        result = {"model": args.output, "config": config.__dict__, "note": "Cards were saved in 4-bit files. Train before expecting useful output."}
+    elif args.command == "card-train":
+        model = CardStackLM.load_for_training(args.model)
+        losses = train_next_character(model, Path(args.text_file).read_text(encoding="utf-8", errors="replace"), steps=args.steps)
+        model.save(args.model)
+        result = {"model": args.model, "steps": args.steps, "loss_samples": losses}
+    elif args.command == "card-generate":
+        result = {"text": CardStackLM.load(args.model).generate(args.prompt, new_characters=args.characters)}
+    else:
+        result = PRESETS[args.name]
+    print(json.dumps(result, indent=2))
+
+if __name__ == "__main__":
+    main()

appbinary_inspector-0.4.0/src/appbinary_inspector/extract.py

@@ -0,0 +1,78 @@
+from __future__ import annotations
+
+import io
+import json
+import struct
+import zipfile
+from pathlib import Path
+
+from .formats import EMBEDDED_SIGNATURES
+
+
+def extract_binary_parts(path: str | Path, output_dir: str | Path) -> dict:
+    """Write raw PE sections and a manifest. Reads bytes only; never executes code."""
+    source, destination = Path(path), Path(output_dir)
+    destination.mkdir(parents=True, exist_ok=True)
+    data = source.read_bytes()
+    manifest: dict = {"source": str(source), "size_bytes": len(data), "parts": []}
+    if not data.startswith(b"MZ") or len(data) < 0x40:
+        raise ValueError("binary-parts currently supports Windows PE files beginning with MZ")
+    pe_offset = struct.unpack_from("<I", data, 0x3C)[0]
+    if data[pe_offset:pe_offset + 4] != b"PE\0\0":
+        raise ValueError("invalid PE header")
+    _, section_count, _, _, _, optional_size, _ = struct.unpack_from("<HHIIIHH", data, pe_offset + 4)
+    header_end = pe_offset + 24 + optional_size + section_count * 40
+    header_path = destination / "pe_headers.bin"
+    header_path.write_bytes(data[:min(header_end, len(data))])
+    manifest["parts"].append({"name": "headers", "offset": 0, "size": header_path.stat().st_size, "path": str(header_path)})
+    table_offset = pe_offset + 24 + optional_size
+    for index in range(section_count):
+        entry = table_offset + index * 40
+        name = data[entry:entry + 8].split(b"\0", 1)[0].decode("ascii", "replace") or f"section_{index}"
+        raw_size, raw_offset = struct.unpack_from("<II", data, entry + 16)
+        content = data[raw_offset:min(raw_offset + raw_size, len(data))] if raw_offset < len(data) else b""
+        safe_name = "".join(char if char.isalnum() or char in "._-" else "_" for char in name)
+        section_path = destination / f"{index:02d}_{safe_name}.bin"
+        section_path.write_bytes(content)
+        manifest["parts"].append({"name": name, "offset": raw_offset, "size": len(content), "path": str(section_path)})
+    (destination / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
+    return manifest
+
+def extract_embedded(path: str | Path, output_dir: str | Path, min_offset: int = 1) -> list[dict]:
+    """Carve recognizable embedded files without executing or decrypting content.
+
+    ZIP data is extracted only when it starts at a detected signature and passes
+    ZIP validation. Other formats are preserved from their signature to EOF for
+    analyst review, avoiding guesses about encrypted or packed payload bounds.
+    """
+    source = Path(path)
+    destination = Path(output_dir)
+    destination.mkdir(parents=True, exist_ok=True)
+    data = source.read_bytes()
+    found: list[dict] = []
+    seen: set[tuple[int, str]] = set()
+    for signature, (kind, extension) in EMBEDDED_SIGNATURES.items():
+        start = 0
+        while True:
+            index = data.find(signature, start)
+            if index < 0:
+                break
+            start = index + 1
+            if index < min_offset or (index, kind) in seen:
+                continue
+            seen.add((index, kind))
+            stem = f"{source.stem}_offset_{index:08x}"
+            if kind == "zip":
+                try:
+                    with zipfile.ZipFile(io.BytesIO(data[index:])) as archive:
+                        member_dir = destination / stem
+                        member_dir.mkdir(exist_ok=True)
+                        archive.extractall(member_dir)
+                        found.append({"offset": index, "type": kind, "path": str(member_dir), "members": archive.namelist()})
+                except (zipfile.BadZipFile, OSError):
+                    pass
+            else:
+                carved = destination / f"{stem}{extension}"
+                carved.write_bytes(data[index:])
+                found.append({"offset": index, "type": kind, "path": str(carved), "note": "carved from signature through EOF"})
+    return found

appbinary_inspector-0.4.0/src/appbinary_inspector/tokenizer.py

@@ -0,0 +1,48 @@
+"""Small, dependency-free tokenizer for source, logs, and extracted text."""
+from __future__ import annotations
+
+import json
+import re
+from collections import Counter
+from pathlib import Path
+
+TOKEN_PATTERN = re.compile(r"[A-Za-z_][A-Za-z_0-9]*|\d+(?:\.\d+)?|[^\s]", re.UNICODE)
+SPECIAL_TOKENS = ("<pad>", "<unk>", "<bos>", "<eos>")
+
+
+class WordTokenizer:
+    """Train a compact word/code tokenizer without external model downloads."""
+
+    def __init__(self, vocabulary: dict[str, int] | None = None) -> None:
+        self.vocabulary = vocabulary or {token: index for index, token in enumerate(SPECIAL_TOKENS)}
+        self.inverse_vocabulary = {index: token for token, index in self.vocabulary.items()}
+
+    @staticmethod
+    def tokenize(text: str) -> list[str]:
+        return TOKEN_PATTERN.findall(text)
+
+    def fit(self, texts: list[str], max_vocab: int = 20_000, min_frequency: int = 1) -> "WordTokenizer":
+        if max_vocab < len(SPECIAL_TOKENS):
+            raise ValueError("max_vocab must allow the special tokens")
+        counts = Counter(token for text in texts for token in self.tokenize(text))
+        self.vocabulary = {token: index for index, token in enumerate(SPECIAL_TOKENS)}
+        for token, count in counts.most_common(max_vocab - len(SPECIAL_TOKENS)):
+            if count < min_frequency:
+                continue
+            self.vocabulary[token] = len(self.vocabulary)
+        self.inverse_vocabulary = {index: token for token, index in self.vocabulary.items()}
+        return self
+
+    def encode(self, text: str, add_boundaries: bool = False) -> list[int]:
+        ids = [self.vocabulary.get(token, self.vocabulary["<unk>"]) for token in self.tokenize(text)]
+        return ([self.vocabulary["<bos>"]] + ids + [self.vocabulary["<eos>"]]) if add_boundaries else ids
+
+    def decode(self, ids: list[int]) -> list[str]:
+        return [self.inverse_vocabulary.get(index, "<unk>") for index in ids]
+
+    def save(self, path: str | Path) -> None:
+        Path(path).write_text(json.dumps({"vocabulary": self.vocabulary}, indent=2), encoding="utf-8")
+
+    @classmethod
+    def load(cls, path: str | Path) -> "WordTokenizer":
+        return cls(json.loads(Path(path).read_text(encoding="utf-8"))["vocabulary"])

appbinary_inspector-0.4.0/src/appbinary_inspector/torch_builder.py

@@ -0,0 +1,70 @@
+"""Config-first PyTorch model construction with no hidden training framework."""
+from __future__ import annotations
+
+from copy import deepcopy
+
+PRESETS = {
+    "classifier": {"input_features": 128, "layers": [{"type": "linear", "out_features": 256}, {"type": "relu"}, {"type": "dropout", "p": 0.2}, {"type": "linear", "out_features": 10}]},
+    "autoencoder": {"input_features": 128, "layers": [{"type": "linear", "out_features": 64}, {"type": "relu"}, {"type": "linear", "out_features": 16}, {"type": "relu"}, {"type": "linear", "out_features": 64}, {"type": "relu"}, {"type": "linear", "out_features": 128}]},
+    "text_transformer": {"vocab_size": 20_000, "embedding_dim": 128, "sequence_length": 128, "transformer_layers": 2, "heads": 4, "classes": 2},
+}
+
+
+def preset(name: str) -> dict:
+    if name not in PRESETS:
+        raise ValueError(f"Unknown preset: {name}. Choose from {', '.join(PRESETS)}")
+    return deepcopy(PRESETS[name])
+
+
+def create_model(config: dict):
+    """Build an nn.Module from a transparent config. Requires ``pip install appbinary-inspector[torch]``."""
+    try:
+        import torch.nn as nn
+    except ImportError as error:
+        raise RuntimeError("PyTorch is optional. Install it with: pip install appbinary-inspector[torch]") from error
+
+    if config.get("kind") == "transformer" or "vocab_size" in config:
+        required = ("vocab_size", "embedding_dim", "sequence_length", "transformer_layers", "heads", "classes")
+        missing = [key for key in required if key not in config]
+        if missing:
+            raise ValueError(f"Missing transformer settings: {', '.join(missing)}")
+        d_model = config["embedding_dim"]
+        encoder = nn.TransformerEncoder(
+            nn.TransformerEncoderLayer(d_model=d_model, nhead=config["heads"], batch_first=True),
+            num_layers=config["transformer_layers"],
+        )
+
+        class TokenClassifier(nn.Module):
+            def __init__(self) -> None:
+                super().__init__()
+                self.embedding = nn.Embedding(config["vocab_size"], d_model)
+                self.encoder = encoder
+                self.head = nn.Linear(d_model, config["classes"])
+
+            def forward(self, token_ids):
+                encoded = self.encoder(self.embedding(token_ids))
+                return self.head(encoded.mean(dim=1))
+
+        return TokenClassifier()
+
+    features = config.get("input_features")
+    if not isinstance(features, int) or features < 1:
+        raise ValueError("input_features must be a positive integer")
+    layers = []
+    for spec in config.get("layers", []):
+        kind = spec.get("type", "").lower()
+        if kind == "linear":
+            out_features = spec.get("out_features")
+            if not isinstance(out_features, int) or out_features < 1:
+                raise ValueError("linear layers require positive out_features")
+            layers.append(nn.Linear(features, out_features))
+            features = out_features
+        elif kind in {"relu", "gelu", "tanh", "sigmoid", "flatten"}:
+            layers.append({"relu": nn.ReLU, "gelu": nn.GELU, "tanh": nn.Tanh, "sigmoid": nn.Sigmoid, "flatten": nn.Flatten}[kind]())
+        elif kind == "dropout":
+            layers.append(nn.Dropout(float(spec.get("p", 0.5))))
+        else:
+            raise ValueError(f"Unsupported layer type: {kind}")
+    if not layers:
+        raise ValueError("config requires at least one layer")
+    return nn.Sequential(*layers)

appbinary_inspector-0.1.0/PKG-INFO

@@ -1,41 +0,0 @@
-Metadata-Version: 2.5
-Name: appbinary-inspector
-Version: 0.1.0
-Summary: Safe static inspection and embedded-file extraction for application binaries.
-Author: William Laurento
-License: MIT
-License-File: LICENSE
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-
-# appbinary-inspector
-
-Static inspection of application binaries without executing them.
-
-## Features
-
-- Detects PE, ELF, Mach-O, ZIP, 7z, and RAR headers
-- Calculates SHA-256 and SHA-1 hashes
-- Extracts readable strings and entropy
-- Reads basic PE header metadata
-- Extracts embedded ZIP archives and carves recognizable binary signatures
-
-## Install
-
-```bash
-pip install appbinary-inspector
-```
-
-## Use
-
-```bash
-appbin inspect program.exe
-appbin extract program.exe --output extracted
-```
-
-## Safety
-
-Targets are read as bytes only and never executed. The package does not defeat encryption, DRM, licensing controls, passwords, or code protections. It can analyze plaintext and extract recognizable embedded content from files you are authorized to inspect.

appbinary_inspector-0.1.0/README.md

@@ -1,28 +0,0 @@
-# appbinary-inspector
-
-Static inspection of application binaries without executing them.
-
-## Features
-
-- Detects PE, ELF, Mach-O, ZIP, 7z, and RAR headers
-- Calculates SHA-256 and SHA-1 hashes
-- Extracts readable strings and entropy
-- Reads basic PE header metadata
-- Extracts embedded ZIP archives and carves recognizable binary signatures
-
-## Install
-
-```bash
-pip install appbinary-inspector
-```
-
-## Use
-
-```bash
-appbin inspect program.exe
-appbin extract program.exe --output extracted
-```
-
-## Safety
-
-Targets are read as bytes only and never executed. The package does not defeat encryption, DRM, licensing controls, passwords, or code protections. It can analyze plaintext and extract recognizable embedded content from files you are authorized to inspect.

appbinary_inspector-0.1.0/pyproject.toml

@@ -1,20 +0,0 @@
-[build-system]
-requires = ["hatchling>=1.25"]
-build-backend = "hatchling.build"
-
-[project]
-name = "appbinary-inspector"
-version = "0.1.0"
-description = "Safe static inspection and embedded-file extraction for application binaries."
-readme = "README.md"
-requires-python = ">=3.10"
-license = {text = "MIT"}
-authors = [{name = "William Laurento"}]
-classifiers = ["Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent"]
-dependencies = []
-
-[project.scripts]
-appbin = "appbinary_inspector.cli:main"
-
-[tool.hatch.build.targets.wheel]
-packages = ["src/appbinary_inspector"]

appbinary_inspector-0.1.0/src/appbinary_inspector/__init__.py

@@ -1,7 +0,0 @@
-"""Static binary inspection without executing the analyzed file."""
-
-from .analyzer import inspect_file
-from .extract import extract_embedded
-
-__all__ = ["inspect_file", "extract_embedded"]
-__version__ = "0.1.0"

appbinary_inspector-0.1.0/src/appbinary_inspector/cli.py

@@ -1,23 +0,0 @@
-from __future__ import annotations
-
-import argparse
-import json
-
-from .analyzer import inspect_file
-from .extract import extract_embedded
-
-def main() -> None:
-    parser = argparse.ArgumentParser(description="Static application-binary inspector. Never executes targets.")
-    commands = parser.add_subparsers(dest="command", required=True)
-    inspect_cmd = commands.add_parser("inspect", help="inspect a binary")
-    inspect_cmd.add_argument("file")
-    inspect_cmd.add_argument("--strings", type=int, default=200)
-    extract_cmd = commands.add_parser("extract", help="extract recognizable embedded files")
-    extract_cmd.add_argument("file")
-    extract_cmd.add_argument("--output", required=True)
-    args = parser.parse_args()
-    result = inspect_file(args.file, args.strings) if args.command == "inspect" else extract_embedded(args.file, args.output)
-    print(json.dumps(result, indent=2))
-
-if __name__ == "__main__":
-    main()

appbinary_inspector-0.1.0/src/appbinary_inspector/extract.py

@@ -1,41 +0,0 @@
-from __future__ import annotations
-
-import io
-import zipfile
-from pathlib import Path
-
-from .formats import EMBEDDED_SIGNATURES
-
-def extract_embedded(path: str | Path, output_dir: str | Path, min_offset: int = 1) -> list[dict]:
-    """Carve recognizable embedded files without executing or decrypting content."""
-    source = Path(path)
-    destination = Path(output_dir)
-    destination.mkdir(parents=True, exist_ok=True)
-    data = source.read_bytes()
-    found: list[dict] = []
-    seen: set[tuple[int, str]] = set()
-    for signature, (kind, extension) in EMBEDDED_SIGNATURES.items():
-        start = 0
-        while True:
-            index = data.find(signature, start)
-            if index < 0:
-                break
-            start = index + 1
-            if index < min_offset or (index, kind) in seen:
-                continue
-            seen.add((index, kind))
-            stem = f"{source.stem}_offset_{index:08x}"
-            if kind == "zip":
-                try:
-                    with zipfile.ZipFile(io.BytesIO(data[index:])) as archive:
-                        member_dir = destination / stem
-                        member_dir.mkdir(exist_ok=True)
-                        archive.extractall(member_dir)
-                        found.append({"offset": index, "type": kind, "path": str(member_dir), "members": archive.namelist()})
-                except (zipfile.BadZipFile, OSError):
-                    pass
-            else:
-                carved = destination / f"{stem}{extension}"
-                carved.write_bytes(data[index:])
-                found.append({"offset": index, "type": kind, "path": str(carved), "note": "carved from signature through EOF"})
-    return found