python-skills 1.0.0__py3-none-any.whl

skills/security/unsafe_deserialization.md

@@ -0,0 +1,164 @@
+# Security: Unsafe Deserialization Prevention
+
+**Purpose**: Prevent code execution via deserialization of untrusted data.
+
+**When to use**: Loading pickles, YAML, JSON with custom decoders, any serialized data.
+
+---
+
+## Core Rules
+
+### Never Deserialize Untrusted Data with Unsafe Formats
+```python
+# NEVER — pickle executes arbitrary code
+import pickle
+data = pickle.loads(untrusted_bytes)  # RCE!
+
+# NEVER — yaml.load without SafeLoader
+import yaml
+data = yaml.load(untrusted_string)  # RCE!
+
+# NEVER — shelve (uses pickle)
+import shelve
+db = shelve.open(untrusted_file)
+
+# NEVER — dill, cloudpickle, marshal
+```
+
+### Safe Alternatives
+| Unsafe | Safe Replacement |
+|--------|------------------|
+| `pickle` | `json`, `msgpack`, `orjson`, `cbor2` |
+| `yaml.load()` | `yaml.safe_load()` |
+| `shelve` | `sqlite3` + `json` |
+| Custom `__reduce__` | Don't accept serialized objects |
+
+### If You Must Use Pickle (Internal Only)
+```python
+# ONLY for trusted internal data
+# Add integrity verification
+import hmac
+import hashlib
+
+def sign_data(data: bytes, key: bytes) -> bytes:
+    return hmac.new(key, data, hashlib.sha256).digest()
+
+def verify_and_load(data: bytes, signature: bytes, key: bytes) -> Any:
+    if not hmac.compare_digest(sign(data, key), signature):
+        raise SecurityError("Invalid signature")
+    return pickle.loads(data)  # Still risky if key compromised
+```
+
+### YAML Safe Loading
+```python
+import yaml
+
+# ALWAYS use safe_load
+data = yaml.safe_load(untrusted_string)
+
+# Or explicit SafeLoader
+data = yaml.load(untrusted_string, Loader=yaml.SafeLoader)
+
+# NEVER
+data = yaml.load(untrusted_string)  # Default Loader is unsafe!
+data = yaml.load(untrusted_string, Loader=yaml.Loader)  # Unsafe!
+data = yaml.load(untrusted_string, Loader=yaml.FullLoader)  # Still unsafe!
+```
+
+### JSON Safety
+```python
+import json
+
+# json module is SAFE — no code execution
+data = json.loads(untrusted_string)
+
+# But: watch for DoS (deeply nested, huge objects)
+# Use limits if needed
+import sys
+json.loads(huge_string)  # Can consume memory
+```
+
+### Custom Decoders (Risk)
+```python
+# Dangerous if hook executes code
+def dangerous_hook(d):
+    if "__class__" in d:
+        return globals()[d["__class__"]](**d)  # RCE!
+    return d
+
+json.loads(untrusted, object_hook=dangerous_hook)  # NEVER
+
+# Safe: only transform data, no execution
+def safe_hook(d):
+    if "created_at" in d:
+        d["created_at"] = datetime.fromisoformat(d["created_at"])
+    return d
+```
+
+---
+
+## Decision Rules
+
+| Format | Trusted Internal | Untrusted External |
+|--------|------------------|-------------------|
+| JSON | ✓ | ✓ (with size limits) |
+| YAML (safe_load) | ✓ | ✓ |
+| MessagePack | ✓ | ✓ |
+| CBOR | ✓ | ✓ |
+| Pickle | ✓ (with signing) | ✗ NEVER |
+| Shelve | ✓ | ✗ |
+| Custom pickle | ✓ (with signing) | ✗ NEVER |
+
+---
+
+## Preferred Patterns
+
+```python
+# Config loading — use safe format
+def load_config(path: Path) -> Config:
+    if path.suffix == ".json":
+        return Config.model_validate(json.loads(path.read_text()))
+    elif path.suffix in (".yaml", ".yml"):
+        return Config.model_validate(yaml.safe_load(path.read_text()))
+    elif path.suffix == ".toml":
+        import tomllib
+        return Config.model_validate(tomllib.loads(path.read_text()))
+    else:
+        raise ValueError("Unsupported config format")
+
+# Cache — use safe serialization
+def cache_get(key: str) -> Any | None:
+    data = redis.get(key)
+    if data:
+        return json.loads(data)
+    return None
+
+def cache_set(key: str, value: Any, ttl: int) -> None:
+    redis.setex(key, ttl, json.dumps(value, default=json_default))
+```
+
+---
+
+## Avoid
+
+- `pickle` for any external data
+- `yaml.load()` without `SafeLoader`
+- Custom `object_hook` that instantiates classes
+- Storing serialized objects in database for later deserialization
+- Assuming "internal" data stays internal forever
+
+---
+
+## Validation Considerations
+
+- `bandit` B301, B506 checks
+- Dependency scan for pickle usage
+- Penetration testing deserialization endpoints
+
+---
+
+## Related Skills
+
+- `security/input_validation.md`
+- `stdlib/json.md`
+- `engineering/configuration.md`

skills/stdlib/argparse.md

@@ -0,0 +1,178 @@
+# Stdlib: argparse
+
+**Purpose**: Command-line argument parsing.
+
+**When to use**: CLI applications. For simple scripts, consider `sys.argv` or `click`/`typer` (external).
+
+---
+
+## Core Rules
+
+### Basic Setup
+```python
+import argparse
+
+parser = argparse.ArgumentParser(
+    description="My CLI tool",
+    formatter_class=argparse.ArgumentDefaultsHelpFormatter,  # Shows defaults
+)
+
+# Positional argument
+parser.add_argument("input", type=Path, help="Input file")
+
+# Optional argument
+parser.add_argument("-o", "--output", type=Path, help="Output file")
+parser.add_argument("-v", "--verbose", action="count", default=0)  # -v, -vv, -vvv
+
+# Flag
+parser.add_argument("--force", action="store_true")
+
+# Choice
+parser.add_argument("--format", choices=["json", "yaml", "txt"], default="json")
+
+# Type with validation
+def positive_int(s):
+    v = int(s)
+    if v <= 0:
+        raise argparse.ArgumentTypeError("Must be positive")
+    return v
+
+parser.add_argument("--count", type=positive_int, default=10)
+
+# Append (multiple values)
+parser.add_argument("--tags", action="append", default=[])
+
+# Mutually exclusive group
+group = parser.add_mutually_exclusive_group(required=True)
+group.add_argument("--fast", action="store_true")
+group.add_argument("--thorough", action="store_true")
+
+# Subcommands
+subparsers = parser.add_subparsers(dest="command", required=True)
+init_parser = subparsers.add_parser("init")
+init_parser.add_argument("name")
+run_parser = subparsers.add_parser("run")
+run_parser.add_argument("--port", type=int, default=8000)
+
+args = parser.parse_args()
+```
+
+### Namespace Access
+```python
+args.input           # Path object
+args.output
+args.verbose         # 0, 1, 2, 3
+args.force           # True/False
+args.format          # "json" | "yaml" | "txt"
+args.command         # "init" | "run"
+args.name            # For init subcommand
+```
+
+### Custom Types
+```python
+# Path validation
+def existing_file(s):
+    path = Path(s)
+    if not path.is_file():
+        raise argparse.ArgumentTypeError(f"Not a file: {s}")
+    return path
+
+parser.add_argument("file", type=existing_file)
+
+# Comma-separated list
+def csv_list(s):
+    return [x.strip() for x in s.split(",") if x.strip()]
+
+parser.add_argument("--tags", type=csv_list, default=[])
+```
+
+### Subcommand Pattern (Recommended)
+```python
+def main():
+    parser = create_parser()
+    args = parser.parse_args()
+    
+    # Dispatch to handler
+    handlers = {
+        "init": handle_init,
+        "run": handle_run,
+    }
+    return handlers[args.command](args)
+
+def handle_init(args):
+    print(f"Initializing {args.name}")
+
+def handle_run(args):
+    print(f"Running on port {args.port}")
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+---
+
+## Decision Rules
+
+| Need | Pattern |
+|------|---------|
+| Required input | Positional argument |
+| Optional with default | `--option` with `default` |
+| Boolean flag | `action="store_true"` |
+| Count verbosity | `action="count"` |
+| Multiple values | `action="append"` or `nargs="*"` |
+| Choice validation | `choices=[...]` |
+| Custom validation | `type=callable` |
+| Subcommands | `add_subparsers()` |
+| Mutually exclusive | `add_mutually_exclusive_group()` |
+
+---
+
+## Preferred Patterns
+
+```python
+# Main entry point pattern
+def create_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="mytool",
+        description="Description",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    # ... add arguments
+    return parser
+
+def main(argv: list[str] | None = None) -> int:
+    parser = create_parser()
+    args = parser.parse_args(argv)
+    return dispatch(args)
+
+if __name__ == "__main__":
+    sys.exit(main())
+```
+
+---
+
+## Avoid
+
+- `sys.argv` parsing manually (error-prone)
+- Global parser state
+- Complex logic in `type=` callables (keep simple)
+- Required optional arguments (confusing UX)
+- Too many positional arguments (>2-3)
+
+---
+
+## Validation Considerations
+
+- `parser.parse_args(["--help"])` exits 0
+- Invalid args exit 2 with usage
+- Test with `argv` parameter for unit testing
+- `ArgumentDefaultsHelpFormatter` shows defaults in help
+
+---
+
+## Related Skills
+
+- `engineering/cli_apps.md`
+- `engineering/configuration.md`
+- `stdlib/os_sys.md`
+- `generation/type_hints.md`

skills/stdlib/collections.md

@@ -0,0 +1,212 @@
+# Stdlib: collections
+
+**Purpose**: Specialized container datatypes beyond built-ins.
+
+**When to use**: When built-in list/dict/set don't fit the problem.
+
+---
+
+## Core Rules
+
+### `namedtuple` / `NamedTuple`
+```python
+# Classic
+from collections import namedtuple
+Point = namedtuple("Point", ["x", "y"])
+
+# Modern (Python 3.6+) — preferred
+from typing import NamedTuple
+class Point(NamedTuple):
+    x: float
+    y: float
+
+# Usage
+p = Point(1.0, 2.0)
+p.x, p[0]  # Both work
+```
+
+- Immutable, hashable, lightweight
+- NamedTuple supports type hints, methods, defaults
+
+### `dataclass` vs `NamedTuple`
+| Feature | `NamedTuple` | `@dataclass` |
+|---------|--------------|--------------|
+| Mutability | Immutable | Mutable (or `frozen=True`) |
+| Methods | Yes | Yes |
+| Defaults | Yes | Yes |
+| Inheritance | Limited | Full |
+| Performance | Slightly faster | Flexible |
+
+### `defaultdict`
+```python
+from collections import defaultdict
+
+# Auto-create missing keys
+grouped = defaultdict(list)
+for item in items:
+    grouped[key(item)].append(item)
+
+# Counter pattern
+counts = defaultdict(int)
+for item in items:
+    counts[item] += 1
+
+# Custom factory
+def default_factory():
+    return {"count": 0, "items": []}
+```
+
+### `Counter`
+```python
+from collections import Counter
+
+c = Counter(["a", "b", "a", "c", "b", "a"])
+# Counter({'a': 3, 'b': 2, 'c': 1})
+
+c.most_common(2)        # [('a', 3), ('b', 2)]
+c.total()               # 6 (Python 3.10+)
+c.elements()            # Iterator over elements
+c.update(["a", "d"])    # Add counts
+c.subtract({"a": 1})    # Subtract counts
+
+# Arithmetic
+c1 + c2   # Add (keep positive)
+c1 - c2   # Subtract (keep positive)
+c1 & c2   # Intersection (min)
+c1 | c2   # Union (max)
+```
+
+### `deque` (Double-Ended Queue)
+```python
+from collections import deque
+
+d = deque([1, 2, 3], maxlen=5)  # Bounded (auto-discards old)
+
+d.append(4)       # Right
+d.appendleft(0)   # Left
+d.pop()           # Right
+d.popleft()       # Left
+d.extend([5, 6])
+d.extendleft([-1, -2])
+d.rotate(1)       # Rotate right
+d.rotate(-1)      # Rotate left
+
+# Use cases: queue, stack, sliding window, BFS
+```
+
+### `OrderedDict`
+```python
+from collections import OrderedDict
+
+# Python 3.7+: regular dict preserves insertion order
+# OrderedDict still useful for:
+# - move_to_end(key, last=True)
+# - popitem(last=True) — LIFO
+# - Equality considers order
+```
+
+### `ChainMap`
+```python
+from collections import ChainMap
+
+# Layered mappings (config layers)
+defaults = {"color": "blue", "size": 10}
+user = {"size": 12}
+env = {"color": "red"}
+
+config = ChainMap(env, user, defaults)
+config["color"]  # "red" (first match)
+config["size"]   # 12
+
+# Mutable — writes go to first mapping
+config["new"] = "value"  # Added to env
+```
+
+### `UserDict`, `UserList`, `UserString`
+```python
+from collections import UserDict
+
+class ValidatedDict(UserDict):
+    def __setitem__(self, key, value):
+        if not isinstance(key, str):
+            raise TypeError("Keys must be strings")
+        super().__setitem__(key, value)
+```
+
+- Subclassable wrappers for built-in types
+- Easier than inheriting from `dict`/`list`/`str` directly
+
+---
+
+## Decision Rules
+
+| Need | Type |
+|------|------|
+| Immutable record with names | `NamedTuple` |
+| Mutable record with methods | `@dataclass` |
+| Auto-create missing keys | `defaultdict` |
+| Count hashable items | `Counter` |
+| Queue (FIFO) | `deque` |
+| Stack (LIFO) | `list` or `deque` |
+| Bounded buffer | `deque(maxlen=N)` |
+| Sliding window | `deque(maxlen=N)` |
+| Layered config | `ChainMap` |
+| Custom dict behavior | `UserDict` |
+| LRU cache | `functools.lru_cache` (not collections) |
+
+---
+
+## Preferred Patterns
+
+```python
+# Frequency analysis
+def top_words(text: str, n: int = 10) -> list[tuple[str, int]]:
+    words = re.findall(r"\w+", text.lower())
+    return Counter(words).most_common(n)
+
+# Sliding window average
+def moving_avg(values: list[float], window: int) -> list[float]:
+    from collections import deque
+    d = deque(maxlen=window)
+    result = []
+    for v in values:
+        d.append(v)
+        if len(d) == window:
+            result.append(sum(d) / window)
+    return result
+
+# Config with layers
+def load_config() -> ChainMap:
+    return ChainMap(
+        os.environ,           # Highest priority
+        read_yaml("config.yaml"),
+        DEFAULTS,             # Lowest priority
+    )
+```
+
+---
+
+## Avoid
+
+- `OrderedDict` when regular `dict` works (Python 3.7+)
+- `namedtuple` (legacy) — use `NamedTuple` from `typing`
+- `ChainMap` for deep nesting (only looks at first level)
+- `deque` for random access (O(n)) — use `list`
+- `Counter` for non-hashable items
+
+---
+
+## Validation Considerations
+
+- Type checkers understand `NamedTuple`, `TypedDict` better than `namedtuple`
+- `Counter` arithmetic returns new `Counter`
+- `defaultdict` converts to regular `dict` via `dict(dd)`
+
+---
+
+## Related Skills
+
+- `core/data_structures.md`
+- `core/oop.md` (dataclasses)
+- `stdlib/itertools.md`
+- `generation/type_hints.md`