firegex 4.0.6__tar.gz → 5.0.0__tar.gz

firegex-5.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+recursive-include llhttp *.c *.h
+include llhttp/PROVENANCE.md
+include llhttp/LICENSE.pyllhttp
+include llhttp/lib/LICENSE.llhttp

firegex-5.0.0/PKG-INFO

@@ -0,0 +1,69 @@
+Metadata-Version: 2.4
+Name: firegex
+Version: 5.0.0
+Summary: Firegex client
+Author-email: Pwnzer0tt1 <pwnzer0tt1@poliba.it>
+Project-URL: Homepage, https://github.com/pwnzer0tt1/firegex
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: typer
+Requires-Dist: rich
+Requires-Dist: pydantic>=2
+Requires-Dist: typing-extensions>=4.7.1
+Requires-Dist: pycryptodome
+Requires-Dist: watchfiles
+Requires-Dist: websockets
+Requires-Dist: brotli>=1.2.0
+Requires-Dist: zstandard; python_version < "3.14"
+
+# Firegex Python Library and CLI
+
+This is the Python library for [Firegex](https://github.com/Pwnzer0tt1/firegex), a firewall built for CTF Attack-Defense competitions. It provides `firegex.pyfilters` — the package you write [Python filters](../docs/pyfilter.md) against — `firegex.regex` for pattern rulesets, the `fgex` CLI, and local simulators for both, so a filter can be tried before it ever reaches a running Firegex.
+
+## Installation
+
+```bash
+pip install -U firegex
+```
+
+`fgex` is an alias package for `firegex`: installing either one gives you the same `firegex` module and the `fgex`/`firegex` CLI commands.
+
+## Documentation
+
+It carries `firegex.regex`, which binds the same hyperscan firegex matches with, so
+a **pattern ruleset** can be checked and tried without a running instance:
+
+```bash
+fgex regex check rules.json                    # does it compile? what is in it?
+fgex regex test  rules.json --sample req.txt   # what would it do to this?
+fgex regex proxy rules.json 10.60.3.1 8080     # a local proxy applying it
+```
+
+The ruleset file is the same shape firegex uses. Matching needs `libhs` (vectorscan)
+installed; without it the tools say so and stop rather than answering with a different
+engine, because a tester that disagrees with production is worse than none.
+
+**➡️ [Writing and testing `@pyfilter` scripts](../docs/pyfilter.md)** — the `pyfilter` decorator, every data structure (`RawPacket`, `TCPInputStream`/`TCPOutputStream`, `HttpRequest`/`HttpResponse` and their variants, `HttpHistory`), packet statements, stream limits, the `fgex pyfilters` CLI simulator, and a full worked example.
+
+That page is also what's shown in the Firegex web UI itself (the docs button on the service pages) — both are generated from that single file, so they can never drift out of sync.
+
+## The rest of Firegex
+
+This package is the part you write filters *with*. What runs them is a Firegex instance,
+started from a clone with `python3 run.py` — there is deliberately no deploy command here,
+because any second description of that script's arguments would drift from it.
+
+Two guides cover everything an instance does:
+
+- **[Services](../docs/services.md)** — a service is a **network layer** (NFQUEUE, the
+  terminating proxy, or a hand-off to a proxy you wrote) plus an ordered chain of
+  **filters** (hyperscan patterns, or the Python you write with this library). TLS, QUIC
+  and every version of HTTP are protocols a service can speak rather than separate things
+  to set up.
+- **[Firewall Rules](../docs/firewall.md)** — plain nftables allow/drop/reject rules.
+
+The modules this library used to list — `nfregex`, `nfproxy`, `porthijack` and `tls` —
+were folded into services as choices rather than modules in 5.0.0, which is also when
+`firegex.nfproxy` became `firegex.pyfilters`.

firegex-5.0.0/README.md

@@ -0,0 +1,49 @@
+# Firegex Python Library and CLI
+
+This is the Python library for [Firegex](https://github.com/Pwnzer0tt1/firegex), a firewall built for CTF Attack-Defense competitions. It provides `firegex.pyfilters` — the package you write [Python filters](../docs/pyfilter.md) against — `firegex.regex` for pattern rulesets, the `fgex` CLI, and local simulators for both, so a filter can be tried before it ever reaches a running Firegex.
+
+## Installation
+
+```bash
+pip install -U firegex
+```
+
+`fgex` is an alias package for `firegex`: installing either one gives you the same `firegex` module and the `fgex`/`firegex` CLI commands.
+
+## Documentation
+
+It carries `firegex.regex`, which binds the same hyperscan firegex matches with, so
+a **pattern ruleset** can be checked and tried without a running instance:
+
+```bash
+fgex regex check rules.json                    # does it compile? what is in it?
+fgex regex test  rules.json --sample req.txt   # what would it do to this?
+fgex regex proxy rules.json 10.60.3.1 8080     # a local proxy applying it
+```
+
+The ruleset file is the same shape firegex uses. Matching needs `libhs` (vectorscan)
+installed; without it the tools say so and stop rather than answering with a different
+engine, because a tester that disagrees with production is worse than none.
+
+**➡️ [Writing and testing `@pyfilter` scripts](../docs/pyfilter.md)** — the `pyfilter` decorator, every data structure (`RawPacket`, `TCPInputStream`/`TCPOutputStream`, `HttpRequest`/`HttpResponse` and their variants, `HttpHistory`), packet statements, stream limits, the `fgex pyfilters` CLI simulator, and a full worked example.
+
+That page is also what's shown in the Firegex web UI itself (the docs button on the service pages) — both are generated from that single file, so they can never drift out of sync.
+
+## The rest of Firegex
+
+This package is the part you write filters *with*. What runs them is a Firegex instance,
+started from a clone with `python3 run.py` — there is deliberately no deploy command here,
+because any second description of that script's arguments would drift from it.
+
+Two guides cover everything an instance does:
+
+- **[Services](../docs/services.md)** — a service is a **network layer** (NFQUEUE, the
+  terminating proxy, or a hand-off to a proxy you wrote) plus an ordered chain of
+  **filters** (hyperscan patterns, or the Python you write with this library). TLS, QUIC
+  and every version of HTTP are protocols a service can speak rather than separate things
+  to set up.
+- **[Firewall Rules](../docs/firewall.md)** — plain nftables allow/drop/reject rules.
+
+The modules this library used to list — `nfregex`, `nfproxy`, `porthijack` and `tls` —
+were folded into services as choices rather than modules in 5.0.0, which is also when
+`firegex.nfproxy` became `firegex.pyfilters`.

{firegex-4.0.6 → firegex-5.0.0}/firegex/__init__.py

@@ -1,6 +1,6 @@
 
 def _get_version():
-    v = "4.0.6" if "{" not in "4.0.6" else "0.0.0"
+    v = "5.0.0" if "{" not in "5.0.0" else "0.0.0"
     if v == "0.0.0":
         import os
         env_v = os.getenv("FIREGEX_VERSION")

firegex-5.0.0/firegex/cli.py

@@ -0,0 +1,219 @@
+
+#!/usr/bin/env python3
+
+import json
+import os
+import socket
+import sys
+
+import typer
+from rich import print
+from rich.markup import escape
+from typer import Exit
+
+from firegex import __version__
+from firegex.pyfilters.proxysim import run_proxy_simulation
+
+app = typer.Typer(
+    no_args_is_help=True,
+    context_settings={"help_option_names": ["-h", "--help"]}
+)
+
+def close_cli(code:int=1):
+    raise Exit(code)
+
+DEV_MODE = __version__ == "0.0.0"
+
+def test_connection(host, port, use_ipv6=False):
+    family = socket.AF_INET6 if use_ipv6 else socket.AF_INET
+    sock = socket.socket(family, socket.SOCK_STREAM)
+    
+    try:
+        sock.settimeout(3)
+        sock.connect((host, port))
+        return True
+    except Exception:
+        return False
+    finally:
+        sock.close()
+
+@app.command("pyfilters", help="Run your Python filters against a real service, locally")
+def pyfilters(
+    filter_file: str = typer.Argument(..., help="The path to the filter file"),
+    address: str = typer.Argument(..., help="The address of the target to proxy"),
+    port: int = typer.Argument(..., help="The port of the target to proxy"),
+    
+    from_address: str = typer.Option(None, help="The address of the local server"),
+    from_port: int = typer.Option(7474, help="The port of the local server"),    
+    ipv6: bool = typer.Option(False, "-6", help="Use IPv6 for the connection"),
+):
+    if from_address is None:
+        from_address = "::1" if ipv6 else "127.0.0.1"
+    if not os.path.isfile(filter_file):
+        print(f"[bold red]'{escape(os.path.abspath(filter_file))}' not found[/]")
+        close_cli()
+    if not test_connection(address, port, ipv6):
+        print(f"[bold red]Can't connect to {escape(address)}:{port}[/]")
+        close_cli()
+    # No protocol to pass: the filter file says which one it speaks by what it asks for.
+    run_proxy_simulation(filter_file, address, port, from_address, from_port, ipv6)
+
+regex_app = typer.Typer(
+    no_args_is_help=True,
+    help="Try a regex ruleset with the engine firegex matches with",
+)
+app.add_typer(regex_app, name="regex")
+
+
+def _load_ruleset(path: str):
+    """Read a ruleset, with the errors an operator can act on.
+
+    The file is the same shape firegex itself uses, so a ruleset can be written here and
+    pasted there, or exported from there and replayed here.
+    """
+    from firegex.regex import HyperscanMissing, Rule, available, validate
+
+    if not available():
+        print(f"[bold red]{escape(str(HyperscanMissing()))}[/]")
+        close_cli()
+    if not os.path.isfile(path):
+        print(f"[bold red]'{escape(os.path.abspath(path))}' not found[/]")
+        close_cli()
+    try:
+        with open(path) as f:
+            raw = json.load(f)
+    except json.JSONDecodeError as e:
+        print(f"[bold red]{escape(path)} is not valid JSON: {escape(str(e))}[/]")
+        close_cli()
+    if not isinstance(raw, list):
+        print("[bold red]A ruleset is a list of rules[/]")
+        close_cli()
+
+    rules = []
+    broken = False
+    for index, item in enumerate(raw):
+        try:
+            rule = Rule.from_dict(item)
+        except (KeyError, ValueError) as e:
+            print(f"[bold red]rule {index} is malformed: {escape(str(e))}[/]")
+            broken = True
+            continue
+        # Judged against the mode it would actually run in, so what is accepted here is
+        # what firegex will accept.
+        why = validate(rule)
+        if why:
+            print(f"[bold red]{escape(rule.id)}: {escape(why)}[/]")
+            broken = True
+            continue
+        rules.append(rule)
+    if broken:
+        close_cli()
+    if not rules:
+        print("[bold red]The ruleset is empty[/]")
+        close_cli()
+    return rules
+
+
+@regex_app.command("check", help="Check that a ruleset compiles, and say what is in it")
+def regex_check(
+    rules_file: str = typer.Argument(..., help="The path to the ruleset (JSON)"),
+):
+    rules = _load_ruleset(rules_file)
+    print(f"[bold green]{len(rules)} rule(s), all valid[/]")
+    for rule in rules:
+        case = "" if rule.case_sensitive else ", any case"
+        print(f"  [bold]{escape(rule.id)}[/]  /{escape(rule.pattern)}/  "
+              f"[dim]{rule.direction.value}{case}, block[/]")
+
+
+@regex_app.command("test", help="Run a ruleset over a sample and show what it would do")
+def regex_test(
+    rules_file: str = typer.Argument(..., help="The path to the ruleset (JSON)"),
+    sample_file: str = typer.Option(None, "--sample", "-s",
+                                    help="File to match against; stdin when omitted"),
+    from_service: bool = typer.Option(False, "--from-service",
+                                      help="Treat the sample as traffic from the service"),
+):
+    from firegex.regex import Ruleset
+
+    rules = _load_ruleset(rules_file)
+    if sample_file:
+        if not os.path.isfile(sample_file):
+            print(f"[bold red]'{escape(os.path.abspath(sample_file))}' not found[/]")
+            close_cli()
+        with open(sample_file, "rb") as f:
+            sample = f.read()
+    else:
+        sample = sys.stdin.buffer.read()
+    if not sample:
+        print("[bold red]Nothing to match against[/]")
+        close_cli()
+
+    ruleset = Ruleset(rules)
+    is_input = not from_service
+    blocked_by = ruleset.apply(sample, is_input)
+
+    # Said before the verdict, because it changes what the verdict means: "passed
+    # through" from a run that could not try every rule is not the same answer.
+    for rule_id, why in ruleset.unscannable.items():
+        print(f"[bold yellow]not tried here[/] [bold]{escape(rule_id)}[/]: {escape(why)}. "
+              f"It is valid and will run on a real service — only matching it against a "
+              f"sample needs a mode hyperscan does not offer for it.")
+
+    if blocked_by is not None:
+        print(f"[bold red]blocked[/] by [bold]{escape(blocked_by)}[/]")
+        close_cli(0)
+    print("[bold green]passed through[/]"
+          + (" by the rules that could be tried" if ruleset.unscannable else ""))
+
+
+@regex_app.command("proxy", help="Run a local proxy applying a ruleset, like `fgex pyfilters`")
+def regex_proxy(
+    rules_file: str = typer.Argument(..., help="The path to the ruleset (JSON)"),
+    address: str = typer.Argument(..., help="The address of the target to proxy"),
+    port: int = typer.Argument(..., help="The port of the target to proxy"),
+    from_address: str = typer.Option(None, help="The address of the local server"),
+    from_port: int = typer.Option(7474, help="The port of the local server"),
+    ipv6: bool = typer.Option(False, "-6", help="Use IPv6 for the connection"),
+):
+    from firegex.regex import Ruleset
+    from firegex.regex.sim import run_regex_simulation
+
+    rules = _load_ruleset(rules_file)
+    if from_address is None:
+        from_address = "::1" if ipv6 else "127.0.0.1"
+    if not test_connection(address, port, ipv6):
+        print(f"[bold red]Can't connect to {escape(address)}:{port}[/]")
+        close_cli()
+    print("[dim]Blocking matches per chunk here, but across the whole stream on a real "
+          "service: a pattern split over two reads is caught there and not here.[/]")
+    ruleset = Ruleset(rules)
+    # Built before the first connection rather than on it, so the warning is on screen
+    # while the operator is still reading the banner instead of appearing mid-traffic.
+    for is_input in (True, False):
+        ruleset._db(is_input)
+    for rule_id, why in ruleset.unscannable.items():
+        print(f"[bold yellow]not applied here[/] [bold]{escape(rule_id)}[/]: {escape(why)}. "
+              f"It is valid and will run on a real service.")
+    run_regex_simulation(ruleset, address, port, from_address, from_port, ipv6)
+
+
+def version_callback(verison: bool):
+    if verison:
+        print(__version__, "Development Mode" if DEV_MODE else "Release")
+        raise typer.Exit()
+
+@app.callback()
+def main(
+    verison: bool = typer.Option(False, "--version", "-v", help="Show the version of the client", callback=version_callback),
+):
+    pass
+
+def run():
+    try:
+        app()
+    except KeyboardInterrupt:
+        print("[bold yellow]Operation cancelled[/]") 
+ 
+if __name__ == "__main__":
+    run()

firegex-5.0.0/firegex/pyfilters/__init__.py

@@ -0,0 +1,58 @@
+from firegex.pyfilters.models import (
+    RawPacket,
+    TCPInputStream,
+    TCPOutputStream,
+    TCPClientStream,
+    TCPServerStream,
+    HttpHistory,
+    HttpStreamHistory,
+)
+from firegex.pyfilters.internals.models import Action, FullStreamAction, ExceptionAction
+
+ACCEPT = Action.ACCEPT
+DROP = Action.DROP
+REJECT = Action.REJECT
+
+#: Set on a function by `@pyfilter`. Marking the function *is* the registration — there
+#: is no registry to keep, to clear between files, or to race two compilations against.
+PYFILTER_MARK = "__firegex_pyfilter__"
+
+
+def pyfilter(func):
+    """Mark a function as a filter, and hand it back unchanged.
+
+    The mark lives on the function, so the file that defines it holds the whole list:
+    `collect_pyfilters` reads a module's globals in definition order. What that replaced
+    was a set of *names* on a module-global attribute of this decorator, which had three
+    consequences worth not having back — a set has no order, so the order filters ran in
+    was whatever hashing produced; the state outlived the file, so every caller had to
+    clear it before and after; and a name registered from somewhere the module globals
+    could not reach (a nested `def`) was registered and then not found.
+
+    The function is returned as it is, not wrapped. The wrapper it used to get forwarded
+    `*args` and did nothing else, at the cost of a stack frame in every traceback the
+    operator has to read.
+    """
+    setattr(func, PYFILTER_MARK, True)
+    return func
+
+
+def collect_pyfilters(namespace: dict) -> list[str]:
+    """The filters a module defines, in the order it defines them.
+
+    Definition order is the order they run in, which is the only order a reader of the
+    file can predict.
+    """
+    seen, names = set(), []
+    for name, obj in namespace.items():
+        if callable(obj) and getattr(obj, PYFILTER_MARK, False) and id(obj) not in seen:
+            seen.add(id(obj))
+            names.append(name)
+    return names
+
+__all__ = [
+    "ACCEPT", "DROP", "REJECT",
+    "Action", "FullStreamAction", "ExceptionAction", "pyfilter", "collect_pyfilters",
+    "RawPacket", "TCPInputStream", "TCPOutputStream", "TCPClientStream", "TCPServerStream",
+    "HttpHistory", "HttpStreamHistory"
+]