mail-parser 4.2.1__tar.gz → 4.4.0__tar.gz

{mail_parser-4.2.1 → mail_parser-4.4.0}/.github/workflows/main.yml

@@ -29,7 +29,7 @@ jobs:
           curl -LsSf https://astral.sh/uv/install.sh | sh
           sudo apt-get -qq update
           sudo apt-get install -y libemail-outlook-message-perl
-          uv sync
+          uv sync --all-extras
           export PERL_MM_USE_DEFAULT=1
           sudo cpan -f -i Email::Outlook::Message
 
@@ -41,6 +41,7 @@ jobs:
           uv run mail-parser -v
           uv run mail-parser -h
           uv run mail-parser -f tests/mails/mail_malformed_3 -j
+          uv run mail-parser -f tests/mails/mail_outlook_1 -o -j
           cat tests/mails/mail_malformed_3 | uv run mail-parser -k -j
 
       - name: Run pre-commit

mail_parser-4.4.0/CLAUDE.md

@@ -0,0 +1,97 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Engineering Principles
+
+- **DRY** — extract repeated logic; no copy-paste code.
+- **KISS** — simplest solution that works; no clever tricks.
+- **YAGNI** — no code for hypothetical future needs.
+- **SRP** — one function/class, one responsibility.
+- **Fail fast** — validate at boundaries; trust internal code.
+- No dead code, commented-out blocks, or unused imports.
+- Every third-party import must have an explicit entry in `pyproject.toml`. Use extras when the
+  runtime feature requires them (e.g. `elasticsearch[async]`).
+
+## Python Style
+
+- Follow PEP 8. Use 4 spaces, never tabs.
+- Lines: 79 chars for code, 72 for comments/docstrings.
+- `snake_case` for functions/variables, `CapWords` for classes, `UPPER_CASE` for constants.
+- Imports order: stdlib → third-party → local.
+- Add docstrings to all public functions, methods, classes, and modules (params, return values,
+  exceptions).
+
+## Commands
+
+```bash
+uv sync                  # install all dependencies
+uv run pytest            # run all tests
+uv run pytest tests/test_mail_parser.py::TestMailParser::test_name  # single test
+uv run ruff check .      # lint
+uv run ruff format .     # format
+make check               # lint + tests
+make build               # clean + build wheel/sdist
+```
+
+## Architecture
+
+**Package layout**: `src/mailparser/` (src layout, no runtime deps — stdlib only).
+
+### Module responsibilities
+
+| Module          | Role                                                                  |
+| --------------- | --------------------------------------------------------------------- |
+| `core.py`       | `MailParser` class + `parse_from_*` factory functions                 |
+| `utils.py`      | Stateless parsing helpers (address, received headers, date, encoding) |
+| `const.py`      | Compiled regexes, header sets, clause splitter                        |
+| `exceptions.py` | Custom exception hierarchy                                            |
+| `__main__.py`   | CLI entry point (`mail-parser` command)                               |
+| `__init__.py`   | Re-exports public API                                                 |
+
+### Key design patterns
+
+**Dynamic attribute access via `__getattr__`**: `MailParser` exposes every header dynamically.
+Three access modes are supported for any attribute `X`:
+
+- `parser.X` → Python object (str or list)
+- `parser.X_json` → JSON string
+- `parser.X_raw` → raw JSON via `message.get_all()`
+
+Address headers listed in `const.ADDRESSES_HEADERS` (`from`, `to`, `cc`, `bcc`, `reply-to`,
+`delivered-to`) return `list[tuple[str, str]]` (display_name, email) instead of plain strings.
+
+**RFC non-compliance fallback in `get_addresses()`**: Python's
+`email.utils.getaddresses(strict=True)` (hardened against CVE-2023-27043) rejects headers where
+the display name contains `@`. Since this is a forensics tool, `utils.py` applies
+`_ADDR_FALLBACK_RE` whenever strict parsing returns empty results — always surfacing what is
+actually in the header.
+
+**Received header parsing**: `utils.receiveds_parsing()` tokenizes on RFC 5321 clause keywords
+(`from`, `by`, `via`, `with`, `id`, `for`, `envelope-from`) using `const._CLAUSE_SPLITTER`.
+Output list is ordered first-hop first. Unparseable headers fall back to `{"raw": ...}`.
+
+**Defect detection**: During `parse()`, every MIME part is walked and `_append_defects()` records
+RFC violations. `EPILOGUE_DEFECTS` triggers special epilogue extraction to recover hidden payloads
+in malformed boundaries.
+
+**Outlook support**: `parse_from_file_msg()` shells out to the system `msgconvert` Perl tool
+(`libemail-outlook-message-perl`) to convert `.msg` → `.eml`, then parses normally.
+
+**Partial vs full mail**: `_make_mail(complete=True)` includes all headers found in the message;
+`complete=False` restricts to `const.ADDRESSES_HEADERS | const.OTHERS_PARTS` (the "main" headers).
+Accessible as `parser.mail` / `parser.mail_partial`.
+
+### Adding new headers
+
+To make a header always appear in partial output, add its lowercase name to `OTHERS_PARTS` in
+`const.py`. Address-type headers (returning parsed name/email tuples) go in `ADDRESSES_HEADERS`.
+
+## Workflow
+
+After every change:
+
+1. Add/update unittests to cover the change.
+1. Update README.md if the change affects usage, API, or setup.
+1. Stage changes and run pre-commit; fix all reported issues before proceeding.
+1. Run full test suite; fix all failures before reporting done.

mail_parser-4.2.1/README.md → mail_parser-4.4.0/PKG-INFO

@@ -1,3 +1,30 @@
+Metadata-Version: 2.4
+Name: mail-parser
+Version: 4.4.0
+Summary: A tool that parses emails by enhancing the Python standard library, extracting all details into a comprehensive object.
+Author-email: Fedele Mantuano <mantuano.fedele@gmail.com>
+Maintainer-email: Fedele Mantuano <mantuano.fedele@gmail.com>
+License-Expression: Apache-2.0
+License-File: LICENSE.txt
+License-File: NOTICE.txt
+Keywords: email,forensics,mail,malware,parser,phishing,security,spam,threat detection
+Classifier: Natural Language :: English
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: Unix
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: <3.15,>=3.9
+Provides-Extra: outlook
+Requires-Dist: extract-msg>=0.54; extra == 'outlook'
+Description-Content-Type: text/markdown
+
 [![PyPI - Version](https://img.shields.io/pypi/v/mail-parser)](https://pypi.org/project/mail-parser/)
 [![Coverage Status](https://coveralls.io/repos/github/SpamScope/mail-parser/badge.svg?branch=develop)](https://coveralls.io/github/SpamScope/mail-parser?branch=develop)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/mail-parser?color=blue)](https://pypistats.org/packages/mail-parser)
@@ -36,20 +63,44 @@ formats, making it versatile for diverse email ecosystems.
 **⚡ Production-Ready**: Trusted by security professionals and developers worldwide, with extensive
 test coverage and proven reliability in high-stakes environments.
 
-Additionally, mail-parser provides full support for parsing Outlook email formats (.msg). To enable
-this functionality on Debian-based systems, simply install the required system package:
+mail-parser is fully compatible with Python 3, ensuring modern performance and reliability.
 
-```bash
-apt-get install libemail-outlook-message-perl
-```
+## Parsing Outlook `.msg` files
 
-For further details about the package, you can run:
+mail-parser converts Outlook `.msg` files to standard `.eml` before parsing.
+Two conversion backends are supported:
 
-```bash
-apt-cache show libemail-outlook-message-perl
-```
+1. **`extract-msg` (recommended, pure Python).** No external tools required.
+   Install the optional extra:
 
-mail-parser is fully compatible with Python 3, ensuring modern performance and reliability.
+   ```bash
+   pip install mail-parser[outlook]
+   ```
+
+1. **`msgconvert` (deprecated, external Perl tool).** Requires the
+   `libemail-outlook-message-perl` system package:
+
+   ```bash
+   apt-get install libemail-outlook-message-perl   # Debian-based systems
+   apt-cache show libemail-outlook-message-perl     # package details
+   ```
+
+**Backend precedence:** when `extract-msg` is installed it is used first.
+Only when it is *not* available does mail-parser fall back to the `msgconvert`
+external tool, logging a deprecation warning. If neither backend is available,
+`parse_from_file_msg()` raises `MailParserOSError` telling you to install
+either path.
+
+> **⚠️ Deprecated:** the `msgconvert` external-tool backend is deprecated and
+> will be removed in a future release. Migrate to the pure-Python backend with
+> `pip install mail-parser[outlook]`.
+
+**💥 BREAKING CHANGE:** the default `.msg` conversion backend changed.
+When `extract-msg` is installed it is now preferred over `msgconvert`. The two
+converters produce different intermediate `.eml` output, so some parsed fields
+(header ordering, encoding edge cases, attachment naming) can differ from the
+previous `msgconvert`-only behavior. Downstream code asserting on exact
+`.msg`-derived output may need updating.
 
 # Apache 2 Open Source License
 

mail_parser-4.2.1/PKG-INFO → mail_parser-4.4.0/README.md

@@ -1,28 +1,3 @@
-Metadata-Version: 2.4
-Name: mail-parser
-Version: 4.2.1
-Summary: A tool that parses emails by enhancing the Python standard library, extracting all details into a comprehensive object.
-Author-email: Fedele Mantuano <mantuano.fedele@gmail.com>
-Maintainer-email: Fedele Mantuano <mantuano.fedele@gmail.com>
-License-Expression: Apache-2.0
-License-File: LICENSE.txt
-License-File: NOTICE.txt
-Keywords: email,forensics,mail,malware,parser,phishing,security,spam,threat detection
-Classifier: Natural Language :: English
-Classifier: Operating System :: MacOS
-Classifier: Operating System :: Microsoft :: Windows
-Classifier: Operating System :: Unix
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
-Requires-Python: <3.15,>=3.9
-Description-Content-Type: text/markdown
-
 [![PyPI - Version](https://img.shields.io/pypi/v/mail-parser)](https://pypi.org/project/mail-parser/)
 [![Coverage Status](https://coveralls.io/repos/github/SpamScope/mail-parser/badge.svg?branch=develop)](https://coveralls.io/github/SpamScope/mail-parser?branch=develop)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/mail-parser?color=blue)](https://pypistats.org/packages/mail-parser)
@@ -61,20 +36,44 @@ formats, making it versatile for diverse email ecosystems.
 **⚡ Production-Ready**: Trusted by security professionals and developers worldwide, with extensive
 test coverage and proven reliability in high-stakes environments.
 
-Additionally, mail-parser provides full support for parsing Outlook email formats (.msg). To enable
-this functionality on Debian-based systems, simply install the required system package:
+mail-parser is fully compatible with Python 3, ensuring modern performance and reliability.
 
-```bash
-apt-get install libemail-outlook-message-perl
-```
+## Parsing Outlook `.msg` files
 
-For further details about the package, you can run:
+mail-parser converts Outlook `.msg` files to standard `.eml` before parsing.
+Two conversion backends are supported:
 
-```bash
-apt-cache show libemail-outlook-message-perl
-```
+1. **`extract-msg` (recommended, pure Python).** No external tools required.
+   Install the optional extra:
 
-mail-parser is fully compatible with Python 3, ensuring modern performance and reliability.
+   ```bash
+   pip install mail-parser[outlook]
+   ```
+
+1. **`msgconvert` (deprecated, external Perl tool).** Requires the
+   `libemail-outlook-message-perl` system package:
+
+   ```bash
+   apt-get install libemail-outlook-message-perl   # Debian-based systems
+   apt-cache show libemail-outlook-message-perl     # package details
+   ```
+
+**Backend precedence:** when `extract-msg` is installed it is used first.
+Only when it is *not* available does mail-parser fall back to the `msgconvert`
+external tool, logging a deprecation warning. If neither backend is available,
+`parse_from_file_msg()` raises `MailParserOSError` telling you to install
+either path.
+
+> **⚠️ Deprecated:** the `msgconvert` external-tool backend is deprecated and
+> will be removed in a future release. Migrate to the pure-Python backend with
+> `pip install mail-parser[outlook]`.
+
+**💥 BREAKING CHANGE:** the default `.msg` conversion backend changed.
+When `extract-msg` is installed it is now preferred over `msgconvert`. The two
+converters produce different intermediate `.eml` output, so some parsed fields
+(header ordering, encoding edge cases, attachment naming) can differ from the
+previous `msgconvert`-only behavior. Downstream code asserting on exact
+`.msg`-derived output may need updating.
 
 # Apache 2 Open Source License
 

{mail_parser-4.2.1 → mail_parser-4.4.0}/pyproject.toml

@@ -28,6 +28,9 @@ maintainers = [
 ]
 dependencies = []
 
+[project.optional-dependencies]
+outlook = ["extract-msg>=0.54"]
+
 [dependency-groups]
 dev = [
     "build>=1.2.2.post1",

{mail_parser-4.2.1 → mail_parser-4.4.0}/src/mailparser/core.py

@@ -18,6 +18,7 @@ limitations under the License.
 
 import base64
 import email
+import importlib.util
 import ipaddress
 import json
 import logging
@@ -27,6 +28,7 @@ from mailparser.const import ADDRESSES_HEADERS, EPILOGUE_DEFECTS, REGXIP, REGXIP
 from mailparser.utils import (
     convert_mail_date,
     decode_header_part,
+    extract_msg_convert,
     find_between,
     get_addresses,
     get_header,
@@ -186,14 +188,32 @@ class MailParser:
         Init a new object from a Outlook message file,
         mime type: application/vnd.ms-outlook
 
+        Conversion backend precedence:
+          1. ``extract-msg`` (pure Python, optional ``outlook`` extra).
+          2. ``msgconvert`` external Perl tool — **deprecated** fallback,
+             used only when ``extract-msg`` is not installed.
+
         Args:
             fp (string): file path of raw Outlook email
 
         Returns:
             Instance of MailParser
+
+        Raises:
+            MailParserOSError: if no conversion backend is available
         """
         log.debug("Parsing email from file Outlook")
-        f, _ = msgconvert(fp)
+
+        if importlib.util.find_spec("extract_msg") is not None:
+            f, _ = extract_msg_convert(fp)
+        else:
+            log.warning(
+                "msgconvert backend is deprecated and will be removed "
+                "in a future release. Install the pure-Python Outlook "
+                "support with 'pip install mail-parser[outlook]'."
+            )
+            f, _ = msgconvert(fp)
+
         return cls.from_file(f, True)
 
     @classmethod
@@ -429,22 +449,29 @@ class MailParser:
                 else:
                     log.debug(f"Email part {i!r} is not an attachment")
 
-                    # Get the payload using get_payload method with decode=True
-                    # As Python truly decodes only 'base64',
-                    # 'quoted-printable', 'x-uuencode',
-                    # 'uuencode', 'uue', 'x-uue'
-                    # And for other encodings it breaks the characters so
-                    # we need to decode them with encoding python is appying
-                    # To maintain the characters
                     payload = p.get_payload(decode=True)
                     cte = p.get("Content-Transfer-Encoding")
                     if cte:
                         cte = cte.lower()
 
                     if not cte or cte in ["7bit", "8bit"]:
-                        try:
-                            payload = payload.decode("raw-unicode-escape")
-                        except UnicodeDecodeError:
+                        # message_from_bytes stores non-ASCII body bytes via
+                        # ascii+surrogateescape, producing surrogates in the
+                        # payload string.  message_from_string stores a proper
+                        # Unicode str (no surrogates).  Detect which case we
+                        # have via get_payload(decode=False) and decode
+                        # accordingly so the declared charset is honoured.
+                        raw_str = p.get_payload(decode=False)
+                        if isinstance(raw_str, str):
+                            try:
+                                # Raises if surrogates present (from_bytes path)
+                                raw_str.encode("utf-8")
+                                payload = raw_str
+                            except UnicodeEncodeError:
+                                # Recover original bytes then decode with charset
+                                orig_bytes = raw_str.encode("ascii", "surrogateescape")
+                                payload = ported_string(orig_bytes, encoding=charset)
+                        else:
                             payload = ported_string(payload, encoding=charset)
                     else:
                         payload = ported_string(payload, encoding=charset)

{mail_parser-4.2.1 → mail_parser-4.4.0}/src/mailparser/utils.py

@@ -16,6 +16,8 @@ See the License for the specific language governing permissions and
 limitations under the License.
 """
 
+from __future__ import annotations
+
 import base64
 import datetime
 import email
@@ -75,7 +77,9 @@ _ADDR_FALLBACK_RE = re.compile(
 )
 
 
-def get_addresses(raw_header):
+def get_addresses(
+    raw_header: str | email.header.Header | None,
+) -> list[tuple[str, str]]:
     """
     Parse email addresses from a raw address header with a fallback for
     RFC-non-compliant but real-world-common formats.
@@ -100,13 +104,36 @@ def get_addresses(raw_header):
     that was actually present in the header.
 
     Args:
-        raw_header (str): raw value of an address header
-            (e.g. ``From``, ``To``, ``CC`` …)
+        raw_header (str | email.header.Header | None): raw value of an
+            address header (e.g. ``From``, ``To``, ``CC`` …). Accepts a
+            plain ``str``, an ``email.header.Header`` instance (returned
+            by ``email.message.Message.get`` for headers containing
+            RFC 2047 encoded-words such as non-ASCII display names), or
+            ``None``.
 
     Returns:
         list[tuple[str, str]]: list of ``(display_name, email_addr)`` tuples.
             ``display_name`` is an empty string when absent.
     """
+    # ``Message.get(name)`` returns an ``email.header.Header`` for any header
+    # whose value contains RFC 2047 encoded-words (typical for non-ASCII
+    # display names like ``=?utf-8?q?=C3=81rp=C3=A1d?=``). ``Header`` does
+    # not implement string methods such as ``.strip()`` and is not a valid
+    # input to ``email.utils.getaddresses``.
+    #
+    # Important: decode ``Header`` values into a plain parseable string first.
+    # In practice, strict address parsing can treat raw encoded-word tokens like
+    # ``=?unknown-8bit?...?=`` as the *address* itself, producing output such as
+    # ``To: =?unknown-8bit?...?=``.  Decoding first gives
+    # ``Álpám Longsom <recipient@example.com>`` so getaddresses() can split
+    # name/address correctly.
+    if raw_header is None:
+        return []
+    if isinstance(raw_header, email.header.Header):
+        raw_header = decode_header_part(raw_header.encode())
+    elif not isinstance(raw_header, str):
+        raw_header = str(raw_header)
+
     parsed = email.utils.getaddresses([raw_header], strict=True)
 
     # If every result from the strict parser has an empty address — while the
@@ -119,7 +146,7 @@ def get_addresses(raw_header):
                 results.append((m.group(1).strip(), m.group(2).strip()))
             elif m.group(4):  # Any Name <email>  (incl. email-as-display-name)
                 results.append((m.group(3).strip(), m.group(4).strip()))
-            elif m.group(5):  # bare email
+            elif m.group(5):  # bare email  # pragma: no branch
                 results.append(("", m.group(5).strip()))
         if results:
             log.debug(
@@ -291,6 +318,66 @@ def fingerprints(data):
     return hashes(md5, sha1, sha256, sha512)
 
 
+def _new_outlook_tempfile():
+    """
+    Create an empty temporary file to hold a converted Outlook email.
+
+    The OS-level file handle is closed immediately; callers write to the
+    returned path with their own handle (a subprocess ``--outfile`` for
+    ``msgconvert`` or a plain ``open`` for the pure-Python backend).
+
+    Returns:
+        str: path of the new temporary ``.eml`` file
+    """
+    handle, path = tempfile.mkstemp(prefix="outlook_")
+    os.close(handle)
+    return path
+
+
+def extract_msg_convert(fp):
+    """
+    Convert an Outlook ``.msg`` file to ``.eml`` using the pure-Python
+    ``extract-msg`` library (no external Perl tool required).
+
+    The ``extract_msg`` import is performed lazily inside this function so
+    that the package keeps importing with zero runtime dependencies when
+    the optional ``outlook`` extra is not installed.
+
+    Args:
+        fp (string): file path of the Outlook ``.msg`` mail
+
+    Returns:
+        tuple: ``(eml_path, info)`` where ``eml_path`` is the path of the
+        converted ``.eml`` file and ``info`` is a short descriptive string
+
+    Raises:
+        ImportError: if the ``extract-msg`` library is not installed
+        MailParserOSError: if the ``.msg`` is not a convertible email
+            message (e.g. a contact or calendar item)
+    """
+    import extract_msg  # lazy: keep package import stdlib-only
+
+    log.debug("Started converting Outlook email with extract-msg")
+    msg = extract_msg.openMsg(fp)
+    try:
+        # openMsg() may return a non-email MSGFile (contact, calendar,
+        # task...) which cannot be rendered as an email message.
+        as_email_message = getattr(msg, "asEmailMessage", None)
+        if as_email_message is None:
+            raise MailParserOSError(
+                f"Outlook file {fp!r} is not a convertible email "
+                f"message (type {type(msg).__name__})"
+            )
+        eml = as_email_message()
+        info = f"{eml.get('From', '')} | {eml.get('Subject', '')}".strip()
+        temp = _new_outlook_tempfile()
+        with open(temp, "wb") as f:
+            f.write(eml.as_bytes())
+        return temp, info
+    finally:
+        msg.close()
+
+
 def msgconvert(email):
     """
     Exec msgconvert tool, to convert msg Outlook
@@ -302,9 +389,12 @@ def msgconvert(email):
     Returns:
         tuple with file path of mail converted and
         standard output data (str)
+
+    Raises:
+        MailParserOSError: if the ``msgconvert`` tool is not installed
     """
     log.debug("Started converting Outlook email")
-    temph, temp = tempfile.mkstemp(prefix="outlook_")
+    temp = _new_outlook_tempfile()
     command = ["msgconvert", "--outfile", temp, email]
 
     try:
@@ -316,7 +406,13 @@ def msgconvert(email):
         )
 
     except OSError as e:
-        message = f"Check if 'msgconvert' tool is installed / {e!r}"
+        message = (
+            "Cannot convert Outlook .msg: no conversion backend "
+            "available. Install pure-Python support with "
+            "'pip install mail-parser[outlook]', or install the "
+            "'msgconvert' Perl tool "
+            f"(libemail-outlook-message-perl). {e!r}"
+        )
         log.exception(message)
         raise MailParserOSError(message)
 
@@ -324,9 +420,6 @@ def msgconvert(email):
         stdoutdata, _ = out.communicate()
         return temp, stdoutdata.decode("utf-8").strip()
 
-    finally:
-        os.close(temph)
-
 
 def parse_received(received):
     """
@@ -447,7 +540,7 @@ def receiveds_parsing(receiveds):
 
     log.debug("len(receiveds) %s, len(parsed) %s" % (len(receiveds), len(parsed)))
 
-    if len(receiveds) != len(parsed):
+    if len(receiveds) != len(parsed):  # pragma: no cover
         # something really bad happened,
         # so just return raw receiveds with hop indices
         log.error(

{mail_parser-4.2.1 → mail_parser-4.4.0}/src/mailparser/version.py

@@ -16,4 +16,4 @@ See the License for the specific language governing permissions and
 limitations under the License.
 """
 
-__version__ = "4.2.1"
+__version__ = "4.4.0"