mail-parser 4.2.1__tar.gz → 4.3.0__tar.gz

mail_parser-4.3.0/CLAUDE.md

@@ -0,0 +1,103 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Engineering Principles
+
+- **DRY** — extract repeated logic; no copy-paste code.
+- **KISS** — simplest solution that works; no clever tricks.
+- **YAGNI** — no code for hypothetical future needs.
+- **SRP** — one function/class, one responsibility.
+- **Fail fast** — validate at boundaries; trust internal code.
+- No dead code, commented-out blocks, or unused imports.
+- Every third-party import must have an explicit entry in `pyproject.toml`. Use extras when the
+  runtime feature requires them (e.g. `elasticsearch[async]`).
+
+## Python Style
+
+- Follow PEP 8. Use 4 spaces, never tabs.
+- Lines: 79 chars for code, 72 for comments/docstrings.
+- `snake_case` for functions/variables, `CapWords` for classes, `UPPER_CASE` for constants.
+- Imports order: stdlib → third-party → local.
+- Add docstrings to all public functions, methods, classes, and modules (params, return values,
+  exceptions).
+
+## Commands
+
+```bash
+uv sync                  # install all dependencies
+uv run pytest            # run all tests
+uv run pytest tests/test_mail_parser.py::TestMailParser::test_name  # single test
+uv run ruff check .      # lint
+uv run ruff format .     # format
+make check               # lint + tests
+make build               # clean + build wheel/sdist
+```
+
+## Architecture
+
+**Package layout**: `src/mailparser/` (src layout, no runtime deps — stdlib only).
+
+### Module responsibilities
+
+| Module          | Role                                                                  |
+| --------------- | --------------------------------------------------------------------- |
+| `core.py`       | `MailParser` class + `parse_from_*` factory functions                 |
+| `utils.py`      | Stateless parsing helpers (address, received headers, date, encoding) |
+| `const.py`      | Compiled regexes, header sets, clause splitter                        |
+| `exceptions.py` | Custom exception hierarchy                                            |
+| `__main__.py`   | CLI entry point (`mail-parser` command)                               |
+| `__init__.py`   | Re-exports public API                                                 |
+
+### Key design patterns
+
+**Dynamic attribute access via `__getattr__`**: `MailParser` exposes every header dynamically.
+Three access modes are supported for any attribute `X`:
+
+- `parser.X` → Python object (str or list)
+- `parser.X_json` → JSON string
+- `parser.X_raw` → raw JSON via `message.get_all()`
+
+Address headers listed in `const.ADDRESSES_HEADERS` (`from`, `to`, `cc`, `bcc`, `reply-to`,
+`delivered-to`) return `list[tuple[str, str]]` (display_name, email) instead of plain strings.
+
+**RFC non-compliance fallback in `get_addresses()`**: Python's
+`email.utils.getaddresses(strict=True)` (hardened against CVE-2023-27043) rejects headers where
+the display name contains `@`. Since this is a forensics tool, `utils.py` applies
+`_ADDR_FALLBACK_RE` whenever strict parsing returns empty results — always surfacing what is
+actually in the header.
+
+**Received header parsing**: `utils.receiveds_parsing()` tokenizes on RFC 5321 clause keywords
+(`from`, `by`, `via`, `with`, `id`, `for`, `envelope-from`) using `const._CLAUSE_SPLITTER`.
+Output list is ordered first-hop first. Unparseable headers fall back to `{"raw": ...}`.
+
+**Defect detection**: During `parse()`, every MIME part is walked and `_append_defects()` records
+RFC violations. `EPILOGUE_DEFECTS` triggers special epilogue extraction to recover hidden payloads
+in malformed boundaries.
+
+**Outlook support**: `parse_from_file_msg()` shells out to the system `msgconvert` Perl tool
+(`libemail-outlook-message-perl`) to convert `.msg` → `.eml`, then parses normally.
+
+**Partial vs full mail**: `_make_mail(complete=True)` includes all headers found in the message;
+`complete=False` restricts to `const.ADDRESSES_HEADERS | const.OTHERS_PARTS` (the "main" headers).
+Accessible as `parser.mail` / `parser.mail_partial`.
+
+### Adding new headers
+
+To make a header always appear in partial output, add its lowercase name to `OTHERS_PARTS` in
+`const.py`. Address-type headers (returning parsed name/email tuples) go in `ADDRESSES_HEADERS`.
+
+## Workflow
+
+After every change:
+
+1. Add/update unittests to cover the change.
+1. Update README.md if the change affects usage, API, or setup.
+1. Stage changes and run pre-commit; fix all reported issues before proceeding.
+1. Run full test suite; fix all failures before reporting done.
+
+### Test fixtures
+
+Raw email files in `tests/mails/` are the fixtures. `mail_malformed_*` files exercise defect
+detection; `mail_outlook_*` require `msgconvert` installed. Tests that need the Outlook tool should
+be marked `@pytest.mark.integration`.

{mail_parser-4.2.1 → mail_parser-4.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mail-parser
-Version: 4.2.1
+Version: 4.3.0
 Summary: A tool that parses emails by enhancing the Python standard library, extracting all details into a comprehensive object.
 Author-email: Fedele Mantuano <mantuano.fedele@gmail.com>
 Maintainer-email: Fedele Mantuano <mantuano.fedele@gmail.com>

{mail_parser-4.2.1 → mail_parser-4.3.0}/src/mailparser/core.py

@@ -429,22 +429,29 @@ class MailParser:
                 else:
                     log.debug(f"Email part {i!r} is not an attachment")
 
-                    # Get the payload using get_payload method with decode=True
-                    # As Python truly decodes only 'base64',
-                    # 'quoted-printable', 'x-uuencode',
-                    # 'uuencode', 'uue', 'x-uue'
-                    # And for other encodings it breaks the characters so
-                    # we need to decode them with encoding python is appying
-                    # To maintain the characters
                     payload = p.get_payload(decode=True)
                     cte = p.get("Content-Transfer-Encoding")
                     if cte:
                         cte = cte.lower()
 
                     if not cte or cte in ["7bit", "8bit"]:
-                        try:
-                            payload = payload.decode("raw-unicode-escape")
-                        except UnicodeDecodeError:
+                        # message_from_bytes stores non-ASCII body bytes via
+                        # ascii+surrogateescape, producing surrogates in the
+                        # payload string.  message_from_string stores a proper
+                        # Unicode str (no surrogates).  Detect which case we
+                        # have via get_payload(decode=False) and decode
+                        # accordingly so the declared charset is honoured.
+                        raw_str = p.get_payload(decode=False)
+                        if isinstance(raw_str, str):
+                            try:
+                                # Raises if surrogates present (from_bytes path)
+                                raw_str.encode("utf-8")
+                                payload = raw_str
+                            except UnicodeEncodeError:
+                                # Recover original bytes then decode with charset
+                                orig_bytes = raw_str.encode("ascii", "surrogateescape")
+                                payload = ported_string(orig_bytes, encoding=charset)
+                        else:
                             payload = ported_string(payload, encoding=charset)
                     else:
                         payload = ported_string(payload, encoding=charset)

{mail_parser-4.2.1 → mail_parser-4.3.0}/src/mailparser/utils.py

@@ -16,6 +16,8 @@ See the License for the specific language governing permissions and
 limitations under the License.
 """
 
+from __future__ import annotations
+
 import base64
 import datetime
 import email
@@ -75,7 +77,9 @@ _ADDR_FALLBACK_RE = re.compile(
 )
 
 
-def get_addresses(raw_header):
+def get_addresses(
+    raw_header: str | email.header.Header | None,
+) -> list[tuple[str, str]]:
     """
     Parse email addresses from a raw address header with a fallback for
     RFC-non-compliant but real-world-common formats.
@@ -100,13 +104,36 @@ def get_addresses(raw_header):
     that was actually present in the header.
 
     Args:
-        raw_header (str): raw value of an address header
-            (e.g. ``From``, ``To``, ``CC`` …)
+        raw_header (str | email.header.Header | None): raw value of an
+            address header (e.g. ``From``, ``To``, ``CC`` …). Accepts a
+            plain ``str``, an ``email.header.Header`` instance (returned
+            by ``email.message.Message.get`` for headers containing
+            RFC 2047 encoded-words such as non-ASCII display names), or
+            ``None``.
 
     Returns:
         list[tuple[str, str]]: list of ``(display_name, email_addr)`` tuples.
             ``display_name`` is an empty string when absent.
     """
+    # ``Message.get(name)`` returns an ``email.header.Header`` for any header
+    # whose value contains RFC 2047 encoded-words (typical for non-ASCII
+    # display names like ``=?utf-8?q?=C3=81rp=C3=A1d?=``). ``Header`` does
+    # not implement string methods such as ``.strip()`` and is not a valid
+    # input to ``email.utils.getaddresses``.
+    #
+    # Important: decode ``Header`` values into a plain parseable string first.
+    # In practice, strict address parsing can treat raw encoded-word tokens like
+    # ``=?unknown-8bit?...?=`` as the *address* itself, producing output such as
+    # ``To: =?unknown-8bit?...?=``.  Decoding first gives
+    # ``Álpám Longsom <recipient@example.com>`` so getaddresses() can split
+    # name/address correctly.
+    if raw_header is None:
+        return []
+    if isinstance(raw_header, email.header.Header):
+        raw_header = decode_header_part(raw_header.encode())
+    elif not isinstance(raw_header, str):
+        raw_header = str(raw_header)
+
     parsed = email.utils.getaddresses([raw_header], strict=True)
 
     # If every result from the strict parser has an empty address — while the
@@ -119,7 +146,7 @@ def get_addresses(raw_header):
                 results.append((m.group(1).strip(), m.group(2).strip()))
             elif m.group(4):  # Any Name <email>  (incl. email-as-display-name)
                 results.append((m.group(3).strip(), m.group(4).strip()))
-            elif m.group(5):  # bare email
+            elif m.group(5):  # bare email  # pragma: no branch
                 results.append(("", m.group(5).strip()))
         if results:
             log.debug(
@@ -447,7 +474,7 @@ def receiveds_parsing(receiveds):
 
     log.debug("len(receiveds) %s, len(parsed) %s" % (len(receiveds), len(parsed)))
 
-    if len(receiveds) != len(parsed):
+    if len(receiveds) != len(parsed):  # pragma: no cover
         # something really bad happened,
         # so just return raw receiveds with hop indices
         log.error(

{mail_parser-4.2.1 → mail_parser-4.3.0}/src/mailparser/version.py

@@ -16,4 +16,4 @@ See the License for the specific language governing permissions and
 limitations under the License.
 """
 
-__version__ = "4.2.1"
+__version__ = "4.3.0"

{mail_parser-4.2.1 → mail_parser-4.3.0}/tests/test_mail_parser.py

@@ -1087,6 +1087,62 @@ This is plain text with 8bit encoding."""
         # Should have parsed successfully and body contains the text
         self.assertIn("hello", mail.body)
 
+    def _make_utf8_raw(self, cte_header=""):
+        """Return a minimal raw email bytes with UTF-8 body (em-dash U+2014)."""
+        cte_line = f"Content-Transfer-Encoding: {cte_header}\n" if cte_header else ""
+        raw = (
+            "From: a@example.com\n"
+            "To: b@example.com\n"
+            "Subject: probe\n"
+            f"Content-Type: text/plain; charset=utf-8\n"
+            "MIME-Version: 1.0\n"
+            f"{cte_line}"
+            "\n"
+            "Hello — world\n"
+        )
+        return raw.encode("utf-8")
+
+    def test_utf8_body_from_bytes_no_cte(self):
+        """parse_from_bytes must not mojibake UTF-8 body with no CTE (issue #152)."""
+        raw = self._make_utf8_raw()
+        mail = mailparser.parse_from_bytes(raw)
+        self.assertEqual(mail.text_plain[0], "Hello — world\n")
+
+    def test_utf8_body_from_bytes_cte_8bit(self):
+        """parse_from_bytes must not mojibake UTF-8 body with CTE: 8bit (issue #152)."""
+        raw = self._make_utf8_raw("8bit")
+        mail = mailparser.parse_from_bytes(raw)
+        self.assertEqual(mail.text_plain[0], "Hello — world\n")
+
+    def test_utf8_body_from_bytes_cte_7bit(self):
+        """parse_from_bytes must not mojibake ASCII body with CTE: 7bit (issue #152)."""
+        raw = (
+            b"From: a@example.com\n"
+            b"To: b@example.com\n"
+            b"Content-Type: text/plain; charset=utf-8\n"
+            b"Content-Transfer-Encoding: 7bit\n"
+            b"\n"
+            b"Hello world\n"
+        )
+        mail = mailparser.parse_from_bytes(raw)
+        self.assertEqual(mail.text_plain[0], "Hello world\n")
+
+    def test_utf8_body_from_bytes_matches_from_string(self):
+        """parse_from_bytes and parse_from_string: identical text_plain (issue #152)."""
+        raw_bytes = self._make_utf8_raw()
+        raw_str = raw_bytes.decode("utf-8")
+        mail_b = mailparser.parse_from_bytes(raw_bytes)
+        mail_s = mailparser.parse_from_string(raw_str)
+        self.assertEqual(mail_b.text_plain[0], mail_s.text_plain[0])
+
+    def test_utf8_body_from_bytes_8bit_matches_from_string(self):
+        """parse_from_bytes (8bit CTE) matches parse_from_string (issue #152)."""
+        raw_bytes = self._make_utf8_raw("8bit")
+        raw_str = raw_bytes.decode("utf-8")
+        mail_b = mailparser.parse_from_bytes(raw_bytes)
+        mail_s = mailparser.parse_from_string(raw_str)
+        self.assertEqual(mail_b.text_plain[0], mail_s.text_plain[0])
+
 
 class TestEmailAsDisplayName(unittest.TestCase):
     """

{mail_parser-4.2.1 → mail_parser-4.3.0}/tests/test_utils.py

@@ -26,6 +26,7 @@ from mailparser.exceptions import MailParserOSError, MailParserReceivedParsingEr
 from mailparser.utils import (
     decode_header_part,
     find_between,
+    get_addresses,
     msgconvert,
     parse_received,
     ported_open,
@@ -240,23 +241,14 @@ class TestUtilsEdgeCases(unittest.TestCase):
         self.assertEqual(len(result.sha256), 64)
         self.assertEqual(len(result.sha512), 128)
 
-    def test_parse_received_with_multiple_matches_error(self):
-        """Test parse_received raises error on multiple matches for same pattern"""
-        # This tests the error branch when multiple matches are found
-        # We need a received header that triggers duplicate matches
+    def test_parse_received_with_multiple_from_clauses(self):
+        """parse_received keeps only first 'from' clause (RFC 5321 one-from rule)"""
         received = (
             "from server.example.com from server2.example.com by mail.example.com"
         )
-
-        # Depending on the patterns, this might raise an error or succeed
-        # We test that it handles the scenario correctly
-        try:
-            result = parse_received(received)
-            # If it succeeds, it should return a dict
-            self.assertIsInstance(result, dict)
-        except MailParserReceivedParsingError as e:
-            # This is the expected path for multiple matches
-            self.assertIn("More than one match", str(e))
+        result = parse_received(received)
+        self.assertIsInstance(result, dict)
+        self.assertEqual(result["from"], "server.example.com")
 
     def test_get_to_domains_with_keyerror(self):
         """Test get_to_domains handles KeyError gracefully"""
@@ -442,10 +434,8 @@ class TestUtilsEdgeCases(unittest.TestCase):
 
         result = receiveds_format(parsed)
         self.assertIsInstance(result, list)
-        # Should have date_utc and delay calculated
-        if result[0].get("date_utc"):
-            self.assertIn("date_utc", result[0])
-            self.assertIn("delay", result[1])
+        self.assertIn("date_utc", result[0])
+        self.assertIn("delay", result[1])
 
     def test_receiveds_format_with_invalid_date(self):
         """Test receiveds_format handles invalid dates"""
@@ -600,6 +590,105 @@ class TestUtilsEdgeCases(unittest.TestCase):
         self.assertIsInstance(result, str)
         self.assertEqual(result, raw_val)
 
+    def test_get_addresses_handles_header_object(self):
+        """
+        Test that get_addresses accepts an email.header.Header instance
+        without raising AttributeError on `.strip()`.
+
+        Regression for the case where Message.get(name) returns a Header
+        for address headers containing RFC 2047 encoded-words (e.g.
+        non-ASCII display names like ``=?utf-8?q?=C3=81lp=C3=A1m_Longsom?=``).
+        Before the fix, this raised:
+
+            AttributeError: 'Header' object has no attribute 'strip'
+        """
+        from email.header import Header
+
+        header_obj = Header("Álpám Longsom", charset="utf-8")
+        header_obj.append(" <recipient@example.com>", charset="us-ascii")
+        result = get_addresses(header_obj)
+        self.assertIsInstance(result, list)
+        self.assertEqual(len(result), 1)
+        display_name, addr = result[0]
+        self.assertEqual(addr, "recipient@example.com")
+        # get_addresses returns encoded-word form for the display name.
+        # Decoding to Unicode happens in core.py via decode_header_part.
+        self.assertEqual(display_name, "Álpám Longsom")
+
+    def test_get_addresses_handles_unknown_8bit_header_object(self):
+        """
+        Regression for real-world unknown-8bit encoded-word headers.
+        Address parsing must not treat the encoded-word token itself as
+        the address.
+        """
+        from email.header import Header
+
+        encoded_name = "=?unknown-8bit?b?w4FscMOhbSBMb25nc29t?="
+        header_obj = Header()
+        header_obj.append(encoded_name, charset="us-ascii")
+        header_obj.append(" <recipient@example.com>", charset="us-ascii")
+
+        result = get_addresses(header_obj)
+        self.assertEqual(result, [("Álpám Longsom", "recipient@example.com")])
+
+    def test_get_addresses_handles_none(self):
+        """
+        Test that get_addresses returns an empty list when given None,
+        rather than crashing on attribute access.
+        """
+        self.assertEqual(get_addresses(None), [])
+
+    def test_get_addresses_plain_string_unchanged(self):
+        """
+        Test that the existing plain-string path still works. This guards
+        against accidentally regressing the common case while adding
+        Header / None handling.
+        """
+        result = get_addresses("Plain Name <plain@example.com>")
+        self.assertEqual(result, [("Plain Name", "plain@example.com")])
+
+    def test_mailparser_from_bytes_preserves_unicode_display_name(self):
+        """
+        Regression: Header objects from Message.get(name) must round-trip
+        through get_addresses() without introducing replacement characters.
+
+        The parser should expose the decoded Unicode display name on
+        MailParser.to.
+        """
+        from mailparser.core import MailParser
+
+        raw_email = (
+            b"From: Sender <sender@example.com>\r\n"
+            b"To: =?utf-8?b?w4FscMOhbSBMb25nc29t?= <recipient@example.com>\r\n"
+            b"Subject: Test\r\n"
+            b"Date: Tue, 12 May 2026 18:00:00 +0000\r\n"
+            b"Content-Type: text/plain; charset=utf-8\r\n"
+            b"\r\n"
+            b"hello\r\n"
+        )
+
+        mail = MailParser.from_bytes(raw_email)
+        self.assertEqual(mail.to, [("Álpám Longsom", "recipient@example.com")])
+
+    def test_mailparser_from_bytes_unknown_8bit_display_name(self):
+        """
+        End-to-end regression for unknown-8bit encoded-word in To header.
+        """
+        from mailparser.core import MailParser
+
+        raw_email = (
+            b"From: Sender <sender@example.com>\r\n"
+            b"To: =?unknown-8bit?b?w4FscMOhbSBMb25nc29t?= <recipient@example.com>\r\n"
+            b"Subject: Test\r\n"
+            b"Date: Tue, 12 May 2026 18:00:00 +0000\r\n"
+            b"Content-Type: text/plain; charset=utf-8\r\n"
+            b"\r\n"
+            b"hello\r\n"
+        )
+
+        mail = MailParser.from_bytes(raw_email)
+        self.assertEqual(mail.to, [("Álpám Longsom", "recipient@example.com")])
+
     def test_parse_received_envelope_from_with_angle_brackets(self):
         """Test utils.py:294-296 — envelope-from clause with angle-bracket match"""
         # When envelope-from keyword is present AND its value has angle
@@ -644,3 +733,69 @@ class TestUtilsEdgeCases(unittest.TestCase):
         self.assertIsInstance(result, dict)
         # envelope_from must NOT be set
         self.assertNotIn("envelope_from", result)
+
+    def test_get_addresses_non_str_non_header(self):
+        """get_addresses coerces unexpected types via str() (utils.py:135)"""
+        result = get_addresses(42)  # type: ignore[arg-type]
+        self.assertIsInstance(result, list)
+
+    def test_get_addresses_fallback_regex_bare_email(self):
+        """fallback regex matches bare email (utils.py:149-150)"""
+        with patch(
+            "mailparser.utils.email.utils.getaddresses", return_value=[("", "")]
+        ):
+            result = get_addresses("user@example.com")
+        self.assertEqual(result, [("", "user@example.com")])
+
+    def test_get_addresses_fallback_regex_quoted_name(self):
+        """fallback regex matches quoted-name format (utils.py:145-146)"""
+        with patch(
+            "mailparser.utils.email.utils.getaddresses", return_value=[("", "")]
+        ):
+            result = get_addresses('"Quoted Name" <user@example.com>')
+        self.assertEqual(result, [("Quoted Name", "user@example.com")])
+
+    def test_get_addresses_fallback_regex_any_name(self):
+        """fallback regex matches any-name format (utils.py:147-148)"""
+        with patch(
+            "mailparser.utils.email.utils.getaddresses", return_value=[("", "")]
+        ):
+            result = get_addresses("Any Name <user@example.com>")
+        self.assertEqual(result, [("Any Name", "user@example.com")])
+
+    def test_get_addresses_fallback_regex_no_matches(self):
+        """fallback regex finds no addresses, falls through to return parsed"""
+        with patch(
+            "mailparser.utils.email.utils.getaddresses", return_value=[("", "")]
+        ):
+            result = get_addresses("not an email address at all")
+        self.assertEqual(result, [("", "")])
+
+    def test_parse_received_sendgrid_date(self):
+        """parse_received extracts SendGrid non-standard date (utils.py:389-390)"""
+        received = (
+            "from sendgrid.net by mx.example.com with SMTP"
+            " 2024-01-15 10:30:45.123456789 +0000 UTC m=+1234.567890123"
+        )
+        result = parse_received(received)
+        self.assertIn("date", result)
+        self.assertIn("2024-01-15", result["date"])
+
+    def test_parse_received_for_clause(self):
+        """parse_received captures 'for' clause (utils.py:411)"""
+        received = (
+            "from mail.example.com by mx.example.com"
+            " for <user@example.com>; Mon, 15 Jan 2024 10:30:45 +0000"
+        )
+        result = parse_received(received)
+        self.assertIn("for", result)
+        self.assertIn("user@example.com", result["for"])
+
+    def test_parse_received_envelope_from_embedded_in_clause(self):
+        """parse_received extracts envelope-from embedded in clause value"""
+        received = (
+            "from mail.example.com (envelope-from <sender@example.com>)"
+            " by mx.example.com; Mon, 15 Jan 2024 10:30:45 +0000"
+        )
+        result = parse_received(received)
+        self.assertEqual(result.get("envelope_from"), "sender@example.com")