twitwi 0.21.2__tar.gz → 0.22.0__tar.gz

{twitwi-0.21.2/twitwi.egg-info → twitwi-0.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: twitwi
-Version: 0.21.2
+Version: 0.22.0
 Summary: A collection of Twitter-related helper functions for python.
 Home-page: http://github.com/medialab/twitwi
 Author: Béatrice Mazoyer, Guillaume Plique, Benjamin Ooghe-Tabanou
@@ -56,18 +56,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -179,7 +183,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -209,6 +224,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -221,6 +242,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -229,7 +256,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS
 

{twitwi-0.21.2 → twitwi-0.22.0}/README.md

@@ -29,18 +29,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -152,7 +156,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -182,6 +197,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -194,6 +215,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -202,7 +229,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS
 

{twitwi-0.21.2 → twitwi-0.22.0}/setup.py

@@ -5,7 +5,7 @@ with open("./README.md", "r") as f:
 
 setup(
     name="twitwi",
-    version="0.21.2",
+    version="0.22.0",
     description="A collection of Twitter-related helper functions for python.",
     long_description=long_description,
     long_description_content_type="text/markdown",

{twitwi-0.21.2 → twitwi-0.22.0}/test/bluesky/formatters_test.py

@@ -2,11 +2,13 @@ import csv
 from io import StringIO
 from twitwi.bluesky import (
     format_profile_as_csv_row,
+    format_partial_profile_as_csv_row,
     format_post_as_csv_row,
     transform_profile_into_csv_dict,
+    transform_partial_profile_into_csv_dict,
     transform_post_into_csv_dict,
 )
-from twitwi.bluesky.constants import PROFILE_FIELDS, POST_FIELDS
+from twitwi.bluesky.constants import PROFILE_FIELDS, PARTIAL_PROFILE_FIELDS, POST_FIELDS
 from test.utils import get_json_resource, open_resource
 
 
@@ -56,6 +58,51 @@ class TestFormatters:
             buffer.seek(0)
             assert list(csv.DictReader(buffer)) == list(csv.DictReader(f))
 
+    def test_format_partial_profile_as_csv_row(self):
+        normalized_partial_profiles = get_json_resource(
+            "bluesky-normalized-partial-profiles.json"
+        )
+
+        buffer = StringIO(newline=None)
+        writer = csv.writer(buffer, quoting=csv.QUOTE_MINIMAL)
+        writer.writerow(PARTIAL_PROFILE_FIELDS)
+
+        for profile in normalized_partial_profiles:
+            writer.writerow(format_partial_profile_as_csv_row(profile))
+
+        if OVERWRITE_TESTS:
+            written = buffer.getvalue()
+
+            with open("test/resources/bluesky-partial-profiles-export.csv", "w") as f:
+                f.write(written)
+
+        with open_resource("bluesky-partial-profiles-export.csv") as f:
+            buffer.seek(0)
+            assert list(csv.reader(buffer)) == list(csv.reader(f))
+
+    def test_transform_partial_profile_into_csv_dict(self):
+        normalized_partial_profiles = get_json_resource(
+            "bluesky-normalized-partial-profiles.json"
+        )
+
+        buffer = StringIO(newline=None)
+        writer = csv.DictWriter(
+            buffer,
+            fieldnames=PARTIAL_PROFILE_FIELDS,
+            extrasaction="ignore",
+            restval="",
+            quoting=csv.QUOTE_MINIMAL,
+        )
+        writer.writeheader()
+
+        for profile in normalized_partial_profiles:
+            transform_partial_profile_into_csv_dict(profile)
+            writer.writerow(profile)
+
+        with open_resource("bluesky-partial-profiles-export.csv") as f:
+            buffer.seek(0)
+            assert list(csv.DictReader(buffer)) == list(csv.DictReader(f))
+
     def test_format_post_as_csv_row(self):
         normalized_posts = get_json_resource("bluesky-normalized-posts.json")
 

{twitwi-0.21.2 → twitwi-0.22.0}/test/bluesky/normalizers_test.py

@@ -5,7 +5,7 @@ from functools import partial
 from pytz import timezone
 from copy import deepcopy
 
-from twitwi.bluesky import normalize_profile, normalize_post
+from twitwi.bluesky import normalize_profile, normalize_partial_profile, normalize_post
 
 from test.utils import get_json_resource
 
@@ -74,6 +74,42 @@ class TestNormalizers:
 
         assert profile == original_arg
 
+    def test_normalize_partial_profile(self):
+        tz = timezone("Europe/Paris")
+
+        profiles = get_json_resource("bluesky-partial-profiles.json")
+        fn = partial(normalize_partial_profile, locale=tz)
+
+        if OVERWRITE_TESTS:
+            from test.utils import dump_json_resource
+
+            normalized_profiles = [
+                set_fake_collection_time(fn(profile)) for profile in profiles
+            ]
+            dump_json_resource(
+                normalized_profiles, "bluesky-normalized-partial-profiles.json"
+            )
+
+        expected = get_json_resource("bluesky-normalized-partial-profiles.json")
+
+        for idx, profile in enumerate(profiles):
+            result = fn(profile)
+            assert isinstance(result, dict)
+            assert "collection_time" in result and isinstance(
+                result["collection_time"], str
+            )
+
+            compare_dicts(profile["handle"], result, expected[idx])
+
+    def test_normalize_partial_profile_should_not_mutate(self):
+        profile = get_json_resource("bluesky-partial-profiles.json")[0]
+
+        original_arg = deepcopy(profile)
+
+        normalize_partial_profile(profile)
+
+        assert profile == original_arg
+
     def test_normalize_post(self):
         tz = timezone("Europe/Paris")
 

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/__init__.py

@@ -1,7 +1,13 @@
-from twitwi.bluesky.normalizers import normalize_profile, normalize_post
+from twitwi.bluesky.normalizers import (
+    normalize_profile,
+    normalize_partial_profile,
+    normalize_post,
+)
 from twitwi.bluesky.formatters import (
     transform_profile_into_csv_dict,
     format_profile_as_csv_row,
+    transform_partial_profile_into_csv_dict,
+    format_partial_profile_as_csv_row,
     transform_post_into_csv_dict,
     format_post_as_csv_row,
 )
@@ -9,8 +15,11 @@ from twitwi.bluesky.formatters import (
 __all__ = [
     "transform_profile_into_csv_dict",
     "format_profile_as_csv_row",
+    "transform_partial_profile_into_csv_dict",
+    "format_partial_profile_as_csv_row",
     "transform_post_into_csv_dict",
     "format_post_as_csv_row",
     "normalize_profile",
+    "normalize_partial_profile",
     "normalize_post",
 ]

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/constants.py

@@ -1,9 +1,11 @@
 from typing import List, Optional
 
-from twitwi.bluesky.types import BlueskyProfile, BlueskyPost
+from twitwi.bluesky.types import BlueskyProfile, BlueskyPartialProfile, BlueskyPost
 
 PROFILE_FIELDS = list(BlueskyProfile.__annotations__.keys())
 
+PARTIAL_PROFILE_FIELDS = list(BlueskyPartialProfile.__annotations__.keys())
+
 POST_FIELDS = list(BlueskyPost.__annotations__.keys())
 
 POST_PLURAL_FIELDS = [

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/formatters.py

@@ -1,6 +1,7 @@
 from twitwi.formatters import make_transform_into_csv_dict, make_format_as_csv_row
 from twitwi.bluesky.constants import (
     PROFILE_FIELDS,
+    PARTIAL_PROFILE_FIELDS,
     POST_FIELDS,
     POST_PLURAL_FIELDS,
     POST_BOOLEAN_FIELDS,
@@ -20,10 +21,18 @@ transform_profile_into_csv_dict = make_transform_into_csv_dict([], [])
 
 format_profile_as_csv_row = make_format_as_csv_row(PROFILE_FIELDS, [], [])
 
+transform_partial_profile_into_csv_dict = make_transform_into_csv_dict([], [])
+
+format_partial_profile_as_csv_row = make_format_as_csv_row(
+    PARTIAL_PROFILE_FIELDS, [], []
+)
+
 
 __all__ = [
     "transform_post_into_csv_dict",
     "format_post_as_csv_row",
     "transform_profile_into_csv_dict",
     "format_profile_as_csv_row",
+    "transform_partial_profile_into_csv_dict",
+    "format_partial_profile_as_csv_row",
 ]

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/normalizers.py

@@ -1,11 +1,13 @@
 from copy import deepcopy
-from typing import List, Dict, Union, Optional, Literal, overload
+from typing import List, Dict, Union, Optional, Literal, Any, overload
+
+from ural import is_url
 
 from twitwi.exceptions import BlueskyPayloadError
 from twitwi.utils import (
     get_collection_time,
     get_dates,
-    custom_normalize_url,
+    safe_normalize_url,
     custom_get_normalized_hostname,
 )
 from twitwi.bluesky.utils import (
@@ -18,10 +20,10 @@ from twitwi.bluesky.utils import (
     format_starterpack_url,
     format_media_url,
 )
-from twitwi.bluesky.types import BlueskyProfile, BlueskyPost
+from twitwi.bluesky.types import BlueskyProfile, BlueskyPartialProfile, BlueskyPost
 
 
-def normalize_profile(data: Dict, locale: Optional[str] = None) -> BlueskyProfile:
+def normalize_profile(data: Dict, locale: Optional[Any] = None) -> BlueskyProfile:
     associated = data["associated"]
 
     pinned_post_uri = None
@@ -38,23 +40,48 @@ def normalize_profile(data: Dict, locale: Optional[str] = None) -> BlueskyProfil
         "did": data["did"],
         "url": format_profile_url(data["handle"]),
         "handle": data["handle"],
-        "display_name": data.get("displayName", ""),
+        "display_name": data.get("displayName"),
         "created_at": created_at,
         "timestamp_utc": timestamp_utc,
-        "description": data["description"],
-        "avatar": data.get("avatar", ""),
+        "description": data.get("description"),
+        "avatar": data.get("avatar"),
         "posts": data["postsCount"],
         "followers": data["followersCount"],
         "follows": data["followsCount"],
         "lists": associated["lists"],
         "feedgens": associated["feedgens"],
         "starter_packs": associated["starterPacks"],
-        "banner": data["banner"],
+        "banner": data.get("banner"),
         "pinned_post_uri": pinned_post_uri,
         "collection_time": get_collection_time(),
     }
 
 
+def normalize_partial_profile(
+    data: Dict, locale: Optional[Any] = None
+) -> BlueskyPartialProfile:
+    associated = data["associated"]
+
+    timestamp_utc, created_at = get_dates(
+        data["createdAt"], locale=locale, source="bluesky"
+    )
+
+    return {
+        "did": data["did"],
+        "url": format_profile_url(data["handle"]),
+        "handle": data["handle"],
+        "display_name": data.get("displayName"),
+        "created_at": created_at,
+        "timestamp_utc": timestamp_utc,
+        "description": data.get("description"),
+        "avatar": data.get("avatar"),
+        "lists": associated.get("lists"),
+        "feedgens": associated.get("feedgens"),
+        "starter_packs": associated.get("starterPacks"),
+        "collection_time": get_collection_time(),
+    }
+
+
 def prepare_native_gif_as_media(gif_data, user_did, source):
     if "thumb" in gif_data:
         media_cid = gif_data["thumb"]["ref"]["$link"]
@@ -125,7 +152,14 @@ def prepare_quote_data(embed_quote, card_data, post, links):
     )
 
     # First store ugly quoted url with user did in case full quote data is missing (recursion > 3 or detached quote)
-    post["quoted_url"] = format_post_url(post["quoted_user_did"], post["quoted_did"])
+    # Handling special posts types (only lists for now, for example: https://bsky.app/profile/lanana421.bsky.social/lists/3lxdgjtpqhf2z)
+    if "/app.bsky.graph.list/" in post["quoted_uri"]:
+        post_splitter = "/lists/"
+    else:
+        post_splitter = "/post/"
+    post["quoted_url"] = format_post_url(
+        post["quoted_user_did"], post["quoted_did"], post_splitter=post_splitter
+    )
 
     quoted_data = None
     if card_data:
@@ -202,7 +236,7 @@ def normalize_post(
 
 def normalize_post(
     payload: Dict,
-    locale: Optional[str] = None,
+    locale: Optional[Any] = None,
     extract_referenced_posts: bool = False,
     collection_source: Optional[str] = None,
 ) -> Union[BlueskyPost, List[BlueskyPost]]:
@@ -347,16 +381,28 @@ def normalize_post(
             # Handle native polls
             if "https://poll.blue/" in feat["uri"]:
                 if feat["uri"].endswith("/0"):
-                    links.add(custom_normalize_url(feat["uri"]))
+                    link = safe_normalize_url(feat["uri"])
+                    if is_url(link):
+                        links.add(link)
+                    else:
+                        continue
                     text += b" %s" % feat["uri"].encode("utf-8")
                 continue
 
-            links.add(custom_normalize_url(feat["uri"]))
+            link = safe_normalize_url(feat["uri"])
+            if is_url(link):
+                links.add(link)
+            else:
+                continue
             # Check & fix occasional errored link positioning
-            # example: https://bsky.app/profile/ecrime.ch/post/3lqotmopayr23
+            # examples: https://bsky.app/profile/ecrime.ch/post/3lqotmopayr23
+            #           https://bsky.app/profile/clustz.com/post/3lqfi7mnto52w
             byteStart = facet["index"]["byteStart"]
-            if b" " in text[byteStart : facet["index"]["byteEnd"]]:
-                byteStart = text.find(b"http", byteStart)
+
+            if not text[byteStart : facet["index"]["byteEnd"]].startswith(b"http"):
+                new_byteStart = text.find(b"http", byteStart, facet["index"]["byteEnd"])
+                if new_byteStart != -1:
+                    byteStart = new_byteStart
 
             links_to_replace.append(
                 {
@@ -440,7 +486,7 @@ def normalize_post(
 
             # Extra card links sometimes missing from facets & text due to manual action in post form
             else:
-                extra_links.append(embed["external"]["uri"])
+                extra_links.append(link)
                 # Handle link card metadata
                 if "embed" in data:
                     post = process_card_data(data["embed"]["external"], post)
@@ -515,9 +561,10 @@ def normalize_post(
 
         # Process extra links
         for link in extra_links:
-            norm_link = custom_normalize_url(link)
+            norm_link = safe_normalize_url(link)
             if norm_link not in links:
-                links.add(norm_link)
+                if is_url(norm_link):
+                    links.add(norm_link)
                 text += b" " + link.encode("utf-8")
 
         # Process medias

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/types.py

@@ -10,7 +10,7 @@ class BlueskyProfile(TypedDict):
     url: str                            # URL of the profile accessible on the web
     handle: str                         # updatable human-readable username of the account (usually like username.bsky.social or username.com)
     display_name: Optional[str]         # updatable human-readable name of the account
-    description: str                    # profile short description written by the user
+    description: Optional[str]          # profile short description written by the user
     posts: int                          # total number of posts submitted by the user (at collection time)
     followers: int                      # total number of followers of the user (at collection time)
     follows: int                        # total number of other users followed by the user (at collection time)
@@ -18,12 +18,26 @@ class BlueskyProfile(TypedDict):
     feedgens: int                       # total number of custom feeds created by the user (at collection time)
     starter_packs: int                  # total number of starter packs created by the user (at collection time)
     avatar: Optional[str]               # URL to the image serving as avatar to the user
-    banner: str                         # URL to the image serving as profile banner to the user
+    banner: Optional[str]               # URL to the image serving as profile banner to the user
     pinned_post_uri: Optional[str]      # ATProto's internal URI to the post potentially pinned by the user to appear at the top of his posts on his profile
     created_at: str                     # datetime (potentially timezoned) of when the user created the account
     timestamp_utc: int                  # Unix UTC timestamp of when the user created the account
     collection_time: Optional[str]      # datetime (potentially timezoned) of when the data was normalized
 
+class BlueskyPartialProfile(TypedDict): # A partial version of the profile found in follower/follow profile payloads
+    did: str                            # persistent long-term identifier of the account
+    url: str                            # URL of the profile accessible on the web
+    handle: str                         # updatable human-readable username of the account (usually like username.bsky.social or username.com)
+    display_name: Optional[str]         # updatable human-readable name of the account
+    description: Optional[str]          # profile short description written by the user
+    lists: Optional[int]                # total number of lists created by the user (at collection time)
+    feedgens: Optional[int]             # total number of custom feeds created by the user (at collection time)
+    starter_packs: Optional[int]        # total number of starter packs created by the user (at collection time)
+    avatar: Optional[str]               # URL to the image serving as avatar to the user
+    created_at: str                     # datetime (potentially timezoned) of when the user created the account
+    timestamp_utc: int                  # Unix UTC timestamp of when the user created the account
+    collection_time: Optional[str]      # datetime (potentially timezoned) of when the data was normalized
+
 
 
 class BlueskyPost(TypedDict):

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/bluesky/utils.py

@@ -55,7 +55,9 @@ def validate_post_payload(data):
     return True, None
 
 
-re_embed_types = re.compile(r"\.(record|recordWithMedia|images|video|external)$")
+re_embed_types = re.compile(
+    r"\.(record|recordWithMedia|images|video|external)(?:#.*)?$"
+)
 
 
 def valid_embed_type(embed_type):
@@ -66,29 +68,39 @@ def format_profile_url(user_handle_or_did):
     return f"https://bsky.app/profile/{user_handle_or_did}"
 
 
-def format_post_url(user_handle_or_did, post_did):
-    return f"https://bsky.app/profile/{user_handle_or_did}/post/{post_did}"
+def format_post_url(user_handle_or_did, post_did, post_splitter="/post/"):
+    return f"https://bsky.app/profile/{user_handle_or_did}{post_splitter}{post_did}"
 
 
 def parse_post_url(url, source):
     """Returns a tuple of (author_handle/did, post_did) from an https://bsky.app post URL"""
 
-    if not url.startswith("https://bsky.app/profile/") and "/post/" not in url:
-        raise BlueskyPayloadError(source, f"{url} is not a usual Bluesky post url")
-    return url[25:].split("/post/")
+    known_splits = ["/post/", "/lists/"]
+
+    if url.startswith("https://bsky.app/profile/"):
+        for split in known_splits:
+            if split in url[25:]:
+                return url[25:].split(split)
+
+    raise BlueskyPayloadError(source, f"{url} is not a usual Bluesky post url")
 
 
 def parse_post_uri(uri, source=None):
     """Returns a tuple of (author_did, post_did) from an at:// post URI"""
 
-    if uri.startswith("at://") and "/app.bsky.graph.starterpack/" in uri:
-        return uri[5:].split("/app.bsky.graph.starterpack/")
+    known_splits = [
+        "/app.bsky.feed.post/",
+        "/app.bsky.graph.starterpack/",
+        "/app.bsky.feed.generator/",
+        "/app.bsky.graph.list/",
+    ]
 
-    if not uri.startswith("at://") and "/app.bsky.feed.post/" not in uri:
-        raise BlueskyPayloadError(
-            source or uri, f"{uri} is not a usual Bluesky post uri"
-        )
-    return uri[5:].split("/app.bsky.feed.post/")
+    if uri.startswith("at://"):
+        for split in known_splits:
+            if split in uri:
+                return uri[5:].split(split)
+
+    raise BlueskyPayloadError(source or uri, f"{uri} is not a usual Bluesky post uri")
 
 
 def format_starterpack_url(user_handle_or_did, record_did):
@@ -105,6 +117,13 @@ def format_media_url(user_did, media_cid, mime_type, source):
         media_thumb = (
             f"https://video.bsky.app/watch/{user_did}/{media_cid}/thumbnail.jpg"
         )
+    elif mime_type == "application/octet-stream":
+        media_url = (
+            f"https://cdn.bsky.app/img/feed_fullsize/plain/{user_did}/{media_cid}@jpeg"
+        )
+        media_thumb = (
+            f"https://cdn.bsky.app/img/feed_thumbnail/plain/{user_did}/{media_cid}@jpeg"
+        )
     else:
-        raise BlueskyPayloadError(source, f"{mime_type} is an usual media mimeType")
+        raise BlueskyPayloadError(source, f"{mime_type} is an unusual media mimeType")
     return media_url, media_thumb

{twitwi-0.21.2 → twitwi-0.22.0}/twitwi/utils.py

@@ -4,6 +4,8 @@
 #
 # Miscellaneous utility functions.
 #
+from typing import Tuple
+
 from pytz import timezone
 from dateutil.parser import parse as parse_date
 from ural import normalize_url, get_normalized_hostname
@@ -25,6 +27,21 @@ UTC_TIMEZONE = timezone("UTC")
 
 custom_normalize_url = partial(normalize_url, **CANONICAL_URL_KWARGS)
 
+
+def safe_normalize_url(url):
+    # We avoid normalizing bluesky urls containing specific uri parts because
+    # Bluesky servers don't handle quoting correctly...
+    # See https://github.com/medialab/twitwi/issues/72
+    if "/did:plc:" in url:
+        return url
+
+    try:
+        return custom_normalize_url(url)
+    except Exception:
+        # In case of error, return the original URL. Possibly not a valid URL, e.g. url containing double slashes
+        return url
+
+
 custom_get_normalized_hostname = partial(
     get_normalized_hostname, **CANONICAL_HOSTNAME_KWARGS
 )
@@ -34,7 +51,9 @@ def get_collection_time():
     return datetime.now().strftime(FORMATTED_FULL_DATETIME_FORMAT)
 
 
-def get_dates(date_str, locale=None, source="v1"):
+def get_dates(
+    date_str: str, locale=None, source: str = "v1", millisecond_timestamp: bool = False
+) -> Tuple[int, str]:
     if source not in ["v1", "v2", "bluesky"]:
         raise Exception("source should be one of v1, v2 or bluesky")
 
@@ -56,8 +75,14 @@ def get_dates(date_str, locale=None, source="v1"):
         utc_datetime = UTC_TIMEZONE.localize(parsed_datetime)
     locale_datetime = utc_datetime.astimezone(locale)
 
+    timestamp = int(utc_datetime.timestamp())
+
+    if millisecond_timestamp:
+        timestamp *= 1000
+        timestamp += utc_datetime.microsecond / 1000
+
     return (
-        int(utc_datetime.timestamp()),
+        int(timestamp),
         datetime.strftime(
             locale_datetime,
             FORMATTED_FULL_DATETIME_FORMAT
@@ -106,6 +131,7 @@ def get_dates_from_id(tweet_id, locale=None):
         locale = UTC_TIMEZONE
 
     timestamp = get_timestamp_from_id(tweet_id)
+    assert timestamp is not None
 
     locale_datetime = datetime.fromtimestamp(timestamp, locale)
 

{twitwi-0.21.2 → twitwi-0.22.0/twitwi.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: twitwi
-Version: 0.21.2
+Version: 0.22.0
 Summary: A collection of Twitter-related helper functions for python.
 Home-page: http://github.com/medialab/twitwi
 Author: Béatrice Mazoyer, Guillaume Plique, Benjamin Ooghe-Tabanou
@@ -56,18 +56,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -179,7 +183,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -209,6 +224,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -221,6 +242,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -229,7 +256,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS