twitwi 0.21.1__tar.gz → 0.22.0__tar.gz

{twitwi-0.21.1/twitwi.egg-info → twitwi-0.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: twitwi
-Version: 0.21.1
+Version: 0.22.0
 Summary: A collection of Twitter-related helper functions for python.
 Home-page: http://github.com/medialab/twitwi
 Author: Béatrice Mazoyer, Guillaume Plique, Benjamin Ooghe-Tabanou
@@ -56,18 +56,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -95,7 +99,7 @@ for post_data in posts_payload_from_API:
 
 # Then, saving normalized profiles into a CSV using DictWriter:
 
-from csv import DictWriter
+import csv
 from twitwi.bluesky.constants import POST_FIELDS
 from twitwi.bluesky import transform_post_into_csv_dict
 
@@ -108,7 +112,6 @@ with open("normalized_bluesky_posts.csv", "w") as f:
 
 # Or using the basic CSV writer:
 
-from csv import writer
 from twitwi.bluesky import format_post_as_csv_row
 
 with open("normalized_bluesky_posts.csv", "w") as f:
@@ -180,7 +183,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -210,6 +224,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -222,6 +242,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -230,7 +256,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS
 

{twitwi-0.21.1 → twitwi-0.22.0}/README.md

@@ -29,18 +29,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -68,7 +72,7 @@ for post_data in posts_payload_from_API:
 
 # Then, saving normalized profiles into a CSV using DictWriter:
 
-from csv import DictWriter
+import csv
 from twitwi.bluesky.constants import POST_FIELDS
 from twitwi.bluesky import transform_post_into_csv_dict
 
@@ -81,7 +85,6 @@ with open("normalized_bluesky_posts.csv", "w") as f:
 
 # Or using the basic CSV writer:
 
-from csv import writer
 from twitwi.bluesky import format_post_as_csv_row
 
 with open("normalized_bluesky_posts.csv", "w") as f:
@@ -153,7 +156,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -183,6 +197,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -195,6 +215,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -203,7 +229,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS
 

{twitwi-0.21.1 → twitwi-0.22.0}/setup.py

@@ -5,7 +5,7 @@ with open("./README.md", "r") as f:
 
 setup(
     name="twitwi",
-    version="0.21.1",
+    version="0.22.0",
     description="A collection of Twitter-related helper functions for python.",
     long_description=long_description,
     long_description_content_type="text/markdown",

{twitwi-0.21.1 → twitwi-0.22.0}/test/bluesky/formatters_test.py

@@ -2,11 +2,13 @@ import csv
 from io import StringIO
 from twitwi.bluesky import (
     format_profile_as_csv_row,
+    format_partial_profile_as_csv_row,
     format_post_as_csv_row,
     transform_profile_into_csv_dict,
+    transform_partial_profile_into_csv_dict,
     transform_post_into_csv_dict,
 )
-from twitwi.bluesky.constants import PROFILE_FIELDS, POST_FIELDS
+from twitwi.bluesky.constants import PROFILE_FIELDS, PARTIAL_PROFILE_FIELDS, POST_FIELDS
 from test.utils import get_json_resource, open_resource
 
 
@@ -56,6 +58,51 @@ class TestFormatters:
             buffer.seek(0)
             assert list(csv.DictReader(buffer)) == list(csv.DictReader(f))
 
+    def test_format_partial_profile_as_csv_row(self):
+        normalized_partial_profiles = get_json_resource(
+            "bluesky-normalized-partial-profiles.json"
+        )
+
+        buffer = StringIO(newline=None)
+        writer = csv.writer(buffer, quoting=csv.QUOTE_MINIMAL)
+        writer.writerow(PARTIAL_PROFILE_FIELDS)
+
+        for profile in normalized_partial_profiles:
+            writer.writerow(format_partial_profile_as_csv_row(profile))
+
+        if OVERWRITE_TESTS:
+            written = buffer.getvalue()
+
+            with open("test/resources/bluesky-partial-profiles-export.csv", "w") as f:
+                f.write(written)
+
+        with open_resource("bluesky-partial-profiles-export.csv") as f:
+            buffer.seek(0)
+            assert list(csv.reader(buffer)) == list(csv.reader(f))
+
+    def test_transform_partial_profile_into_csv_dict(self):
+        normalized_partial_profiles = get_json_resource(
+            "bluesky-normalized-partial-profiles.json"
+        )
+
+        buffer = StringIO(newline=None)
+        writer = csv.DictWriter(
+            buffer,
+            fieldnames=PARTIAL_PROFILE_FIELDS,
+            extrasaction="ignore",
+            restval="",
+            quoting=csv.QUOTE_MINIMAL,
+        )
+        writer.writeheader()
+
+        for profile in normalized_partial_profiles:
+            transform_partial_profile_into_csv_dict(profile)
+            writer.writerow(profile)
+
+        with open_resource("bluesky-partial-profiles-export.csv") as f:
+            buffer.seek(0)
+            assert list(csv.DictReader(buffer)) == list(csv.DictReader(f))
+
     def test_format_post_as_csv_row(self):
         normalized_posts = get_json_resource("bluesky-normalized-posts.json")
 

{twitwi-0.21.1 → twitwi-0.22.0}/test/bluesky/normalizers_test.py

@@ -5,7 +5,7 @@ from functools import partial
 from pytz import timezone
 from copy import deepcopy
 
-from twitwi.bluesky import normalize_profile, normalize_post
+from twitwi.bluesky import normalize_profile, normalize_partial_profile, normalize_post
 
 from test.utils import get_json_resource
 
@@ -15,6 +15,8 @@ OVERWRITE_TESTS = False
 
 
 FAKE_COLLECTION_TIME = "2025-01-01T00:00:00.000000"
+
+
 def set_fake_collection_time(dico):
     if "collection_time" in dico:
         dico["collection_time"] = FAKE_COLLECTION_TIME
@@ -47,7 +49,9 @@ class TestNormalizers:
         if OVERWRITE_TESTS:
             from test.utils import dump_json_resource
 
-            normalized_profiles = [set_fake_collection_time(fn(profile)) for profile in profiles]
+            normalized_profiles = [
+                set_fake_collection_time(fn(profile)) for profile in profiles
+            ]
             dump_json_resource(normalized_profiles, "bluesky-normalized-profiles.json")
 
         expected = get_json_resource("bluesky-normalized-profiles.json")
@@ -70,6 +74,42 @@ class TestNormalizers:
 
         assert profile == original_arg
 
+    def test_normalize_partial_profile(self):
+        tz = timezone("Europe/Paris")
+
+        profiles = get_json_resource("bluesky-partial-profiles.json")
+        fn = partial(normalize_partial_profile, locale=tz)
+
+        if OVERWRITE_TESTS:
+            from test.utils import dump_json_resource
+
+            normalized_profiles = [
+                set_fake_collection_time(fn(profile)) for profile in profiles
+            ]
+            dump_json_resource(
+                normalized_profiles, "bluesky-normalized-partial-profiles.json"
+            )
+
+        expected = get_json_resource("bluesky-normalized-partial-profiles.json")
+
+        for idx, profile in enumerate(profiles):
+            result = fn(profile)
+            assert isinstance(result, dict)
+            assert "collection_time" in result and isinstance(
+                result["collection_time"], str
+            )
+
+            compare_dicts(profile["handle"], result, expected[idx])
+
+    def test_normalize_partial_profile_should_not_mutate(self):
+        profile = get_json_resource("bluesky-partial-profiles.json")[0]
+
+        original_arg = deepcopy(profile)
+
+        normalize_partial_profile(profile)
+
+        assert profile == original_arg
+
     def test_normalize_post(self):
         tz = timezone("Europe/Paris")
 
@@ -79,7 +119,13 @@ class TestNormalizers:
         if OVERWRITE_TESTS:
             from test.utils import dump_json_resource
 
-            normalized_posts = [[set_fake_collection_time(p) for p in fn(post, extract_referenced_posts=True)] for post in posts]
+            normalized_posts = [
+                [
+                    set_fake_collection_time(p)
+                    for p in fn(post, extract_referenced_posts=True)
+                ]
+                for post in posts
+            ]
             dump_json_resource(normalized_posts, "bluesky-normalized-posts.json")
 
         expected = get_json_resource("bluesky-normalized-posts.json")

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/__init__.py

@@ -1,7 +1,13 @@
-from twitwi.bluesky.normalizers import normalize_profile, normalize_post
+from twitwi.bluesky.normalizers import (
+    normalize_profile,
+    normalize_partial_profile,
+    normalize_post,
+)
 from twitwi.bluesky.formatters import (
     transform_profile_into_csv_dict,
     format_profile_as_csv_row,
+    transform_partial_profile_into_csv_dict,
+    format_partial_profile_as_csv_row,
     transform_post_into_csv_dict,
     format_post_as_csv_row,
 )
@@ -9,8 +15,11 @@ from twitwi.bluesky.formatters import (
 __all__ = [
     "transform_profile_into_csv_dict",
     "format_profile_as_csv_row",
+    "transform_partial_profile_into_csv_dict",
+    "format_partial_profile_as_csv_row",
     "transform_post_into_csv_dict",
     "format_post_as_csv_row",
     "normalize_profile",
+    "normalize_partial_profile",
     "normalize_post",
 ]

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/constants.py

@@ -1,9 +1,11 @@
 from typing import List, Optional
 
-from twitwi.bluesky.types import BlueskyProfile, BlueskyPost
+from twitwi.bluesky.types import BlueskyProfile, BlueskyPartialProfile, BlueskyPost
 
 PROFILE_FIELDS = list(BlueskyProfile.__annotations__.keys())
 
+PARTIAL_PROFILE_FIELDS = list(BlueskyPartialProfile.__annotations__.keys())
+
 POST_FIELDS = list(BlueskyPost.__annotations__.keys())
 
 POST_PLURAL_FIELDS = [

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/formatters.py

@@ -1,6 +1,7 @@
 from twitwi.formatters import make_transform_into_csv_dict, make_format_as_csv_row
 from twitwi.bluesky.constants import (
     PROFILE_FIELDS,
+    PARTIAL_PROFILE_FIELDS,
     POST_FIELDS,
     POST_PLURAL_FIELDS,
     POST_BOOLEAN_FIELDS,
@@ -20,10 +21,18 @@ transform_profile_into_csv_dict = make_transform_into_csv_dict([], [])
 
 format_profile_as_csv_row = make_format_as_csv_row(PROFILE_FIELDS, [], [])
 
+transform_partial_profile_into_csv_dict = make_transform_into_csv_dict([], [])
+
+format_partial_profile_as_csv_row = make_format_as_csv_row(
+    PARTIAL_PROFILE_FIELDS, [], []
+)
+
 
 __all__ = [
     "transform_post_into_csv_dict",
     "format_post_as_csv_row",
     "transform_profile_into_csv_dict",
     "format_profile_as_csv_row",
+    "transform_partial_profile_into_csv_dict",
+    "format_partial_profile_as_csv_row",
 ]

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/normalizers.py

@@ -1,11 +1,13 @@
 from copy import deepcopy
-from typing import List, Dict, Union, Optional, Literal, overload
+from typing import List, Dict, Union, Optional, Literal, Any, overload
+
+from ural import is_url
 
 from twitwi.exceptions import BlueskyPayloadError
 from twitwi.utils import (
     get_collection_time,
     get_dates,
-    custom_normalize_url,
+    safe_normalize_url,
     custom_get_normalized_hostname,
 )
 from twitwi.bluesky.utils import (
@@ -18,10 +20,10 @@ from twitwi.bluesky.utils import (
     format_starterpack_url,
     format_media_url,
 )
-from twitwi.bluesky.types import BlueskyProfile, BlueskyPost
+from twitwi.bluesky.types import BlueskyProfile, BlueskyPartialProfile, BlueskyPost
 
 
-def normalize_profile(data: Dict, locale: Optional[str] = None) -> BlueskyProfile:
+def normalize_profile(data: Dict, locale: Optional[Any] = None) -> BlueskyProfile:
     associated = data["associated"]
 
     pinned_post_uri = None
@@ -38,23 +40,48 @@ def normalize_profile(data: Dict, locale: Optional[str] = None) -> BlueskyProfil
         "did": data["did"],
         "url": format_profile_url(data["handle"]),
         "handle": data["handle"],
-        "display_name": data.get("displayName", ""),
+        "display_name": data.get("displayName"),
         "created_at": created_at,
         "timestamp_utc": timestamp_utc,
-        "description": data["description"],
-        "avatar": data.get("avatar", ""),
+        "description": data.get("description"),
+        "avatar": data.get("avatar"),
         "posts": data["postsCount"],
         "followers": data["followersCount"],
         "follows": data["followsCount"],
         "lists": associated["lists"],
         "feedgens": associated["feedgens"],
         "starter_packs": associated["starterPacks"],
-        "banner": data["banner"],
+        "banner": data.get("banner"),
         "pinned_post_uri": pinned_post_uri,
         "collection_time": get_collection_time(),
     }
 
 
+def normalize_partial_profile(
+    data: Dict, locale: Optional[Any] = None
+) -> BlueskyPartialProfile:
+    associated = data["associated"]
+
+    timestamp_utc, created_at = get_dates(
+        data["createdAt"], locale=locale, source="bluesky"
+    )
+
+    return {
+        "did": data["did"],
+        "url": format_profile_url(data["handle"]),
+        "handle": data["handle"],
+        "display_name": data.get("displayName"),
+        "created_at": created_at,
+        "timestamp_utc": timestamp_utc,
+        "description": data.get("description"),
+        "avatar": data.get("avatar"),
+        "lists": associated.get("lists"),
+        "feedgens": associated.get("feedgens"),
+        "starter_packs": associated.get("starterPacks"),
+        "collection_time": get_collection_time(),
+    }
+
+
 def prepare_native_gif_as_media(gif_data, user_did, source):
     if "thumb" in gif_data:
         media_cid = gif_data["thumb"]["ref"]["$link"]
@@ -73,8 +100,12 @@ def prepare_native_gif_as_media(gif_data, user_did, source):
 
 
 def prepare_image_as_media(image_data):
+    if "ref" not in image_data["image"] or "$link" not in image_data["image"]["ref"]:
+        image_id = image_data["image"]["cid"]
+    else:
+        image_id = image_data["image"]["ref"]["$link"]
     return {
-        "id": image_data["image"]["ref"]["$link"],
+        "id": image_id,
         "type": image_data["image"]["mimeType"],
         "alt": image_data["alt"],
     }
@@ -92,7 +123,9 @@ def process_starterpack_card(embed_data, post):
 
     card = embed_data.get("record", {})
     creator_did, pack_did = parse_post_uri(embed_data["uri"])
-    post["card_link"] = format_starterpack_url(embed_data.get("creator", {}).get("handle") or creator_did, pack_did)
+    post["card_link"] = format_starterpack_url(
+        embed_data.get("creator", {}).get("handle") or creator_did, pack_did
+    )
     post["card_title"] = card.get("name", "")
     post["card_description"] = card.get("description", "")
     post["card_thumbnail"] = card.get("thumb", "")
@@ -119,7 +152,14 @@ def prepare_quote_data(embed_quote, card_data, post, links):
     )
 
     # First store ugly quoted url with user did in case full quote data is missing (recursion > 3 or detached quote)
-    post["quoted_url"] = format_post_url(post["quoted_user_did"], post["quoted_did"])
+    # Handling special posts types (only lists for now, for example: https://bsky.app/profile/lanana421.bsky.social/lists/3lxdgjtpqhf2z)
+    if "/app.bsky.graph.list/" in post["quoted_uri"]:
+        post_splitter = "/lists/"
+    else:
+        post_splitter = "/post/"
+    post["quoted_url"] = format_post_url(
+        post["quoted_user_did"], post["quoted_did"], post_splitter=post_splitter
+    )
 
     quoted_data = None
     if card_data:
@@ -145,7 +185,9 @@ def prepare_quote_data(embed_quote, card_data, post, links):
 
         # Extract user handle from url
         if "did:plc:" not in post["quoted_url"]:
-            post["quoted_user_handle"], _ = parse_post_url(post["quoted_url"], post["url"])
+            post["quoted_user_handle"], _ = parse_post_url(
+                post["quoted_url"], post["url"]
+            )
 
     return (post, quoted_data, links)
 
@@ -176,7 +218,7 @@ def merge_nested_posts(referenced_posts, nested, source):
 
 @overload
 def normalize_post(
-    data: Dict,
+    payload: Dict,
     locale: Optional[str] = ...,
     extract_referenced_posts: Literal[True] = ...,
     collection_source: Optional[str] = ...,
@@ -185,7 +227,7 @@ def normalize_post(
 
 @overload
 def normalize_post(
-    data: Dict,
+    payload: Dict,
     locale: Optional[str] = ...,
     extract_referenced_posts: Literal[False] = ...,
     collection_source: Optional[str] = ...,
@@ -194,7 +236,7 @@ def normalize_post(
 
 def normalize_post(
     payload: Dict,
-    locale: Optional[str] = None,
+    locale: Optional[Any] = None,
     extract_referenced_posts: bool = False,
     collection_source: Optional[str] = None,
 ) -> Union[BlueskyPost, List[BlueskyPost]]:
@@ -308,7 +350,7 @@ def normalize_post(
         feat = facet["features"][0]
 
         # Hashtags
-        if feat["$type"].endswith("#tag"):
+        if feat["$type"].endswith("#tag") or feat["$type"].endswith("#hashtag"):
             hashtags.add(feat["tag"].strip().lower())
 
         # Mentions
@@ -323,7 +365,11 @@ def normalize_post(
                     byteStart = text.find(b"@", byteStart)
 
                 handle = (
-                    text[byteStart + 1 : facet["index"]["byteEnd"] + byteStart - facet["index"]["byteStart"]]
+                    text[
+                        byteStart + 1 : facet["index"]["byteEnd"]
+                        + byteStart
+                        - facet["index"]["byteStart"]
+                    ]
                     .strip()
                     .lower()
                     .decode("utf-8")
@@ -335,22 +381,36 @@ def normalize_post(
             # Handle native polls
             if "https://poll.blue/" in feat["uri"]:
                 if feat["uri"].endswith("/0"):
-                    links.add(custom_normalize_url(feat["uri"]))
+                    link = safe_normalize_url(feat["uri"])
+                    if is_url(link):
+                        links.add(link)
+                    else:
+                        continue
                     text += b" %s" % feat["uri"].encode("utf-8")
                 continue
 
-            links.add(custom_normalize_url(feat["uri"]))
+            link = safe_normalize_url(feat["uri"])
+            if is_url(link):
+                links.add(link)
+            else:
+                continue
             # Check & fix occasional errored link positioning
-            # example: https://bsky.app/profile/ecrime.ch/post/3lqotmopayr23
+            # examples: https://bsky.app/profile/ecrime.ch/post/3lqotmopayr23
+            #           https://bsky.app/profile/clustz.com/post/3lqfi7mnto52w
             byteStart = facet["index"]["byteStart"]
-            if b" " in text[byteStart : facet["index"]["byteEnd"]]:
-                byteStart = text.find(b"http", byteStart)
+
+            if not text[byteStart : facet["index"]["byteEnd"]].startswith(b"http"):
+                new_byteStart = text.find(b"http", byteStart, facet["index"]["byteEnd"])
+                if new_byteStart != -1:
+                    byteStart = new_byteStart
 
             links_to_replace.append(
                 {
                     "uri": feat["uri"].encode("utf-8"),
                     "start": byteStart,
-                    "end": byteStart - facet["index"]["byteStart"] + facet["index"]["byteEnd"],
+                    "end": byteStart
+                    - facet["index"]["byteStart"]
+                    + facet["index"]["byteEnd"],
                 }
             )
 
@@ -426,7 +486,7 @@ def normalize_post(
 
             # Extra card links sometimes missing from facets & text due to manual action in post form
             else:
-                extra_links.append(embed["external"]["uri"])
+                extra_links.append(link)
                 # Handle link card metadata
                 if "embed" in data:
                     post = process_card_data(data["embed"]["external"], post)
@@ -442,7 +502,9 @@ def normalize_post(
         # Quote & Starter-packs
         if embed["$type"].endswith(".record"):
             if "app.bsky.graph.starterpack" in embed["record"]["uri"]:
-                post = process_starterpack_card(data.get("embed", {}).get("record"), post)
+                post = process_starterpack_card(
+                    data.get("embed", {}).get("record"), post
+                )
                 if post["card_link"]:
                     extra_links.append(post["card_link"])
             else:
@@ -499,9 +561,10 @@ def normalize_post(
 
         # Process extra links
         for link in extra_links:
-            norm_link = custom_normalize_url(link)
+            norm_link = safe_normalize_url(link)
             if norm_link not in links:
-                links.add(norm_link)
+                if is_url(norm_link):
+                    links.add(norm_link)
                 text += b" " + link.encode("utf-8")
 
         # Process medias
@@ -523,7 +586,10 @@ def normalize_post(
 
                 # Rewrite post's text to include links to medias within
                 text += b" " + (
-                    media_thumb if media_type.startswith("video") and not media_type.endswith("/gif") else media_url
+                    media_thumb
+                    if media_type.startswith("video")
+                    and not media_type.endswith("/gif")
+                    else media_url
                 ).encode("utf-8")
 
         # Process quotes

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/types.py

@@ -10,7 +10,7 @@ class BlueskyProfile(TypedDict):
     url: str                            # URL of the profile accessible on the web
     handle: str                         # updatable human-readable username of the account (usually like username.bsky.social or username.com)
     display_name: Optional[str]         # updatable human-readable name of the account
-    description: str                    # profile short description written by the user
+    description: Optional[str]          # profile short description written by the user
     posts: int                          # total number of posts submitted by the user (at collection time)
     followers: int                      # total number of followers of the user (at collection time)
     follows: int                        # total number of other users followed by the user (at collection time)
@@ -18,12 +18,26 @@ class BlueskyProfile(TypedDict):
     feedgens: int                       # total number of custom feeds created by the user (at collection time)
     starter_packs: int                  # total number of starter packs created by the user (at collection time)
     avatar: Optional[str]               # URL to the image serving as avatar to the user
-    banner: str                         # URL to the image serving as profile banner to the user
+    banner: Optional[str]               # URL to the image serving as profile banner to the user
     pinned_post_uri: Optional[str]      # ATProto's internal URI to the post potentially pinned by the user to appear at the top of his posts on his profile
     created_at: str                     # datetime (potentially timezoned) of when the user created the account
     timestamp_utc: int                  # Unix UTC timestamp of when the user created the account
     collection_time: Optional[str]      # datetime (potentially timezoned) of when the data was normalized
 
+class BlueskyPartialProfile(TypedDict): # A partial version of the profile found in follower/follow profile payloads
+    did: str                            # persistent long-term identifier of the account
+    url: str                            # URL of the profile accessible on the web
+    handle: str                         # updatable human-readable username of the account (usually like username.bsky.social or username.com)
+    display_name: Optional[str]         # updatable human-readable name of the account
+    description: Optional[str]          # profile short description written by the user
+    lists: Optional[int]                # total number of lists created by the user (at collection time)
+    feedgens: Optional[int]             # total number of custom feeds created by the user (at collection time)
+    starter_packs: Optional[int]        # total number of starter packs created by the user (at collection time)
+    avatar: Optional[str]               # URL to the image serving as avatar to the user
+    created_at: str                     # datetime (potentially timezoned) of when the user created the account
+    timestamp_utc: int                  # Unix UTC timestamp of when the user created the account
+    collection_time: Optional[str]      # datetime (potentially timezoned) of when the data was normalized
+
 
 
 class BlueskyPost(TypedDict):
@@ -64,7 +78,7 @@ class BlueskyPost(TypedDict):
     # user_lists: int                   # not available from posts payloads
     user_langs: List[str]               # languages in which the author of the posts usually writes posts (declarative)
     user_avatar: Optional[str]          # URL to the image serving as avatar to the user who authored the post
-    user_created_at: str                # datetime (potentially timezoned) ofwhen the user who authored the post created the account
+    user_created_at: str                # datetime (potentially timezoned) of when the user who authored the post created the account
     user_timestamp_utc: int             # Unix UTC timestamp of when the user who authored the post created the account
 
     # Parent post identifying fields
@@ -102,27 +116,27 @@ class BlueskyPost(TypedDict):
     quoted_user_handle: Optional[str]   # updatable human-readable username of the account who authored the quoted post
     quoted_created_at: Optional[int]    # datetime (potentially timezoned) of when the quoted post was submitted
     quoted_timestamp_utc: Optional[int] # Unix UTC timestamp of when the quoted post was submitted
-    quoted_status: Optional[str]        # empty or "detached" when the author of the quoted post intentionnally required the quoting post not to be accessible from their own
+    quoted_status: Optional[str]        # empty or "detached" when the author of the quoted post intentionnally required the quoting post not to appear in the list of this post's quotes
 
     # Embedded elements metadata fields
     links: List[str]                    # list of URLs of all links shared within the post (including potentially the embedded card detailed below, but not the link to a potential quoted post)
-    domains: List[str]                  # list of domains of the links shared within the post (here a domain refer to a full hostname, including subdomains, for instance bluesky.com or medialab.sciencespo.fr)
+    domains: List[str]                  # list of domains of the links shared within the post (here a domain refers to a full hostname, including subdomains, for instance bluesky.com or medialab.sciencespo.fr)
     card_link: Optional[str]            # URL of the link displayed as a card within the post if any
     card_title: Optional[str]           # title of the webpage corresponding to the linkg diplayed as a card within the post if any
     card_description: Optional[str]     # description of the webpage corresponding to the linkg diplayed as a card within the post if any
-    card_thumbnail: Optional[str]       # image displayed as an illustration of the webpage corresponding to the linkg diplayed as a card within the post if any
-    media_urls: List[str]               # list of URLs to all medias (images, videos, gifs) embedded in the post
-    media_thumbnails: List[str]         # list of URLs to small thumbnail version of all medias (images, videos, gifs) embedded in the post
-    media_types: List[str]              # MIME types (such as image/jpeg, image/gif, video/mp4, etc.) of all medias (images, videos, gifs) embedded in the post
-    media_alt_texts: List[str]          # description texts of all medias (images, videos, gifs) embedded in the post
-    mentioned_user_dids: List[str]      # list of all persistent long-term identifier of the accounts adressed within the post (does not include users to which the post replied)
-    mentioned_user_handles: List[str]   # list of all updatable human-readable username of the accounts adressed within the post (does not include users to which the post replied)
+    card_thumbnail: Optional[str]       # image displayed as an illustration of the webpage corresponding to the link diplayed as a card within the post if any
+    media_urls: List[str]               # list of URLs to all media (images, videos, gifs) embedded in the post
+    media_thumbnails: List[str]         # list of URLs to small thumbnail version of all media (images, videos, gifs) embedded in the post
+    media_types: List[str]              # MIME types (such as image/jpeg, image/gif, video/mp4, etc.) of all media (images, videos, gifs) embedded in the post
+    media_alt_texts: List[str]          # description texts of all media (images, videos, gifs) embedded in the post
+    mentioned_user_dids: List[str]      # list of all persistent long-term identifiers of the accounts adressed within the post (does not include users to which the post replied)
+    mentioned_user_handles: List[str]   # list of all updatable human-readable usernames of the accounts adressed within the post (does not include users to which the post replied)
     hashtags: List[str]                 # list of all unique lowercased hashtags found within the post's text
 
     # Conversation rules fields
     replies_rules: Optional[List[str]]          # list of specific conversation rules set by the author for the current post (can be one or a combination of: disallow, allow_from_follower, allow_from_following, allow_from_mention, or allow_from_list: followed by a list of user DIDs)
     replies_rules_created_at: Optional[str]     # datetime (potentially timezoned) of when the user set the replies_rules
-    replies_rules_timestamp_utc: Optional[int]  # Unix UTC timestamp of when the userset the replies_rules
+    replies_rules_timestamp_utc: Optional[int]  # Unix UTC timestamp of when the user set the replies_rules
     hidden_replies_uris: Optional[List[str]]    # list of ATProto's internal URIs to posts who replied to the post, but where intentionnally marked as hidden by the current post's author
     # quotes_rule: Optional[str]                # not available from posts payloads, cf https://github.com/bluesky-social/atproto/issues/3712
     # quotes_rules_created_at: Optional[str]    # not available from posts payloads, cf https://github.com/bluesky-social/atproto/issues/3712
@@ -131,5 +145,5 @@ class BlueskyPost(TypedDict):
 
     # Extra fields linked to the data collection and processing
     collection_time: Optional[str]      # datetime (potentially timezoned) of when the data was normalized
-    collected_via: Optional[List[str]]  # extra field added by the normalization process to express how the data collection was ran, will be "quote" or "thread" when a post was grabbed as a referenced post within a really collected post using the "extract_referenced_posts" option of "normalize_post"
-    match_query: Optional[bool]         # extra field added by the normalization process to express whether the post was an intentionnally collected one or only came as a referenced post within a really collected post using the "extract_referenced_posts" option of "normalize_post"
+    collected_via: Optional[List[str]]  # extra field added by the normalization process to express how the data collection was ran, will be "quote" or "thread" when a post was grabbed as a referenced post within the originally collected post using the "extract_referenced_posts" option of "normalize_post"
+    match_query: Optional[bool]         # extra field added by the normalization process to express whether the post was an intentionnally collected one or only came as a referenced post within the originally collected post using the "extract_referenced_posts" option of "normalize_post"

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/bluesky/utils.py

@@ -55,7 +55,9 @@ def validate_post_payload(data):
     return True, None
 
 
-re_embed_types = re.compile(r"\.(record|recordWithMedia|images|video|external)$")
+re_embed_types = re.compile(
+    r"\.(record|recordWithMedia|images|video|external)(?:#.*)?$"
+)
 
 
 def valid_embed_type(embed_type):
@@ -66,29 +68,39 @@ def format_profile_url(user_handle_or_did):
     return f"https://bsky.app/profile/{user_handle_or_did}"
 
 
-def format_post_url(user_handle_or_did, post_did):
-    return f"https://bsky.app/profile/{user_handle_or_did}/post/{post_did}"
+def format_post_url(user_handle_or_did, post_did, post_splitter="/post/"):
+    return f"https://bsky.app/profile/{user_handle_or_did}{post_splitter}{post_did}"
 
 
 def parse_post_url(url, source):
     """Returns a tuple of (author_handle/did, post_did) from an https://bsky.app post URL"""
 
-    if not url.startswith("https://bsky.app/profile/") and "/post/" not in url:
-        raise BlueskyPayloadError(source, f"{url} is not a usual Bluesky post url")
-    return url[25:].split("/post/")
+    known_splits = ["/post/", "/lists/"]
+
+    if url.startswith("https://bsky.app/profile/"):
+        for split in known_splits:
+            if split in url[25:]:
+                return url[25:].split(split)
+
+    raise BlueskyPayloadError(source, f"{url} is not a usual Bluesky post url")
 
 
 def parse_post_uri(uri, source=None):
     """Returns a tuple of (author_did, post_did) from an at:// post URI"""
 
-    if uri.startswith("at://") and "/app.bsky.graph.starterpack/" in uri:
-        return uri[5:].split("/app.bsky.graph.starterpack/")
+    known_splits = [
+        "/app.bsky.feed.post/",
+        "/app.bsky.graph.starterpack/",
+        "/app.bsky.feed.generator/",
+        "/app.bsky.graph.list/",
+    ]
 
-    if not uri.startswith("at://") and "/app.bsky.feed.post/" not in uri:
-        raise BlueskyPayloadError(
-            source or uri, f"{uri} is not a usual Bluesky post uri"
-        )
-    return uri[5:].split("/app.bsky.feed.post/")
+    if uri.startswith("at://"):
+        for split in known_splits:
+            if split in uri:
+                return uri[5:].split(split)
+
+    raise BlueskyPayloadError(source or uri, f"{uri} is not a usual Bluesky post uri")
 
 
 def format_starterpack_url(user_handle_or_did, record_did):
@@ -105,6 +117,13 @@ def format_media_url(user_did, media_cid, mime_type, source):
         media_thumb = (
             f"https://video.bsky.app/watch/{user_did}/{media_cid}/thumbnail.jpg"
         )
+    elif mime_type == "application/octet-stream":
+        media_url = (
+            f"https://cdn.bsky.app/img/feed_fullsize/plain/{user_did}/{media_cid}@jpeg"
+        )
+        media_thumb = (
+            f"https://cdn.bsky.app/img/feed_thumbnail/plain/{user_did}/{media_cid}@jpeg"
+        )
     else:
-        raise BlueskyPayloadError(source, f"{mime_type} is an usual media mimeType")
+        raise BlueskyPayloadError(source, f"{mime_type} is an unusual media mimeType")
     return media_url, media_thumb

{twitwi-0.21.1 → twitwi-0.22.0}/twitwi/utils.py

@@ -4,6 +4,8 @@
 #
 # Miscellaneous utility functions.
 #
+from typing import Tuple
+
 from pytz import timezone
 from dateutil.parser import parse as parse_date
 from ural import normalize_url, get_normalized_hostname
@@ -25,6 +27,21 @@ UTC_TIMEZONE = timezone("UTC")
 
 custom_normalize_url = partial(normalize_url, **CANONICAL_URL_KWARGS)
 
+
+def safe_normalize_url(url):
+    # We avoid normalizing bluesky urls containing specific uri parts because
+    # Bluesky servers don't handle quoting correctly...
+    # See https://github.com/medialab/twitwi/issues/72
+    if "/did:plc:" in url:
+        return url
+
+    try:
+        return custom_normalize_url(url)
+    except Exception:
+        # In case of error, return the original URL. Possibly not a valid URL, e.g. url containing double slashes
+        return url
+
+
 custom_get_normalized_hostname = partial(
     get_normalized_hostname, **CANONICAL_HOSTNAME_KWARGS
 )
@@ -34,7 +51,9 @@ def get_collection_time():
     return datetime.now().strftime(FORMATTED_FULL_DATETIME_FORMAT)
 
 
-def get_dates(date_str, locale=None, source="v1"):
+def get_dates(
+    date_str: str, locale=None, source: str = "v1", millisecond_timestamp: bool = False
+) -> Tuple[int, str]:
     if source not in ["v1", "v2", "bluesky"]:
         raise Exception("source should be one of v1, v2 or bluesky")
 
@@ -56,8 +75,14 @@ def get_dates(date_str, locale=None, source="v1"):
         utc_datetime = UTC_TIMEZONE.localize(parsed_datetime)
     locale_datetime = utc_datetime.astimezone(locale)
 
+    timestamp = int(utc_datetime.timestamp())
+
+    if millisecond_timestamp:
+        timestamp *= 1000
+        timestamp += utc_datetime.microsecond / 1000
+
     return (
-        int(utc_datetime.timestamp()),
+        int(timestamp),
         datetime.strftime(
             locale_datetime,
             FORMATTED_FULL_DATETIME_FORMAT
@@ -106,6 +131,7 @@ def get_dates_from_id(tweet_id, locale=None):
         locale = UTC_TIMEZONE
 
     timestamp = get_timestamp_from_id(tweet_id)
+    assert timestamp is not None
 
     locale_datetime = datetime.fromtimestamp(timestamp, locale)
 

{twitwi-0.21.1 → twitwi-0.22.0/twitwi.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: twitwi
-Version: 0.21.1
+Version: 0.22.0
 Summary: A collection of Twitter-related helper functions for python.
 Home-page: http://github.com/medialab/twitwi
 Author: Béatrice Mazoyer, Guillaume Plique, Benjamin Ooghe-Tabanou
@@ -56,18 +56,22 @@ pip install twitwi
 *Normalization functions*
 
 * [normalize_profile](#normalize_profile)
+* [normalize_partial_profile](#normalize_partial_profile)
 * [normalize_post](#normalize_post)
 
 *Formatting functions*
 
 * [transform_profile_into_csv_dict](#transform_profile_into_csv_dict)
+* [transform_partial_profile_into_csv_dict](#transform_partial_profile_into_csv_dict)
 * [transform_post_into_csv_dict](#transform_post_into_csv_dict)
 * [format_profile_as_csv_row](#format_profile_as_csv_row)
+* [format_partial_profile_as_csv_row](#format_partial_profile_as_csv_row)
 * [format_post_as_csv_row](#format_post_as_csv_row)
 
 *Useful constants (under `twitwi.bluesky.constants`)*
 
 * [PROFILE_FIELDS](#profile_fields)
+* [PARTIAL_PROFILE_FIELDS](#partial_profile_fields)
 * [POST_FIELDS](#post_fields)
 
 *Examples*
@@ -95,7 +99,7 @@ for post_data in posts_payload_from_API:
 
 # Then, saving normalized profiles into a CSV using DictWriter:
 
-from csv import DictWriter
+import csv
 from twitwi.bluesky.constants import POST_FIELDS
 from twitwi.bluesky import transform_post_into_csv_dict
 
@@ -108,7 +112,6 @@ with open("normalized_bluesky_posts.csv", "w") as f:
 
 # Or using the basic CSV writer:
 
-from csv import writer
 from twitwi.bluesky import format_post_as_csv_row
 
 with open("normalized_bluesky_posts.csv", "w") as f:
@@ -180,7 +183,18 @@ with open("normalized_bluesky_profiles.csv", "w") as f:
 
 ### normalize_profile
 
-Function taking a nested dict describing a user profile from Bluesky's JSON payload and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys.
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses)) and returning a flat "normalized" dict composed of all [PROFILE_FIELDS](#profile_fields) keys. Be careful not to confuse with the [normalize_partial_profile](#normalize_partial_profile) function which operate on a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
+
+*Arguments*
+
+* **data** *(dict)*: user profile data payload coming from Bluesky API.
+* **locale** *(pytz.timezone as str, optional)*: timezone used to convert dates. If not given, will default to UTC.
+
+### normalize_partial_profile
+
+Function taking a nested dict describing a user profile from Bluesky's JSON payload (with the same format as retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses)) and returning a flat "normalized" dict composed of all [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) keys. Be careful not to confuse with the [normalize_profile](#normalize_profile) function which operate on the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 Will return datetimes as UTC but can take an optional second `locale` argument as a [`pytz`](https://pypi.org/project/pytz/) string timezone.
 
@@ -210,6 +224,12 @@ Function transforming (i.e. mutating, so beware) a given normalized Bluesky prof
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### transform_partial_profile_into_csv_dict
+
+Function transforming (i.e. mutating, so beware) a given normalized Bluesky partial profile into a suitable dict able to be written by a `csv.DictWriter` as a row.
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### transform_post_into_csv_dict
 
 Function transforming (i.e. mutating, so beware) a given normalized Bluesky post into a suitable dict able to be written by a `csv.DictWriter` as a row.
@@ -222,6 +242,12 @@ Function formatting the given normalized Bluesky profile as a list able to be wr
 
 Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
 
+### format_partial_profile_as_csv_row
+
+Function formatting the given normalized Bluesky partial profile as a list able to be written by a `csv.writer` as a row in the order of [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) (which can therefore be used as header row of the CSV).
+
+Will convert list elements of the normalized data into a string with all elements separated by the `|` character, which can be changed using an optional `plural_separator` argument.
+
 ### format_post_as_csv_row
 
 Function formatting the given normalized tBluesky post as a list able to be written by a `csv.writer` as a row in the order of [POST_FIELDS](#post_fields) (which can therefore be used as header row of the CSV).
@@ -230,7 +256,11 @@ Will convert list elements of the normalized data into a string with all element
 
 ### PROFILE_FIELDS
 
-List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers.
+List of a Bluesky user profile's normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PARTIAL_PROFILE_FIELDS](#partial_profile_fields) which correspond to a lighter version of the profile data, retrieved from [follower/follow profile payloads](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example.
+
+### PARTIAL_PROFILE_FIELDS
+
+List of a Bluesky user partial profile's (retrieved from [`app.bsky.graph.getFollowers` HTTP endpoint](https://docs.bsky.app/docs/api/app-bsky-graph-get-followers#responses) for example) normalized field names. Useful to declare headers with csv writers. Be careful not to confuse with [PROFILE_FIELDS](#profile_fields) which correspond to the full version of the profile data, retrieved from [`app.bsky.actor.getProfiles` HTTP endpoint](docs.bsky.app/docs/api/app-bsky-actor-get-profiles#responses) for example.
 
 ### POST_FIELDS