mapillary-downloader 0.7.7__tar.gz → 0.8.0__tar.gz

{mapillary_downloader-0.7.7 → mapillary_downloader-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mapillary_downloader
-Version: 0.7.7
+Version: 0.8.0
 Summary: Archive user data from Mapillary
 Author-email: Gareth Davidson <gaz@bitplane.net>
 Requires-Python: >=3.10
@@ -32,7 +32,7 @@ Provides-Extra: dev
 
 Download your Mapillary data before it's gone.
 
-## Installation
+## ▶️ Installation
 
 Installation is optional, you can prefix the command with `uvx` or `pipx` to
 download and run it. Or if you're oldskool you can do:
@@ -41,7 +41,7 @@ download and run it. Or if you're oldskool you can do:
 pip install mapillary-downloader
 ```
 
-## Usage
+## ❓ Usage
 
 First, get your Mapillary API access token from
 [the developer dashboard](https://www.mapillary.com/dashboard/developers)
@@ -75,12 +75,14 @@ The downloader will:
 * 🏛️ Check Internet Archive to avoid duplicate downloads
 * 📷 Download multiple users' images organized by sequence
 * 📜 Inject EXIF metadata (GPS coordinates, camera info, timestamps,
-  compass direction)
+     compass direction) and XMP data for panoramas.
 * 🗜️ Convert to WebP (by default) to save ~70% disk space
-* 🛟 Save progress so you can safely resume if interrupted
-* 📦 Tar sequence directories (by default) for faster uploads to Internet Archive
+* 🛟 Save progress every 5 minutes so you can safely resume if interrupted
+     ()
+* 📦 Tar sequence directories (by default) for faster uploads to Internet
+     Archive
 
-## WebP Conversion
+## 🖼️ WebP Conversion
 
 You'll need the `cwebp` binary installed:
 
@@ -94,11 +96,7 @@ brew install webp
 
 To disable WebP conversion and keep original JPEGs, use `--no-webp`:
 
-```bash
-mapillary-downloader --no-webp USERNAME
-```
-
-## Tarballs
+## 📦 Tarballs
 
 Images are organized by capture date (YYYY-MM-DD) for incremental archiving:
 
@@ -116,16 +114,20 @@ mapillary-username-quality/
 ```
 
 By default, these date directories are automatically tarred after download
-(resulting in `2024-01-15.tar`, `2024-01-16.tar`, etc.). This date-based
-organization enables:
+(`2024-01-15.tar`, `2024-01-16.tar`, etc.). Reasons:
 
-- **Incremental uploads** - Upload each day's tar as soon as it's ready
-- **Manageable file counts** - ~365 days/year × 10 years = 3,650 tars max
-- **Chronological organization** - Natural sorting and progress tracking
+* ⤴️ Incremental uploads. Add more to a collection. Well, eventually anyway.
+     This won't work yet unless you delete the jsonl file and start again.
+* 📂 Fewer files - ~365 days/year × 10 years = 3,650 tars max. IA only want
+     5k items per collection
+* 🧨 Avoids blowing up IA's derive workers. We don't want Brewster's computers
+     to create thumbs for 2 billion images.
+* 💾 I like to have a few inodes available for things other than this. I'm sure
+     you do too.
 
 To keep individual files instead of creating tars, use the `--no-tar` flag.
 
-## Internet Archive upload
+## 🏛️ Internet Archive upload
 
 I've written a bash tool to rip media then tag, queue, and upload to The
 Internet Archive. The metadata is in the same format. If you symlink your
@@ -139,15 +141,11 @@ See inlay for details:
 
 To see overall project progress, or an estimate, use `--stats`
 
-```bash
-mapillary-downloader --stats
-```
-
 ## 🚧 Development
 
 ```bash
 make dev      # Setup dev environment
-make test     # Run tests
+make test     # Run tests. Note: requires `exiftool`
 make dist     # Build the distribution
 make help     # See other make options
 ```
@@ -160,12 +158,12 @@ make help     # See other make options
 * [🐱 github](https://github.com/bitplane/mapillary_downloader)
 * [📀 rip](https://bitplane.net/dev/sh/rip)
 
-## License
+## ⚖️ License
 
 WTFPL with one additional clause
 
 1. Don't blame me
 
 Do wtf you want, but don't blame me if it makes jokes about the size of your
-disk drive.
+disk.
 

{mapillary_downloader-0.7.7 → mapillary_downloader-0.8.0}/README.md

@@ -2,7 +2,7 @@
 
 Download your Mapillary data before it's gone.
 
-## Installation
+## ▶️ Installation
 
 Installation is optional, you can prefix the command with `uvx` or `pipx` to
 download and run it. Or if you're oldskool you can do:
@@ -11,7 +11,7 @@ download and run it. Or if you're oldskool you can do:
 pip install mapillary-downloader
 ```
 
-## Usage
+## ❓ Usage
 
 First, get your Mapillary API access token from
 [the developer dashboard](https://www.mapillary.com/dashboard/developers)
@@ -45,12 +45,14 @@ The downloader will:
 * 🏛️ Check Internet Archive to avoid duplicate downloads
 * 📷 Download multiple users' images organized by sequence
 * 📜 Inject EXIF metadata (GPS coordinates, camera info, timestamps,
-  compass direction)
+     compass direction) and XMP data for panoramas.
 * 🗜️ Convert to WebP (by default) to save ~70% disk space
-* 🛟 Save progress so you can safely resume if interrupted
-* 📦 Tar sequence directories (by default) for faster uploads to Internet Archive
+* 🛟 Save progress every 5 minutes so you can safely resume if interrupted
+     ()
+* 📦 Tar sequence directories (by default) for faster uploads to Internet
+     Archive
 
-## WebP Conversion
+## 🖼️ WebP Conversion
 
 You'll need the `cwebp` binary installed:
 
@@ -64,11 +66,7 @@ brew install webp
 
 To disable WebP conversion and keep original JPEGs, use `--no-webp`:
 
-```bash
-mapillary-downloader --no-webp USERNAME
-```
-
-## Tarballs
+## 📦 Tarballs
 
 Images are organized by capture date (YYYY-MM-DD) for incremental archiving:
 
@@ -86,16 +84,20 @@ mapillary-username-quality/
 ```
 
 By default, these date directories are automatically tarred after download
-(resulting in `2024-01-15.tar`, `2024-01-16.tar`, etc.). This date-based
-organization enables:
+(`2024-01-15.tar`, `2024-01-16.tar`, etc.). Reasons:
 
-- **Incremental uploads** - Upload each day's tar as soon as it's ready
-- **Manageable file counts** - ~365 days/year × 10 years = 3,650 tars max
-- **Chronological organization** - Natural sorting and progress tracking
+* ⤴️ Incremental uploads. Add more to a collection. Well, eventually anyway.
+     This won't work yet unless you delete the jsonl file and start again.
+* 📂 Fewer files - ~365 days/year × 10 years = 3,650 tars max. IA only want
+     5k items per collection
+* 🧨 Avoids blowing up IA's derive workers. We don't want Brewster's computers
+     to create thumbs for 2 billion images.
+* 💾 I like to have a few inodes available for things other than this. I'm sure
+     you do too.
 
 To keep individual files instead of creating tars, use the `--no-tar` flag.
 
-## Internet Archive upload
+## 🏛️ Internet Archive upload
 
 I've written a bash tool to rip media then tag, queue, and upload to The
 Internet Archive. The metadata is in the same format. If you symlink your
@@ -109,15 +111,11 @@ See inlay for details:
 
 To see overall project progress, or an estimate, use `--stats`
 
-```bash
-mapillary-downloader --stats
-```
-
 ## 🚧 Development
 
 ```bash
 make dev      # Setup dev environment
-make test     # Run tests
+make test     # Run tests. Note: requires `exiftool`
 make dist     # Build the distribution
 make help     # See other make options
 ```
@@ -130,11 +128,11 @@ make help     # See other make options
 * [🐱 github](https://github.com/bitplane/mapillary_downloader)
 * [📀 rip](https://bitplane.net/dev/sh/rip)
 
-## License
+## ⚖️ License
 
 WTFPL with one additional clause
 
 1. Don't blame me
 
 Do wtf you want, but don't blame me if it makes jokes about the size of your
-disk drive.
+disk.

{mapillary_downloader-0.7.7 → mapillary_downloader-0.8.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "mapillary_downloader"
 description = "Archive user data from Mapillary"
-version = "0.7.7"
+version = "0.8.0"
 authors = [
     { name = "Gareth Davidson", email = "gaz@bitplane.net" }
 ]

{mapillary_downloader-0.7.7 → mapillary_downloader-0.8.0}/src/mapillary_downloader/exif_writer.py

@@ -72,9 +72,6 @@ def write_exif_to_image(image_path, metadata):
         if "model" in metadata and metadata["model"]:
             exif_dict["0th"][piexif.ImageIFD.Model] = metadata["model"].encode("utf-8")
 
-        if "exif_orientation" in metadata and metadata["exif_orientation"]:
-            exif_dict["0th"][piexif.ImageIFD.Orientation] = metadata["exif_orientation"]
-
         if "width" in metadata and metadata["width"]:
             exif_dict["0th"][piexif.ImageIFD.ImageWidth] = metadata["width"]
 
@@ -88,6 +85,8 @@ def write_exif_to_image(image_path, metadata):
             exif_dict["0th"][piexif.ImageIFD.DateTime] = datetime_bytes
             exif_dict["Exif"][piexif.ExifIFD.DateTimeOriginal] = datetime_bytes
             exif_dict["Exif"][piexif.ExifIFD.DateTimeDigitized] = datetime_bytes
+            exif_dict["Exif"][piexif.ExifIFD.SubSecTimeOriginal] = ("000" + str(metadata["captured_at"] % 1000))[-3:]
+            exif_dict["Exif"][piexif.ExifIFD.SubSecTimeDigitized] = ("000" + str(metadata["captured_at"] % 1000))[-3:]
 
         # GPS data - prefer computed_geometry over geometry
         geometry = metadata.get("computed_geometry") or metadata.get("geometry")
@@ -102,8 +101,8 @@ def write_exif_to_image(image_path, metadata):
             exif_dict["GPS"][piexif.GPSIFD.GPSLongitude] = decimal_to_dms(lon)
             exif_dict["GPS"][piexif.GPSIFD.GPSLongitudeRef] = b"E" if lon >= 0 else b"W"
 
-        # GPS Altitude - prefer computed_altitude over altitude
-        altitude = metadata.get("computed_altitude") or metadata.get("altitude")
+        # GPS Altitude - prefer raw altitude (photogrammetry can't compute elevation)
+        altitude = metadata.get("altitude") or metadata.get("computed_altitude")
         if altitude is not None:
             altitude_val = int(abs(altitude) * 100)
             logger.debug(f"Raw altitude value: {altitude}, calculated: {altitude_val}")

{mapillary_downloader-0.7.7 → mapillary_downloader-0.8.0}/src/mapillary_downloader/worker.py

@@ -7,6 +7,7 @@ from datetime import datetime
 from pathlib import Path
 import requests
 from mapillary_downloader.exif_writer import write_exif_to_image
+from mapillary_downloader.xmp_writer import write_xmp_to_image
 from mapillary_downloader.webp_converter import convert_to_webp
 from mapillary_downloader.utils import http_get_with_retry
 
@@ -117,6 +118,9 @@ def download_and_convert_image(image_data, output_dir, quality, convert_webp, se
         # Write EXIF metadata
         write_exif_to_image(jpg_path, image_data)
 
+        # Write XMP metadata for panoramas
+        write_xmp_to_image(jpg_path, image_data)
+
         # Convert to WebP if requested
         if convert_webp:
             webp_path = convert_to_webp(jpg_path, output_path=final_path, delete_original=False)

mapillary_downloader-0.8.0/src/mapillary_downloader/xmp_writer.py

@@ -0,0 +1,154 @@
+"""XMP metadata writer for panoramic Mapillary images."""
+
+import logging
+
+logger = logging.getLogger("mapillary_downloader")
+
+# XMP namespace identifier for APP1 segment
+XMP_NAMESPACE = b"http://ns.adobe.com/xap/1.0/\x00"
+
+# XMP packet template for GPano metadata
+XMP_TEMPLATE = """<?xpacket begin="\ufeff" id="W5M0MpCehiHzreSzNTczkc9d"?>
+<x:xmpmeta xmlns:x="adobe:ns:meta/">
+  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+    <rdf:Description rdf:about=""
+      xmlns:GPano="http://ns.google.com/photos/1.0/panorama/"
+      GPano:ProjectionType="equirectangular"
+      GPano:UsePanoramaViewer="True"
+      GPano:FullPanoWidthPixels="{width}"
+      GPano:FullPanoHeightPixels="{height}"
+      GPano:CroppedAreaImageWidthPixels="{width}"
+      GPano:CroppedAreaImageHeightPixels="{height}"
+      GPano:CroppedAreaLeftPixels="0"
+      GPano:CroppedAreaTopPixels="0"{pose_heading}/>
+  </rdf:RDF>
+</x:xmpmeta>
+<?xpacket end="w"?>"""
+
+
+def build_xmp_packet(metadata):
+    """Build XMP packet with GPano metadata.
+
+    Args:
+        metadata: Dictionary with width, height, and optionally compass_angle
+
+    Returns:
+        XMP XML string
+    """
+    width = metadata.get("width", 0)
+    height = metadata.get("height", 0)
+
+    # Get compass angle (prefer computed)
+    compass = metadata.get("computed_compass_angle") or metadata.get("compass_angle")
+
+    # Build pose heading attribute if available
+    if compass is not None:
+        pose_heading = f'\n      GPano:PoseHeadingDegrees="{compass:.1f}"'
+    else:
+        pose_heading = ""
+
+    return XMP_TEMPLATE.format(
+        width=width,
+        height=height,
+        pose_heading=pose_heading,
+    )
+
+
+def write_xmp_to_image(image_path, metadata):
+    """Write XMP GPano metadata to a JPEG image for panoramas.
+
+    Only writes metadata if is_pano is True in the metadata dict.
+
+    Args:
+        image_path: Path to the JPEG image file
+        metadata: Dictionary of metadata from Mapillary API
+
+    Returns:
+        True if XMP was written, False if skipped or failed
+    """
+    # Only write XMP for panoramas
+    if not metadata.get("is_pano"):
+        return False
+
+    # Need dimensions to write meaningful GPano data
+    if not metadata.get("width") or not metadata.get("height"):
+        logger.warning(f"Skipping XMP for {image_path}: missing dimensions")
+        return False
+
+    try:
+        # Read the JPEG file
+        with open(image_path, "rb") as f:
+            data = f.read()
+
+        # Verify JPEG signature
+        if data[:2] != b"\xff\xd8":
+            logger.warning(f"Skipping XMP for {image_path}: not a valid JPEG")
+            return False
+
+        # Build XMP packet
+        xmp_xml = build_xmp_packet(metadata)
+        xmp_bytes = xmp_xml.encode("utf-8")
+
+        # Build APP1 segment with XMP namespace
+        xmp_segment = XMP_NAMESPACE + xmp_bytes
+        segment_length = len(xmp_segment) + 2  # +2 for length bytes
+
+        if segment_length > 65535:
+            logger.warning(f"Skipping XMP for {image_path}: XMP too large")
+            return False
+
+        # APP1 marker (0xFFE1) + length + data
+        app1_marker = b"\xff\xe1"
+        length_bytes = segment_length.to_bytes(2, byteorder="big")
+        full_segment = app1_marker + length_bytes + xmp_segment
+
+        # Find insertion point - after SOI (0xFFD8) and any existing APP0/APP1 segments
+        # We want to insert after EXIF APP1 but before other segments
+        pos = 2  # Skip SOI
+
+        while pos < len(data) - 1:
+            if data[pos] != 0xFF:
+                break
+
+            marker = data[pos + 1]
+
+            # Stop at SOS (start of scan) or non-marker data
+            if marker == 0xDA or marker == 0x00:
+                break
+
+            # Check if this is an APP1 with XMP namespace (skip if exists)
+            if marker == 0xE1:  # APP1
+                seg_len = int.from_bytes(data[pos + 2 : pos + 4], byteorder="big")
+                seg_data = data[pos + 4 : pos + 2 + seg_len]
+                if seg_data.startswith(XMP_NAMESPACE):
+                    # XMP already exists, replace it
+                    new_data = data[:pos] + full_segment + data[pos + 2 + seg_len :]
+                    with open(image_path, "wb") as f:
+                        f.write(new_data)
+                    logger.debug(f"Replaced XMP in {image_path}")
+                    return True
+                # Skip this APP1 (probably EXIF)
+                pos += 2 + seg_len
+                continue
+
+            # Skip APP0 (JFIF) segments
+            if marker == 0xE0:  # APP0
+                seg_len = int.from_bytes(data[pos + 2 : pos + 4], byteorder="big")
+                pos += 2 + seg_len
+                continue
+
+            # Found a different marker, insert XMP here
+            break
+
+        # Insert XMP segment at current position
+        new_data = data[:pos] + full_segment + data[pos:]
+
+        with open(image_path, "wb") as f:
+            f.write(new_data)
+
+        logger.debug(f"Wrote XMP GPano metadata to {image_path}")
+        return True
+
+    except Exception as e:
+        logger.warning(f"Failed to write XMP to {image_path}: {e}")
+        return False