iporigin 1.1.0__tar.gz → 1.1.2__tar.gz

{iporigin-1.1.0 → iporigin-1.1.2}/NOTICE

@@ -28,6 +28,14 @@ consumed programmatically. No separate permission is needed or claimed.
 
 CC0 is a public-domain dedication, so these carry no conditions.
 
+The AI-company bot slugs probed for outbound crawler coverage
+(OpenAI, Perplexity AI, DuckAssistBot, Apple Intelligence Proxy,
+Meta) come from these same two repositories. The set above is
+what was probed; only Meta adds ranges the disjoint sweep does not
+collapse into an existing bot or vpn label, and the rest overlap
+exactly with GPTBot / PerplexityBot / DuckDuckBot / Apple Private
+Relay and answer under those provider names.
+
 
 3. Used with the maintainer's permission
 ----------------------------------------

{iporigin-1.1.0 → iporigin-1.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: iporigin
-Version: 1.1.0
+Version: 1.1.2
 Summary: Is this IP a datacenter, VPN, Tor exit or a home connection? Offline lookup, no API key.
 Author-email: Yuix Networks <info@yuix.org>
 License: MIT
@@ -91,7 +91,7 @@ origin.is_datacenter # True
 | `cdn` | edge infrastructure fronting other people's sites | Cloudflare, Akamai |
 | `vpn` | a consumer VPN or private relay exit | Mullvad, ProtonVPN, Apple Private Relay |
 | `tor` | a Tor exit node | |
-| `bot` | a declared crawler | Googlebot, GPTBot, ClaudeBot |
+| `bot` | a declared crawler or an AI company's outbound IP | Googlebot, GPTBot, GPTBot (OpenAI's crawler), Meta, ... |
 | `reserved` | private, loopback, link-local, documentation | `127.0.0.1`, `10.0.0.0/8` |
 | `unknown` | in none of our lists — most often a residential ISP | |
 
@@ -185,7 +185,11 @@ and are second-hand by definition:
 
 Covering Hetzner, OVHcloud, Scaleway, Alibaba Cloud, Leaseweb, UpCloud, IBM
 Cloud, Huawei Cloud, Tencent Cloud, Rackspace, Akamai, Gcore, Mullvad,
-ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers.
+ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers. AI-company
+outbound ranges (OpenAI, Perplexity AI, DuckAssistBot, Apple Intelligence
+Proxy, Meta) come from these same two repositories. Most of them overlap
+exactly with an existing bot label, so they contribute label diversity
+rather than new bot coverage — only Meta adds distinct segments.
 
 Both are CC0, a public-domain dedication, so they carry no conditions.
 
@@ -207,15 +211,24 @@ covered. It is a subset of the provider feeds it was itself built from, and
 a source that adds nothing is still another endpoint that can break the
 weekly rebuild.
 
-About 454,000 published prefixes collapse into 38,379 disjoint ranges
-(21,901 IPv4, 16,478 IPv6) covering 287 million IPv4 addresses. Rebuild it
-yourself at any time:
+About 454,000 published prefixes collapse into roughly 38,400 disjoint
+ranges (about 21,900 IPv4 and 16,500 IPv6) covering 287 million IPv4
+addresses. The exact figures move every week with the feeds; the build
+prints them. Rebuild it yourself at any time:
 
 ```
 python tools/build_dataset.py
 ```
 
-A GitHub Action re-runs that weekly and opens a PR when the ranges move.
+A GitHub Action re-runs that weekly and commits the result when the ranges
+move, and a second one cuts a patch release on the 6th of each month if the
+dataset changed since the last release — so `pip install --upgrade iporigin`
+is never more than about a month behind the feeds, and a quiet month
+produces no release rather than a version whose only content is a new
+number. The build refuses to replace the committed dataset if it shrinks by
+more than 20% — a feed that starts answering with an empty body looks
+exactly like a provider giving up its address space, and nothing else
+would catch it. Pass `--allow-shrink` when the drop is genuine.
 
 ### Known gaps
 

{iporigin-1.1.0 → iporigin-1.1.2}/README.md

@@ -56,7 +56,7 @@ origin.is_datacenter # True
 | `cdn` | edge infrastructure fronting other people's sites | Cloudflare, Akamai |
 | `vpn` | a consumer VPN or private relay exit | Mullvad, ProtonVPN, Apple Private Relay |
 | `tor` | a Tor exit node | |
-| `bot` | a declared crawler | Googlebot, GPTBot, ClaudeBot |
+| `bot` | a declared crawler or an AI company's outbound IP | Googlebot, GPTBot, GPTBot (OpenAI's crawler), Meta, ... |
 | `reserved` | private, loopback, link-local, documentation | `127.0.0.1`, `10.0.0.0/8` |
 | `unknown` | in none of our lists — most often a residential ISP | |
 
@@ -150,7 +150,11 @@ and are second-hand by definition:
 
 Covering Hetzner, OVHcloud, Scaleway, Alibaba Cloud, Leaseweb, UpCloud, IBM
 Cloud, Huawei Cloud, Tencent Cloud, Rackspace, Akamai, Gcore, Mullvad,
-ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers.
+ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers. AI-company
+outbound ranges (OpenAI, Perplexity AI, DuckAssistBot, Apple Intelligence
+Proxy, Meta) come from these same two repositories. Most of them overlap
+exactly with an existing bot label, so they contribute label diversity
+rather than new bot coverage — only Meta adds distinct segments.
 
 Both are CC0, a public-domain dedication, so they carry no conditions.
 
@@ -172,15 +176,24 @@ covered. It is a subset of the provider feeds it was itself built from, and
 a source that adds nothing is still another endpoint that can break the
 weekly rebuild.
 
-About 454,000 published prefixes collapse into 38,379 disjoint ranges
-(21,901 IPv4, 16,478 IPv6) covering 287 million IPv4 addresses. Rebuild it
-yourself at any time:
+About 454,000 published prefixes collapse into roughly 38,400 disjoint
+ranges (about 21,900 IPv4 and 16,500 IPv6) covering 287 million IPv4
+addresses. The exact figures move every week with the feeds; the build
+prints them. Rebuild it yourself at any time:
 
 ```
 python tools/build_dataset.py
 ```
 
-A GitHub Action re-runs that weekly and opens a PR when the ranges move.
+A GitHub Action re-runs that weekly and commits the result when the ranges
+move, and a second one cuts a patch release on the 6th of each month if the
+dataset changed since the last release — so `pip install --upgrade iporigin`
+is never more than about a month behind the feeds, and a quiet month
+produces no release rather than a version whose only content is a new
+number. The build refuses to replace the committed dataset if it shrinks by
+more than 20% — a feed that starts answering with an empty body looks
+exactly like a provider giving up its address space, and nothing else
+would catch it. Pass `--allow-shrink` when the drop is genuine.
 
 ### Known gaps
 

{iporigin-1.1.0 → iporigin-1.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "iporigin"
-version = "1.1.0"
+version = "1.1.2"
 description = "Is this IP a datacenter, VPN, Tor exit or a home connection? Offline lookup, no API key."
 readme = "README.md"
 requires-python = ">=3.8"

{iporigin-1.1.0 → iporigin-1.1.2}/src/iporigin/__init__.py

@@ -29,7 +29,7 @@ from .core import (
     is_datacenter,
 )
 
-__version__ = "1.1.0"
+__version__ = "1.1.2"
 
 __all__ = [
     "ANONYMIZER_KINDS",

{iporigin-1.1.0 → iporigin-1.1.2}/src/iporigin.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: iporigin
-Version: 1.1.0
+Version: 1.1.2
 Summary: Is this IP a datacenter, VPN, Tor exit or a home connection? Offline lookup, no API key.
 Author-email: Yuix Networks <info@yuix.org>
 License: MIT
@@ -91,7 +91,7 @@ origin.is_datacenter # True
 | `cdn` | edge infrastructure fronting other people's sites | Cloudflare, Akamai |
 | `vpn` | a consumer VPN or private relay exit | Mullvad, ProtonVPN, Apple Private Relay |
 | `tor` | a Tor exit node | |
-| `bot` | a declared crawler | Googlebot, GPTBot, ClaudeBot |
+| `bot` | a declared crawler or an AI company's outbound IP | Googlebot, GPTBot, GPTBot (OpenAI's crawler), Meta, ... |
 | `reserved` | private, loopback, link-local, documentation | `127.0.0.1`, `10.0.0.0/8` |
 | `unknown` | in none of our lists — most often a residential ISP | |
 
@@ -185,7 +185,11 @@ and are second-hand by definition:
 
 Covering Hetzner, OVHcloud, Scaleway, Alibaba Cloud, Leaseweb, UpCloud, IBM
 Cloud, Huawei Cloud, Tencent Cloud, Rackspace, Akamai, Gcore, Mullvad,
-ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers.
+ProtonVPN, Apple Private Relay, Tor, and ten declared crawlers. AI-company
+outbound ranges (OpenAI, Perplexity AI, DuckAssistBot, Apple Intelligence
+Proxy, Meta) come from these same two repositories. Most of them overlap
+exactly with an existing bot label, so they contribute label diversity
+rather than new bot coverage — only Meta adds distinct segments.
 
 Both are CC0, a public-domain dedication, so they carry no conditions.
 
@@ -207,15 +211,24 @@ covered. It is a subset of the provider feeds it was itself built from, and
 a source that adds nothing is still another endpoint that can break the
 weekly rebuild.
 
-About 454,000 published prefixes collapse into 38,379 disjoint ranges
-(21,901 IPv4, 16,478 IPv6) covering 287 million IPv4 addresses. Rebuild it
-yourself at any time:
+About 454,000 published prefixes collapse into roughly 38,400 disjoint
+ranges (about 21,900 IPv4 and 16,500 IPv6) covering 287 million IPv4
+addresses. The exact figures move every week with the feeds; the build
+prints them. Rebuild it yourself at any time:
 
 ```
 python tools/build_dataset.py
 ```
 
-A GitHub Action re-runs that weekly and opens a PR when the ranges move.
+A GitHub Action re-runs that weekly and commits the result when the ranges
+move, and a second one cuts a patch release on the 6th of each month if the
+dataset changed since the last release — so `pip install --upgrade iporigin`
+is never more than about a month behind the feeds, and a quiet month
+produces no release rather than a version whose only content is a new
+number. The build refuses to replace the committed dataset if it shrinks by
+more than 20% — a feed that starts answering with an empty body looks
+exactly like a provider giving up its address space, and nothing else
+would catch it. Pass `--allow-shrink` when the drop is genuine.
 
 ### Known gaps
 

{iporigin-1.1.0 → iporigin-1.1.2}/src/iporigin.egg-info/SOURCES.txt

@@ -19,4 +19,5 @@ src/iporigin.egg-info/top_level.txt
 src/iporigin/data/ranges.bin
 tests/test_iporigin.py
 tools/build_dataset.py
+tools/bump_version.py
 tools/sources.py

{iporigin-1.1.0 → iporigin-1.1.2}/tests/test_iporigin.py

@@ -175,6 +175,61 @@ def test_a_known_bot_range_is_classified_as_bot():
     pytest.fail("no bot ranges in the dataset")
 
 
+@pytest.mark.parametrize(
+    "provider",
+    [
+        # Updated whenever a new bot provider is added in tools/sources.py.
+        # Slugs that returned 404 upstream are omitted from the dataset
+        # and skipped here. Slugs whose ranges overlap exactly with an
+        # existing bot label are also skipped (they add label diversity
+        # without contributing distinct segments to the disjoint sweep).
+        "OpenAI",
+        "Perplexity AI",
+        "DuckAssistBot",
+        "Apple Intelligence Proxy",
+        "Meta",
+    ],
+)
+def test_known_ai_bot_provider_is_classified_as_bot(provider):
+    """One sample per AI-company bot provider, taken from the dataset.
+
+    Sampled rather than hardcoded because ranges move with the feeds.
+    """
+    data = _data.dataset()
+    for start, label_idx in zip(data.v4_starts, data.v4_labels):
+        if data.labels[label_idx][0] == provider:
+            origin = iporigin.classify(str(ipaddress.ip_address(start)))
+            assert origin.kind == core.BOT
+            assert origin.provider == provider
+            assert origin.is_datacenter
+            return
+    pytest.skip("provider %r has no ranges in this build" % provider)
+
+
+_AI_BOT_PROVIDERS = [
+    "OpenAI",
+    "Perplexity AI",
+    "DuckAssistBot",
+    "Apple Intelligence Proxy",
+    "Meta",
+]
+
+
+def test_at_least_one_ai_bot_provider_contributes_ranges():
+    """Catches a future regression where every new AI-bot slug turns
+    out to be a strict subset of an existing bot label — the
+    parametrized test above would quietly skip every case. At least
+    one of the probed providers must add distinct segments.
+    """
+    data = _data.dataset()
+    bot_providers = {p for p, _ in data.labels if p in _AI_BOT_PROVIDERS}
+    assert bot_providers, (
+        "no AI-company bot provider contributed distinct ranges "
+        "to this build — every probed slug was absorbed into an "
+        "existing label"
+    )
+
+
 def test_a_known_vpn_range_is_classified_as_vpn():
     data = _data.dataset()
     for start, label in zip(data.v4_starts, data.v4_labels):
@@ -363,3 +418,148 @@ def _stub_api(monkeypatch, payload):
         yield Response()
 
     monkeypatch.setattr(online.urllib.request, "urlopen", fake_urlopen)
+
+
+# --- the build guard -------------------------------------------------------
+#
+# The weekly refresh commits straight to main, so this check is the only
+# thing standing between a broken upstream feed and a shipped dataset.
+
+
+def _build_module():
+    import importlib.util
+    import pathlib
+
+    path = pathlib.Path(__file__).parent.parent / "tools" / "build_dataset.py"
+    spec = importlib.util.spec_from_file_location("build_dataset", path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_a_large_shrink_is_refused():
+    build = _build_module()
+    assert build.shrink_complaint(30000, 5000) is not None
+
+
+def test_a_normal_week_is_allowed():
+    """Feeds move by a few percent all the time."""
+    build = _build_module()
+    assert build.shrink_complaint(30000, 29000) is None
+    assert build.shrink_complaint(30000, 31500) is None
+
+
+def test_growth_is_never_refused():
+    build = _build_module()
+    assert build.shrink_complaint(1000, 100000) is None
+
+
+def test_the_first_ever_build_is_allowed():
+    """No committed dataset to compare against."""
+    build = _build_module()
+    assert build.shrink_complaint(None, 10) is None
+    assert build.shrink_complaint(0, 10) is None
+
+
+def test_the_threshold_is_the_documented_one():
+    build = _build_module()
+    assert build.MAX_SHRINK == 0.20
+    # Exactly at the limit passes; a hair below does not.
+    assert build.shrink_complaint(1000, 800) is None
+    assert build.shrink_complaint(1000, 799) is not None
+
+
+def test_the_complaint_says_what_to_do():
+    build = _build_module()
+    message = build.shrink_complaint(30000, 5000)
+    assert "--allow-shrink" in message
+    assert "5000" in message and "30000" in message
+
+
+def test_existing_range_count_reads_the_committed_dataset():
+    build = _build_module()
+    count = build.existing_range_count()
+    assert count and count > 1000
+
+
+def test_existing_range_count_survives_a_missing_or_corrupt_file(tmp_path, monkeypatch):
+    build = _build_module()
+    monkeypatch.setattr(build, "OUT", tmp_path / "absent.bin")
+    assert build.existing_range_count() is None
+
+    corrupt = tmp_path / "ranges.bin"
+    corrupt.write_bytes(b"nope")
+    monkeypatch.setattr(build, "OUT", corrupt)
+    assert build.existing_range_count() is None
+
+
+# --- the release bump ------------------------------------------------------
+#
+# Releases are cut by a scheduled job, so a bump that goes half-way lands on
+# PyPI before anyone looks at it.
+
+
+def _bump_module():
+    import importlib.util
+    import pathlib
+
+    path = pathlib.Path(__file__).parent.parent / "tools" / "bump_version.py"
+    spec = importlib.util.spec_from_file_location("bump_version", path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_the_two_declared_versions_agree():
+    """pyproject.toml and __init__.py each hold the version separately."""
+    bump = _bump_module()
+    declared, exported = bump.read_versions(
+        bump.PYPROJECT.read_text(), bump.INIT.read_text()
+    )
+    assert declared == exported == iporigin.__version__
+
+
+def test_patch_bump():
+    bump = _bump_module()
+    assert bump.next_patch("1.1.0") == "1.1.1"
+    assert bump.next_patch("1.1.9") == "1.1.10"
+    assert bump.next_patch("0.0.0") == "0.0.1"
+
+
+def test_a_non_numeric_version_is_left_alone():
+    bump = _bump_module()
+    for bad in ("2.0.0rc1", "1.1", "1.1.0.post1", "v1.1.0"):
+        with pytest.raises(ValueError):
+            bump.next_patch(bad)
+
+
+def test_bump_rewrites_both_files():
+    bump = _bump_module()
+    pyproject, init, new = bump.bump(
+        'name = "iporigin"\nversion = "1.2.3"\n', '__version__ = "1.2.3"\n'
+    )
+    assert new == "1.2.4"
+    assert 'version = "1.2.4"' in pyproject
+    assert '__version__ = "1.2.4"' in init
+
+
+def test_bump_refuses_when_the_files_disagree():
+    bump = _bump_module()
+    with pytest.raises(ValueError, match="fix that before releasing"):
+        bump.bump('version = "1.2.3"\n', '__version__ = "1.0.0"\n')
+
+
+def test_bump_refuses_when_a_declaration_is_missing():
+    bump = _bump_module()
+    with pytest.raises(ValueError, match="pyproject"):
+        bump.read_versions("name = \"iporigin\"\n", '__version__ = "1.0.0"\n')
+    with pytest.raises(ValueError, match="__init__"):
+        bump.read_versions('version = "1.0.0"\n', "x = 1\n")
+
+
+def test_bump_only_touches_the_version_line():
+    """pyproject holds other quoted values; a greedy substitution eats them."""
+    bump = _bump_module()
+    source = 'version = "1.2.3"\nrequires-python = ">=3.8"\n'
+    pyproject, _init, _new = bump.bump(source, '__version__ = "1.2.3"\n')
+    assert 'requires-python = ">=3.8"' in pyproject

{iporigin-1.1.0 → iporigin-1.1.2}/tools/build_dataset.py

@@ -155,7 +155,48 @@ def pack(v4, v6):
     return bytes(out)
 
 
-def main():
+#: Refuse to replace the dataset if it shrinks by more than this. A source
+#: that starts answering 200 with an empty body, or quietly changes its JSON
+#: shape, looks exactly like a provider giving up its address space — and
+#: nothing else in the pipeline would notice.
+MAX_SHRINK = 0.20
+
+
+def existing_range_count():
+    """How many ranges the committed dataset holds, or None if there is none."""
+    if not OUT.exists():
+        return None
+    try:
+        blob = OUT.read_bytes()
+        magic, _version, _built, _labels, n_v4, n_v6 = struct.unpack_from("<8sHQHII", blob, 0)
+    except Exception:
+        return None
+    if magic != MAGIC:
+        return None
+    return n_v4 + n_v6
+
+
+def shrink_complaint(previous, current, max_shrink=None):
+    """Why this build must not replace the committed one, or None.
+
+    A source that starts answering 200 with an empty body, or quietly
+    changes its JSON shape, looks exactly like a provider giving up its
+    address space. Nothing else in the pipeline notices, and since the
+    refresh now commits straight to main there is no review that would.
+    """
+    limit = MAX_SHRINK if max_shrink is None else max_shrink
+    if not previous or current >= previous * (1 - limit):
+        return None
+    return (
+        "Refusing to write: %d ranges is %.0f%% below the %d already "
+        "committed. Either several sources failed at once, or one changed "
+        "shape and is now parsing to nothing. Check the per-source counts "
+        "above, then override with --allow-shrink if it is genuine."
+        % (current, 100 * (1 - current / previous), previous)
+    )
+
+
+def main(allow_shrink=False):
     print("Fetching provider feeds...")
     v4, v6 = collect()
     if not v4:
@@ -169,6 +210,14 @@ def main():
     v4, v6 = flatten(v4), flatten(v6)
     blob = pack(v4, v6)
 
+    # The last line of defence now that nothing downstream reviews this.
+    complaint = None if allow_shrink else shrink_complaint(
+        existing_range_count(), len(v4) + len(v6)
+    )
+    if complaint:
+        print(complaint, file=sys.stderr)
+        return 1
+
     OUT.parent.mkdir(parents=True, exist_ok=True)
     OUT.write_bytes(blob)
 
@@ -179,4 +228,4 @@ def main():
 
 
 if __name__ == "__main__":
-    raise SystemExit(main())
+    raise SystemExit(main(allow_shrink="--allow-shrink" in sys.argv))

iporigin-1.1.2/tools/bump_version.py

@@ -0,0 +1,84 @@
+"""Bump the patch version, in both of the places that hold it.
+
+The version lives in pyproject.toml (what PyPI sees) and in
+src/iporigin/__init__.py (what `iporigin --version` and any caller sees).
+Nothing enforces that they agree, and a release that bumps one of them
+ships a package that lies about itself — so this refuses to touch either
+unless they start out identical.
+"""
+import pathlib
+import re
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+PYPROJECT = ROOT / "pyproject.toml"
+INIT = ROOT / "src" / "iporigin" / "__init__.py"
+
+PYPROJECT_RE = re.compile(r'^version = "([^"]+)"', re.M)
+INIT_RE = re.compile(r'^__version__ = "([^"]+)"', re.M)
+
+
+def read_versions(pyproject_text, init_text):
+    """The declared version in each file, as a pair."""
+    found = []
+    for pattern, text, where in (
+        (PYPROJECT_RE, pyproject_text, "pyproject.toml"),
+        (INIT_RE, init_text, "__init__.py"),
+    ):
+        match = pattern.search(text)
+        if not match:
+            raise ValueError("no version declaration found in %s" % where)
+        found.append(match.group(1))
+    return tuple(found)
+
+
+def next_patch(version):
+    """1.1.0 -> 1.1.1. Refuses anything that is not three plain numbers,
+    because guessing at what comes after 2.0.0rc1 is not this script's job."""
+    parts = version.split(".")
+    if len(parts) != 3 or not all(p.isdigit() for p in parts):
+        raise ValueError(
+            "%r is not a plain major.minor.patch version; bump it by hand" % version
+        )
+    major, minor, patch = parts
+    return "%s.%s.%d" % (major, minor, int(patch) + 1)
+
+
+def bump(pyproject_text, init_text):
+    """Both files' new contents, plus the version they now declare."""
+    current_pyproject, current_init = read_versions(pyproject_text, init_text)
+    if current_pyproject != current_init:
+        raise ValueError(
+            "pyproject.toml says %s but __init__.py says %s; fix that before "
+            "releasing" % (current_pyproject, current_init)
+        )
+    new = next_patch(current_pyproject)
+    return (
+        PYPROJECT_RE.sub('version = "%s"' % new, pyproject_text, count=1),
+        INIT_RE.sub('__version__ = "%s"' % new, init_text, count=1),
+        new,
+    )
+
+
+def main(argv):
+    pyproject_text = PYPROJECT.read_text()
+    init_text = INIT.read_text()
+
+    if "--current" in argv:
+        print(read_versions(pyproject_text, init_text)[0])
+        return 0
+
+    new_pyproject, new_init, new = bump(pyproject_text, init_text)
+    if "--dry-run" not in argv:
+        PYPROJECT.write_text(new_pyproject)
+        INIT.write_text(new_init)
+    print(new)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main(sys.argv[1:]))
+    except ValueError as exc:
+        print("error: %s" % exc, file=sys.stderr)
+        raise SystemExit(1)

{iporigin-1.1.0 → iporigin-1.1.2}/tools/sources.py

@@ -14,6 +14,7 @@ import csv
 import io
 import json
 import re
+import sys
 import urllib.request
 
 USER_AGENT = "iporigin-dataset-builder/1.0 (+https://github.com/Yuix-Networks/iporigin)"
@@ -214,6 +215,41 @@ def _lord_alfred(slug):
     return fetch
 
 
+def _optional_rezmoss(slug):
+    """Like _rezmoss, but a missing upstream file is logged and skipped,
+    not raised. Use for community slugs whose availability is not certain."""
+    def fetch():
+        for version in (4, 6):
+            url = REZMOSS % (slug, slug, version)
+            try:
+                text = _get_text(url)
+            except Exception as exc:
+                print("  !! %-18s skipped: %s" % (slug, exc), file=sys.stderr)
+                continue
+            for line in text.splitlines():
+                line = line.strip()
+                if line and not line.startswith("#") and "/" in line:
+                    yield line
+    return fetch
+
+
+def _optional_lord_alfred(slug):
+    """Like _lord_alfred, but a missing upstream file is logged and skipped."""
+    def fetch():
+        for version in (4, 6):
+            url = LORD_ALFRED % (slug, version)
+            try:
+                text = _get_text(url)
+            except Exception as exc:
+                print("  !! %-18s skipped: %s" % (slug, exc), file=sys.stderr)
+                continue
+            for line in text.splitlines():
+                line = line.strip()
+                if line and not line.startswith("#") and "/" in line:
+                    yield line
+    return fetch
+
+
 COMMUNITY = {
     # Hosting with no official feed.
     "Hetzner": ("hosting", _rezmoss("hetzner")),
@@ -249,6 +285,11 @@ COMMUNITY = {
     "Applebot": ("bot", _rezmoss("applebot")),
     "Common Crawl": ("bot", _rezmoss("commoncrawl")),
     "Internet Archive": ("bot", _rezmoss("internetarchive")),
+    "OpenAI": ("bot", _optional_lord_alfred("openai")),
+    "Perplexity AI": ("bot", _optional_lord_alfred("perplexity")),
+    "DuckAssistBot": ("bot", _optional_lord_alfred("duckassistbot")),
+    "Apple Intelligence Proxy": ("bot", _optional_lord_alfred("apple-proxy")),
+    "Meta": ("bot", _optional_rezmoss("meta")),
 }
 
 SOURCES.update(COMMUNITY)