django-bots 1.0.0__tar.gz

django_bots-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 M4p4
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

django_bots-1.0.0/PKG-INFO

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: django-bots
+Version: 1.0.0
+Summary: User-agent parsing, crawler detection and AI bot blocking for Django.
+Author-email: M4p4 <homejr@protonmail.com>
+License-Expression: MIT
+Project-URL: Changelog, https://github.com/M4p4/django-bots/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://djangobots.readthedocs.io/
+Project-URL: Issues, https://github.com/M4p4/django-bots/issues
+Project-URL: Repository, https://github.com/M4p4/django-bots
+Keywords: ai,bots,crawlers,django,robots.txt,user-agents
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 5.2
+Classifier: Framework :: Django :: 6.0
+Classifier: Framework :: Django :: 6.1
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: crawler-user-agents>=1.62
+Requires-Dist: django>=5.2
+Requires-Dist: ua-parser<2,>=1
+Dynamic: license-file
+
+# django-bots
+
+[![PyPI version](https://img.shields.io/pypi/v/django-bots.svg)](https://pypi.org/project/django-bots/)
+[![Python versions](https://img.shields.io/pypi/pyversions/django-bots.svg)](https://pypi.org/project/django-bots/)
+[![CI](https://github.com/M4p4/django-bots/actions/workflows/main.yml/badge.svg)](https://github.com/M4p4/django-bots/actions/workflows/main.yml)
+[![Documentation](https://readthedocs.org/projects/djangobots/badge/?version=stable)](https://djangobots.readthedocs.io/)
+
+More and more of the traffic on a website comes from crawlers, scrapers and AI bots
+instead of people. I wrote django-bots to tell them apart in Django, and to keep out
+the AI bots I don't want.
+
+django-bots tells you who is on the other end of a request: which browser, operating
+system and device, whether it's a crawler, and whether it's an AI bot. It can serve a
+robots.txt that disallows AI bots and turn them away with a 403. Each feature is
+opt-in, and the package adds no models or migrations.
+
+## Installation
+
+```console
+python -m pip install django-bots
+```
+
+Add the app to `INSTALLED_APPS`:
+
+```python
+INSTALLED_APPS = [
+    ...,
+    "django_bots",
+]
+```
+
+## User agents
+
+Add the middleware, and every request gets a lazily parsed `request.user_agent`:
+
+```python
+MIDDLEWARE = [
+    ...,
+    "django_bots.middleware.UserAgentMiddleware",
+]
+```
+
+```python
+def home(request):
+    if request.user_agent.is_mobile:
+        ...
+    request.user_agent.browser  # Browser(family="Mobile Safari", version=(5, 1), version_string="5.1")
+    request.user_agent.os  # OperatingSystem(family="iOS", version=(5, 1), version_string="5.1")
+    request.user_agent.device  # Device(family="iPhone", brand="Apple", model="iPhone")
+```
+
+Templates get the same checks as filters:
+
+```django
+{% load bots %}
+
+{% if request|is_mobile %}
+  <a href="/app/">Get the app</a>
+{% endif %}
+```
+
+Parsing uses [ua-parser](https://github.com/ua-parser/uap-python) 1.x with an
+in-process cache, and works for sync and async views.
+
+## Crawlers
+
+`is_crawler` matches the patterns from
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents), which cover
+search engines, SEO tools, uptime monitors and HTTP libraries:
+
+```python
+if request.user_agent.is_crawler:
+    ...
+```
+
+```django
+{% if not request|is_crawler %}
+  <script src="/analytics.js"></script>
+{% endif %}
+```
+
+## AI bots
+
+`is_ai_bot` and `ai_bot` match the list from
+[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), which is bundled with
+the package:
+
+```python
+if request.user_agent.is_ai_bot:
+    logger.info("AI bot: %s", request.user_agent.ai_bot)  # "GPTBot"
+```
+
+Serve a robots.txt that disallows every AI bot:
+
+```python
+from django.urls import path
+
+from django_bots.views import robots_txt
+
+urlpatterns = [
+    path("robots.txt", robots_txt),
+]
+```
+
+robots.txt only asks. To refuse AI bots outright, add the blocking middleware near
+the top of `MIDDLEWARE`:
+
+```python
+MIDDLEWARE = [
+    "django.middleware.security.SecurityMiddleware",
+    "django_bots.middleware.AIBotBlockMiddleware",
+    ...,
+]
+```
+
+`BOTS_AI_ALLOW` keeps the bots you want, for both robots.txt and the middleware:
+
+```python
+BOTS_AI_ALLOW = ["OAI-SearchBot"]
+```
+
+A daily workflow checks for new ai.robots.txt releases, and data updates ship as patch
+releases.
+
+## Upgrading from django-user-agents
+
+The `request.user_agent` attributes, the `django_bots.utils` helpers and the
+`user_agents` template filters work the same way as in django-user-agents, so
+upgrading comes down to new app, middleware and import paths. One thing behaves
+differently: `is_bot` is also true for crawlers and AI bots. The
+[migration guide](https://djangobots.readthedocs.io/en/stable/migration.html) walks
+through each step.
+
+## Compatibility
+
+| Python | Django |
+|---|---|
+| 3.10, 3.11 | 5.2 |
+| 3.12, 3.13, 3.14 | 5.2, 6.0, 6.1 |
+
+## Data sources and licenses
+
+django-bots is released under the MIT license. It uses:
+
+- [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT), bundled in
+  `django_bots/data/` with its license
+- [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT),
+  installed as a dependency
+- [ua-parser](https://github.com/ua-parser/uap-python) (Apache-2.0), installed as a
+  dependency
+- the device detection rules of
+  [user-agents](https://github.com/selwin/python-user-agents) (MIT), ported into
+  `django_bots/useragent.py`
+
+`django_bots.DATA_VERSIONS` shows which versions are in use.
+
+## Documentation
+
+The full documentation is at [djangobots.readthedocs.io](https://djangobots.readthedocs.io/):
+
+- [User agents](https://djangobots.readthedocs.io/en/stable/user-agents.html)
+- [Crawler detection](https://djangobots.readthedocs.io/en/stable/crawlers.html)
+- [AI bots](https://djangobots.readthedocs.io/en/stable/ai-bots.html)
+- [Settings](https://djangobots.readthedocs.io/en/stable/settings.html)
+- [Limits](https://djangobots.readthedocs.io/en/stable/limits.html)
+- [Contributing](https://djangobots.readthedocs.io/en/stable/contributing.html)

django_bots-1.0.0/README.md

@@ -0,0 +1,166 @@
+# django-bots
+
+[![PyPI version](https://img.shields.io/pypi/v/django-bots.svg)](https://pypi.org/project/django-bots/)
+[![Python versions](https://img.shields.io/pypi/pyversions/django-bots.svg)](https://pypi.org/project/django-bots/)
+[![CI](https://github.com/M4p4/django-bots/actions/workflows/main.yml/badge.svg)](https://github.com/M4p4/django-bots/actions/workflows/main.yml)
+[![Documentation](https://readthedocs.org/projects/djangobots/badge/?version=stable)](https://djangobots.readthedocs.io/)
+
+More and more of the traffic on a website comes from crawlers, scrapers and AI bots
+instead of people. I wrote django-bots to tell them apart in Django, and to keep out
+the AI bots I don't want.
+
+django-bots tells you who is on the other end of a request: which browser, operating
+system and device, whether it's a crawler, and whether it's an AI bot. It can serve a
+robots.txt that disallows AI bots and turn them away with a 403. Each feature is
+opt-in, and the package adds no models or migrations.
+
+## Installation
+
+```console
+python -m pip install django-bots
+```
+
+Add the app to `INSTALLED_APPS`:
+
+```python
+INSTALLED_APPS = [
+    ...,
+    "django_bots",
+]
+```
+
+## User agents
+
+Add the middleware, and every request gets a lazily parsed `request.user_agent`:
+
+```python
+MIDDLEWARE = [
+    ...,
+    "django_bots.middleware.UserAgentMiddleware",
+]
+```
+
+```python
+def home(request):
+    if request.user_agent.is_mobile:
+        ...
+    request.user_agent.browser  # Browser(family="Mobile Safari", version=(5, 1), version_string="5.1")
+    request.user_agent.os  # OperatingSystem(family="iOS", version=(5, 1), version_string="5.1")
+    request.user_agent.device  # Device(family="iPhone", brand="Apple", model="iPhone")
+```
+
+Templates get the same checks as filters:
+
+```django
+{% load bots %}
+
+{% if request|is_mobile %}
+  <a href="/app/">Get the app</a>
+{% endif %}
+```
+
+Parsing uses [ua-parser](https://github.com/ua-parser/uap-python) 1.x with an
+in-process cache, and works for sync and async views.
+
+## Crawlers
+
+`is_crawler` matches the patterns from
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents), which cover
+search engines, SEO tools, uptime monitors and HTTP libraries:
+
+```python
+if request.user_agent.is_crawler:
+    ...
+```
+
+```django
+{% if not request|is_crawler %}
+  <script src="/analytics.js"></script>
+{% endif %}
+```
+
+## AI bots
+
+`is_ai_bot` and `ai_bot` match the list from
+[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), which is bundled with
+the package:
+
+```python
+if request.user_agent.is_ai_bot:
+    logger.info("AI bot: %s", request.user_agent.ai_bot)  # "GPTBot"
+```
+
+Serve a robots.txt that disallows every AI bot:
+
+```python
+from django.urls import path
+
+from django_bots.views import robots_txt
+
+urlpatterns = [
+    path("robots.txt", robots_txt),
+]
+```
+
+robots.txt only asks. To refuse AI bots outright, add the blocking middleware near
+the top of `MIDDLEWARE`:
+
+```python
+MIDDLEWARE = [
+    "django.middleware.security.SecurityMiddleware",
+    "django_bots.middleware.AIBotBlockMiddleware",
+    ...,
+]
+```
+
+`BOTS_AI_ALLOW` keeps the bots you want, for both robots.txt and the middleware:
+
+```python
+BOTS_AI_ALLOW = ["OAI-SearchBot"]
+```
+
+A daily workflow checks for new ai.robots.txt releases, and data updates ship as patch
+releases.
+
+## Upgrading from django-user-agents
+
+The `request.user_agent` attributes, the `django_bots.utils` helpers and the
+`user_agents` template filters work the same way as in django-user-agents, so
+upgrading comes down to new app, middleware and import paths. One thing behaves
+differently: `is_bot` is also true for crawlers and AI bots. The
+[migration guide](https://djangobots.readthedocs.io/en/stable/migration.html) walks
+through each step.
+
+## Compatibility
+
+| Python | Django |
+|---|---|
+| 3.10, 3.11 | 5.2 |
+| 3.12, 3.13, 3.14 | 5.2, 6.0, 6.1 |
+
+## Data sources and licenses
+
+django-bots is released under the MIT license. It uses:
+
+- [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT), bundled in
+  `django_bots/data/` with its license
+- [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT),
+  installed as a dependency
+- [ua-parser](https://github.com/ua-parser/uap-python) (Apache-2.0), installed as a
+  dependency
+- the device detection rules of
+  [user-agents](https://github.com/selwin/python-user-agents) (MIT), ported into
+  `django_bots/useragent.py`
+
+`django_bots.DATA_VERSIONS` shows which versions are in use.
+
+## Documentation
+
+The full documentation is at [djangobots.readthedocs.io](https://djangobots.readthedocs.io/):
+
+- [User agents](https://djangobots.readthedocs.io/en/stable/user-agents.html)
+- [Crawler detection](https://djangobots.readthedocs.io/en/stable/crawlers.html)
+- [AI bots](https://djangobots.readthedocs.io/en/stable/ai-bots.html)
+- [Settings](https://djangobots.readthedocs.io/en/stable/settings.html)
+- [Limits](https://djangobots.readthedocs.io/en/stable/limits.html)
+- [Contributing](https://djangobots.readthedocs.io/en/stable/contributing.html)

django_bots-1.0.0/pyproject.toml

@@ -0,0 +1,156 @@
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = [ "setuptools>=77" ]
+
+[project]
+name = "django-bots"
+version = "1.0.0"
+description = "User-agent parsing, crawler detection and AI bot blocking for Django."
+readme = "README.md"
+keywords = [ "ai", "bots", "crawlers", "django", "robots.txt", "user-agents" ]
+license = "MIT"
+license-files = [ "LICENSE" ]
+authors = [
+  { name = "M4p4", email = "homejr@protonmail.com" },
+]
+requires-python = ">=3.10"
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "Framework :: Django",
+  "Framework :: Django :: 5.2",
+  "Framework :: Django :: 6.0",
+  "Framework :: Django :: 6.1",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Topic :: Internet :: WWW/HTTP",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Typing :: Typed",
+]
+dependencies = [
+  "crawler-user-agents>=1.62",
+  "django>=5.2",
+  "ua-parser>=1,<2",
+]
+urls.Changelog = "https://github.com/M4p4/django-bots/blob/main/CHANGELOG.md"
+urls.Documentation = "https://djangobots.readthedocs.io/"
+urls.Issues = "https://github.com/M4p4/django-bots/issues"
+urls.Repository = "https://github.com/M4p4/django-bots"
+
+[dependency-groups]
+test = [
+  "coverage[toml]",
+  "pytest",
+  "pytest-django",
+  "pytest-randomly",
+  "user-agents==2.2",
+]
+# Sphinx 9 needs Python 3.12 or newer.
+docs = [
+  "furo; python_version>='3.12'",
+  "myst-parser; python_version>='3.12'",
+  "sphinx; python_version>='3.12'",
+]
+django52 = [ "django>=5.2,<5.3" ]
+# Django 6.0 and later need Python 3.12 or newer.
+django60 = [ "django>=6,<6.1; python_version>='3.12'" ]
+django61 = [ "django>=6.1,<6.2; python_version>='3.12'" ]
+
+[tool.setuptools]
+packages.find.where = [ "src" ]
+package-data.django_bots = [ "data/*", "py.typed", "templates/django_bots/*" ]
+
+[tool.uv]
+# uv only auto-installs the `dev` group, so name the groups a bare `uv run`
+# needs. The full version matrix is exercised by tox, not by the default env.
+default-groups = [ "test", "django52" ]
+conflicts = [
+  [
+    { group = "django52" },
+    { group = "django60" },
+    { group = "django61" },
+  ],
+]
+
+[tool.ruff]
+lint.select = [
+  # flake8-bugbear
+  "B",
+  # flake8-comprehensions
+  "C4",
+  # pycodestyle
+  "E",
+  # Pyflakes errors
+  "F",
+  # isort
+  "I",
+  # flake8-simplify
+  "SIM",
+  # flake8-tidy-imports
+  "TID",
+  # pyupgrade
+  "UP",
+  # Pyflakes warnings
+  "W",
+]
+lint.ignore = [
+  # flake8-bugbear opinionated rules
+  "B9",
+  # line-too-long
+  "E501",
+  # suppressible-exception
+  "SIM105",
+  # if-else-block-instead-of-if-exp
+  "SIM108",
+]
+lint.extend-safe-fixes = [
+  # non-pep585-annotation
+  "UP006",
+]
+lint.isort.required-imports = [ "from __future__ import annotations" ]
+
+[tool.pyproject-fmt]
+max_supported_python = "3.14"
+
+[tool.mypy]
+mypy_path = "src/"
+namespace_packages = false
+warn_unreachable = true
+enable_error_code = [
+  "ignore-without-code",
+  "redundant-expr",
+  "truthy-bool",
+]
+strict = true
+overrides = [
+  { module = "tests.*", allow_untyped_defs = true, allow_untyped_calls = true },
+  { module = "crawleruseragents", ignore_missing_imports = true },
+]
+
+[tool.pytest]
+ini_options.testpaths = [ "tests" ]
+ini_options.pythonpath = [ "." ]
+ini_options.addopts = "-ra --strict-config --strict-markers"
+ini_options.django_find_project = false
+ini_options.DJANGO_SETTINGS_MODULE = "tests.settings"
+
+[tool.coverage]
+run.branch = true
+run.data_file = ".coverage/cov"
+run.parallel = true
+run.source = [
+  "django_bots",
+  "tests",
+]
+paths.source = [
+  "src",
+  ".tox/**/site-packages",
+]
+report.fail_under = 100
+report.show_missing = true
+report.skip_covered = true
+report.skip_empty = true

django_bots-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

django_bots-1.0.0/src/django_bots/__init__.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+import json
+from importlib.metadata import version
+from importlib.resources import files
+
+__version__ = version("django-bots")
+
+DATA_VERSIONS: dict[str, str] = {
+    **json.loads(files(__name__).joinpath("data", "VERSIONS.json").read_text("utf-8")),
+    "crawler_user_agents": version("crawler-user-agents"),
+    "ua_parser": version("ua-parser"),
+    "uap_core": version("ua-parser-builtins"),
+}
+"""Versions of the bundled and installed data sources."""

django_bots-1.0.0/src/django_bots/ai.py

@@ -0,0 +1,94 @@
+"""Detect AI bots by user-agent string.
+
+The bot list is vendored from `ai.robots.txt
+<https://github.com/ai-robots-txt/ai.robots.txt>`_, copyright (c) 2024
+ai.robots.txt, released under the MIT license.
+
+Each name matches case-insensitively at word boundaries in the user-agent string,
+after URLs and email addresses are removed from it. Results are cached per string in
+an LRU of ``BOTS_UA_CACHE_SIZE`` entries.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from collections.abc import Callable
+from functools import lru_cache
+from importlib.resources import files
+from typing import Any
+
+from django_bots.conf import bots_settings
+
+__all__ = ["ai_bot_names", "ai_bots", "is_ai_bot", "match_ai_bot", "robots_rules"]
+
+
+def _read_data(name: str) -> Any:
+    return json.loads(files("django_bots").joinpath("data", name).read_text("utf-8"))
+
+
+@lru_cache(maxsize=1)
+def ai_bots() -> dict[str, dict[str, Any]]:
+    """Return the vendored ai.robots.txt entries, keyed by bot name."""
+    bots: dict[str, dict[str, Any]] = _read_data("ai_robots.json")
+    return bots
+
+
+def ai_bot_names() -> list[str]:
+    """Return the AI bot names in use, after ``BOTS_AI_ALLOW`` and ``BOTS_AI_EXTRA``."""
+    allow = {name.lower() for name in bots_settings.AI_ALLOW}
+    names = [*ai_bots(), *bots_settings.AI_EXTRA]
+    return list(dict.fromkeys(name for name in names if name.lower() not in allow))
+
+
+# Contact links name the operator, not the bot: "openai.com" would match "OpenAI".
+URL_OR_EMAIL = re.compile(r"(?:https?://|www\.)[^\s;)]+|[\w.+-]+@[\w.-]+")
+
+
+@lru_cache(maxsize=1)
+def _get_matcher(
+    cache_size: int, allow: tuple[str, ...], extra: tuple[str, ...]
+) -> Callable[[str], str | None]:
+    """Build the matcher, rebuilt whenever one of the settings changes."""
+    # Upstream lists some names in two spellings, so the first one is reported.
+    canonical: dict[str, str] = {}
+    for name in ai_bot_names():
+        canonical.setdefault(name.lower(), name)
+    # Longest first, so a name wins over a shorter name it starts with.
+    alternation = "|".join(map(re.escape, sorted(canonical, key=len, reverse=True)))
+    # An empty alternation matches everything, so fall back to a never-matching regex.
+    regex = re.compile(rf"\b(?:{alternation})\b" if canonical else "(?!)", re.I)
+
+    def match(ua_string: str) -> str | None:
+        found = regex.search(URL_OR_EMAIL.sub(" ", ua_string))
+        return canonical[found.group().lower()] if found else None
+
+    if cache_size > 0:
+        return lru_cache(maxsize=cache_size)(match)
+    return match
+
+
+def match_ai_bot(ua_string: str) -> str | None:
+    """Return the name of the AI bot the user-agent string matches, or ``None``."""
+    matcher = _get_matcher(
+        bots_settings.UA_CACHE_SIZE,
+        tuple(bots_settings.AI_ALLOW),
+        tuple(bots_settings.AI_EXTRA),
+    )
+    return matcher(ua_string)
+
+
+def is_ai_bot(ua_string: str) -> bool:
+    """Whether the user-agent string matches an AI bot."""
+    return match_ai_bot(ua_string) is not None
+
+
+def robots_rules() -> str:
+    """Return a robots.txt group that disallows every AI bot in ``ai_bot_names()``.
+
+    Returns an empty string when no names are left.
+    """
+    names = ai_bot_names()
+    if not names:
+        return ""
+    return "\n".join([*(f"User-agent: {name}" for name in names), "Disallow: /"])

django_bots-1.0.0/src/django_bots/apps.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+from django.apps import AppConfig
+from django.core.checks import Tags, register
+
+from django_bots.checks import (
+    check_ai_block_view,
+    check_django_user_agents_installed,
+    check_user_agents_cache,
+)
+
+
+class DjangoBotsConfig(AppConfig):
+    name = "django_bots"
+    verbose_name = "Bots"
+
+    def ready(self) -> None:
+        register(check_user_agents_cache, Tags.compatibility)
+        register(check_django_user_agents_installed, Tags.compatibility)
+        register(check_ai_block_view)