django-robots-manager 6.2.1__py3-none-any.whl

django_robots_manager-6.2.1.dist-info/METADATA

@@ -0,0 +1,175 @@
+Metadata-Version: 2.4
+Name: django-robots-manager
+Version: 6.2.1
+Summary: Robots exclusion application for Django, complementing Sitemaps. Continuation of django-robots 6.x after a long pause in upstream releases.
+License-Expression: BSD-3-Clause
+License-File: LICENSE.txt
+Author: Artem Fabrikov
+Author-email: a.fabrikov1406@yandex.ru
+Requires-Python: >=3.10
+Classifier: Environment :: Web Environment
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 4.2
+Classifier: Framework :: Django :: 5.2
+Classifier: Framework :: Django :: 6.0
+Classifier: Framework :: Django :: 6.1
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
+Requires-Dist: Django (>=4.2)
+Project-URL: Homepage, https://github.com/KitKat-ru/django-robots-manager
+Project-URL: Issues, https://github.com/KitKat-ru/django-robots-manager/issues
+Project-URL: Repository, https://github.com/KitKat-ru/django-robots-manager
+Project-URL: Upstream, https://github.com/jazzband/django-robots/
+Description-Content-Type: text/markdown
+
+# django-robots-manager
+
+[![Tests](https://github.com/KitKat-ru/django-robots-manager/actions/workflows/tests.yml/badge.svg)](https://github.com/KitKat-ru/django-robots-manager/actions/workflows/tests.yml)
+[![GitHub tag](https://img.shields.io/github/v/tag/KitKat-ru/django-robots-manager?sort=semver)](https://github.com/KitKat-ru/django-robots-manager/tags)
+[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://github.com/KitKat-ru/django-robots-manager/actions/workflows/tests.yml)
+[![Django](https://img.shields.io/badge/django-4.2%20%7C%205.2%20%7C%206.0%20%7C%206.1-0C4B33)](https://github.com/KitKat-ru/django-robots-manager/actions/workflows/tests.yml)
+[![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](https://github.com/KitKat-ru/django-robots-manager/blob/main/LICENSE.txt)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+
+A continuation of [django-robots](https://github.com/jazzband/django-robots/),
+picking up from the 6.x series (based on release 6.1), after a long pause in releases
+of the original library.
+
+The import path and app label stay `robots`, so it is a drop-in replacement for
+django-robots 5.0 and 6.x: existing tables and migration history are reused.
+
+## Changes from upstream 6.1
+
+- `__version__` is read via `importlib.metadata` only; the `pkg_resources` fallback
+  and `default_app_config` (Django < 3.2) are removed.
+- The South guard in `robots.migrations` is removed.
+- `RuleAdminForm` rejects a rule whose allowed and disallowed URLs share a pattern.
+- `RuleAdminForm` rejects a second rule for the same robot (case-insensitive) on the
+  same site.
+- `ROBOTS_USE_HOST` now defaults to `False`: Yandex
+  [stopped using](https://webmaster.yandex.ru/blog/301-y-redirekt-polnostyu-zamenil-direktivu-host)
+  the `Host` directive in 2018, and Google
+  [never supported it](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
+  (it is not part of [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) either). Set it
+  to `True` to keep the old output.
+- `ROBOTS_SITE_BY_REQUEST` looks the site up the same way as Django's sites framework:
+  case-insensitively, retrying without the port (`example.com:8000` matches
+  `example.com`).
+- `robots.txt` is served as `text/plain; charset=utf-8`
+  ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) requires UTF-8), with rules sorted
+  by robot and URLs by pattern, in a fixed number of queries.
+- `Rule.comment`: an optional single-line note rendered as a `# ...` line above the
+  rule's group in `robots.txt`.
+- URL patterns are percent-encoded on output: non-ASCII characters (e.g. Cyrillic),
+  whitespace and `#` become `%XX`, as
+  [Yandex requires](https://yandex.ru/support/webmaster/ru/controlling-robot/robots-txt).
+  Patterns are stored as entered, and raw and encoded forms of the same path count as
+  the same pattern when checking allowed/disallowed conflicts.
+- [Clean-param](https://yandex.ru/support/webmaster/ru/robot-workings/clean-param)
+  directives (Yandex): see below.
+- System checks for `ROBOTS_SITEMAP_URLS`: `robots.E001` if it is a string instead of
+  a list, `robots.W001` for URLs that are not absolute, `robots.W002` for non-ASCII
+  domains.
+- Only the `ru` locale is shipped.
+
+## Installation
+
+Requires Python 3.10+ and Django 4.2+ (tested with Django 4.2, 5.2, 6.0 and 6.1).
+Projects on older Python or Django versions can stay on django-robots 6.1.
+
+```python
+INSTALLED_APPS = [
+    "django.contrib.sites",
+    ...
+    "robots",
+]
+```
+
+```python
+urlpatterns = [
+    re_path(r"^robots\.txt", include("robots.urls")),
+]
+```
+
+Settings (`ROBOTS_SITEMAP_URLS`, `ROBOTS_USE_SITEMAP`, `ROBOTS_USE_HOST`,
+`ROBOTS_CACHE_TIMEOUT`, `ROBOTS_SITE_BY_REQUEST`, `ROBOTS_USE_SCHEME_IN_HOST`,
+`ROBOTS_SITEMAP_VIEW_NAME`) are the same as upstream, except for the changes listed
+above.
+
+### Internationalized domains
+
+Store non-ASCII domains in `Site.domain` (and in `ROBOTS_SITEMAP_URLS`) in Punycode,
+e.g. `xn--d1aqf.xn--p1ai` instead of `дом.рф`:
+
+- the domain is written to `Sitemap:` (and `Host:`) as stored, and
+  [Yandex requires](https://yandex.ru/support/webmaster/ru/controlling-robot/robots-txt)
+  Punycode there;
+- with `ROBOTS_SITE_BY_REQUEST = True` the site is looked up by the request `Host`
+  header, which is always Punycode, so a site stored as `дом.рф` is not found and
+  `robots.txt` responds with an error.
+
+## Clean-param
+
+`Clean-param` tells Yandex which URL parameters do not change the page content, so
+`/catalog/?ref=vk` and `/catalog/?sid=1` are crawled and indexed as `/catalog/`. Other
+search engines ignore it.
+
+Add directives in the admin under *Clean-param directives* and attach them to sites:
+
+| Parameters | Path | Output |
+|---|---|---|
+| `ref&sid` | `/catalog/` | `Clean-param: ref&sid /catalog/` |
+| `sort` | *(empty)* | `Clean-param: sort` (whole site) |
+
+The directive is cross-sectional, so it is rendered once per file, next to `Sitemap`,
+not inside a `User-agent` group. Following the Yandex rules, parameter names are
+case-sensitive, the path may contain only `A-Za-z0-9.-/*_` (a leading `/` is added
+if missing, as for URL patterns), and the whole line is limited to 500 characters.
+Unlike `Allow`/`Disallow`, the path is not percent-encoded, so pages with Cyrillic
+paths (e.g. `/о-компании/`) can only be covered by a directive without a path.
+
+## Development
+
+`tests/` holds a minimal Django project (SQLite) used both for the test suite and for
+trying the app locally. It is not part of the distributed package. Run the commands
+from the repository root.
+
+```bash
+python -m venv .venv
+. .venv/bin/activate
+pip install -e .
+export DJANGO_SETTINGS_MODULE=tests.settings
+
+python -m django test tests          # run the test suite
+
+pip install "coverage[toml]"         # test coverage, as in CI
+python -m coverage run -m django test tests
+python -m coverage report
+
+python -m django migrate
+python -m django createsuperuser
+python -m django runserver           # http://localhost:8000/admin/, http://localhost:8000/robots.txt
+
+python -m django makemigrations robots --check --dry-run   # after model changes
+```
+
+The demo project uses `SITE_ID = 1` (`example.com`), so rules must be attached to that
+site to appear in `/robots.txt`.
+
+Linting and formatting use [ruff](https://docs.astral.sh/ruff/) via
+[pre-commit](https://pre-commit.com/); neither is a dependency of the package.
+
+```bash
+pip install pre-commit
+pre-commit install                   # run the hooks on every commit
+pre-commit run --all-files           # run them on the whole repository
+```
+

django_robots_manager-6.2.1.dist-info/RECORD

@@ -0,0 +1,23 @@
+robots/__init__.py,sha256=lZfKHv3FyvT128TfuNhk5czZNVgxSnT7PsflmdQ6ulU,87
+robots/admin.py,sha256=KrL7gaKi9M-0XGjU-C6s1QcjNPmkbQpXKvwhQFTGets,1174
+robots/apps.py,sha256=rVl-CjRkAqOs1js6q-D6IlnZUlo0w5X94qJoczrxznI,287
+robots/checks.py,sha256=1JoBMJGVl447YE-KPpVnNUda3x_azkFJxMo1QoikSSw,1411
+robots/forms.py,sha256=wqz_GD7g_n3n2XbxkDVNh-P-LjjLcqvUBaKYulTRy08,2341
+robots/locale/ru/LC_MESSAGES/django.mo,sha256=ZGZHbOSqCVvf-luGkNsrqMyS8KCpsa3g4gKrZXjRZgE,7564
+robots/locale/ru/LC_MESSAGES/django.po,sha256=mljy3_004mdjwS5NyKOWD9HvUs4xjlFejiPdR5GgwdI,8546
+robots/migrations/0001_initial.py,sha256=-61NoEIujKMlW6Nsu3D_Sned-kjIQpnwUVcSr_Ez_mo,4142
+robots/migrations/0002_alter_id_fields.py,sha256=uHED78W8Ol17OLAKjQ6qsN19TQG-wcZwJtkyj547cEw,693
+robots/migrations/0003_rule_comment.py,sha256=x08veYyK6Vu7Stfp_LxbAgBf4ifm8pwe6K5qFA0ojC8,732
+robots/migrations/0004_cleanparam.py,sha256=u0y4ADtrr5Dd54ZO0i3BtjPm1LY-0LNNyUfywcZCjC0,1868
+robots/migrations/0005_alter_rule_disallowed.py,sha256=Kpna96r7I65jh5GuXiby_kLqk_XMZzajUmkL74i11d8,768
+robots/migrations/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+robots/models.py,sha256=aWTeOZSp29RM4HXH7dGYv8d4bgSmLRnN1bOpatxKCUk,6951
+robots/settings.py,sha256=SLIaTKZqUsSxDzKkpPgr048DqgYuV7c3Br7qv_F-zl0,790
+robots/templates/robots/rule_list.html,sha256=qp42wvWI9LCVYsmBd14p6HC04govRfl8RUDZcdTzLxA,830
+robots/urls.py,sha256=dEIQ_G64GdyCSM142XztOOX9pQ5gKOeLSgP27ISKZvI,136
+robots/validators.py,sha256=KBhkCKNs6s596lV0zQ0nGRTsNoWkuA3eq0HJDyGe5nA,1700
+robots/views.py,sha256=ytjWUxufw6EkgHCqfh97BqDwkv2xMr3LVNia_9SpijE,3980
+django_robots_manager-6.2.1.dist-info/METADATA,sha256=j3HHfaoSC0nz2-2T3loMeQBHBglPsJFREVydYCs3M94,8279
+django_robots_manager-6.2.1.dist-info/WHEEL,sha256=EGEvSphFYqXKs23-kQBeyNoJP1nrT8ZJKQoi5p5DYL8,88
+django_robots_manager-6.2.1.dist-info/licenses/LICENSE.txt,sha256=IBDZ1PU5a4nJOc47Q2U-XMwHENfPB83G-vsFQoPjXBE,1555
+django_robots_manager-6.2.1.dist-info/RECORD,,

django_robots_manager-6.2.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.4.0
+Root-Is-Purelib: true
+Tag: py3-none-any

django_robots_manager-6.2.1.dist-info/licenses/LICENSE.txt

@@ -0,0 +1,29 @@
+Copyright (c) 2008-, Jannis Leidel
+Copyright (c) 2026-, Artem Fabrikov
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above
+      copyright notice, this list of conditions and the following
+      disclaimer in the documentation and/or other materials provided
+      with the distribution.
+    * Neither the name of the author nor the names of other
+      contributors may be used to endorse or promote products derived
+      from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

robots/__init__.py

@@ -0,0 +1,3 @@
+from importlib.metadata import version
+
+__version__ = version("django-robots-manager")

robots/admin.py

@@ -0,0 +1,36 @@
+from django.contrib import admin
+from django.http import HttpRequest
+from django.utils.translation import gettext_lazy as _
+
+from robots.forms import RuleAdminForm
+from robots.models import CleanParam, Rule, Url
+
+
+class RuleAdmin(admin.ModelAdmin):
+    form = RuleAdminForm
+    fieldsets = (
+        (None, {"fields": ("robot", "sites", "comment")}),
+        (_("URL patterns"), {"fields": ("allowed", "disallowed")}),
+        (
+            _("Advanced options"),
+            {"classes": ("collapse",), "fields": ("crawl_delay",)},
+        ),
+    )
+    list_filter = ("sites",)
+    list_display = ("robot", "allowed_urls", "disallowed_urls")
+    search_fields = ("robot", "allowed__pattern", "disallowed__pattern")
+    filter_horizontal = ("sites", "allowed", "disallowed")
+
+    def get_queryset(self, request: HttpRequest):
+        return super().get_queryset(request).prefetch_related("allowed", "disallowed")
+
+
+class CleanParamAdmin(admin.ModelAdmin):
+    list_display = ("parameters", "path")
+    list_filter = ("sites",)
+    filter_horizontal = ("sites",)
+
+
+admin.site.register(Url)
+admin.site.register(Rule, RuleAdmin)
+admin.site.register(CleanParam, CleanParamAdmin)

robots/apps.py

@@ -0,0 +1,12 @@
+from django.apps import AppConfig
+from django.core import checks
+
+from robots.checks import check_sitemap_urls
+
+
+class RobotsConfig(AppConfig):
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "robots"
+
+    def ready(self):
+        checks.register(check_sitemap_urls)

robots/checks.py

@@ -0,0 +1,39 @@
+from urllib.parse import SplitResult, urlsplit
+
+from django.core import checks
+
+from robots import settings
+
+
+def check_sitemap_urls(app_configs, **kwargs):
+    """Validate that ROBOTS_SITEMAP_URLS is a list of absolute ASCII URLs."""
+    sitemap_urls = settings.SITEMAP_URLS
+    if isinstance(sitemap_urls, str):
+        return [
+            checks.Error(
+                "ROBOTS_SITEMAP_URLS must be a list or tuple of URLs, not a string.",
+                hint="Wrap the URL in a list: ['https://example.com/sitemap.xml'].",
+                id="robots.E001",
+            )
+        ]
+
+    messages = []
+    for url in sitemap_urls:
+        parts: SplitResult = urlsplit(url)
+        if parts.scheme not in ("http", "https") or not parts.netloc:
+            messages.append(
+                checks.Warning(
+                    f"ROBOTS_SITEMAP_URLS has a URL that is not absolute: {url!r}.",
+                    hint="Use an absolute URL, e.g. 'https://example.com/sitemap.xml'.",
+                    id="robots.W001",
+                )
+            )
+        elif not parts.netloc.isascii():
+            messages.append(
+                checks.Warning(
+                    f"ROBOTS_SITEMAP_URLS contains a non-ASCII domain: {url!r}.",
+                    hint="Write the domain in Punycode, e.g. 'xn--d1aqf.xn--p1ai'.",
+                    id="robots.W002",
+                )
+            )
+    return messages

robots/forms.py

@@ -0,0 +1,64 @@
+from django import forms
+from django.contrib.sites.models import Site
+from django.utils.translation import gettext_lazy as _
+
+from robots.models import Rule, encode_pattern
+
+
+class RuleAdminForm(forms.ModelForm):
+    class Meta:
+        model = Rule
+        fields = "__all__"
+
+    def clean(self):
+        if not self.cleaned_data.get("disallowed", False) and not self.cleaned_data.get(
+            "allowed", False
+        ):
+            raise forms.ValidationError(
+                _("Please specify at least one allowed or disallowed URL.")
+            )
+
+        allowed = self.cleaned_data.get("allowed")
+        disallowed = self.cleaned_data.get("disallowed")
+        if allowed and disallowed:
+            allowed_patterns = {
+                encode_pattern(pattern): pattern
+                for pattern in allowed.values_list("pattern", flat=True)
+            }
+            conflicts = allowed_patterns.keys() & {
+                encode_pattern(pattern)
+                for pattern in disallowed.values_list("pattern", flat=True)
+            }
+            if conflicts:
+                raise forms.ValidationError(
+                    _(
+                        "URL patterns cannot be both allowed and disallowed: "
+                        "%(patterns)s."
+                    ),
+                    params={
+                        "patterns": ", ".join(
+                            sorted(allowed_patterns[key] for key in conflicts)
+                        )
+                    },
+                )
+
+        robot = self.cleaned_data.get("robot")
+        sites = self.cleaned_data.get("sites")
+        if robot and sites:
+            other_rules = Rule.objects.filter(robot__iexact=robot).exclude(
+                pk=self.instance.pk
+            )
+            duplicate_domains = (
+                Site.objects.filter(pk__in=sites, rule__in=other_rules)
+                .values_list("domain", flat=True)
+                .distinct()
+            )
+            if duplicate_domains:
+                raise forms.ValidationError(
+                    _("A rule for robot %(robot)s already exists on sites: %(sites)s."),
+                    params={
+                        "robot": robot,
+                        "sites": ", ".join(sorted(duplicate_domains)),
+                    },
+                )
+        return self.cleaned_data

robots/locale/ru/LC_MESSAGES/django.po

@@ -0,0 +1,188 @@
+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
+# This file is distributed under the same license as the PACKAGE package.
+#
+# Translators:
+# alekam <alex@kamedov.ru>, 2011
+# Jannis Leidel <jannis@leidel.info>, 2011
+msgid ""
+msgstr ""
+"Project-Id-Version: django-robots\n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2011-02-08 12:09+0100\n"
+"PO-Revision-Date: 2013-11-20 09:28+0000\n"
+"Last-Translator: Jannis Leidel <jannis@leidel.info>\n"
+"Language-Team: Russian (http://www.transifex.com/projects/p/django-robots/language/ru/)\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Language: ru\n"
+"Plural-Forms: nplurals=3; plural=(n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);\n"
+
+#: admin.py:11
+msgid "URL patterns"
+msgstr "Шаблоны URL"
+
+#: admin.py:12
+msgid "Advanced options"
+msgstr "Расширенные настройки"
+
+#: forms.py:18
+msgid "Please specify at least one allowed or disallowed URL."
+msgstr "Укажите как минимум один URL-адрес (неважно разрешенный или нет)"
+
+#: forms.py:29
+#, python-format
+msgid "URL patterns cannot be both allowed and disallowed: %(patterns)s."
+msgstr "Шаблоны URL не могут быть одновременно разрешёнными и запрещёнными: %(patterns)s."
+
+#: forms.py:48
+#, python-format
+msgid "A rule for robot %(robot)s already exists on sites: %(sites)s."
+msgstr "Правило для робота %(robot)s уже существует на сайтах: %(sites)s."
+
+#: models.py:11
+msgid "pattern"
+msgstr "шаблон"
+
+#: models.py:12
+msgid ""
+"Case-sensitive. A missing trailing slash does also match to files which "
+"start with the name of the pattern, e.g., '/admin' matches /admin.html too. "
+"Some major search engines allow an asterisk (*) as a wildcard and a dollar "
+"sign ($) to match the end of the URL, e.g., '/*.jpg$'."
+msgstr "Внимание: учитывается регистр!<br />Если в конце пропущен слэш, то под шаблон попадут все файлы, путь к которым начинается с таких же символов. Например: под шаблон \"/admin\" так же попадет \"/admin.html\".<br />Часть поисковых систем понимают звездочку (*) как произвольное количество любых символов и знак доллара ($) как символ конца URL. Например: \"/*.jpg$\""
+
+#: models.py:19 models.py:20
+msgid "url"
+msgstr "URL-адрес"
+
+#: models.py:37
+msgid "robot"
+msgstr "робот"
+
+#: models.py:38
+msgid ""
+"This should be a user agent string like 'Googlebot'. Enter an asterisk (*) "
+"for all user agents. For a full list look at the <a target=_blank "
+"href='http://www.robotstxt.org/db.html'> database of Web Robots</a>."
+msgstr "Название робота (User agent). Введите звездочку (*) для применения правил ко всем роботов. Полный список можно посмотреть в <a target=_blank href='http://www.robotstxt.org/db.html'>базе данных веб-ботов</a>."
+
+#: models.py:46 models.py:83
+msgid "allowed"
+msgstr "разрешенные URL"
+
+#: models.py:47
+msgid "The URLs which are allowed to be accessed by bots."
+msgstr "URL адреса разрешенные для индексации поисковыми роботами."
+
+#: models.py:51 models.py:87
+msgid "disallowed"
+msgstr "запрещенные URL"
+
+#: models.py:98
+msgid ""
+"The URLs which are not allowed to be accessed by bots. robots.txt is public "
+"and only asks well-behaved crawlers to stay away: it does not protect "
+"content, and listing a URL here makes it visible to everyone."
+msgstr ""
+"URL-адреса, запрещённые для обхода поисковыми роботами. Файл robots.txt "
+"публичный и лишь просит добросовестных роботов не заходить на эти адреса: "
+"он не защищает содержимое, а перечисленные здесь URL увидит любой."
+
+#: models.py:55
+msgid "sites"
+msgstr ""
+
+#: models.py:57
+msgid "crawl delay"
+msgstr "частота обновления"
+
+#: models.py:59
+msgid ""
+"Between 0.1 and 99.0. This field is supported by some search engines and "
+"defines the delay between successive crawler accesses in seconds. If the "
+"crawler rate is a problem for your server, you can set the delay up to 5 or "
+"10 or a comfortable value for your server, but it's suggested to start with "
+"small values (0.5-1), and increase as needed to an acceptable value for your"
+" server. Larger delay values add more delay between successive crawl "
+"accesses and decrease the maximum crawl rate to your web server."
+msgstr "Введите значение между 0.1 и 99.0. Этот параметр поддерживается некоторыми поисковыми системами и определяет задержку в секундах до следующего запроса робота. Если робот обнаруживает проблемы на вашем сервер, вы можите установить задержку от 5, 10 или более подходяще значение для вашего сервера, но лучше начинать с небольших значений (0.5-1), и постепенно увеличивать до достижения оптимальных значений для вашего сервера.  Большие значения увеличивают время выгрузки роботом вашего сайта и могут отрицательно влиять на ваш рейтинг в данной системе."
+
+#: models.py:75
+msgid "rule"
+msgstr "правило индексации"
+
+#: models.py:76
+msgid "rules"
+msgstr "правила индексации"
+
+#: models.py:82 models.py:86
+msgid "and"
+msgstr "и"
+
+#: models.py:106
+msgid "comment"
+msgstr "комментарий"
+
+#: models.py:111
+msgid ""
+"Optional note shown as a '# ...' line above this rule in robots.txt. "
+"robots.txt is public: do not put anything confidential here."
+msgstr ""
+"Необязательная заметка, которая выводится строкой «# ...» перед правилом в "
+"robots.txt. Файл robots.txt публичный: не пишите сюда ничего "
+"конфиденциального."
+
+#: validators.py:18
+msgid "Use a single line without line breaks or control characters."
+msgstr "Используйте одну строку без переносов и управляющих символов."
+
+#: models.py:162
+msgid "parameters"
+msgstr "параметры"
+
+#: models.py:165
+msgid "Parameter names separated by &, e.g. ref&sid. Case-sensitive."
+msgstr "Имена параметров через &, например ref&sid. Регистр учитывается."
+
+#: models.py:168
+msgid "path"
+msgstr "путь"
+
+#: models.py:173
+msgid ""
+"Optional path prefix, e.g. '/catalog/'. Leave empty to apply to the whole "
+"site."
+msgstr ""
+"Необязательный префикс пути, например '/catalog/'. Оставьте пустым, чтобы "
+"директива действовала на весь сайт."
+
+#: models.py:180
+msgid "Clean-param directive"
+msgstr "директива Clean-param"
+
+#: models.py:181
+msgid "Clean-param directives"
+msgstr "директивы Clean-param"
+
+#: models.py:196
+#, python-format
+msgid "The Clean-param directive must not exceed %(limit)d characters."
+msgstr "Директива Clean-param не должна быть длиннее %(limit)d символов."
+
+#: validators.py:33
+msgid ""
+"Enter parameter names separated by \"&\", without spaces, \"=\", \"#\" or "
+"non-ASCII characters."
+msgstr ""
+"Введите имена параметров через «&», без пробелов, «=», «#» и символов вне "
+"ASCII."
+
+#: validators.py:45
+msgid ""
+"The path may contain only Latin letters, digits and the characters \".\", "
+"\"-\", \"/\", \"*\" and \"_\"."
+msgstr ""
+"Путь может содержать только латинские буквы, цифры и символы «.», «-», «/», "
+"«*» и «_»."

robots/migrations/0001_initial.py

@@ -0,0 +1,103 @@
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("sites", "0001_initial"),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name="Rule",
+            fields=[
+                (
+                    "id",
+                    models.AutoField(
+                        auto_created=True,
+                        serialize=False,
+                        verbose_name="ID",
+                        primary_key=True,
+                    ),
+                ),
+                (
+                    "robot",
+                    models.CharField(
+                        max_length=255,
+                        help_text="This should be a user agent string like 'Googlebot'. Enter an asterisk (*) for all user agents. For a full list look at the <a target=_blank href='http://www.robotstxt.org/db.html'> database of Web Robots</a>.",
+                        verbose_name="robot",
+                    ),
+                ),
+                (
+                    "crawl_delay",
+                    models.DecimalField(
+                        blank=True,
+                        help_text="Between 0.1 and 99.0. This field is supported by some search engines and defines the delay between successive crawler accesses in seconds. If the crawler rate is a problem for your server, you can set the delay up to 5 or 10 or a comfortable value for your server, but it's suggested to start with small values (0.5-1), and increase as needed to an acceptable value for your server. Larger delay values add more delay between successive crawl accesses and decrease the maximum crawl rate to your web server.",
+                        verbose_name="crawl delay",
+                        decimal_places=1,
+                        max_digits=3,
+                        null=True,
+                    ),
+                ),
+                (
+                    "sites",
+                    models.ManyToManyField(to="sites.Site", verbose_name="sites"),
+                ),
+            ],
+            options={
+                "verbose_name_plural": "rules",
+                "verbose_name": "rule",
+            },
+            bases=(models.Model,),
+        ),
+        migrations.CreateModel(
+            name="Url",
+            fields=[
+                (
+                    "id",
+                    models.AutoField(
+                        auto_created=True,
+                        serialize=False,
+                        verbose_name="ID",
+                        primary_key=True,
+                    ),
+                ),
+                (
+                    "pattern",
+                    models.CharField(
+                        max_length=255,
+                        help_text="Case-sensitive. A missing trailing slash does also match to files which start with the name of the pattern, e.g., '/admin' matches /admin.html too. Some major search engines allow an asterisk (*) as a wildcard and a dollar sign ($) to match the end of the URL, e.g., '/*.jpg$'.",
+                        verbose_name="pattern",
+                    ),
+                ),
+            ],
+            options={
+                "verbose_name_plural": "url",
+                "verbose_name": "url",
+            },
+            bases=(models.Model,),
+        ),
+        migrations.AddField(
+            model_name="rule",
+            name="disallowed",
+            field=models.ManyToManyField(
+                to="robots.Url",
+                blank=True,
+                related_name="disallowed",
+                verbose_name="disallowed",
+                help_text="The URLs which are not allowed to be accessed by bots.",
+            ),
+            preserve_default=True,
+        ),
+        migrations.AddField(
+            model_name="rule",
+            name="allowed",
+            field=models.ManyToManyField(
+                to="robots.Url",
+                blank=True,
+                related_name="allowed",
+                verbose_name="allowed",
+                help_text="The URLs which are allowed to be accessed by bots.",
+            ),
+            preserve_default=True,
+        ),
+    ]

robots/migrations/0002_alter_id_fields.py

@@ -0,0 +1,26 @@
+# Generated by Django 3.2 on 2021-09-23 12:14
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("robots", "0001_initial"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="rule",
+            name="id",
+            field=models.BigAutoField(
+                auto_created=True, primary_key=True, serialize=False, verbose_name="ID"
+            ),
+        ),
+        migrations.AlterField(
+            model_name="url",
+            name="id",
+            field=models.BigAutoField(
+                auto_created=True, primary_key=True, serialize=False, verbose_name="ID"
+            ),
+        ),
+    ]

robots/migrations/0003_rule_comment.py

@@ -0,0 +1,25 @@
+# Generated by Django 6.1.1 on 2026-10-06 20:29
+
+from django.db import migrations, models
+
+import robots.validators
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("robots", "0002_alter_id_fields"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="rule",
+            name="comment",
+            field=models.CharField(
+                blank=True,
+                help_text="Optional note shown as a '# ...' line above this rule in robots.txt. robots.txt is public: do not put anything confidential here.",
+                max_length=255,
+                validators=[robots.validators.validate_single_line],
+                verbose_name="comment",
+            ),
+        ),
+    ]

robots/migrations/0004_cleanparam.py

@@ -0,0 +1,56 @@
+# Generated by Django 6.1.1 on 2026-10-06 21:54
+
+from django.db import migrations, models
+
+import robots.validators
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("robots", "0003_rule_comment"),
+        ("sites", "0002_alter_domain_unique"),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name="CleanParam",
+            fields=[
+                (
+                    "id",
+                    models.BigAutoField(
+                        auto_created=True,
+                        primary_key=True,
+                        serialize=False,
+                        verbose_name="ID",
+                    ),
+                ),
+                (
+                    "parameters",
+                    models.CharField(
+                        help_text="Parameter names separated by &, e.g. ref&sid. Case-sensitive.",
+                        max_length=255,
+                        validators=[robots.validators.validate_clean_param_parameters],
+                        verbose_name="parameters",
+                    ),
+                ),
+                (
+                    "path",
+                    models.CharField(
+                        blank=True,
+                        help_text="Optional path prefix, e.g. '/catalog/'. Leave empty to apply to the whole site.",
+                        max_length=255,
+                        validators=[robots.validators.validate_clean_param_path],
+                        verbose_name="path",
+                    ),
+                ),
+                (
+                    "sites",
+                    models.ManyToManyField(to="sites.site", verbose_name="sites"),
+                ),
+            ],
+            options={
+                "verbose_name": "Clean-param directive",
+                "verbose_name_plural": "Clean-param directives",
+            },
+        ),
+    ]

robots/migrations/0005_alter_rule_disallowed.py

@@ -0,0 +1,23 @@
+# Generated by Django 6.1.1 on 2026-10-06 22:43
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("robots", "0004_cleanparam"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="rule",
+            name="disallowed",
+            field=models.ManyToManyField(
+                blank=True,
+                help_text="The URLs which are not allowed to be accessed by bots. robots.txt is public and only asks well-behaved crawlers to stay away: it does not protect content, and listing a URL here makes it visible to everyone.",
+                related_name="disallowed",
+                to="robots.url",
+                verbose_name="disallowed",
+            ),
+        ),
+    ]

robots/models.py

@@ -0,0 +1,216 @@
+from urllib.parse import quote
+
+from django.contrib import admin
+from django.contrib.sites.models import Site
+from django.core.exceptions import ValidationError
+from django.db import models
+from django.utils.text import get_text_list
+from django.utils.translation import gettext_lazy as _
+
+from robots.validators import (
+    validate_clean_param_parameters,
+    validate_clean_param_path,
+    validate_single_line,
+)
+
+# Printable ASCII except "#", which starts a comment in robots.txt.
+PATTERN_SAFE_CHARS = "".join(
+    chr(code) for code in range(0x21, 0x7F) if chr(code) != "#"
+)
+
+
+def encode_pattern(pattern):
+    """Percent-encode non-ASCII characters, whitespace and "#" for robots.txt."""
+    return quote(pattern, safe=PATTERN_SAFE_CHARS)
+
+
+class Url(models.Model):
+    """
+    Defines a URL pattern for use with a robot exclusion rule. It's
+    case-sensitive and exact, e.g., "/admin" and "/admin/" are different URLs.
+    """
+
+    pattern = models.CharField(
+        _("pattern"),
+        max_length=255,
+        help_text=_(
+            "Case-sensitive. A missing trailing slash does al"
+            "so match to files which start with the name of "
+            "the pattern, e.g., '/admin' matches /admin.html "
+            "too. Some major search engines allow an asterisk"
+            " (*) as a wildcard and a dollar sign ($) to "
+            "match the end of the URL, e.g., '/*.jpg$'."
+        ),
+    )
+
+    class Meta:
+        verbose_name = _("url")
+        verbose_name_plural = _("url")
+
+    def __str__(self):
+        return self.pattern
+
+    @property
+    def encoded_pattern(self):
+        return encode_pattern(self.pattern)
+
+    def save(self, *args, **kwargs):
+        if not self.pattern.startswith("/"):
+            self.pattern = "/" + self.pattern
+        super().save(*args, **kwargs)
+
+
+class Rule(models.Model):
+    """
+    Defines an abstract rule which is used to respond to crawling web robots,
+    using the robot exclusion standard, a.k.a. robots.txt. It allows or
+    disallows the robot identified by its user agent to access the given URLs.
+    The Site contrib app is used to enable multiple robots.txt per instance.
+    """
+
+    robot = models.CharField(
+        _("robot"),
+        max_length=255,
+        help_text=_(
+            "This should be a user agent string like "
+            "'Googlebot'. Enter an asterisk (*) for all "
+            "user agents. For a full list look at the "
+            "<a target=_blank href='"
+            "http://www.robotstxt.org/db.html"
+            "'> database of Web Robots</a>."
+        ),
+    )
+
+    allowed = models.ManyToManyField(
+        Url,
+        blank=True,
+        related_name="allowed",
+        verbose_name=_("allowed"),
+        help_text=_("The URLs which are allowed to be accessed by bots."),
+    )
+
+    disallowed = models.ManyToManyField(
+        Url,
+        blank=True,
+        related_name="disallowed",
+        verbose_name=_("disallowed"),
+        help_text=_(
+            "The URLs which are not allowed to be accessed by bots. robots.txt is "
+            "public and only asks well-behaved crawlers to stay away: it does not "
+            "protect content, and listing a URL here makes it visible to everyone."
+        ),
+    )
+    sites = models.ManyToManyField(Site, verbose_name=_("sites"))
+
+    crawl_delay = models.DecimalField(
+        _("crawl delay"),
+        blank=True,
+        null=True,
+        max_digits=3,
+        decimal_places=1,
+        help_text=_(
+            "Between 0.1 and 99.0. This field is "
+            "supported by some search engines and "
+            "defines the delay between successive "
+            "crawler accesses in seconds. If the "
+            "crawler rate is a problem for your "
+            "server, you can set the delay up to 5 "
+            "or 10 or a comfortable value for your "
+            "server, but it's suggested to start "
+            "with small values (0.5-1), and "
+            "increase as needed to an acceptable "
+            "value for your server. Larger delay "
+            "values add more delay between "
+            "successive crawl accesses and "
+            "decrease the maximum crawl rate to "
+            "your web server."
+        ),
+    )
+
+    comment = models.CharField(
+        _("comment"),
+        max_length=255,
+        blank=True,
+        validators=[validate_single_line],
+        help_text=_(
+            "Optional note shown as a '# ...' line above this rule in robots.txt. "
+            "robots.txt is public: do not put anything confidential here."
+        ),
+    )
+
+    class Meta:
+        verbose_name = _("rule")
+        verbose_name_plural = _("rules")
+
+    def __str__(self):
+        return self.robot
+
+    @admin.display(description=_("allowed"))
+    def allowed_urls(self):
+        return get_text_list(list(self.allowed.all()), _("and"))
+
+    @admin.display(description=_("disallowed"))
+    def disallowed_urls(self):
+        return get_text_list(list(self.disallowed.all()), _("and"))
+
+
+class CleanParam(models.Model):
+    """
+    Defines a Yandex Clean-param directive: URL parameters that do not change
+    the page content and should be ignored under the given path prefix.
+    """
+
+    MAX_DIRECTIVE_LENGTH = 500
+
+    parameters = models.CharField(
+        _("parameters"),
+        max_length=255,
+        validators=[validate_clean_param_parameters],
+        help_text=_("Parameter names separated by &, e.g. ref&sid. Case-sensitive."),
+    )
+    path = models.CharField(
+        _("path"),
+        max_length=255,
+        blank=True,
+        validators=[validate_clean_param_path],
+        help_text=_(
+            "Optional path prefix, e.g. '/catalog/'. Leave empty to apply to the "
+            "whole site."
+        ),
+    )
+    sites = models.ManyToManyField(Site, verbose_name=_("sites"))
+
+    class Meta:
+        verbose_name = _("Clean-param directive")
+        verbose_name_plural = _("Clean-param directives")
+
+    def __str__(self):
+        return self.directive
+
+    @property
+    def directive(self):
+        if self.path:
+            return f"Clean-param: {self.parameters} {self.path}"
+        return f"Clean-param: {self.parameters}"
+
+    def add_leading_slash(self):
+        """Prefix a non-empty path with "/" unless it starts with "/" or "*"."""
+        if self.path and not self.path.startswith(("/", "*")):
+            self.path = "/" + self.path
+
+    def clean_fields(self, exclude=None):
+        self.add_leading_slash()
+        super().clean_fields(exclude=exclude)
+
+    def clean(self):
+        super().clean()
+        if len(self.directive) > self.MAX_DIRECTIVE_LENGTH:
+            raise ValidationError(
+                _("The Clean-param directive must not exceed %(limit)d characters."),
+                params={"limit": self.MAX_DIRECTIVE_LENGTH},
+                code="max_length",
+            )
+
+    def save(self, *args, **kwargs):
+        self.add_leading_slash()
+        super().save(*args, **kwargs)

robots/settings.py

@@ -0,0 +1,24 @@
+import sys
+from typing import ClassVar
+
+
+class Settings:
+    defaults: ClassVar[dict] = {
+        #: A list of one or more sitemaps to inform robots about:
+        "SITEMAP_URLS": ("ROBOTS_SITEMAP_URLS", []),
+        "USE_SITEMAP": ("ROBOTS_USE_SITEMAP", True),
+        "USE_HOST": ("ROBOTS_USE_HOST", False),
+        "CACHE_TIMEOUT": ("ROBOTS_CACHE_TIMEOUT", None),
+        "SITE_BY_REQUEST": ("ROBOTS_SITE_BY_REQUEST", False),
+        "USE_SCHEME_IN_HOST": ("ROBOTS_USE_SCHEME_IN_HOST", False),
+        "SITEMAP_VIEW_NAME": ("ROBOTS_SITEMAP_VIEW_NAME", False),
+    }
+
+    def __getattr__(self, attribute):
+        from django.conf import settings
+
+        if attribute in self.defaults:
+            return getattr(settings, *self.defaults[attribute])
+
+
+sys.modules[__name__] = Settings()

robots/templates/robots/rule_list.html

@@ -0,0 +1,13 @@
+{% load l10n %}{% if rules %}{% for rule in rules %}{% ifchanged rule.robot %}{% if not forloop.first %}
+{% endif %}{% if rule.comment %}# {{ rule.comment|safe }}
+{% endif %}User-agent: {{ rule.robot }}{% endifchanged %}
+{% for url in rule.allowed.all %}Allow: {{ url.encoded_pattern|safe }}
+{% endfor %}{% for url in rule.disallowed.all %}Disallow: {{ url.encoded_pattern|safe }}
+{% endfor %}{% if rule.crawl_delay %}Crawl-delay: {% localize off %}{{ rule.crawl_delay|floatformat:'0' }}{% endlocalize %}
+{% endif %}{% endfor %}{% else %}User-agent: *
+Disallow:
+{% endif %}{% if host or clean_params or sitemap_urls %}
+{% if host %}Host: {{ host }}
+{% endif %}{% for clean_param in clean_params %}{{ clean_param.directive|safe }}
+{% endfor %}{% for sitemap_url in sitemap_urls %}Sitemap: {{ sitemap_url }}
+{% endfor %}{% endif %}

robots/urls.py

@@ -0,0 +1,7 @@
+from django.urls import path
+
+from robots.views import rules_list
+
+urlpatterns = [
+    path("", rules_list, name="robots_rule_list"),
+]

robots/validators.py

@@ -0,0 +1,49 @@
+import re
+import unicodedata
+
+from django.core.exceptions import ValidationError
+from django.utils.translation import gettext_lazy as _
+
+# Characters Yandex allows in the Clean-param path prefix.
+CLEAN_PARAM_PATH_RE = re.compile(r"[A-Za-z0-9.\-/*_]+")
+
+# Control characters and Unicode line/paragraph separators. Tab is allowed: RFC 9309
+# permits whitespace inside comments.
+FORBIDDEN_CATEGORIES = {"Cc", "Zl", "Zp"}
+
+
+def validate_single_line(value):
+    """Reject line breaks and other control characters except tab."""
+    if any(
+        char != "\t" and unicodedata.category(char) in FORBIDDEN_CATEGORIES
+        for char in value
+    ):
+        raise ValidationError(
+            _("Use a single line without line breaks or control characters."),
+            code="invalid",
+        )
+
+
+def validate_clean_param_parameters(value):
+    """Require "&"-separated parameter names of printable ASCII without "=" and "#"."""
+    for name in value.split("&"):
+        if not name or any(not "!" <= char <= "~" or char in "=#" for char in name):
+            raise ValidationError(
+                _(
+                    'Enter parameter names separated by "&", without spaces, '
+                    '"=", "#" or non-ASCII characters.'
+                ),
+                code="invalid",
+            )
+
+
+def validate_clean_param_path(value):
+    """Allow only the characters Yandex accepts in a Clean-param path prefix."""
+    if not CLEAN_PARAM_PATH_RE.fullmatch(value):
+        raise ValidationError(
+            _(
+                "The path may contain only Latin letters, digits and the characters "
+                '".", "-", "/", "*" and "_".'
+            ),
+            code="invalid",
+        )

robots/views.py

@@ -0,0 +1,112 @@
+from django.contrib.sitemaps import views as sitemap_views
+from django.contrib.sites.models import Site
+from django.db.models import Prefetch
+from django.http import HttpRequest
+from django.http.request import split_domain_port
+from django.urls import NoReverseMatch, reverse
+from django.views.decorators.cache import cache_page
+from django.views.generic import ListView
+
+from robots import settings
+from robots.models import CleanParam, Rule, Url
+
+
+class RuleList(ListView):
+    """
+    Returns a generated robots.txt file with correct mimetype (text/plain),
+    status code (200 or 404), sitemap url (automatically).
+    """
+
+    model = Rule
+    context_object_name = "rules"
+    cache_timeout = settings.CACHE_TIMEOUT
+
+    def get_current_site(self, request: HttpRequest):
+        if settings.SITE_BY_REQUEST:
+            host = request.get_host()
+            try:
+                return Site.objects.get(domain__iexact=host)
+            except Site.DoesNotExist:
+                domain, _port = split_domain_port(host)
+                return Site.objects.get(domain__iexact=domain)
+        else:
+            return Site.objects.get_current()
+
+    def reverse_sitemap_url(self):
+        try:
+            if settings.SITEMAP_VIEW_NAME:
+                return reverse(settings.SITEMAP_VIEW_NAME)
+            else:
+                return reverse(sitemap_views.index)
+        except NoReverseMatch:
+            try:
+                return reverse(sitemap_views.sitemap)
+            except NoReverseMatch:
+                pass
+
+    def get_domain(self):
+        scheme = self.request.is_secure() and "https" or "http"
+        if not self.current_site.domain.startswith(("http", "https")):
+            return f"{scheme}://{self.current_site.domain}"
+        return self.current_site.domain
+
+    def get_sitemap_urls(self):
+        sitemap_urls = list(settings.SITEMAP_URLS)
+
+        if not sitemap_urls and settings.USE_SITEMAP:
+            sitemap_url = self.reverse_sitemap_url()
+
+            if sitemap_url is not None:
+                if not sitemap_url.startswith(("http", "https")):
+                    sitemap_url = f"{self.get_domain()}{sitemap_url}"
+                if sitemap_url not in sitemap_urls:
+                    sitemap_urls.append(sitemap_url)
+
+        return sitemap_urls
+
+    def get_queryset(self):
+        urls = Url.objects.order_by("pattern")
+        return (
+            Rule.objects.filter(sites=self.current_site)
+            .order_by("robot")
+            .prefetch_related(
+                Prefetch("allowed", queryset=urls),
+                Prefetch("disallowed", queryset=urls),
+            )
+        )
+
+    def get_context_data(self, **kwargs):
+        context = super().get_context_data(**kwargs)
+        context["sitemap_urls"] = self.get_sitemap_urls()
+        context["clean_params"] = CleanParam.objects.filter(
+            sites=self.current_site
+        ).order_by("path", "parameters")
+        if settings.USE_HOST:
+            if settings.USE_SCHEME_IN_HOST:
+                context["host"] = self.get_domain()
+            else:
+                context["host"] = self.current_site.domain
+        else:
+            context["host"] = None
+        return context
+
+    def render_to_response(self, context, **kwargs):
+        return super().render_to_response(
+            context, content_type="text/plain; charset=utf-8", **kwargs
+        )
+
+    def get_cache_timeout(self):
+        return self.cache_timeout
+
+    def dispatch(self, request: HttpRequest, *args, **kwargs):
+        cache_timeout = self.get_cache_timeout()
+        self.current_site = self.get_current_site(request)
+        super_dispatch = super().dispatch
+        if not cache_timeout:
+            return super_dispatch(request, *args, **kwargs)
+        key_prefix = self.current_site.domain
+        cache_decorator = cache_page(cache_timeout, key_prefix=key_prefix)
+        return cache_decorator(super_dispatch)(request, *args, **kwargs)
+
+
+rules_list = RuleList.as_view()