ms-cloud-endpoints 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,281 @@
1
+ Metadata-Version: 2.5
2
+ Name: ms-cloud-endpoints
3
+ Version: 0.1.0
4
+ Summary: Collect Microsoft 365, Entra and Azure domains and IP ranges from the documented Microsoft sources and export them as JSON/YAML.
5
+ Project-URL: Homepage, https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter
6
+ Project-URL: Source, https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints
7
+ Project-URL: Documentation, https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints/blob/main/README.md
8
+ Project-URL: Issues, https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints/issues
9
+ Project-URL: Changelog, https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints/blob/main/CHANGELOG.md
10
+ Author-email: Jayonics <jameshopwood1998@gmail.com>
11
+ License-Expression: MIT
12
+ License-File: LICENSE
13
+ Keywords: azure,endpoints,entra,ip-ranges,microsoft-365,office-365,private-endpoint,service-tags,split-tunnel
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Environment :: Console
16
+ Classifier: Intended Audience :: System Administrators
17
+ Classifier: Operating System :: OS Independent
18
+ Classifier: Programming Language :: Python :: 3
19
+ Classifier: Programming Language :: Python :: 3 :: Only
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Programming Language :: Python :: 3.14
23
+ Classifier: Topic :: System :: Networking
24
+ Classifier: Typing :: Typed
25
+ Requires-Python: >=3.12
26
+ Requires-Dist: pyyaml>=6.0
27
+ Description-Content-Type: text/markdown
28
+
29
+ # ms-cloud-endpoints
30
+
31
+ Collect the **domains and IP ranges of Microsoft 365, Microsoft Entra and Azure** from the sources Microsoft
32
+ documents, and export them as one normalised **JSON and/or YAML** document.
33
+
34
+ It is the data-collection stage of
35
+ [Azure-O365-DNS-UDR-SplitRouter](https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter), which routes
36
+ Microsoft traffic over a dedicated link with `systemd-resolved` routing domains and `systemd-networkd` routes. The
37
+ output is also usable on its own, for example for firewall allow-lists, proxy PAC files or split-tunnel VPN
38
+ profiles.
39
+
40
+ - [Features](#features)
41
+ - [Requirements](#requirements)
42
+ - [Installation](#installation)
43
+ - [Usage](#usage)
44
+ - [Sources and clouds](#sources-and-clouds)
45
+ - [Options](#options)
46
+ - [Output schema](#output-schema)
47
+ - [Notes and limitations](#notes-and-limitations)
48
+ - [Development](#development)
49
+ - [License](#license)
50
+
51
+ ## Features
52
+
53
+ - Fetches all documented sources, finding the weekly Service Tag download link on the download page each run
54
+ instead of hard-coding it.
55
+ - Public cloud by default, with China and US Government (Microsoft 365 GCC High/DoD) selectable.
56
+ - Normalises everything: `snake_case` keys, lower-cased domains, IPv4 and IPv6 prefixes separated.
57
+ - Expands Private Endpoint DNS zone templates (`{regionName}`, `{regionCode}`) into concrete zone names.
58
+ - Optional `summary`: deduplicated domains, wildcard suffixes, root domains and collapsed IPv4/IPv6 prefix lists
59
+ across all sources.
60
+ - Records the upstream versions (Microsoft 365 `version`, Service Tag `changeNumber`) so runs can be compared.
61
+ - The only runtime dependency is [PyYAML](https://pypi.org/project/PyYAML/). HTTP and HTML parsing use the
62
+ standard library.
63
+
64
+ ## Requirements
65
+
66
+ - Python 3.12 or later (tested on 3.12, 3.13 and 3.14)
67
+ - Outbound HTTPS to `endpoints.office.com`, `www.microsoft.com`, `download.microsoft.com` and `learn.microsoft.com`
68
+
69
+ ## Installation
70
+
71
+ From a clone of this repository (or the `ms-cloud-endpoints/` submodule of the
72
+ [parent project](https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter)):
73
+
74
+ ```bash
75
+ pipx install . # isolated command-line install (recommended)
76
+ # or
77
+ pip install . # into the current environment
78
+ ```
79
+
80
+ Directly from GitHub:
81
+
82
+ ```bash
83
+ pipx install "git+https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints.git"
84
+ ```
85
+
86
+ Both install the `ms-cloud-endpoints` command. `python -m ms_cloud_endpoints` works too.
87
+
88
+ For a version-pinned deployment once the release exists on PyPI:
89
+
90
+ ```bash
91
+ pipx install "ms-cloud-endpoints==0.1.0"
92
+ ms-cloud-endpoints --version
93
+ # Upgrade or roll back by explicitly selecting the desired published version:
94
+ pipx install --force "ms-cloud-endpoints==0.1.0"
95
+ ```
96
+
97
+ Do not use `pipx upgrade`/`upgrade-all` for pinned deployments. Pins select the application version, not a full
98
+ transitive dependency lock. For system-wide installation alongside the router, use `sudo pipx install --global`
99
+ (pipx >= 1.5); see the parent's [deployment guide](https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter/blob/main/USAGE.md#version-pinned-deployment-from-pypi).
100
+
101
+ ## Usage
102
+
103
+ ```bash
104
+ # Public cloud, all sources -> output/ms-cloud-endpoints.json and output/ms-cloud-endpoints.yaml
105
+ ms-cloud-endpoints -o output/
106
+
107
+ # Every cloud (Public, China, US Government)
108
+ ms-cloud-endpoints --cloud all -o output/
109
+
110
+ # Only the Entra ID service tags, IPv4 only, as YAML on stdout
111
+ ms-cloud-endpoints --source service-tags --tag 'AzureActiveDirectory*' --no-ipv6 --format yaml
112
+
113
+ # Microsoft 365 only, JSON to a file, keep going if a source fails
114
+ ms-cloud-endpoints --source m365 --source m365-additional -o m365.json --keep-going
115
+ ```
116
+
117
+ ### Exit codes
118
+
119
+ | Code | Meaning |
120
+ |------|---------|
121
+ | `0` | Success |
122
+ | `1` | A source or writing the output failed (nothing is written) |
123
+ | `2` | Usage error (invalid option or `ClientRequestId`) |
124
+ | `3` | Partial output: one or more sources failed with `--keep-going` (failures are listed in `metadata.sources`) |
125
+
126
+ ## Sources and clouds
127
+
128
+ `--cloud` selects which clouds to collect. It is repeatable, and `all` selects every cloud. The default is `public`.
129
+
130
+ | Source (`--source`) | `public` | `china` | `usgov` |
131
+ |---------------------|----------|---------|---------|
132
+ | `m365`: [Microsoft 365 IP Address and URL web service](https://learn.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service) `version` + `endpoints` | `Worldwide` | `China` | `USGovDoD`, `USGovGCCHigh` |
133
+ | `m365-additional`: [Other endpoints not included in the web service](https://learn.microsoft.com/en-us/microsoft-365/enterprise/additional-office365-ip-addresses-and-urls) | one page for every cloud. Rows are split by their cloud labels ("Public Cloud:", "GCC High, DoD …:", "…21Vianet:") | | |
134
+ | `service-tags`: [Azure IP Ranges and Service Tags](https://www.microsoft.com/en-us/download/details.aspx?id=56519) | `Public` (id 56519) | `China` (id 57062) | `AzureGovernment` (id 57063) |
135
+ | `private-dns`: [Private Endpoint DNS zone values](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns) + [Azure Backup geo-codes](https://learn.microsoft.com/en-us/azure/backup/scripts/geo-code-list) | `Commercial` | `China` | `Government` |
136
+
137
+ Microsoft Entra is covered by the Microsoft 365 `Common` endpoint sets, the `AzureActiveDirectory*` service tags and
138
+ the Entra rows on the "other endpoints" page.
139
+
140
+ The Microsoft 365 `changes` method isn't used, because `endpoints` always returns a complete snapshot. Compare the
141
+ recorded versions between runs instead.
142
+
143
+ ## Options
144
+
145
+ | Option | Default | Meaning |
146
+ |--------|---------|---------|
147
+ | `--cloud public\|china\|usgov\|all` | `public` | Clouds to collect (repeatable) |
148
+ | `--source m365\|m365-additional\|service-tags\|private-dns` | all | Sources to collect (repeatable) |
149
+ | `--tag GLOB` | all tags | Only keep service tags whose name matches (case-insensitive glob, repeatable), e.g. `AzureCloud`, `Storage.*` |
150
+ | `--ipv6`, `--no-ipv6` | `--ipv6` | Include IPv6 prefixes, or drop them everywhere. Recorded in `metadata.options.ipv6`. The last one given wins |
151
+ | `--no-summary` | off | Omit the `summary` section |
152
+ | `--format json\|yaml\|both` | `json` on stdout, `both` with `-o` | Output format |
153
+ | `-o`, `--output PATH` | stdout | File, or directory (existing or ending in `/`) that receives `ms-cloud-endpoints.json`/`.yaml`. With `both` and a file path, the suffix is replaced per format |
154
+ | `--client-request-id GUID` | see below | `ClientRequestId` sent to the Microsoft 365 web service |
155
+ | `--keep-going` | off | Record failed sources in `metadata.sources` and still write the output |
156
+ | `--timeout SECONDS` | `60` | HTTP timeout (three attempts with backoff on network errors, 429 and 5xx) |
157
+ | `-v`, `-vv` | warnings | Info or debug logging on stderr |
158
+ | `--version` | | Print the version |
159
+
160
+ **ClientRequestId.** Microsoft asks for one GUID per machine. The value comes from `--client-request-id` first, then
161
+ `$MS_ENDPOINTS_CLIENT_REQUEST_ID`. Otherwise a GUID is generated once and stored in
162
+ `$XDG_CONFIG_HOME/ms-cloud-endpoints/client-request-id` (default `~/.config/…`). It's only created when the `m365`
163
+ source is selected.
164
+
165
+ ## Output schema
166
+
167
+ JSON and YAML have the same structure. Keys are `snake_case`, domains are lower-cased, and IP prefixes are
168
+ normalised and split by family.
169
+
170
+ ```yaml
171
+ metadata: # generator, generator_version, generated_at, clouds, sources_selected, options, client_request_id
172
+ sources: # one record per fetched document: source, scope, url, version, fetched_at, status[, error]
173
+ m365:
174
+ Worldwide:
175
+ version: '2026081400'
176
+ endpoint_sets: # id, service_area, service_area_display_name, category, required, express_route,
177
+ # urls, ipv4, ipv6, tcp_ports, udp_ports, notes
178
+ m365_additional:
179
+ source_url: …
180
+ rows: # row, purpose, source, destination_text (raw), type, domains, ipv4, ipv6,
181
+ # by_cloud.{public,usgov,china}.{domains,ipv4,ipv6}
182
+ azure_service_tags:
183
+ Public: # cloud, change_number, regions, source_url
184
+ tags: # name, id, region, region_id, system_service, platform, network_features, change_number, ipv4, ipv6
185
+ private_dns_zones:
186
+ backup_geo_codes: # geo_code, region_name, region
187
+ Commercial: # category, resource_type, subresources, zone_templates, zones, forwarder_templates,
188
+ # forwarders, unresolved_placeholders, footnotes
189
+ summary:
190
+ domains: # exact, wildcard, wildcard_suffixes, root_domains
191
+ ipv4: [] # collapsed across all selected sources and clouds
192
+ ipv6: []
193
+ counts: {}
194
+ ```
195
+
196
+ ## Notes and limitations
197
+
198
+ - **Private Endpoint templates.**
199
+ - `{regionName}` and `{region}` are expanded with the regions from the same cloud's service tags, so they stay
200
+ unexpanded unless `service-tags` is collected in the same run.
201
+ - `{regionCode}` is expanded with the lower-cased Azure Backup geo-codes.
202
+ - Other placeholders (`{dnsPrefix}`, `{subzone}`, `{partitionId}`, …) can't be expanded and are listed in
203
+ `unresolved_placeholders`.
204
+ - In the summary every placeholder counts as a `*` wildcard, so `{regionName}.kusto.windows.net` adds the suffix
205
+ `kusto.windows.net`.
206
+ - **"Other endpoints" page.** This page is free text, so extraction is heuristic: FQDNs, wildcards, IPs and CIDRs
207
+ are pulled out of the prose. The raw `destination_text` is always kept. When the destination is empty, the
208
+ purpose text is scanned instead.
209
+ - **Third-party domains.** The summary includes the certificate authority and CDN domains Microsoft lists (for
210
+ example `digicert.com`, `akamaihd.net`). Filter them out if you use the list for DNS routing. `root_domains` takes
211
+ the last two labels (three for known multi-label suffixes such as `com.cn`). It doesn't use the Public Suffix
212
+ List.
213
+ - **Upstream HTML changes.** The Learn pages are parsed by section id and table header text. If Microsoft changes
214
+ their structure, the affected source fails with a clear error instead of returning partial data.
215
+ - **Size.** All Public service tags make the JSON about 6 MB. Use `--tag` to narrow it.
216
+ - **Refresh cadence.** Microsoft 365 endpoints change monthly (with 30 days' notice). Service tags are published
217
+ weekly. Check the Microsoft 365 `version` at most once an hour.
218
+
219
+ ## Development
220
+
221
+ Development dependencies are declared as a [PEP 735](https://peps.python.org/pep-0735/) dependency group (`dev`:
222
+ pytest and ruff).
223
+
224
+ ```bash
225
+ python -m venv .venv && . .venv/bin/activate
226
+ pip install -e . --group dev # pip >= 25.1; with uv: `uv sync`
227
+
228
+ pytest # offline tests against the fixtures in tests/fixtures/
229
+ ruff check . && ruff format --check .
230
+ python -m build # sdist + wheel in dist/ (pip install build)
231
+ ```
232
+
233
+ Layout:
234
+
235
+ ```
236
+ src/ms_cloud_endpoints/
237
+ cli.py command line, orchestration, exit codes
238
+ clouds.py cloud name -> upstream identifiers
239
+ fetch.py urllib client with retries (Fetcher protocol)
240
+ html_tables.py html.parser DOM + Learn page section/table helpers
241
+ netutil.py prefix/domain normalisation and extraction
242
+ output.py JSON/YAML writers (atomic file writes)
243
+ summary.py cross-source aggregation
244
+ sources/ one module per upstream source
245
+ tests/ pytest suite; fixtures/ holds trimmed copies of each upstream document
246
+ ```
247
+
248
+ The repository's [pre-commit configuration](.pre-commit-config.yaml)
249
+ runs ruff on this project (`pre-commit install`). CI runs lint, the tests on Python 3.12 to 3.14, and a package
250
+ build.
251
+
252
+ ### Releases
253
+
254
+ The existing workflow also runs on `v*` tag pushes. Stable tags such as `v0.1.0` must exactly match the version
255
+ in the built wheel (sourced from `src/ms_cloud_endpoints/__init__.py`); prerelease or mismatched tags fail.
256
+ After lint, tests and build checks pass, the `pypi` environment gates PyPI Trusted Publishing. A subsequent
257
+ job creates a GitHub Release with the same wheel and source distribution. Pull requests, branch pushes and
258
+ manual runs never publish.
259
+
260
+ Before tagging, configure this repository's `pypi` GitHub environment to allow deployments only from `v*`
261
+ tags (optionally with required reviewers, where the GitHub plan supports them). Register a PyPI Trusted Publisher for project `ms-cloud-endpoints`,
262
+ owner `Jayonics`, repository `Azure-O365-DNS-UDR-SplitRouter.ms-cloud-endpoints`, workflow
263
+ `ms-cloud-endpoints.yml`, environment `pypi`. No PyPI API token is required. Optionally restrict release-tag
264
+ creation and mutation with a tag ruleset.
265
+
266
+ Commit the version and changelog first, then release the reviewed commit:
267
+
268
+ ```bash
269
+ git tag -a v0.1.0 -m "Release 0.1.0"
270
+ git push origin main
271
+ git push origin v0.1.0
272
+ ```
273
+
274
+ Pushing the tag publishes automatically once checks pass (or after approval, if reviewers are required). Never move a published tag or reuse an uploaded PyPI
275
+ version. If GitHub release creation fails after PyPI succeeds, re-run only failed jobs, not the whole workflow.
276
+ See the parent's [release procedure](https://github.com/Jayonics/Azure-O365-DNS-UDR-SplitRouter/blob/main/USAGE.md#publishing-releases)
277
+ for the complete setup and coordinated deployment process.
278
+
279
+ ## License
280
+
281
+ [MIT](LICENSE) © Radux Technologies
@@ -0,0 +1,21 @@
1
+ ms_cloud_endpoints/__init__.py,sha256=9HrJ2I84aneyQINriRXdKal7XY_KYIqTyfOrv57o5Bk,162
2
+ ms_cloud_endpoints/__main__.py,sha256=4JMK66Wj4uLZTKbF-sT3LAxOsr6buig77PmOkJCRRxw,83
3
+ ms_cloud_endpoints/cli.py,sha256=Yhhlpfus4YSwy6H69_Vay2kYMpzlPUlpsOEGS9iwabI,9583
4
+ ms_cloud_endpoints/clouds.py,sha256=VTXa1dJkUi7Gcr_nuFVFZXhlyEBJFXZ5Z50r6dp4RfM,1775
5
+ ms_cloud_endpoints/fetch.py,sha256=NyJuSwyFPqM7xsswHzDr8TzU8rnQOSkzdZdzYKiRfoI,2794
6
+ ms_cloud_endpoints/html_tables.py,sha256=DWHpLYAUmsItJkbYq2ZWPhwdajGGw6MEolZ0XpNlExU,6209
7
+ ms_cloud_endpoints/model.py,sha256=qWAF_TtXeZ0KSSegOwOAWr0zTMwxKrVd2Mf0FSpAYgE,305
8
+ ms_cloud_endpoints/netutil.py,sha256=nb-RYIZ03S65CVwzQ0CrQlmQtTsGpIxXXiYReF_7FyY,3909
9
+ ms_cloud_endpoints/output.py,sha256=5FqqTZURrMKW5a_4IMSRGVM-kw5ZE4zaVqy52SW-qHA,2537
10
+ ms_cloud_endpoints/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
11
+ ms_cloud_endpoints/summary.py,sha256=JF3tXVWuEF1T6N6Gidqdj_nWAuDpaZWbFzsq_iqt7P0,3338
12
+ ms_cloud_endpoints/sources/__init__.py,sha256=_Sk08lNIh8QDG1M8VqDyJ7Zbs3tmBKClTCNIlaJ5IQs,64
13
+ ms_cloud_endpoints/sources/m365.py,sha256=e4_3Z0EL4512RUkPvlfUs7_KJSnGNdrpAikTz1gPvXQ,2719
14
+ ms_cloud_endpoints/sources/m365_additional.py,sha256=tdXaaAlFMgv4A1wlrgsArz-lhBRfN65Nyn2G6dch75w,4454
15
+ ms_cloud_endpoints/sources/private_dns.py,sha256=Hk7eKUaH4cC2ZrMv2C0_ZCil-QTfFRj1LU2Lytpo9Tc,8400
16
+ ms_cloud_endpoints/sources/service_tags.py,sha256=TgxSHfcLrbCNedz5uwTqJZmnWcbj-2soxKqwozY2dgo,3739
17
+ ms_cloud_endpoints-0.1.0.dist-info/METADATA,sha256=UdjT41xsJg9vp318JFWJneyrlpwvos5MR-bdV10TfhY,14792
18
+ ms_cloud_endpoints-0.1.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
19
+ ms_cloud_endpoints-0.1.0.dist-info/entry_points.txt,sha256=cPNQyAG4FFJWZhqKZVXpjeKFgF8ECoH3SpHPX80jEsU,67
20
+ ms_cloud_endpoints-0.1.0.dist-info/licenses/LICENSE,sha256=qUMchGaKzT8fH2lH_5tHa_I5uuiE6uHiB8Qui-OHqUw,1075
21
+ ms_cloud_endpoints-0.1.0.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.33.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ ms-cloud-endpoints = ms_cloud_endpoints.cli:main
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Radux Technologies
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.