pystates-CLI 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pystates_cli-0.1.0/PKG-INFO +193 -0
- pystates_cli-0.1.0/README.md +165 -0
- pystates_cli-0.1.0/pyproject.toml +52 -0
- pystates_cli-0.1.0/pystates/__init__.py +1 -0
- pystates_cli-0.1.0/pystates/__main__.py +4 -0
- pystates_cli-0.1.0/pystates/cli.py +34 -0
- pystates_cli-0.1.0/pystates/exporters.py +43 -0
- pystates_cli-0.1.0/pystates/parser.py +25 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/PKG-INFO +193 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/SOURCES.txt +13 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/dependency_links.txt +1 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/entry_points.txt +2 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/requires.txt +6 -0
- pystates_cli-0.1.0/pystates_CLI.egg-info/top_level.txt +2 -0
- pystates_cli-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pystates-CLI
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: CLI tool to download and save articles from various web resources into Markdown, Text, and other formats.
|
|
5
|
+
Author: Maksym Khlystun
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/yourusername/pystates
|
|
8
|
+
Project-URL: Repository, https://github.com/yourusername/pystates
|
|
9
|
+
Project-URL: Issues, https://github.com/yourusername/pystates/issues
|
|
10
|
+
Keywords: cli,parser,articles,markdown,scraper,text-extractor
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Topic :: Utilities
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
Requires-Dist: requests>=2.31.0
|
|
24
|
+
Requires-Dist: trafilatura>=1.6.0
|
|
25
|
+
Requires-Dist: rich>=13.0.0
|
|
26
|
+
Provides-Extra: pdf
|
|
27
|
+
Requires-Dist: reportlab>=4.0.0; extra == "pdf"
|
|
28
|
+
|
|
29
|
+
<div align="center">
|
|
30
|
+
|
|
31
|
+
# PyStates
|
|
32
|
+
|
|
33
|
+
> **A lightweight, elegant Python CLI tool for fetching clean, distraction-free web articles.**
|
|
34
|
+
|
|
35
|
+
[](https://www.python.org/)
|
|
36
|
+
[](https://opensource.org/licenses/MIT)
|
|
37
|
+
[](https://github.com/adbar/trafilatura)
|
|
38
|
+
|
|
39
|
+
</div>
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## ๐ Description
|
|
43
|
+
|
|
44
|
+
**PyStates** is a fast, minimalist command-line utility that downloads full-length articles from major publications (The New Yorker, Wired, The Atlantic, and hundreds more) and extracts clean, readable text using the industry-leading [Trafilatura](https://github.com/adbar/trafilatura) library.
|
|
45
|
+
|
|
46
|
+
It automatically removes ads, navigation menus, sidebars, comments, and boilerplate, then saves the result as either beautifully formatted **Markdown** (`.md`) or clean **Plain Text** (`.txt`).
|
|
47
|
+
|
|
48
|
+
Designed for researchers, writers, journalists, students, and power users who want to archive high-quality content locally โ without distractions or paywalls interfering with readability.
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
pystates -u https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## โจ Features
|
|
55
|
+
|
|
56
|
+
- ๐ **Blazing Fast** โ Downloads and processes articles in under 3 seconds
|
|
57
|
+
- ๐ก๏ธ **403 Bypass** โ Smart custom `User-Agent` headers to reliably avoid common blocks
|
|
58
|
+
- ๐งผ **Pristine Extraction** โ Powered by Trafilatura for the cleanest possible text
|
|
59
|
+
- ๐ **Dual Format Support** โ Native `.md` (with headings, emphasis, and structure) or `.txt`
|
|
60
|
+
- ๐งน **Intelligent Filename Sanitization** โ Generates safe OS filenames from article titles **without using `re`**
|
|
61
|
+
- โ๏ธ **Custom Filenames** โ Override auto-generated names with the `-o` flag
|
|
62
|
+
- ๐ **Modern Python Stack** โ Built for Python 3.10+ with clean architecture
|
|
63
|
+
- ๐ฆ **One-Command Installation** โ `pip install -e .` for instant local development
|
|
64
|
+
- ๐ **Broad Compatibility** โ Works with virtually any news, magazine, or blog site supported by Trafilatura
|
|
65
|
+
|
|
66
|
+
## ๐ Installation (Local Dev Mode)
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
# 1. Clone the repository
|
|
70
|
+
git clone https://github.com/your-org/pystates.git
|
|
71
|
+
cd pystates
|
|
72
|
+
|
|
73
|
+
# 2. Install in editable/development mode
|
|
74
|
+
pip install -e .
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### Verify Installation
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
pystates --help
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### Requirements
|
|
84
|
+
|
|
85
|
+
- Python โฅ 3.10
|
|
86
|
+
- `pip`
|
|
87
|
+
|
|
88
|
+
All dependencies (`requests`, `trafilatura`, `argparse`) are automatically installed via `pyproject.toml`.
|
|
89
|
+
|
|
90
|
+
## ๐ฅ๏ธ Usage
|
|
91
|
+
|
|
92
|
+
### Command Syntax
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
pystates -u URL [OPTIONS]
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### CLI Flags
|
|
99
|
+
|
|
100
|
+
| Short | Long | Required | Default | Description |
|
|
101
|
+
|-------|-------------|----------|---------|-------------|
|
|
102
|
+
| `-u` | `--url` | **Yes** | โ | The full URL of the article to fetch and process |
|
|
103
|
+
| `-f` | `--format` | No | `md` | Output format. Accepted values: `md` or `txt` |
|
|
104
|
+
| `-o` | `--output` | No | *(auto)* | Custom output filename **without extension**. If omitted, a sanitized version of the article title is used |
|
|
105
|
+
|
|
106
|
+
### Examples
|
|
107
|
+
|
|
108
|
+
#### 1. Basic usage (Markdown, auto-generated filename)
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
pystates -u "https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era"
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
โ Creates `the-new-ai-era.md`
|
|
115
|
+
|
|
116
|
+
#### 2. Save as plain text
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pystates -u "https://www.wired.com/story/the-future-of-coding/" -f txt
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
โ Creates `the-future-of-coding.txt`
|
|
123
|
+
|
|
124
|
+
#### 3. Custom output filename
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
pystates -u "https://www.theatlantic.com/ideas/archive/2025/07/climate-change/" \
|
|
128
|
+
-o "climate_report_july_2025" \
|
|
129
|
+
-f md
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
โ Creates `climate_report_july_2025.md`
|
|
133
|
+
|
|
134
|
+
#### 4. Full-featured command
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
pystates \
|
|
138
|
+
--url "https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it" \
|
|
139
|
+
--format txt \
|
|
140
|
+
--output "end_of_internet"
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
#### 5. View all options
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
pystates --help
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### Sample Output (Markdown)
|
|
150
|
+
|
|
151
|
+
```markdown
|
|
152
|
+
# The New AI Era
|
|
153
|
+
|
|
154
|
+
**By Jane Smith**
|
|
155
|
+
*Published: January 20, 2025*
|
|
156
|
+
*Source: The New Yorker*
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
Artificial intelligence is no longer a distant promise...
|
|
161
|
+
|
|
162
|
+
## A New Paradigm
|
|
163
|
+
|
|
164
|
+
The implications are profound...
|
|
165
|
+
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
## ๐ Project Structure
|
|
169
|
+
|
|
170
|
+
```
|
|
171
|
+
PyStates/
|
|
172
|
+
โโโ pystates/
|
|
173
|
+
โ โโโ __init__.py # Package metadata & version
|
|
174
|
+
โ โโโ __main__.py # Entry point for `python -m pystates`
|
|
175
|
+
โ โโโ cli.py # Argument parsing (argparse)
|
|
176
|
+
โ โโโ exporters.py # Filename sanitization + .md/.txt formatting
|
|
177
|
+
โ โโโ parser.py # HTTP fetching (custom UA) + Trafilatura parsing
|
|
178
|
+
โโโ pyproject.toml # Modern packaging + entry points
|
|
179
|
+
โโโ .gitignore
|
|
180
|
+
โโโ README.md
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
## ๐ License
|
|
184
|
+
|
|
185
|
+
This project is released under the **MIT License**.
|
|
186
|
+
|
|
187
|
+
See [LICENSE](LICENSE) for the full text.
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
**PyStates** โ preserving the web's best writing, one clean file at a time.
|
|
192
|
+
|
|
193
|
+
*Built with care by the open-source community.*
|
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+
# PyStates
|
|
4
|
+
|
|
5
|
+
> **A lightweight, elegant Python CLI tool for fetching clean, distraction-free web articles.**
|
|
6
|
+
|
|
7
|
+
[](https://www.python.org/)
|
|
8
|
+
[](https://opensource.org/licenses/MIT)
|
|
9
|
+
[](https://github.com/adbar/trafilatura)
|
|
10
|
+
|
|
11
|
+
</div>
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## ๐ Description
|
|
15
|
+
|
|
16
|
+
**PyStates** is a fast, minimalist command-line utility that downloads full-length articles from major publications (The New Yorker, Wired, The Atlantic, and hundreds more) and extracts clean, readable text using the industry-leading [Trafilatura](https://github.com/adbar/trafilatura) library.
|
|
17
|
+
|
|
18
|
+
It automatically removes ads, navigation menus, sidebars, comments, and boilerplate, then saves the result as either beautifully formatted **Markdown** (`.md`) or clean **Plain Text** (`.txt`).
|
|
19
|
+
|
|
20
|
+
Designed for researchers, writers, journalists, students, and power users who want to archive high-quality content locally โ without distractions or paywalls interfering with readability.
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
pystates -u https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## โจ Features
|
|
27
|
+
|
|
28
|
+
- ๐ **Blazing Fast** โ Downloads and processes articles in under 3 seconds
|
|
29
|
+
- ๐ก๏ธ **403 Bypass** โ Smart custom `User-Agent` headers to reliably avoid common blocks
|
|
30
|
+
- ๐งผ **Pristine Extraction** โ Powered by Trafilatura for the cleanest possible text
|
|
31
|
+
- ๐ **Dual Format Support** โ Native `.md` (with headings, emphasis, and structure) or `.txt`
|
|
32
|
+
- ๐งน **Intelligent Filename Sanitization** โ Generates safe OS filenames from article titles **without using `re`**
|
|
33
|
+
- โ๏ธ **Custom Filenames** โ Override auto-generated names with the `-o` flag
|
|
34
|
+
- ๐ **Modern Python Stack** โ Built for Python 3.10+ with clean architecture
|
|
35
|
+
- ๐ฆ **One-Command Installation** โ `pip install -e .` for instant local development
|
|
36
|
+
- ๐ **Broad Compatibility** โ Works with virtually any news, magazine, or blog site supported by Trafilatura
|
|
37
|
+
|
|
38
|
+
## ๐ Installation (Local Dev Mode)
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
# 1. Clone the repository
|
|
42
|
+
git clone https://github.com/your-org/pystates.git
|
|
43
|
+
cd pystates
|
|
44
|
+
|
|
45
|
+
# 2. Install in editable/development mode
|
|
46
|
+
pip install -e .
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
### Verify Installation
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
pystates --help
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Requirements
|
|
56
|
+
|
|
57
|
+
- Python โฅ 3.10
|
|
58
|
+
- `pip`
|
|
59
|
+
|
|
60
|
+
All dependencies (`requests`, `trafilatura`, `argparse`) are automatically installed via `pyproject.toml`.
|
|
61
|
+
|
|
62
|
+
## ๐ฅ๏ธ Usage
|
|
63
|
+
|
|
64
|
+
### Command Syntax
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
pystates -u URL [OPTIONS]
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### CLI Flags
|
|
71
|
+
|
|
72
|
+
| Short | Long | Required | Default | Description |
|
|
73
|
+
|-------|-------------|----------|---------|-------------|
|
|
74
|
+
| `-u` | `--url` | **Yes** | โ | The full URL of the article to fetch and process |
|
|
75
|
+
| `-f` | `--format` | No | `md` | Output format. Accepted values: `md` or `txt` |
|
|
76
|
+
| `-o` | `--output` | No | *(auto)* | Custom output filename **without extension**. If omitted, a sanitized version of the article title is used |
|
|
77
|
+
|
|
78
|
+
### Examples
|
|
79
|
+
|
|
80
|
+
#### 1. Basic usage (Markdown, auto-generated filename)
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
pystates -u "https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era"
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
โ Creates `the-new-ai-era.md`
|
|
87
|
+
|
|
88
|
+
#### 2. Save as plain text
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
pystates -u "https://www.wired.com/story/the-future-of-coding/" -f txt
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
โ Creates `the-future-of-coding.txt`
|
|
95
|
+
|
|
96
|
+
#### 3. Custom output filename
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
pystates -u "https://www.theatlantic.com/ideas/archive/2025/07/climate-change/" \
|
|
100
|
+
-o "climate_report_july_2025" \
|
|
101
|
+
-f md
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
โ Creates `climate_report_july_2025.md`
|
|
105
|
+
|
|
106
|
+
#### 4. Full-featured command
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
pystates \
|
|
110
|
+
--url "https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it" \
|
|
111
|
+
--format txt \
|
|
112
|
+
--output "end_of_internet"
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
#### 5. View all options
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
pystates --help
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
### Sample Output (Markdown)
|
|
122
|
+
|
|
123
|
+
```markdown
|
|
124
|
+
# The New AI Era
|
|
125
|
+
|
|
126
|
+
**By Jane Smith**
|
|
127
|
+
*Published: January 20, 2025*
|
|
128
|
+
*Source: The New Yorker*
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
Artificial intelligence is no longer a distant promise...
|
|
133
|
+
|
|
134
|
+
## A New Paradigm
|
|
135
|
+
|
|
136
|
+
The implications are profound...
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
## ๐ Project Structure
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
PyStates/
|
|
144
|
+
โโโ pystates/
|
|
145
|
+
โ โโโ __init__.py # Package metadata & version
|
|
146
|
+
โ โโโ __main__.py # Entry point for `python -m pystates`
|
|
147
|
+
โ โโโ cli.py # Argument parsing (argparse)
|
|
148
|
+
โ โโโ exporters.py # Filename sanitization + .md/.txt formatting
|
|
149
|
+
โ โโโ parser.py # HTTP fetching (custom UA) + Trafilatura parsing
|
|
150
|
+
โโโ pyproject.toml # Modern packaging + entry points
|
|
151
|
+
โโโ .gitignore
|
|
152
|
+
โโโ README.md
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## ๐ License
|
|
156
|
+
|
|
157
|
+
This project is released under the **MIT License**.
|
|
158
|
+
|
|
159
|
+
See [LICENSE](LICENSE) for the full text.
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
**PyStates** โ preserving the web's best writing, one clean file at a time.
|
|
164
|
+
|
|
165
|
+
*Built with care by the open-source community.*
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61.0"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "pystates-CLI"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "CLI tool to download and save articles from various web resources into Markdown, Text, and other formats."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.9"
|
|
11
|
+
license = { text = "MIT" }
|
|
12
|
+
authors = [
|
|
13
|
+
{ name = "Maksym Khlystun" }
|
|
14
|
+
]
|
|
15
|
+
keywords = ["cli", "parser", "articles", "markdown", "scraper", "text-extractor"]
|
|
16
|
+
classifiers = [
|
|
17
|
+
"Development Status :: 3 - Alpha",
|
|
18
|
+
"Environment :: Console",
|
|
19
|
+
"Intended Audience :: Developers",
|
|
20
|
+
"License :: OSI Approved :: MIT License",
|
|
21
|
+
"Programming Language :: Python :: 3",
|
|
22
|
+
"Programming Language :: Python :: 3.9",
|
|
23
|
+
"Programming Language :: Python :: 3.10",
|
|
24
|
+
"Programming Language :: Python :: 3.11",
|
|
25
|
+
"Programming Language :: Python :: 3.12",
|
|
26
|
+
"Topic :: Utilities",
|
|
27
|
+
]
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
dependencies = [
|
|
31
|
+
"requests>=2.31.0",
|
|
32
|
+
"trafilatura>=1.6.0",
|
|
33
|
+
"rich>=13.0.0",
|
|
34
|
+
]
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
[project.optional-dependencies]
|
|
38
|
+
pdf = [
|
|
39
|
+
"reportlab>=4.0.0",
|
|
40
|
+
]
|
|
41
|
+
|
|
42
|
+
[project.urls]
|
|
43
|
+
Homepage = "https://github.com/yourusername/pystates"
|
|
44
|
+
Repository = "https://github.com/yourusername/pystates"
|
|
45
|
+
Issues = "https://github.com/yourusername/pystates/issues"
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
[project.scripts]
|
|
49
|
+
pystates = "pystates.cli:main"
|
|
50
|
+
|
|
51
|
+
[tool.setuptools.packages.find]
|
|
52
|
+
where = ["."]
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "0.1.0"
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import argparse
|
|
2
|
+
import sys
|
|
3
|
+
|
|
4
|
+
|
|
5
|
+
from pystates.parser import fetch_and_parse_article
|
|
6
|
+
from pystates.exporters import save_article
|
|
7
|
+
|
|
8
|
+
|
|
9
|
+
def main():
|
|
10
|
+
parser = argparse.ArgumentParser(description="Download and save web articles")
|
|
11
|
+
parser.add_argument('-u', '--url', required=True, help="URL to article")
|
|
12
|
+
parser.add_argument('-f', '--format', default='md', choices=['md', 'txt'], help="Output format (md or txt)")
|
|
13
|
+
parser.add_argument('-o', '--output', default=None, help="Custom output filename")
|
|
14
|
+
|
|
15
|
+
args = parser.parse_args()
|
|
16
|
+
|
|
17
|
+
try:
|
|
18
|
+
print(f"Downloading article from: {args.url}...")
|
|
19
|
+
article_data = fetch_and_parse_article(args.url)
|
|
20
|
+
|
|
21
|
+
saved_file = save_article(
|
|
22
|
+
title=article_data["title"],
|
|
23
|
+
content=article_data["content"],
|
|
24
|
+
output_format=args.format,
|
|
25
|
+
custom_filename=args.output
|
|
26
|
+
)
|
|
27
|
+
print(f"Saved article to: {saved_file}")
|
|
28
|
+
except Exception as e:
|
|
29
|
+
print(f"Error: {e}")
|
|
30
|
+
sys.exit(1)
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
if __name__ == "__main__":
|
|
34
|
+
main()
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
import os
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
def sanitize_filename(filename: str) -> str:
|
|
5
|
+
FORBIDDEN_CHARS = ['/', '\\', ':', '*', '?', '"', '<', '>', '|', '\n', '\r', '\t']
|
|
6
|
+
|
|
7
|
+
clean_name = filename
|
|
8
|
+
for char in FORBIDDEN_CHARS:
|
|
9
|
+
clean_name = clean_name.replace(char, '')
|
|
10
|
+
|
|
11
|
+
clean_name = clean_name.replace(' ', '_').strip('_')
|
|
12
|
+
|
|
13
|
+
if not clean_name:
|
|
14
|
+
clean_name = "article"
|
|
15
|
+
|
|
16
|
+
return clean_name
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
def save_article(title: str, content: str, output_format: str = "md", custom_filename: str = None) -> str:
|
|
20
|
+
if custom_filename:
|
|
21
|
+
base_name = custom_filename
|
|
22
|
+
else:
|
|
23
|
+
base_name = sanitize_filename(title)
|
|
24
|
+
|
|
25
|
+
fmt = output_format.lower()
|
|
26
|
+
if fmt not in ["md", "txt"]:
|
|
27
|
+
fmt = "md"
|
|
28
|
+
|
|
29
|
+
if not base_name.endswith(f".{fmt}"):
|
|
30
|
+
full_filename = f"{base_name}.{fmt}"
|
|
31
|
+
else:
|
|
32
|
+
full_filename = base_name
|
|
33
|
+
|
|
34
|
+
if fmt == "md":
|
|
35
|
+
formatted_text = f"# {title}\n\n{content}\n"
|
|
36
|
+
else:
|
|
37
|
+
underline = "=" * len(title)
|
|
38
|
+
formatted_text = f"{title}\n{underline}\n\n{content}\n"
|
|
39
|
+
|
|
40
|
+
with open(full_filename, "w", encoding="utf-8") as f:
|
|
41
|
+
f.write(formatted_text)
|
|
42
|
+
|
|
43
|
+
return full_filename
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
import requests
|
|
2
|
+
import trafilatura
|
|
3
|
+
|
|
4
|
+
|
|
5
|
+
def fetch_and_parse_article(url: str) -> dict:
|
|
6
|
+
headers = {
|
|
7
|
+
"User-Agent": (
|
|
8
|
+
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
9
|
+
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
10
|
+
"Chrome/120.0.0.0 Safari/537.36"
|
|
11
|
+
)
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
response = requests.get(url, headers=headers, timeout=15)
|
|
15
|
+
response.raise_for_status()
|
|
16
|
+
|
|
17
|
+
res = trafilatura.bare_extraction(response.text, as_dict=True)
|
|
18
|
+
|
|
19
|
+
if res is None:
|
|
20
|
+
raise ValueError('Failed to extract text')
|
|
21
|
+
|
|
22
|
+
title = res.get("title") or "No name"
|
|
23
|
+
content = res.get("text") or ""
|
|
24
|
+
|
|
25
|
+
return {"title": title, "content": content}
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pystates-CLI
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: CLI tool to download and save articles from various web resources into Markdown, Text, and other formats.
|
|
5
|
+
Author: Maksym Khlystun
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/yourusername/pystates
|
|
8
|
+
Project-URL: Repository, https://github.com/yourusername/pystates
|
|
9
|
+
Project-URL: Issues, https://github.com/yourusername/pystates/issues
|
|
10
|
+
Keywords: cli,parser,articles,markdown,scraper,text-extractor
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Topic :: Utilities
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
Requires-Dist: requests>=2.31.0
|
|
24
|
+
Requires-Dist: trafilatura>=1.6.0
|
|
25
|
+
Requires-Dist: rich>=13.0.0
|
|
26
|
+
Provides-Extra: pdf
|
|
27
|
+
Requires-Dist: reportlab>=4.0.0; extra == "pdf"
|
|
28
|
+
|
|
29
|
+
<div align="center">
|
|
30
|
+
|
|
31
|
+
# PyStates
|
|
32
|
+
|
|
33
|
+
> **A lightweight, elegant Python CLI tool for fetching clean, distraction-free web articles.**
|
|
34
|
+
|
|
35
|
+
[](https://www.python.org/)
|
|
36
|
+
[](https://opensource.org/licenses/MIT)
|
|
37
|
+
[](https://github.com/adbar/trafilatura)
|
|
38
|
+
|
|
39
|
+
</div>
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## ๐ Description
|
|
43
|
+
|
|
44
|
+
**PyStates** is a fast, minimalist command-line utility that downloads full-length articles from major publications (The New Yorker, Wired, The Atlantic, and hundreds more) and extracts clean, readable text using the industry-leading [Trafilatura](https://github.com/adbar/trafilatura) library.
|
|
45
|
+
|
|
46
|
+
It automatically removes ads, navigation menus, sidebars, comments, and boilerplate, then saves the result as either beautifully formatted **Markdown** (`.md`) or clean **Plain Text** (`.txt`).
|
|
47
|
+
|
|
48
|
+
Designed for researchers, writers, journalists, students, and power users who want to archive high-quality content locally โ without distractions or paywalls interfering with readability.
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
pystates -u https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## โจ Features
|
|
55
|
+
|
|
56
|
+
- ๐ **Blazing Fast** โ Downloads and processes articles in under 3 seconds
|
|
57
|
+
- ๐ก๏ธ **403 Bypass** โ Smart custom `User-Agent` headers to reliably avoid common blocks
|
|
58
|
+
- ๐งผ **Pristine Extraction** โ Powered by Trafilatura for the cleanest possible text
|
|
59
|
+
- ๐ **Dual Format Support** โ Native `.md` (with headings, emphasis, and structure) or `.txt`
|
|
60
|
+
- ๐งน **Intelligent Filename Sanitization** โ Generates safe OS filenames from article titles **without using `re`**
|
|
61
|
+
- โ๏ธ **Custom Filenames** โ Override auto-generated names with the `-o` flag
|
|
62
|
+
- ๐ **Modern Python Stack** โ Built for Python 3.10+ with clean architecture
|
|
63
|
+
- ๐ฆ **One-Command Installation** โ `pip install -e .` for instant local development
|
|
64
|
+
- ๐ **Broad Compatibility** โ Works with virtually any news, magazine, or blog site supported by Trafilatura
|
|
65
|
+
|
|
66
|
+
## ๐ Installation (Local Dev Mode)
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
# 1. Clone the repository
|
|
70
|
+
git clone https://github.com/your-org/pystates.git
|
|
71
|
+
cd pystates
|
|
72
|
+
|
|
73
|
+
# 2. Install in editable/development mode
|
|
74
|
+
pip install -e .
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### Verify Installation
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
pystates --help
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### Requirements
|
|
84
|
+
|
|
85
|
+
- Python โฅ 3.10
|
|
86
|
+
- `pip`
|
|
87
|
+
|
|
88
|
+
All dependencies (`requests`, `trafilatura`, `argparse`) are automatically installed via `pyproject.toml`.
|
|
89
|
+
|
|
90
|
+
## ๐ฅ๏ธ Usage
|
|
91
|
+
|
|
92
|
+
### Command Syntax
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
pystates -u URL [OPTIONS]
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### CLI Flags
|
|
99
|
+
|
|
100
|
+
| Short | Long | Required | Default | Description |
|
|
101
|
+
|-------|-------------|----------|---------|-------------|
|
|
102
|
+
| `-u` | `--url` | **Yes** | โ | The full URL of the article to fetch and process |
|
|
103
|
+
| `-f` | `--format` | No | `md` | Output format. Accepted values: `md` or `txt` |
|
|
104
|
+
| `-o` | `--output` | No | *(auto)* | Custom output filename **without extension**. If omitted, a sanitized version of the article title is used |
|
|
105
|
+
|
|
106
|
+
### Examples
|
|
107
|
+
|
|
108
|
+
#### 1. Basic usage (Markdown, auto-generated filename)
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
pystates -u "https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era"
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
โ Creates `the-new-ai-era.md`
|
|
115
|
+
|
|
116
|
+
#### 2. Save as plain text
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pystates -u "https://www.wired.com/story/the-future-of-coding/" -f txt
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
โ Creates `the-future-of-coding.txt`
|
|
123
|
+
|
|
124
|
+
#### 3. Custom output filename
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
pystates -u "https://www.theatlantic.com/ideas/archive/2025/07/climate-change/" \
|
|
128
|
+
-o "climate_report_july_2025" \
|
|
129
|
+
-f md
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
โ Creates `climate_report_july_2025.md`
|
|
133
|
+
|
|
134
|
+
#### 4. Full-featured command
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
pystates \
|
|
138
|
+
--url "https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it" \
|
|
139
|
+
--format txt \
|
|
140
|
+
--output "end_of_internet"
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
#### 5. View all options
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
pystates --help
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### Sample Output (Markdown)
|
|
150
|
+
|
|
151
|
+
```markdown
|
|
152
|
+
# The New AI Era
|
|
153
|
+
|
|
154
|
+
**By Jane Smith**
|
|
155
|
+
*Published: January 20, 2025*
|
|
156
|
+
*Source: The New Yorker*
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
Artificial intelligence is no longer a distant promise...
|
|
161
|
+
|
|
162
|
+
## A New Paradigm
|
|
163
|
+
|
|
164
|
+
The implications are profound...
|
|
165
|
+
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
## ๐ Project Structure
|
|
169
|
+
|
|
170
|
+
```
|
|
171
|
+
PyStates/
|
|
172
|
+
โโโ pystates/
|
|
173
|
+
โ โโโ __init__.py # Package metadata & version
|
|
174
|
+
โ โโโ __main__.py # Entry point for `python -m pystates`
|
|
175
|
+
โ โโโ cli.py # Argument parsing (argparse)
|
|
176
|
+
โ โโโ exporters.py # Filename sanitization + .md/.txt formatting
|
|
177
|
+
โ โโโ parser.py # HTTP fetching (custom UA) + Trafilatura parsing
|
|
178
|
+
โโโ pyproject.toml # Modern packaging + entry points
|
|
179
|
+
โโโ .gitignore
|
|
180
|
+
โโโ README.md
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
## ๐ License
|
|
184
|
+
|
|
185
|
+
This project is released under the **MIT License**.
|
|
186
|
+
|
|
187
|
+
See [LICENSE](LICENSE) for the full text.
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
**PyStates** โ preserving the web's best writing, one clean file at a time.
|
|
192
|
+
|
|
193
|
+
*Built with care by the open-source community.*
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
README.md
|
|
2
|
+
pyproject.toml
|
|
3
|
+
pystates/__init__.py
|
|
4
|
+
pystates/__main__.py
|
|
5
|
+
pystates/cli.py
|
|
6
|
+
pystates/exporters.py
|
|
7
|
+
pystates/parser.py
|
|
8
|
+
pystates_CLI.egg-info/PKG-INFO
|
|
9
|
+
pystates_CLI.egg-info/SOURCES.txt
|
|
10
|
+
pystates_CLI.egg-info/dependency_links.txt
|
|
11
|
+
pystates_CLI.egg-info/entry_points.txt
|
|
12
|
+
pystates_CLI.egg-info/requires.txt
|
|
13
|
+
pystates_CLI.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|