athlinks-races 0.0.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- athlinks_races-0.0.3/LICENSE +21 -0
- athlinks_races-0.0.3/MANIFEST.in +1 -0
- athlinks_races-0.0.3/PKG-INFO +191 -0
- athlinks_races-0.0.3/README.md +146 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/PKG-INFO +191 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/SOURCES.txt +19 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/dependency_links.txt +1 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/entry_points.txt +2 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/requires.txt +9 -0
- athlinks_races-0.0.3/athlinks_races.egg-info/top_level.txt +3 -0
- athlinks_races-0.0.3/pyproject.toml +175 -0
- athlinks_races-0.0.3/scrapy_athlinks/__init__.py +18 -0
- athlinks_races-0.0.3/scrapy_athlinks/cli.py +79 -0
- athlinks_races-0.0.3/scrapy_athlinks/demo.py +74 -0
- athlinks_races-0.0.3/scrapy_athlinks/items.py +66 -0
- athlinks_races-0.0.3/scrapy_athlinks/pipelines.py +108 -0
- athlinks_races-0.0.3/scrapy_athlinks/settings.py +18 -0
- athlinks_races-0.0.3/scrapy_athlinks/spiders/__init__.py +4 -0
- athlinks_races-0.0.3/scrapy_athlinks/spiders/race.py +219 -0
- athlinks_races-0.0.3/setup.cfg +4 -0
- athlinks_races-0.0.3/tests/tests.py +320 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2022 Aaron Schroeder
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
include LICENSE
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: athlinks-races
|
|
3
|
+
Version: 0.0.3
|
|
4
|
+
Summary: Web scraper for race results hosted on Athlinks.
|
|
5
|
+
Author-email: Jose Vicente Nunez <kodegeek.com@protonmail.com>, Aaron Schroeder <aaron@trailzealot.com>
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2022 Aaron Schroeder
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
Project-URL: Homepage, http://kodegeek.com/
|
|
28
|
+
Project-URL: Repository, https://github.com/josevnz/athlinks-races.git
|
|
29
|
+
Keywords: running,race,empire state,racing
|
|
30
|
+
Classifier: Environment :: Console
|
|
31
|
+
Classifier: Development Status :: 3 - Alpha
|
|
32
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
33
|
+
Classifier: Intended Audience :: Developers
|
|
34
|
+
Classifier: Programming Language :: Python :: 3
|
|
35
|
+
Requires-Python: >=3.11
|
|
36
|
+
Description-Content-Type: text/markdown
|
|
37
|
+
License-File: LICENSE
|
|
38
|
+
Requires-Dist: Scrapy==2.6.3
|
|
39
|
+
Provides-Extra: lint
|
|
40
|
+
Requires-Dist: pylint; extra == "lint"
|
|
41
|
+
Provides-Extra: dev
|
|
42
|
+
Requires-Dist: twine; extra == "dev"
|
|
43
|
+
Requires-Dist: wheel; extra == "dev"
|
|
44
|
+
Requires-Dist: build; extra == "dev"
|
|
45
|
+
|
|
46
|
+
# scrapy-athlinks: web scraper for race results hosted on Athlinks
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
## NOTE
|
|
50
|
+
|
|
51
|
+
This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Introduction
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
`scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
|
|
58
|
+
|
|
59
|
+
This spider crawls through all results pages from a race hosted on athlinks.com,
|
|
60
|
+
building and following links to each athlete's individual results page, where it
|
|
61
|
+
collects their split data. It also collects some metadata about the race itself.
|
|
62
|
+
|
|
63
|
+
By default, the spider returns one race metadata object (`RaceItem`), and one
|
|
64
|
+
`AthleteItem` per participant.
|
|
65
|
+
Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
|
|
66
|
+
containing data from each split they recorded.
|
|
67
|
+
|
|
68
|
+
## How to use this package
|
|
69
|
+
|
|
70
|
+
### Option 1: In python scripts
|
|
71
|
+
|
|
72
|
+
Scrapy can be operated entirely from python scripts.
|
|
73
|
+
[See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
|
|
74
|
+
|
|
75
|
+
#### Installation
|
|
76
|
+
|
|
77
|
+
The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
81
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
82
|
+
pip install scrapy-athlinks
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
#### Example usage
|
|
86
|
+
|
|
87
|
+
[A demo script is included in this repo](scrapy_athlinks/demo.py).
|
|
88
|
+
|
|
89
|
+
```python
|
|
90
|
+
"""
|
|
91
|
+
Demonstrate the available classes.
|
|
92
|
+
You can run as python scrapy_athlinks/demo.py
|
|
93
|
+
"""
|
|
94
|
+
from scrapy.crawler import CrawlerProcess
|
|
95
|
+
from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def main():
|
|
99
|
+
# Make settings for two separate output files: one for athlete data,
|
|
100
|
+
# one for race metadata.
|
|
101
|
+
settings = {
|
|
102
|
+
'FEEDS': {
|
|
103
|
+
# Athlete data. Inside this file will be a list of dicts containing
|
|
104
|
+
# data about each athlete's race and splits.
|
|
105
|
+
'athletes.json': {
|
|
106
|
+
'format': 'json',
|
|
107
|
+
'overwrite': True,
|
|
108
|
+
'item_classes': [AthleteItem],
|
|
109
|
+
},
|
|
110
|
+
# Race metadata. Inside this file will be a list with a single dict
|
|
111
|
+
# containing info about the race itself.
|
|
112
|
+
'metadata.json': {
|
|
113
|
+
'format': 'json',
|
|
114
|
+
'overwrite': True,
|
|
115
|
+
'item_classes': [RaceItem],
|
|
116
|
+
},
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
process = CrawlerProcess(settings=settings)
|
|
120
|
+
|
|
121
|
+
# Crawl results for the 2022 Leadville Trail 100 Run
|
|
122
|
+
process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
|
|
123
|
+
process.start()
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
if __name__ == "__main__":
|
|
127
|
+
main()
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
If you do a ```pip install --editable .[lint,dev]``` then you can run as
|
|
131
|
+
|
|
132
|
+
```shell
|
|
133
|
+
scrappy_athlinks_demo
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Then you can build the wheelhouse to install locally if needed:
|
|
137
|
+
|
|
138
|
+
```shell
|
|
139
|
+
python -m build .
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Option 2: Command line
|
|
143
|
+
|
|
144
|
+
Alternatively, you may clone this repo for use like a typical Scrapy project
|
|
145
|
+
that you might create on your own.
|
|
146
|
+
|
|
147
|
+
#### Installation
|
|
148
|
+
|
|
149
|
+
```sh
|
|
150
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
151
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
152
|
+
git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
|
|
153
|
+
cd athlinks-scraper-scrapy
|
|
154
|
+
python install --editable .
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
#### Example usage
|
|
158
|
+
|
|
159
|
+
Run a `RaceSpider`, few races with different years:
|
|
160
|
+
|
|
161
|
+
```shell
|
|
162
|
+
cd scrapy_athlinks
|
|
163
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
|
|
164
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
|
|
165
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
## Dependencies
|
|
169
|
+
|
|
170
|
+
All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
|
|
171
|
+
|
|
172
|
+
## Testing
|
|
173
|
+
|
|
174
|
+
```
|
|
175
|
+
make test
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
## License
|
|
179
|
+
|
|
180
|
+
This project is licensed under the MIT License. See
|
|
181
|
+
[LICENSE](LICENSE) file for details.
|
|
182
|
+
|
|
183
|
+
## Contact
|
|
184
|
+
|
|
185
|
+
You can get in touch here:
|
|
186
|
+
|
|
187
|
+
- GitHub: [https://github.com/josevnz](https://github.com/josevnz)
|
|
188
|
+
|
|
189
|
+
### Original Author
|
|
190
|
+
|
|
191
|
+
- GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# scrapy-athlinks: web scraper for race results hosted on Athlinks
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
## NOTE
|
|
5
|
+
|
|
6
|
+
This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
|
|
7
|
+
|
|
8
|
+
|
|
9
|
+
## Introduction
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
`scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
|
|
13
|
+
|
|
14
|
+
This spider crawls through all results pages from a race hosted on athlinks.com,
|
|
15
|
+
building and following links to each athlete's individual results page, where it
|
|
16
|
+
collects their split data. It also collects some metadata about the race itself.
|
|
17
|
+
|
|
18
|
+
By default, the spider returns one race metadata object (`RaceItem`), and one
|
|
19
|
+
`AthleteItem` per participant.
|
|
20
|
+
Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
|
|
21
|
+
containing data from each split they recorded.
|
|
22
|
+
|
|
23
|
+
## How to use this package
|
|
24
|
+
|
|
25
|
+
### Option 1: In python scripts
|
|
26
|
+
|
|
27
|
+
Scrapy can be operated entirely from python scripts.
|
|
28
|
+
[See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
|
|
29
|
+
|
|
30
|
+
#### Installation
|
|
31
|
+
|
|
32
|
+
The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
36
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
37
|
+
pip install scrapy-athlinks
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
#### Example usage
|
|
41
|
+
|
|
42
|
+
[A demo script is included in this repo](scrapy_athlinks/demo.py).
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
"""
|
|
46
|
+
Demonstrate the available classes.
|
|
47
|
+
You can run as python scrapy_athlinks/demo.py
|
|
48
|
+
"""
|
|
49
|
+
from scrapy.crawler import CrawlerProcess
|
|
50
|
+
from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
def main():
|
|
54
|
+
# Make settings for two separate output files: one for athlete data,
|
|
55
|
+
# one for race metadata.
|
|
56
|
+
settings = {
|
|
57
|
+
'FEEDS': {
|
|
58
|
+
# Athlete data. Inside this file will be a list of dicts containing
|
|
59
|
+
# data about each athlete's race and splits.
|
|
60
|
+
'athletes.json': {
|
|
61
|
+
'format': 'json',
|
|
62
|
+
'overwrite': True,
|
|
63
|
+
'item_classes': [AthleteItem],
|
|
64
|
+
},
|
|
65
|
+
# Race metadata. Inside this file will be a list with a single dict
|
|
66
|
+
# containing info about the race itself.
|
|
67
|
+
'metadata.json': {
|
|
68
|
+
'format': 'json',
|
|
69
|
+
'overwrite': True,
|
|
70
|
+
'item_classes': [RaceItem],
|
|
71
|
+
},
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
process = CrawlerProcess(settings=settings)
|
|
75
|
+
|
|
76
|
+
# Crawl results for the 2022 Leadville Trail 100 Run
|
|
77
|
+
process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
|
|
78
|
+
process.start()
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
if __name__ == "__main__":
|
|
82
|
+
main()
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
If you do a ```pip install --editable .[lint,dev]``` then you can run as
|
|
86
|
+
|
|
87
|
+
```shell
|
|
88
|
+
scrappy_athlinks_demo
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Then you can build the wheelhouse to install locally if needed:
|
|
92
|
+
|
|
93
|
+
```shell
|
|
94
|
+
python -m build .
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### Option 2: Command line
|
|
98
|
+
|
|
99
|
+
Alternatively, you may clone this repo for use like a typical Scrapy project
|
|
100
|
+
that you might create on your own.
|
|
101
|
+
|
|
102
|
+
#### Installation
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
106
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
107
|
+
git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
|
|
108
|
+
cd athlinks-scraper-scrapy
|
|
109
|
+
python install --editable .
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
#### Example usage
|
|
113
|
+
|
|
114
|
+
Run a `RaceSpider`, few races with different years:
|
|
115
|
+
|
|
116
|
+
```shell
|
|
117
|
+
cd scrapy_athlinks
|
|
118
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
|
|
119
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
|
|
120
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Dependencies
|
|
124
|
+
|
|
125
|
+
All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
|
|
126
|
+
|
|
127
|
+
## Testing
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
make test
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
## License
|
|
134
|
+
|
|
135
|
+
This project is licensed under the MIT License. See
|
|
136
|
+
[LICENSE](LICENSE) file for details.
|
|
137
|
+
|
|
138
|
+
## Contact
|
|
139
|
+
|
|
140
|
+
You can get in touch here:
|
|
141
|
+
|
|
142
|
+
- GitHub: [https://github.com/josevnz](https://github.com/josevnz)
|
|
143
|
+
|
|
144
|
+
### Original Author
|
|
145
|
+
|
|
146
|
+
- GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: athlinks-races
|
|
3
|
+
Version: 0.0.3
|
|
4
|
+
Summary: Web scraper for race results hosted on Athlinks.
|
|
5
|
+
Author-email: Jose Vicente Nunez <kodegeek.com@protonmail.com>, Aaron Schroeder <aaron@trailzealot.com>
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2022 Aaron Schroeder
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
Project-URL: Homepage, http://kodegeek.com/
|
|
28
|
+
Project-URL: Repository, https://github.com/josevnz/athlinks-races.git
|
|
29
|
+
Keywords: running,race,empire state,racing
|
|
30
|
+
Classifier: Environment :: Console
|
|
31
|
+
Classifier: Development Status :: 3 - Alpha
|
|
32
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
33
|
+
Classifier: Intended Audience :: Developers
|
|
34
|
+
Classifier: Programming Language :: Python :: 3
|
|
35
|
+
Requires-Python: >=3.11
|
|
36
|
+
Description-Content-Type: text/markdown
|
|
37
|
+
License-File: LICENSE
|
|
38
|
+
Requires-Dist: Scrapy==2.6.3
|
|
39
|
+
Provides-Extra: lint
|
|
40
|
+
Requires-Dist: pylint; extra == "lint"
|
|
41
|
+
Provides-Extra: dev
|
|
42
|
+
Requires-Dist: twine; extra == "dev"
|
|
43
|
+
Requires-Dist: wheel; extra == "dev"
|
|
44
|
+
Requires-Dist: build; extra == "dev"
|
|
45
|
+
|
|
46
|
+
# scrapy-athlinks: web scraper for race results hosted on Athlinks
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
## NOTE
|
|
50
|
+
|
|
51
|
+
This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## Introduction
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
`scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
|
|
58
|
+
|
|
59
|
+
This spider crawls through all results pages from a race hosted on athlinks.com,
|
|
60
|
+
building and following links to each athlete's individual results page, where it
|
|
61
|
+
collects their split data. It also collects some metadata about the race itself.
|
|
62
|
+
|
|
63
|
+
By default, the spider returns one race metadata object (`RaceItem`), and one
|
|
64
|
+
`AthleteItem` per participant.
|
|
65
|
+
Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
|
|
66
|
+
containing data from each split they recorded.
|
|
67
|
+
|
|
68
|
+
## How to use this package
|
|
69
|
+
|
|
70
|
+
### Option 1: In python scripts
|
|
71
|
+
|
|
72
|
+
Scrapy can be operated entirely from python scripts.
|
|
73
|
+
[See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
|
|
74
|
+
|
|
75
|
+
#### Installation
|
|
76
|
+
|
|
77
|
+
The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
81
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
82
|
+
pip install scrapy-athlinks
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
#### Example usage
|
|
86
|
+
|
|
87
|
+
[A demo script is included in this repo](scrapy_athlinks/demo.py).
|
|
88
|
+
|
|
89
|
+
```python
|
|
90
|
+
"""
|
|
91
|
+
Demonstrate the available classes.
|
|
92
|
+
You can run as python scrapy_athlinks/demo.py
|
|
93
|
+
"""
|
|
94
|
+
from scrapy.crawler import CrawlerProcess
|
|
95
|
+
from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def main():
|
|
99
|
+
# Make settings for two separate output files: one for athlete data,
|
|
100
|
+
# one for race metadata.
|
|
101
|
+
settings = {
|
|
102
|
+
'FEEDS': {
|
|
103
|
+
# Athlete data. Inside this file will be a list of dicts containing
|
|
104
|
+
# data about each athlete's race and splits.
|
|
105
|
+
'athletes.json': {
|
|
106
|
+
'format': 'json',
|
|
107
|
+
'overwrite': True,
|
|
108
|
+
'item_classes': [AthleteItem],
|
|
109
|
+
},
|
|
110
|
+
# Race metadata. Inside this file will be a list with a single dict
|
|
111
|
+
# containing info about the race itself.
|
|
112
|
+
'metadata.json': {
|
|
113
|
+
'format': 'json',
|
|
114
|
+
'overwrite': True,
|
|
115
|
+
'item_classes': [RaceItem],
|
|
116
|
+
},
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
process = CrawlerProcess(settings=settings)
|
|
120
|
+
|
|
121
|
+
# Crawl results for the 2022 Leadville Trail 100 Run
|
|
122
|
+
process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
|
|
123
|
+
process.start()
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
if __name__ == "__main__":
|
|
127
|
+
main()
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
If you do a ```pip install --editable .[lint,dev]``` then you can run as
|
|
131
|
+
|
|
132
|
+
```shell
|
|
133
|
+
scrappy_athlinks_demo
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Then you can build the wheelhouse to install locally if needed:
|
|
137
|
+
|
|
138
|
+
```shell
|
|
139
|
+
python -m build .
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Option 2: Command line
|
|
143
|
+
|
|
144
|
+
Alternatively, you may clone this repo for use like a typical Scrapy project
|
|
145
|
+
that you might create on your own.
|
|
146
|
+
|
|
147
|
+
#### Installation
|
|
148
|
+
|
|
149
|
+
```sh
|
|
150
|
+
python -m venv `$HOME/virtualenv/scrapy-athlinks/`
|
|
151
|
+
. $HOME/virtualenv/scrapy-athlinks/bin/activate
|
|
152
|
+
git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
|
|
153
|
+
cd athlinks-scraper-scrapy
|
|
154
|
+
python install --editable .
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
#### Example usage
|
|
158
|
+
|
|
159
|
+
Run a `RaceSpider`, few races with different years:
|
|
160
|
+
|
|
161
|
+
```shell
|
|
162
|
+
cd scrapy_athlinks
|
|
163
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
|
|
164
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
|
|
165
|
+
scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
## Dependencies
|
|
169
|
+
|
|
170
|
+
All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
|
|
171
|
+
|
|
172
|
+
## Testing
|
|
173
|
+
|
|
174
|
+
```
|
|
175
|
+
make test
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
## License
|
|
179
|
+
|
|
180
|
+
This project is licensed under the MIT License. See
|
|
181
|
+
[LICENSE](LICENSE) file for details.
|
|
182
|
+
|
|
183
|
+
## Contact
|
|
184
|
+
|
|
185
|
+
You can get in touch here:
|
|
186
|
+
|
|
187
|
+
- GitHub: [https://github.com/josevnz](https://github.com/josevnz)
|
|
188
|
+
|
|
189
|
+
### Original Author
|
|
190
|
+
|
|
191
|
+
- GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
MANIFEST.in
|
|
3
|
+
README.md
|
|
4
|
+
pyproject.toml
|
|
5
|
+
athlinks_races.egg-info/PKG-INFO
|
|
6
|
+
athlinks_races.egg-info/SOURCES.txt
|
|
7
|
+
athlinks_races.egg-info/dependency_links.txt
|
|
8
|
+
athlinks_races.egg-info/entry_points.txt
|
|
9
|
+
athlinks_races.egg-info/requires.txt
|
|
10
|
+
athlinks_races.egg-info/top_level.txt
|
|
11
|
+
scrapy_athlinks/__init__.py
|
|
12
|
+
scrapy_athlinks/cli.py
|
|
13
|
+
scrapy_athlinks/demo.py
|
|
14
|
+
scrapy_athlinks/items.py
|
|
15
|
+
scrapy_athlinks/pipelines.py
|
|
16
|
+
scrapy_athlinks/settings.py
|
|
17
|
+
scrapy_athlinks/spiders/__init__.py
|
|
18
|
+
scrapy_athlinks/spiders/race.py
|
|
19
|
+
tests/tests.py
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|