athlinks-races 0.0.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2022 Aaron Schroeder
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1 @@
1
+ include LICENSE
@@ -0,0 +1,191 @@
1
+ Metadata-Version: 2.1
2
+ Name: athlinks-races
3
+ Version: 0.0.3
4
+ Summary: Web scraper for race results hosted on Athlinks.
5
+ Author-email: Jose Vicente Nunez <kodegeek.com@protonmail.com>, Aaron Schroeder <aaron@trailzealot.com>
6
+ License: MIT License
7
+
8
+ Copyright (c) 2022 Aaron Schroeder
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+ Project-URL: Homepage, http://kodegeek.com/
28
+ Project-URL: Repository, https://github.com/josevnz/athlinks-races.git
29
+ Keywords: running,race,empire state,racing
30
+ Classifier: Environment :: Console
31
+ Classifier: Development Status :: 3 - Alpha
32
+ Classifier: License :: OSI Approved :: MIT License
33
+ Classifier: Intended Audience :: Developers
34
+ Classifier: Programming Language :: Python :: 3
35
+ Requires-Python: >=3.11
36
+ Description-Content-Type: text/markdown
37
+ License-File: LICENSE
38
+ Requires-Dist: Scrapy==2.6.3
39
+ Provides-Extra: lint
40
+ Requires-Dist: pylint; extra == "lint"
41
+ Provides-Extra: dev
42
+ Requires-Dist: twine; extra == "dev"
43
+ Requires-Dist: wheel; extra == "dev"
44
+ Requires-Dist: build; extra == "dev"
45
+
46
+ # scrapy-athlinks: web scraper for race results hosted on Athlinks
47
+
48
+
49
+ ## NOTE
50
+
51
+ This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
52
+
53
+
54
+ ## Introduction
55
+
56
+
57
+ `scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
58
+
59
+ This spider crawls through all results pages from a race hosted on athlinks.com,
60
+ building and following links to each athlete's individual results page, where it
61
+ collects their split data. It also collects some metadata about the race itself.
62
+
63
+ By default, the spider returns one race metadata object (`RaceItem`), and one
64
+ `AthleteItem` per participant.
65
+ Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
66
+ containing data from each split they recorded.
67
+
68
+ ## How to use this package
69
+
70
+ ### Option 1: In python scripts
71
+
72
+ Scrapy can be operated entirely from python scripts.
73
+ [See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
74
+
75
+ #### Installation
76
+
77
+ The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
78
+
79
+ ```sh
80
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
81
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
82
+ pip install scrapy-athlinks
83
+ ```
84
+
85
+ #### Example usage
86
+
87
+ [A demo script is included in this repo](scrapy_athlinks/demo.py).
88
+
89
+ ```python
90
+ """
91
+ Demonstrate the available classes.
92
+ You can run as python scrapy_athlinks/demo.py
93
+ """
94
+ from scrapy.crawler import CrawlerProcess
95
+ from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
96
+
97
+
98
+ def main():
99
+ # Make settings for two separate output files: one for athlete data,
100
+ # one for race metadata.
101
+ settings = {
102
+ 'FEEDS': {
103
+ # Athlete data. Inside this file will be a list of dicts containing
104
+ # data about each athlete's race and splits.
105
+ 'athletes.json': {
106
+ 'format': 'json',
107
+ 'overwrite': True,
108
+ 'item_classes': [AthleteItem],
109
+ },
110
+ # Race metadata. Inside this file will be a list with a single dict
111
+ # containing info about the race itself.
112
+ 'metadata.json': {
113
+ 'format': 'json',
114
+ 'overwrite': True,
115
+ 'item_classes': [RaceItem],
116
+ },
117
+ }
118
+ }
119
+ process = CrawlerProcess(settings=settings)
120
+
121
+ # Crawl results for the 2022 Leadville Trail 100 Run
122
+ process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
123
+ process.start()
124
+
125
+
126
+ if __name__ == "__main__":
127
+ main()
128
+ ```
129
+
130
+ If you do a ```pip install --editable .[lint,dev]``` then you can run as
131
+
132
+ ```shell
133
+ scrappy_athlinks_demo
134
+ ```
135
+
136
+ Then you can build the wheelhouse to install locally if needed:
137
+
138
+ ```shell
139
+ python -m build .
140
+ ```
141
+
142
+ ### Option 2: Command line
143
+
144
+ Alternatively, you may clone this repo for use like a typical Scrapy project
145
+ that you might create on your own.
146
+
147
+ #### Installation
148
+
149
+ ```sh
150
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
151
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
152
+ git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
153
+ cd athlinks-scraper-scrapy
154
+ python install --editable .
155
+ ```
156
+
157
+ #### Example usage
158
+
159
+ Run a `RaceSpider`, few races with different years:
160
+
161
+ ```shell
162
+ cd scrapy_athlinks
163
+ scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
164
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
165
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
166
+ ```
167
+
168
+ ## Dependencies
169
+
170
+ All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
171
+
172
+ ## Testing
173
+
174
+ ```
175
+ make test
176
+ ```
177
+
178
+ ## License
179
+
180
+ This project is licensed under the MIT License. See
181
+ [LICENSE](LICENSE) file for details.
182
+
183
+ ## Contact
184
+
185
+ You can get in touch here:
186
+
187
+ - GitHub: [https://github.com/josevnz](https://github.com/josevnz)
188
+
189
+ ### Original Author
190
+
191
+ - GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
@@ -0,0 +1,146 @@
1
+ # scrapy-athlinks: web scraper for race results hosted on Athlinks
2
+
3
+
4
+ ## NOTE
5
+
6
+ This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
7
+
8
+
9
+ ## Introduction
10
+
11
+
12
+ `scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
13
+
14
+ This spider crawls through all results pages from a race hosted on athlinks.com,
15
+ building and following links to each athlete's individual results page, where it
16
+ collects their split data. It also collects some metadata about the race itself.
17
+
18
+ By default, the spider returns one race metadata object (`RaceItem`), and one
19
+ `AthleteItem` per participant.
20
+ Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
21
+ containing data from each split they recorded.
22
+
23
+ ## How to use this package
24
+
25
+ ### Option 1: In python scripts
26
+
27
+ Scrapy can be operated entirely from python scripts.
28
+ [See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
29
+
30
+ #### Installation
31
+
32
+ The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
33
+
34
+ ```sh
35
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
36
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
37
+ pip install scrapy-athlinks
38
+ ```
39
+
40
+ #### Example usage
41
+
42
+ [A demo script is included in this repo](scrapy_athlinks/demo.py).
43
+
44
+ ```python
45
+ """
46
+ Demonstrate the available classes.
47
+ You can run as python scrapy_athlinks/demo.py
48
+ """
49
+ from scrapy.crawler import CrawlerProcess
50
+ from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
51
+
52
+
53
+ def main():
54
+ # Make settings for two separate output files: one for athlete data,
55
+ # one for race metadata.
56
+ settings = {
57
+ 'FEEDS': {
58
+ # Athlete data. Inside this file will be a list of dicts containing
59
+ # data about each athlete's race and splits.
60
+ 'athletes.json': {
61
+ 'format': 'json',
62
+ 'overwrite': True,
63
+ 'item_classes': [AthleteItem],
64
+ },
65
+ # Race metadata. Inside this file will be a list with a single dict
66
+ # containing info about the race itself.
67
+ 'metadata.json': {
68
+ 'format': 'json',
69
+ 'overwrite': True,
70
+ 'item_classes': [RaceItem],
71
+ },
72
+ }
73
+ }
74
+ process = CrawlerProcess(settings=settings)
75
+
76
+ # Crawl results for the 2022 Leadville Trail 100 Run
77
+ process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
78
+ process.start()
79
+
80
+
81
+ if __name__ == "__main__":
82
+ main()
83
+ ```
84
+
85
+ If you do a ```pip install --editable .[lint,dev]``` then you can run as
86
+
87
+ ```shell
88
+ scrappy_athlinks_demo
89
+ ```
90
+
91
+ Then you can build the wheelhouse to install locally if needed:
92
+
93
+ ```shell
94
+ python -m build .
95
+ ```
96
+
97
+ ### Option 2: Command line
98
+
99
+ Alternatively, you may clone this repo for use like a typical Scrapy project
100
+ that you might create on your own.
101
+
102
+ #### Installation
103
+
104
+ ```sh
105
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
106
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
107
+ git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
108
+ cd athlinks-scraper-scrapy
109
+ python install --editable .
110
+ ```
111
+
112
+ #### Example usage
113
+
114
+ Run a `RaceSpider`, few races with different years:
115
+
116
+ ```shell
117
+ cd scrapy_athlinks
118
+ scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
119
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
120
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
121
+ ```
122
+
123
+ ## Dependencies
124
+
125
+ All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
126
+
127
+ ## Testing
128
+
129
+ ```
130
+ make test
131
+ ```
132
+
133
+ ## License
134
+
135
+ This project is licensed under the MIT License. See
136
+ [LICENSE](LICENSE) file for details.
137
+
138
+ ## Contact
139
+
140
+ You can get in touch here:
141
+
142
+ - GitHub: [https://github.com/josevnz](https://github.com/josevnz)
143
+
144
+ ### Original Author
145
+
146
+ - GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
@@ -0,0 +1,191 @@
1
+ Metadata-Version: 2.1
2
+ Name: athlinks-races
3
+ Version: 0.0.3
4
+ Summary: Web scraper for race results hosted on Athlinks.
5
+ Author-email: Jose Vicente Nunez <kodegeek.com@protonmail.com>, Aaron Schroeder <aaron@trailzealot.com>
6
+ License: MIT License
7
+
8
+ Copyright (c) 2022 Aaron Schroeder
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+ Project-URL: Homepage, http://kodegeek.com/
28
+ Project-URL: Repository, https://github.com/josevnz/athlinks-races.git
29
+ Keywords: running,race,empire state,racing
30
+ Classifier: Environment :: Console
31
+ Classifier: Development Status :: 3 - Alpha
32
+ Classifier: License :: OSI Approved :: MIT License
33
+ Classifier: Intended Audience :: Developers
34
+ Classifier: Programming Language :: Python :: 3
35
+ Requires-Python: >=3.11
36
+ Description-Content-Type: text/markdown
37
+ License-File: LICENSE
38
+ Requires-Dist: Scrapy==2.6.3
39
+ Provides-Extra: lint
40
+ Requires-Dist: pylint; extra == "lint"
41
+ Provides-Extra: dev
42
+ Requires-Dist: twine; extra == "dev"
43
+ Requires-Dist: wheel; extra == "dev"
44
+ Requires-Dist: build; extra == "dev"
45
+
46
+ # scrapy-athlinks: web scraper for race results hosted on Athlinks
47
+
48
+
49
+ ## NOTE
50
+
51
+ This is a fork of the original [athlinks-scraper-scrapy](https://github.com/josevnz/scrapy-athlinks). I decided to take over as I want to add features that were not originally available on the project.
52
+
53
+
54
+ ## Introduction
55
+
56
+
57
+ `scrapy-athlinks` provides the [`RaceSpider`](scrapy_athlinks/spiders/race.py) class.
58
+
59
+ This spider crawls through all results pages from a race hosted on athlinks.com,
60
+ building and following links to each athlete's individual results page, where it
61
+ collects their split data. It also collects some metadata about the race itself.
62
+
63
+ By default, the spider returns one race metadata object (`RaceItem`), and one
64
+ `AthleteItem` per participant.
65
+ Each `AthleteItem` consists of some basic athlete info and a list of `RaceSplitItem`
66
+ containing data from each split they recorded.
67
+
68
+ ## How to use this package
69
+
70
+ ### Option 1: In python scripts
71
+
72
+ Scrapy can be operated entirely from python scripts.
73
+ [See the scrapy documentation for more info.](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)
74
+
75
+ #### Installation
76
+
77
+ The package is available on [PyPi](https://pypi.org/project/athlinks-races) and can be installed with `pip`:
78
+
79
+ ```sh
80
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
81
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
82
+ pip install scrapy-athlinks
83
+ ```
84
+
85
+ #### Example usage
86
+
87
+ [A demo script is included in this repo](scrapy_athlinks/demo.py).
88
+
89
+ ```python
90
+ """
91
+ Demonstrate the available classes.
92
+ You can run as python scrapy_athlinks/demo.py
93
+ """
94
+ from scrapy.crawler import CrawlerProcess
95
+ from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem
96
+
97
+
98
+ def main():
99
+ # Make settings for two separate output files: one for athlete data,
100
+ # one for race metadata.
101
+ settings = {
102
+ 'FEEDS': {
103
+ # Athlete data. Inside this file will be a list of dicts containing
104
+ # data about each athlete's race and splits.
105
+ 'athletes.json': {
106
+ 'format': 'json',
107
+ 'overwrite': True,
108
+ 'item_classes': [AthleteItem],
109
+ },
110
+ # Race metadata. Inside this file will be a list with a single dict
111
+ # containing info about the race itself.
112
+ 'metadata.json': {
113
+ 'format': 'json',
114
+ 'overwrite': True,
115
+ 'item_classes': [RaceItem],
116
+ },
117
+ }
118
+ }
119
+ process = CrawlerProcess(settings=settings)
120
+
121
+ # Crawl results for the 2022 Leadville Trail 100 Run
122
+ process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
123
+ process.start()
124
+
125
+
126
+ if __name__ == "__main__":
127
+ main()
128
+ ```
129
+
130
+ If you do a ```pip install --editable .[lint,dev]``` then you can run as
131
+
132
+ ```shell
133
+ scrappy_athlinks_demo
134
+ ```
135
+
136
+ Then you can build the wheelhouse to install locally if needed:
137
+
138
+ ```shell
139
+ python -m build .
140
+ ```
141
+
142
+ ### Option 2: Command line
143
+
144
+ Alternatively, you may clone this repo for use like a typical Scrapy project
145
+ that you might create on your own.
146
+
147
+ #### Installation
148
+
149
+ ```sh
150
+ python -m venv `$HOME/virtualenv/scrapy-athlinks/`
151
+ . $HOME/virtualenv/scrapy-athlinks/bin/activate
152
+ git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
153
+ cd athlinks-scraper-scrapy
154
+ python install --editable .
155
+ ```
156
+
157
+ #### Example usage
158
+
159
+ Run a `RaceSpider`, few races with different years:
160
+
161
+ ```shell
162
+ cd scrapy_athlinks
163
+ scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O 1018673.json
164
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1093108 -O 1093108.json
165
+ scrapy crawl race -a url=https://www.athlinks.com/event/382111/results/Event/1062909 -O 1093108.json
166
+ ```
167
+
168
+ ## Dependencies
169
+
170
+ All that is required is [Scrapy](https://scrapy.org/) (and its dependencies).
171
+
172
+ ## Testing
173
+
174
+ ```
175
+ make test
176
+ ```
177
+
178
+ ## License
179
+
180
+ This project is licensed under the MIT License. See
181
+ [LICENSE](LICENSE) file for details.
182
+
183
+ ## Contact
184
+
185
+ You can get in touch here:
186
+
187
+ - GitHub: [https://github.com/josevnz](https://github.com/josevnz)
188
+
189
+ ### Original Author
190
+
191
+ - GitHub: [github.com/aaron-schroeder](https://github.com/aaron-schroeder)
@@ -0,0 +1,19 @@
1
+ LICENSE
2
+ MANIFEST.in
3
+ README.md
4
+ pyproject.toml
5
+ athlinks_races.egg-info/PKG-INFO
6
+ athlinks_races.egg-info/SOURCES.txt
7
+ athlinks_races.egg-info/dependency_links.txt
8
+ athlinks_races.egg-info/entry_points.txt
9
+ athlinks_races.egg-info/requires.txt
10
+ athlinks_races.egg-info/top_level.txt
11
+ scrapy_athlinks/__init__.py
12
+ scrapy_athlinks/cli.py
13
+ scrapy_athlinks/demo.py
14
+ scrapy_athlinks/items.py
15
+ scrapy_athlinks/pipelines.py
16
+ scrapy_athlinks/settings.py
17
+ scrapy_athlinks/spiders/__init__.py
18
+ scrapy_athlinks/spiders/race.py
19
+ tests/tests.py
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ scrapy_athlinks_cli = scrapy_athlinks.cli:main
@@ -0,0 +1,9 @@
1
+ Scrapy==2.6.3
2
+
3
+ [dev]
4
+ twine
5
+ wheel
6
+ build
7
+
8
+ [lint]
9
+ pylint
@@ -0,0 +1,3 @@
1
+ dist
2
+ sample_data
3
+ scrapy_athlinks