JobSelect 0.10.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- jobselect-0.10.2/JobSelect.egg-info/PKG-INFO +247 -0
- jobselect-0.10.2/JobSelect.egg-info/SOURCES.txt +38 -0
- jobselect-0.10.2/JobSelect.egg-info/dependency_links.txt +1 -0
- jobselect-0.10.2/JobSelect.egg-info/entry_points.txt +2 -0
- jobselect-0.10.2/JobSelect.egg-info/requires.txt +7 -0
- jobselect-0.10.2/JobSelect.egg-info/top_level.txt +3 -0
- jobselect-0.10.2/LICENSE +21 -0
- jobselect-0.10.2/PKG-INFO +247 -0
- jobselect-0.10.2/README.md +232 -0
- jobselect-0.10.2/cli/jobselect.py +114 -0
- jobselect-0.10.2/model/Model.py +123 -0
- jobselect-0.10.2/model/__init__.py +10 -0
- jobselect-0.10.2/model/eval.py +106 -0
- jobselect-0.10.2/model/pred.py +81 -0
- jobselect-0.10.2/model/prep/__init__.py +0 -0
- jobselect-0.10.2/model/prep/data_prep.py +95 -0
- jobselect-0.10.2/model/prep/label_vocab.json +50 -0
- jobselect-0.10.2/model/prep/prepared_data.npz +0 -0
- jobselect-0.10.2/model/prep/sym_map.py +106 -0
- jobselect-0.10.2/model/prep/test.py +5 -0
- jobselect-0.10.2/model/prep/vectorizer.pkl +0 -0
- jobselect-0.10.2/model_out/skill_classifier.pt +0 -0
- jobselect-0.10.2/model_out/training_history.json +1 -0
- jobselect-0.10.2/pyproject.toml +32 -0
- jobselect-0.10.2/setup.cfg +4 -0
- jobselect-0.10.2/test/test_model.py +36 -0
|
@@ -0,0 +1,247 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: JobSelect
|
|
3
|
+
Version: 0.10.2
|
|
4
|
+
Author-email: Akshay Babu <akshaysureshbabu100@gmail.com>
|
|
5
|
+
Description-Content-Type: text/markdown
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Requires-Dist: rich==15.0.0
|
|
8
|
+
Requires-Dist: pyfiglet==1.0.4
|
|
9
|
+
Requires-Dist: scikit-learn==1.9.0
|
|
10
|
+
Requires-Dist: uvicorn==0.50.2
|
|
11
|
+
Requires-Dist: pydantic==2.13.4
|
|
12
|
+
Requires-Dist: requests==2.34.2
|
|
13
|
+
Requires-Dist: torch==2.12.1
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# Job Description Skill Classifier [JobSeclect v0.10.2 & JobAnalyze 6k v1.0] (Multi-Label)
|
|
17
|
+
|
|
18
|
+
[](https://www.python.org/)
|
|
19
|
+
[](https://pytorch.org/)
|
|
20
|
+
[](https://scikit-learn.org/)
|
|
21
|
+
[](https://numpy.org/)
|
|
22
|
+
[](https://pandas.pydata.org/)
|
|
23
|
+
|
|
24
|
+
This project builds a lightweight text classification pipeline that predicts **multiple technical skills** from a job posting. Given a job description (optionally augmented with role and job type), the model outputs a ranked list of likely skills.
|
|
25
|
+
|
|
26
|
+
Installation:
|
|
27
|
+
|
|
28
|
+
- `pip install jobselect`
|
|
29
|
+
|
|
30
|
+
It uses:
|
|
31
|
+
|
|
32
|
+
- **TF-IDF** features over the combined text (job description + role + type)
|
|
33
|
+
- A **PyTorch feed-forward neural network** trained as a **multi-label** classifier
|
|
34
|
+
- **Per-skill thresholding** for evaluation and **top-k ranked probabilities** for inference
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## What it does
|
|
39
|
+
|
|
40
|
+
1. **Data preparation** (`model/prep/data_prep.py`)
|
|
41
|
+
- Reads cleaned job description data.
|
|
42
|
+
- Normalizes/repairs common skill typos (e.g., `tesnorflow/pytorch` → `tensorflow/pytorch`).
|
|
43
|
+
- Builds a **multi-hot** target vector of skills.
|
|
44
|
+
- Fits a **TF-IDF** vectorizer (with n-grams) and splits into train/test.
|
|
45
|
+
- Saves:
|
|
46
|
+
- `model/prep/prepared_data.npz` (TF-IDF arrays + labels + indexes)
|
|
47
|
+
- `model/prep/vectorizer.pkl` (fitted TF-IDF vectorizer)
|
|
48
|
+
- `model/prep/label_vocab.json` (skill label vocabulary)
|
|
49
|
+
|
|
50
|
+
2. **Model training** (`model/model.py`)
|
|
51
|
+
- Loads prepared TF-IDF arrays.
|
|
52
|
+
- Defines a simple **MLP**:
|
|
53
|
+
- Linear → ReLU → Dropout → Linear (one logit per skill)
|
|
54
|
+
- Trains with `BCEWithLogitsLoss` (multi-label setting).
|
|
55
|
+
- Saves:
|
|
56
|
+
- `model_out/skill_classifier.pt` (model weights)
|
|
57
|
+
- `model_out/training_history.json` (train/test loss curves)
|
|
58
|
+
|
|
59
|
+
3. **Evaluation** (`model/eval.py`)
|
|
60
|
+
- Loads the trained model.
|
|
61
|
+
- Applies a fixed sigmoid + threshold (**0.5**) to obtain binary skill predictions.
|
|
62
|
+
- Reports:
|
|
63
|
+
- Per-skill precision/recall/F1
|
|
64
|
+
- Micro-F1 and Macro-F1
|
|
65
|
+
- Compares against a simple baseline (frequency-driven / always-predict-most-frequent labels).
|
|
66
|
+
|
|
67
|
+
4. **Prediction / Inference** (`model/pred.py`)
|
|
68
|
+
- Loads the TF-IDF vectorizer and trained model.
|
|
69
|
+
- Creates TF-IDF features for the input text.
|
|
70
|
+
- Outputs the **top-k** skills by probability.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Use cases
|
|
75
|
+
|
|
76
|
+
- **Resume/job-post matching** (first-pass filtering of relevant skills)
|
|
77
|
+
- **Job taxonomy building** (discover recurring skills from postings)
|
|
78
|
+
- **Recruiting analytics** (aggregate predicted skill demand by seniority/role/type)
|
|
79
|
+
- **Prototyping multi-label NLP classifiers** (TF-IDF + MLP baseline)
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## Project structure
|
|
84
|
+
|
|
85
|
+
```text
|
|
86
|
+
.
|
|
87
|
+
├─ data/
|
|
88
|
+
│ ├─ raw/
|
|
89
|
+
│ │ └─ Job_descriptions.csv # Raw input dataset
|
|
90
|
+
│ ├─ sample_data/
|
|
91
|
+
│ │ └─ test.txt # Example JD, Role and Type for testing
|
|
92
|
+
│ └─ clean/
|
|
93
|
+
│ ├─ cleaned_job_descriptions.csv # Cleaned master CSV
|
|
94
|
+
│ ├─ cleaned_job_descriptions_internships.csv
|
|
95
|
+
│ ├─ cleaned_job_descriptions_junior.csv
|
|
96
|
+
│ └─ cleaned_job_descriptions_senior.csv
|
|
97
|
+
│
|
|
98
|
+
├─ model/
|
|
99
|
+
│ ├─ prep/
|
|
100
|
+
│ │ ├─ data_prep.py # TF-IDF + multi-hot label creation + train/test split
|
|
101
|
+
│ │ ├─ sym_map.py # Synonym/phrase normalization map used during prep
|
|
102
|
+
│ │ ├─ vectorizer.pkl # Saved TF-IDF vectorizer (generated by data_prep)
|
|
103
|
+
│ │ ├─ prepared_data.npz # Saved arrays (generated by data_prep)
|
|
104
|
+
│ │ └─ label_vocab.json # Skill label vocabulary (generated by data_prep)
|
|
105
|
+
│ │
|
|
106
|
+
│ ├─ model.py # PyTorch multi-label classifier training
|
|
107
|
+
│ ├─ eval.py # Thresholded evaluation + F1 metrics + baseline comparison
|
|
108
|
+
│ └─ pred.py # Predict top-k skills for new text
|
|
109
|
+
│
|
|
110
|
+
├─ model_out/
|
|
111
|
+
│ ├─ skill_classifier.pt # Trained model weights (generated by model.py)
|
|
112
|
+
│ └─ training_history.json # Training loss history (generated by model.py)
|
|
113
|
+
│
|
|
114
|
+
├─ cli/
|
|
115
|
+
│ └─ jobselect.py # Rich terminal CLI for interactive prediction (uses model.pred.predict)
|
|
116
|
+
│
|
|
117
|
+
├─ test/
|
|
118
|
+
│ └─ test_model.py # Pytest checks expected artifacts exist in model_out/ and model/prep/
|
|
119
|
+
│
|
|
120
|
+
├─ notebooks/
|
|
121
|
+
│ ├─ 01_EDA.ipynb # Exploratory Data Analysis
|
|
122
|
+
│ └─ 02_Data_Engineering.ipynb # Data engineering / cleaning notes
|
|
123
|
+
│
|
|
124
|
+
├─ pipeline.py # Executes notebooks + training/eval steps in order
|
|
125
|
+
├─ pyproject.toml # Installs as a cli tool
|
|
126
|
+
├─ requirements.txt
|
|
127
|
+
└─ README.md
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Requirements
|
|
133
|
+
|
|
134
|
+
See `requirements.txt` for the exact dependencies.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## Getting started
|
|
139
|
+
|
|
140
|
+
### 1) Clone and Install dependencies
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
git clone https://github.com/Ak47xdd/Job-Description-Analysis.git
|
|
144
|
+
pip install -r requirements.txt
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
### 2) Run data preparation (optional)
|
|
148
|
+
|
|
149
|
+
This builds the TF-IDF features and label vocabulary from the cleaned CSV.
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
python model/prep/data_prep.py
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Expected outputs:
|
|
156
|
+
|
|
157
|
+
- `model/prep/prepared_data.npz`
|
|
158
|
+
- `model/prep/vectorizer.pkl`
|
|
159
|
+
- `model/prep/label_vocab.json`
|
|
160
|
+
|
|
161
|
+
### 3) Train the model (optional)
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
python model/model.py
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Expected outputs:
|
|
168
|
+
|
|
169
|
+
- `model_out/skill_classifier.pt`
|
|
170
|
+
- `model_out/training_history.json`
|
|
171
|
+
|
|
172
|
+
### 4) Evaluate performance (optional)
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
python model/eval.py
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Outputs include:
|
|
179
|
+
|
|
180
|
+
- Per-skill metrics (precision/recall/F1)
|
|
181
|
+
- Micro-F1 and Macro-F1
|
|
182
|
+
- Baseline comparison
|
|
183
|
+
|
|
184
|
+
### 5) Predict skills for a new job description
|
|
185
|
+
|
|
186
|
+
#### Option A: Use Python function
|
|
187
|
+
|
|
188
|
+
`from model.pred import predict`
|
|
189
|
+
|
|
190
|
+
`data/sample_data/test.txt` contains an example job description inside. You can also call the `predict(job_desc, role=..., job_type=..., top_k=...)` function from Python.
|
|
191
|
+
|
|
192
|
+
#### Option B: Use the interactive CLI
|
|
193
|
+
|
|
194
|
+
```bash
|
|
195
|
+
python cli/jobselect.py
|
|
196
|
+
|
|
197
|
+
or
|
|
198
|
+
|
|
199
|
+
pip install jobselect
|
|
200
|
+
jobselect
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
The CLI prompts for job description, role, and job type, then prints the top skills with probabilities.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## How predictions work
|
|
208
|
+
|
|
209
|
+
- Text is concatenated as:
|
|
210
|
+
`"{job_desc} {role} {job_type}"`
|
|
211
|
+
- TF-IDF transforms text into a fixed-size vector
|
|
212
|
+
- The network outputs one logit per skill
|
|
213
|
+
- Sigmoid converts logits → probabilities
|
|
214
|
+
- Skills are ranked by probability and the top-k are returned
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## Important implementation notes
|
|
219
|
+
|
|
220
|
+
- **Multi-label learning:** Each skill is treated independently (binary relevance via sigmoid + `BCEWithLogitsLoss`).
|
|
221
|
+
- **Evaluation threshold:** `model/eval.py` uses a fixed threshold of **0.5**. For production use, you may want per-label thresholds tuned on a validation set.
|
|
222
|
+
- **Dataset size:** The included notebooks and evaluation code suggest the dataset may be small; results can be limited by label frequency and data coverage.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## New features / capabilities
|
|
227
|
+
|
|
228
|
+
- **Rich terminal CLI** (`cli/jobauto.py`) using `rich` + `pyfiglet` for interactive top-skill display.
|
|
229
|
+
- **Synonym/phrase normalization hook** (`model/prep/sym_map.py`) applied during data preparation.
|
|
230
|
+
- **Pipeline runner** (`pipeline.py`) to execute notebooks and training steps in sequence.
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
## Customization ideas
|
|
235
|
+
|
|
236
|
+
- Improve text cleaning and skill normalization in `data_prep.py`
|
|
237
|
+
- Tune TF-IDF parameters (`max_features`, `ngram_range`, `min_df`)
|
|
238
|
+
- Replace the simple MLP with a stronger baseline (e.g., logistic regression on TF-IDF)
|
|
239
|
+
- Calibrate thresholds per label using validation data
|
|
240
|
+
- Add a CLI or web service endpoint for prediction
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## References / Inspiration
|
|
245
|
+
|
|
246
|
+
This repository follows a common pattern for multi-label NLP baselines:
|
|
247
|
+
TF-IDF features + a simple neural network + sigmoid-based multi-label outputs.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
./cli/jobselect.py
|
|
5
|
+
./model/Model.py
|
|
6
|
+
./model/__init__.py
|
|
7
|
+
./model/eval.py
|
|
8
|
+
./model/pred.py
|
|
9
|
+
./model/prep/__init__.py
|
|
10
|
+
./model/prep/data_prep.py
|
|
11
|
+
./model/prep/label_vocab.json
|
|
12
|
+
./model/prep/prepared_data.npz
|
|
13
|
+
./model/prep/sym_map.py
|
|
14
|
+
./model/prep/test.py
|
|
15
|
+
./model/prep/vectorizer.pkl
|
|
16
|
+
./model_out/skill_classifier.pt
|
|
17
|
+
./model_out/training_history.json
|
|
18
|
+
JobSelect.egg-info/PKG-INFO
|
|
19
|
+
JobSelect.egg-info/SOURCES.txt
|
|
20
|
+
JobSelect.egg-info/dependency_links.txt
|
|
21
|
+
JobSelect.egg-info/entry_points.txt
|
|
22
|
+
JobSelect.egg-info/requires.txt
|
|
23
|
+
JobSelect.egg-info/top_level.txt
|
|
24
|
+
cli/jobselect.py
|
|
25
|
+
model/Model.py
|
|
26
|
+
model/__init__.py
|
|
27
|
+
model/eval.py
|
|
28
|
+
model/pred.py
|
|
29
|
+
model/prep/__init__.py
|
|
30
|
+
model/prep/data_prep.py
|
|
31
|
+
model/prep/label_vocab.json
|
|
32
|
+
model/prep/prepared_data.npz
|
|
33
|
+
model/prep/sym_map.py
|
|
34
|
+
model/prep/test.py
|
|
35
|
+
model/prep/vectorizer.pkl
|
|
36
|
+
model_out/skill_classifier.pt
|
|
37
|
+
model_out/training_history.json
|
|
38
|
+
test/test_model.py
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
jobselect-0.10.2/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Akshay_Babu
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,247 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: JobSelect
|
|
3
|
+
Version: 0.10.2
|
|
4
|
+
Author-email: Akshay Babu <akshaysureshbabu100@gmail.com>
|
|
5
|
+
Description-Content-Type: text/markdown
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Requires-Dist: rich==15.0.0
|
|
8
|
+
Requires-Dist: pyfiglet==1.0.4
|
|
9
|
+
Requires-Dist: scikit-learn==1.9.0
|
|
10
|
+
Requires-Dist: uvicorn==0.50.2
|
|
11
|
+
Requires-Dist: pydantic==2.13.4
|
|
12
|
+
Requires-Dist: requests==2.34.2
|
|
13
|
+
Requires-Dist: torch==2.12.1
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# Job Description Skill Classifier [JobSeclect v0.10.2 & JobAnalyze 6k v1.0] (Multi-Label)
|
|
17
|
+
|
|
18
|
+
[](https://www.python.org/)
|
|
19
|
+
[](https://pytorch.org/)
|
|
20
|
+
[](https://scikit-learn.org/)
|
|
21
|
+
[](https://numpy.org/)
|
|
22
|
+
[](https://pandas.pydata.org/)
|
|
23
|
+
|
|
24
|
+
This project builds a lightweight text classification pipeline that predicts **multiple technical skills** from a job posting. Given a job description (optionally augmented with role and job type), the model outputs a ranked list of likely skills.
|
|
25
|
+
|
|
26
|
+
Installation:
|
|
27
|
+
|
|
28
|
+
- `pip install jobselect`
|
|
29
|
+
|
|
30
|
+
It uses:
|
|
31
|
+
|
|
32
|
+
- **TF-IDF** features over the combined text (job description + role + type)
|
|
33
|
+
- A **PyTorch feed-forward neural network** trained as a **multi-label** classifier
|
|
34
|
+
- **Per-skill thresholding** for evaluation and **top-k ranked probabilities** for inference
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## What it does
|
|
39
|
+
|
|
40
|
+
1. **Data preparation** (`model/prep/data_prep.py`)
|
|
41
|
+
- Reads cleaned job description data.
|
|
42
|
+
- Normalizes/repairs common skill typos (e.g., `tesnorflow/pytorch` → `tensorflow/pytorch`).
|
|
43
|
+
- Builds a **multi-hot** target vector of skills.
|
|
44
|
+
- Fits a **TF-IDF** vectorizer (with n-grams) and splits into train/test.
|
|
45
|
+
- Saves:
|
|
46
|
+
- `model/prep/prepared_data.npz` (TF-IDF arrays + labels + indexes)
|
|
47
|
+
- `model/prep/vectorizer.pkl` (fitted TF-IDF vectorizer)
|
|
48
|
+
- `model/prep/label_vocab.json` (skill label vocabulary)
|
|
49
|
+
|
|
50
|
+
2. **Model training** (`model/model.py`)
|
|
51
|
+
- Loads prepared TF-IDF arrays.
|
|
52
|
+
- Defines a simple **MLP**:
|
|
53
|
+
- Linear → ReLU → Dropout → Linear (one logit per skill)
|
|
54
|
+
- Trains with `BCEWithLogitsLoss` (multi-label setting).
|
|
55
|
+
- Saves:
|
|
56
|
+
- `model_out/skill_classifier.pt` (model weights)
|
|
57
|
+
- `model_out/training_history.json` (train/test loss curves)
|
|
58
|
+
|
|
59
|
+
3. **Evaluation** (`model/eval.py`)
|
|
60
|
+
- Loads the trained model.
|
|
61
|
+
- Applies a fixed sigmoid + threshold (**0.5**) to obtain binary skill predictions.
|
|
62
|
+
- Reports:
|
|
63
|
+
- Per-skill precision/recall/F1
|
|
64
|
+
- Micro-F1 and Macro-F1
|
|
65
|
+
- Compares against a simple baseline (frequency-driven / always-predict-most-frequent labels).
|
|
66
|
+
|
|
67
|
+
4. **Prediction / Inference** (`model/pred.py`)
|
|
68
|
+
- Loads the TF-IDF vectorizer and trained model.
|
|
69
|
+
- Creates TF-IDF features for the input text.
|
|
70
|
+
- Outputs the **top-k** skills by probability.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Use cases
|
|
75
|
+
|
|
76
|
+
- **Resume/job-post matching** (first-pass filtering of relevant skills)
|
|
77
|
+
- **Job taxonomy building** (discover recurring skills from postings)
|
|
78
|
+
- **Recruiting analytics** (aggregate predicted skill demand by seniority/role/type)
|
|
79
|
+
- **Prototyping multi-label NLP classifiers** (TF-IDF + MLP baseline)
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## Project structure
|
|
84
|
+
|
|
85
|
+
```text
|
|
86
|
+
.
|
|
87
|
+
├─ data/
|
|
88
|
+
│ ├─ raw/
|
|
89
|
+
│ │ └─ Job_descriptions.csv # Raw input dataset
|
|
90
|
+
│ ├─ sample_data/
|
|
91
|
+
│ │ └─ test.txt # Example JD, Role and Type for testing
|
|
92
|
+
│ └─ clean/
|
|
93
|
+
│ ├─ cleaned_job_descriptions.csv # Cleaned master CSV
|
|
94
|
+
│ ├─ cleaned_job_descriptions_internships.csv
|
|
95
|
+
│ ├─ cleaned_job_descriptions_junior.csv
|
|
96
|
+
│ └─ cleaned_job_descriptions_senior.csv
|
|
97
|
+
│
|
|
98
|
+
├─ model/
|
|
99
|
+
│ ├─ prep/
|
|
100
|
+
│ │ ├─ data_prep.py # TF-IDF + multi-hot label creation + train/test split
|
|
101
|
+
│ │ ├─ sym_map.py # Synonym/phrase normalization map used during prep
|
|
102
|
+
│ │ ├─ vectorizer.pkl # Saved TF-IDF vectorizer (generated by data_prep)
|
|
103
|
+
│ │ ├─ prepared_data.npz # Saved arrays (generated by data_prep)
|
|
104
|
+
│ │ └─ label_vocab.json # Skill label vocabulary (generated by data_prep)
|
|
105
|
+
│ │
|
|
106
|
+
│ ├─ model.py # PyTorch multi-label classifier training
|
|
107
|
+
│ ├─ eval.py # Thresholded evaluation + F1 metrics + baseline comparison
|
|
108
|
+
│ └─ pred.py # Predict top-k skills for new text
|
|
109
|
+
│
|
|
110
|
+
├─ model_out/
|
|
111
|
+
│ ├─ skill_classifier.pt # Trained model weights (generated by model.py)
|
|
112
|
+
│ └─ training_history.json # Training loss history (generated by model.py)
|
|
113
|
+
│
|
|
114
|
+
├─ cli/
|
|
115
|
+
│ └─ jobselect.py # Rich terminal CLI for interactive prediction (uses model.pred.predict)
|
|
116
|
+
│
|
|
117
|
+
├─ test/
|
|
118
|
+
│ └─ test_model.py # Pytest checks expected artifacts exist in model_out/ and model/prep/
|
|
119
|
+
│
|
|
120
|
+
├─ notebooks/
|
|
121
|
+
│ ├─ 01_EDA.ipynb # Exploratory Data Analysis
|
|
122
|
+
│ └─ 02_Data_Engineering.ipynb # Data engineering / cleaning notes
|
|
123
|
+
│
|
|
124
|
+
├─ pipeline.py # Executes notebooks + training/eval steps in order
|
|
125
|
+
├─ pyproject.toml # Installs as a cli tool
|
|
126
|
+
├─ requirements.txt
|
|
127
|
+
└─ README.md
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Requirements
|
|
133
|
+
|
|
134
|
+
See `requirements.txt` for the exact dependencies.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## Getting started
|
|
139
|
+
|
|
140
|
+
### 1) Clone and Install dependencies
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
git clone https://github.com/Ak47xdd/Job-Description-Analysis.git
|
|
144
|
+
pip install -r requirements.txt
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
### 2) Run data preparation (optional)
|
|
148
|
+
|
|
149
|
+
This builds the TF-IDF features and label vocabulary from the cleaned CSV.
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
python model/prep/data_prep.py
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Expected outputs:
|
|
156
|
+
|
|
157
|
+
- `model/prep/prepared_data.npz`
|
|
158
|
+
- `model/prep/vectorizer.pkl`
|
|
159
|
+
- `model/prep/label_vocab.json`
|
|
160
|
+
|
|
161
|
+
### 3) Train the model (optional)
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
python model/model.py
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Expected outputs:
|
|
168
|
+
|
|
169
|
+
- `model_out/skill_classifier.pt`
|
|
170
|
+
- `model_out/training_history.json`
|
|
171
|
+
|
|
172
|
+
### 4) Evaluate performance (optional)
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
python model/eval.py
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Outputs include:
|
|
179
|
+
|
|
180
|
+
- Per-skill metrics (precision/recall/F1)
|
|
181
|
+
- Micro-F1 and Macro-F1
|
|
182
|
+
- Baseline comparison
|
|
183
|
+
|
|
184
|
+
### 5) Predict skills for a new job description
|
|
185
|
+
|
|
186
|
+
#### Option A: Use Python function
|
|
187
|
+
|
|
188
|
+
`from model.pred import predict`
|
|
189
|
+
|
|
190
|
+
`data/sample_data/test.txt` contains an example job description inside. You can also call the `predict(job_desc, role=..., job_type=..., top_k=...)` function from Python.
|
|
191
|
+
|
|
192
|
+
#### Option B: Use the interactive CLI
|
|
193
|
+
|
|
194
|
+
```bash
|
|
195
|
+
python cli/jobselect.py
|
|
196
|
+
|
|
197
|
+
or
|
|
198
|
+
|
|
199
|
+
pip install jobselect
|
|
200
|
+
jobselect
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
The CLI prompts for job description, role, and job type, then prints the top skills with probabilities.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## How predictions work
|
|
208
|
+
|
|
209
|
+
- Text is concatenated as:
|
|
210
|
+
`"{job_desc} {role} {job_type}"`
|
|
211
|
+
- TF-IDF transforms text into a fixed-size vector
|
|
212
|
+
- The network outputs one logit per skill
|
|
213
|
+
- Sigmoid converts logits → probabilities
|
|
214
|
+
- Skills are ranked by probability and the top-k are returned
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## Important implementation notes
|
|
219
|
+
|
|
220
|
+
- **Multi-label learning:** Each skill is treated independently (binary relevance via sigmoid + `BCEWithLogitsLoss`).
|
|
221
|
+
- **Evaluation threshold:** `model/eval.py` uses a fixed threshold of **0.5**. For production use, you may want per-label thresholds tuned on a validation set.
|
|
222
|
+
- **Dataset size:** The included notebooks and evaluation code suggest the dataset may be small; results can be limited by label frequency and data coverage.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## New features / capabilities
|
|
227
|
+
|
|
228
|
+
- **Rich terminal CLI** (`cli/jobauto.py`) using `rich` + `pyfiglet` for interactive top-skill display.
|
|
229
|
+
- **Synonym/phrase normalization hook** (`model/prep/sym_map.py`) applied during data preparation.
|
|
230
|
+
- **Pipeline runner** (`pipeline.py`) to execute notebooks and training steps in sequence.
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
## Customization ideas
|
|
235
|
+
|
|
236
|
+
- Improve text cleaning and skill normalization in `data_prep.py`
|
|
237
|
+
- Tune TF-IDF parameters (`max_features`, `ngram_range`, `min_df`)
|
|
238
|
+
- Replace the simple MLP with a stronger baseline (e.g., logistic regression on TF-IDF)
|
|
239
|
+
- Calibrate thresholds per label using validation data
|
|
240
|
+
- Add a CLI or web service endpoint for prediction
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## References / Inspiration
|
|
245
|
+
|
|
246
|
+
This repository follows a common pattern for multi-label NLP baselines:
|
|
247
|
+
TF-IDF features + a simple neural network + sigmoid-based multi-label outputs.
|