nlpkit14 0.3.0__tar.gz

nlpkit14-0.3.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include TEST_RESULTS.md
+recursive-include tests *.py

nlpkit14-0.3.0/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.4
+Name: nlpkit14
+Version: 0.3.0
+Summary: Execute fourteen NLP practical programs with commented source and expected outputs
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: nltk<4,>=3.9
+Requires-Dist: spacy<4,>=3.8
+Requires-Dist: scikit-learn<2,>=1.5
+Requires-Dist: requests<3,>=2.32
+Requires-Dist: beautifulsoup4<5,>=4.12
+Provides-Extra: models
+Requires-Dist: torch<3,>=2.4; extra == "models"
+Requires-Dist: transformers<5,>=4.45; extra == "models"
+Requires-Dist: benepar<0.3,>=0.2.0; extra == "models"
+Requires-Dist: sentencepiece<1,>=0.2; extra == "models"
+
+# nlpkit14 0.3.0
+
+Run the fourteen programs from the supplied IA1 question/code/output document.
+
+```python
+import nlpkit14 as n
+n.q1()       # Actually runs question 1 and prints computed output
+n.source(1)  # Shows the code, question and expected output as comments
+n.questions()
+```
+
+Question numbers 1–7 are B1–B7. Numbers 8–14 are C1–C7. The question, expected output, and extra notes are comments in each bundled standalone Python program. Exercise calls print only the program's output. No expected-output text is substituted for model computation.
+
+## Install and prepare online once
+
+Install the shared wheel, then prepare the basic data and spaCy model:
+
+```sh
+python -m pip install ./nlpkit14-0.3.0-py3-none-any.whl
+python -c "import nlpkit14 as n; n.setup()"
+```
+
+For all exercises, install the model extra and run the complete setup (large downloads):
+
+```sh
+python -m pip install './nlpkit14-0.3.0-py3-none-any.whl[models]'
+python -c "import nlpkit14 as n; n.setup(models=True)"
+```
+
+This is a local release of the new nlpkit14 project, not yet uploaded to PyPI. Do not use pip install nlpkit14 until publication succeeds.
+
+## Offline installation
+
+On an internet-connected machine with the SAME OS, architecture and Python version as the offline machine, download the wheel and dependencies:
+
+```sh
+python -m pip download --dest wheelhouse './nlpkit14-0.3.0-py3-none-any.whl[models]'
+```
+
+Install and run setup online on that matching machine. Copy the wheelhouse, both installed spaCy model packages (or their installation wheels), and the full data/cache folder to the offline machine. Model data defaults to ~/.nlpkit14; set NLPKIT14_DATA before setup/import to choose a portable folder. Copying the entire prepared Python environment is platform specific; do not copy a Mac environment to Windows.
+
+On the offline computer:
+
+```sh
+python -m pip install --no-index --find-links=wheelhouse nlpkit14==0.3.0
+```
+
+Install the spaCy model wheels from local storage too. Keep the copied data folder at ~/.nlpkit14 or point NLPKIT14_DATA to it. Model and dependency archives are not included in this small library wheel.
+
+q8 fetches https://www.bnmit.org/ and requires internet even after setup. Other exercises can run without internet after all their requirements are installed. No downloads run implicitly from q1–q14. Missing packages/data raise their real installation errors rather than producing fake results.
+
+## Output fidelity
+
+The original algorithms and sample input are preserved. Downloads are moved to explicit setup. HTTP errors are checked. BERT uses evaluation mode and local cached model files. The raw Good–Turing demonstration is not a normalized probability estimator. The document's classifier fits its vocabulary before splitting and is an educational example, not an unbiased evaluation pipeline. Model outputs, parse trees, POS tags and similarity values can differ from the commented reference output. The document's B5 reference omits POS lines that the code actually prints. Read TEST_RESULTS.md for exactly which programs were executed.

nlpkit14-0.3.0/README.md

@@ -0,0 +1,54 @@
+# nlpkit14 0.3.0
+
+Run the fourteen programs from the supplied IA1 question/code/output document.
+
+```python
+import nlpkit14 as n
+n.q1()       # Actually runs question 1 and prints computed output
+n.source(1)  # Shows the code, question and expected output as comments
+n.questions()
+```
+
+Question numbers 1–7 are B1–B7. Numbers 8–14 are C1–C7. The question, expected output, and extra notes are comments in each bundled standalone Python program. Exercise calls print only the program's output. No expected-output text is substituted for model computation.
+
+## Install and prepare online once
+
+Install the shared wheel, then prepare the basic data and spaCy model:
+
+```sh
+python -m pip install ./nlpkit14-0.3.0-py3-none-any.whl
+python -c "import nlpkit14 as n; n.setup()"
+```
+
+For all exercises, install the model extra and run the complete setup (large downloads):
+
+```sh
+python -m pip install './nlpkit14-0.3.0-py3-none-any.whl[models]'
+python -c "import nlpkit14 as n; n.setup(models=True)"
+```
+
+This is a local release of the new nlpkit14 project, not yet uploaded to PyPI. Do not use pip install nlpkit14 until publication succeeds.
+
+## Offline installation
+
+On an internet-connected machine with the SAME OS, architecture and Python version as the offline machine, download the wheel and dependencies:
+
+```sh
+python -m pip download --dest wheelhouse './nlpkit14-0.3.0-py3-none-any.whl[models]'
+```
+
+Install and run setup online on that matching machine. Copy the wheelhouse, both installed spaCy model packages (or their installation wheels), and the full data/cache folder to the offline machine. Model data defaults to ~/.nlpkit14; set NLPKIT14_DATA before setup/import to choose a portable folder. Copying the entire prepared Python environment is platform specific; do not copy a Mac environment to Windows.
+
+On the offline computer:
+
+```sh
+python -m pip install --no-index --find-links=wheelhouse nlpkit14==0.3.0
+```
+
+Install the spaCy model wheels from local storage too. Keep the copied data folder at ~/.nlpkit14 or point NLPKIT14_DATA to it. Model and dependency archives are not included in this small library wheel.
+
+q8 fetches https://www.bnmit.org/ and requires internet even after setup. Other exercises can run without internet after all their requirements are installed. No downloads run implicitly from q1–q14. Missing packages/data raise their real installation errors rather than producing fake results.
+
+## Output fidelity
+
+The original algorithms and sample input are preserved. Downloads are moved to explicit setup. HTTP errors are checked. BERT uses evaluation mode and local cached model files. The raw Good–Turing demonstration is not a normalized probability estimator. The document's classifier fits its vocabulary before splitting and is an educational example, not an unbiased evaluation pipeline. Model outputs, parse trees, POS tags and similarity values can differ from the commented reference output. The document's B5 reference omits POS lines that the code actually prints. Read TEST_RESULTS.md for exactly which programs were executed.

nlpkit14-0.3.0/TEST_RESULTS.md

@@ -0,0 +1,15 @@
+# Verification of version 0.3.0
+
+- All fourteen document programs were extracted, reviewed, compiled, and bundled with their question and expected output as Python comments.
+- The wheel built with Python 3.12 and passed `twine check`.
+- A local no-index, no-dependency installation into a Python 3.12 virtual environment succeeded. The installed q9() computed the document's bigram probability. This is not a complete dependency-resolution test.
+- The execution suite ran under an existing Python 3.13 Anaconda environment with NLTK, scikit-learn, requests and BeautifulSoup. Ten tests passed; one was skipped because WordNet data was not fully downloaded. Programs q1, q8 (controlled HTML response), q9, q10, q11 and q12 executed. Numeric/classification exercises were also run with socket connections blocked.
+- q4 has not completed an end-to-end test because the WordNet download was stopped to keep delivery quick. The incomplete test archive is outside the deliverables.
+- q2, q3, q5, q6, q7, q13 and q14 were syntax-checked but not executed with their required models. spaCy/Benepar/BERT model downloads were not completed. These are not asserted to be error-free across all dependency versions.
+- q8's HTML extraction and real NLTK POS tagging were tested with a controlled HTTP response. Fetching the live BNMIT website was not tested.
+- Windows and Linux were not tested. Full setup(), model extras dependency resolution, and transfer of an offline wheelhouse/model cache have not been end-to-end tested.
+- This release has not been uploaded to PyPI. The previously published version remains separate.
+
+Model exercises require their real packages and data. No static sample answer is printed in place of model inference. Unexpected errors are not suppressed.
+
+Rename: public distribution and import are now nlpkit14. No upload has been performed.

nlpkit14-0.3.0/pyproject.toml

@@ -0,0 +1,20 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "nlpkit14"
+version = "0.3.0"
+description = "Execute fourteen NLP practical programs with commented source and expected outputs"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = ["nltk>=3.9,<4", "spacy>=3.8,<4", "scikit-learn>=1.5,<2", "requests>=2.32,<3", "beautifulsoup4>=4.12,<5"]
+
+[project.optional-dependencies]
+models = ["torch>=2.4,<3", "transformers>=4.45,<5", "benepar>=0.2.0,<0.3", "sentencepiece>=0.2,<1"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+nlpkit14 = ["questions.json", "programs/*.py"]

nlpkit14-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

nlpkit14-0.3.0/src/nlpkit14/__init__.py

@@ -0,0 +1,70 @@
+"""Run the fourteen supplied NLP practicals, with source available separately."""
+from importlib.resources import files
+import json
+import os
+
+__version__ = "0.3.0"
+
+def _resource(number):
+    if number not in range(1, 15):
+        raise ValueError("Choose a question from 1 to 14")
+    return files(__package__).joinpath("programs", f"q{number}.py")
+
+def _configure():
+    path = os.path.expanduser(os.environ.get("NLPKIT14_DATA", "~/.nlpkit14"))
+    data = os.path.join(path, "nltk_data")
+    os.environ.setdefault("HF_HOME", os.path.join(path, "huggingface"))
+    try:
+        import nltk
+    except ModuleNotFoundError:
+        return
+    if data not in nltk.data.path:
+        nltk.data.path.insert(0, data)
+
+def source(number):
+    """Print runnable source, including the commented question and expected output."""
+    print(_resource(number).read_text(encoding="utf-8"), end="")
+
+def questions():
+    for row in json.loads(files(__package__).joinpath("questions.json").read_text(encoding="utf-8")):
+        print(f"q{row['number']}(): {row['question']}")
+
+def _run(number):
+    _configure()
+    resource = _resource(number)
+    namespace = {"__name__": f"nlpkit14.question{number}"}
+    exec(compile(resource.read_text(encoding="utf-8"), str(resource), "exec"), namespace)
+
+def _make(number):
+    def question():
+        _run(number)
+    question.__name__ = f"q{number}"
+    question.__qualname__ = question.__name__
+    question.__doc__ = f"Execute question {number}. Use source({number}) to see code and comments."
+    return question
+
+for _number in range(1, 15):
+    globals()[f"q{_number}"] = _make(_number)
+
+def setup(models=False):
+    """Explicit online preparation. models=True also downloads large parsing/BERT models."""
+    _configure()
+    import nltk
+    import subprocess
+    import sys
+    directory = os.path.join(os.path.expanduser(os.environ.get("NLPKIT14_DATA", "~/.nlpkit14")), "nltk_data")
+    os.makedirs(directory, exist_ok=True)
+    for name in ("punkt_tab", "stopwords", "averaged_perceptron_tagger_eng", "wordnet", "omw-1.4"):
+        if not nltk.download(name, download_dir=directory, quiet=True):
+            raise RuntimeError(f"Could not download {name}; check internet connectivity")
+    subprocess.run([sys.executable, "-m", "spacy", "download", "en_core_web_sm"], check=True)
+    if models:
+        subprocess.run([sys.executable, "-m", "spacy", "download", "en_core_web_md"], check=True)
+        import benepar
+        if not benepar.download("benepar_en3", download_dir=directory):
+            raise RuntimeError("Could not download benepar_en3; check internet connectivity")
+        from transformers import BertTokenizer, BertModel
+        BertTokenizer.from_pretrained("bert-base-uncased")
+        BertModel.from_pretrained("bert-base-uncased")
+
+__all__ = ["questions", "source", "setup"] + [f"q{i}" for i in range(1, 15)]

nlpkit14-0.3.0/src/nlpkit14/programs/q1.py

@@ -0,0 +1,30 @@
+# Question 1 (B1): Tokenization and Stop-word Removal using NLTK
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Sentences: ['This is a simple example demonstrating stopword removal in NLP.', 'NLP is interesting.']
+# Words: ['This', 'is', 'a', 'simple', 'example', 'demonstrating', 'stopword', 'removal', 'in', 'NLP', '.', 'NLP', 'is', 'interesting', '.']
+# Filtered Tokens: ['simple', 'example', 'demonstrating', 'stopword', 'removal', 'NLP', 'NLP', 'interesting']
+
+import nltk
+from nltk.tokenize import word_tokenize, sent_tokenize
+from nltk.corpus import stopwords
+
+# Download required resources
+# Resource download moved to n.setup().
+# Resource download moved to n.setup().
+# Input text
+text = "This is a simple example demonstrating stopword removal in NLP. NLP is interesting."
+
+# Sentence tokenization
+sentences = sent_tokenize(text)
+
+# Word tokenization
+words = word_tokenize(text)
+
+# Stop-word removal
+stop_words = set(stopwords.words('english'))
+filtered = [w for w in words if w.lower() not in stop_words and w.isalnum()]
+
+print("Sentences:", sentences)
+print("Words:", words)
+print("Filtered Tokens:", filtered)

nlpkit14-0.3.0/src/nlpkit14/programs/q10.py

@@ -0,0 +1,41 @@
+# Question 10 (C3): Add-One (Laplace) Smoothing for an N-gram Model
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# P(i|<s>) = 0.272727
+# P(like|i) = 0.300000
+# P(apples|like) = 0.200000
+# P(</s>|apples) = 0.300000
+# Vocabulary Size = 8
+# Add-One Sentence Probability = 0.00490909
+
+from collections import Counter
+
+corpus = [
+    ['i', 'like', 'apples'],
+    ['i', 'like', 'bananas'],
+    ['you', 'eat', 'apples']
+]
+
+unigrams = Counter()
+bigrams = Counter()
+
+for sentence in corpus:
+    tokens = ['<s>'] + sentence + ['</s>']
+    unigrams.update(tokens)
+    bigrams.update(zip(tokens[:-1], tokens[1:]))
+
+V = len(unigrams)   # vocabulary size including boundary tokens
+
+def add_one(w1, w2):
+    return (bigrams[(w1, w2)] + 1) / (unigrams[w1] + V)
+
+test = ['<s>', 'i', 'like', 'apples', '</s>']
+probability = 1.0
+
+for w1, w2 in zip(test[:-1], test[1:]):
+    p = add_one(w1, w2)
+    probability *= p
+    print(f"P({w2}|{w1}) = {p:.6f}")
+
+print("Vocabulary Size =", V)
+print("Add-One Sentence Probability =", round(probability, 8))

nlpkit14-0.3.0/src/nlpkit14/programs/q11.py

@@ -0,0 +1,39 @@
+# Question 11 (C4): Calculate Good-Turing Frequency
+# Output below is the document example; actual model and website results may differ.
+# Raw Good-Turing adjusted frequencies only; this is not a normalized probability model.
+# Expected output:
+# r    C*(r)
+# -------------
+# 0    0.0
+# 1    1.3333
+# 2    1.5
+# 3    0.0
+
+from collections import Counter
+
+def good_turing(counts):
+    # counts contains observed frequencies of different events
+    freq_of_freq = Counter(counts)
+    max_r = max(freq_of_freq.keys())
+    adjusted_counts = {}
+
+    for r in range(0, max_r + 1):
+        Nr = freq_of_freq.get(r, 0)
+        Nr_plus_1 = freq_of_freq.get(r + 1, 0)
+
+        if Nr > 0:
+            c_star = (r + 1) * (Nr_plus_1 / Nr)
+            adjusted_counts[r] = round(c_star, 4)
+        else:
+            adjusted_counts[r] = 0.0
+
+    return adjusted_counts
+
+# 3 events occur once, 2 events occur twice, 1 event occurs 3 times
+sample_counts = [1, 1, 1, 2, 2, 3]
+result = good_turing(sample_counts)
+
+print(f"{'r':<5}{'C*(r)':<10}")
+print("-" * 13)
+for r in sorted(result):
+    print(f"{r:<5}{result[r]:<10}")

nlpkit14-0.3.0/src/nlpkit14/programs/q12.py

@@ -0,0 +1,50 @@
+# Question 12 (C5): Text Classification
+# Output below is the document example; actual model and website results may differ.
+# Preserves the document example; fit the vectorizer on training data only for unbiased evaluation.
+# Expected output:
+# precision    recall  f1-score   support
+# 
+#       sports       0.00      0.00      0.00         2
+#   technology       0.33      1.00      0.50         1
+# 
+#     accuracy                           0.33         3
+#    macro avg       0.17      0.50      0.25         3
+# weighted avg       0.11      0.33      0.17         3
+
+from sklearn.feature_extraction.text import CountVectorizer
+from sklearn.model_selection import train_test_split
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import classification_report
+
+texts = [
+    "The CPU and memory are important in a computer",
+    "Basketball is a popular sport",
+    "Python and Java are programming languages",
+    "Football players need strength and speed",
+    "Hard disk and RAM are types of hardware",
+    "Tennis is played with a racket",
+    "Motherboard connects all components",
+    "He scored a goal in the last minute"
+]
+
+labels = [
+    "technology", "sports", "technology", "sports",
+    "technology", "sports", "technology", "sports"
+]
+
+# Convert text into Bag-of-Words vectors
+vectorizer = CountVectorizer()
+X = vectorizer.fit_transform(texts)
+
+# Train-test split
+X_train, X_test, y_train, y_test = train_test_split(
+    X, labels, test_size=0.3, random_state=42
+)
+
+# Train classifier
+clf = LogisticRegression()
+clf.fit(X_train, y_train)
+
+# Predict and evaluate
+y_pred = clf.predict(X_test)
+print(classification_report(y_test, y_pred, zero_division=0))

nlpkit14-0.3.0/src/nlpkit14/programs/q13.py

@@ -0,0 +1,25 @@
+# Question 13 (C6): Calculate Vector Similarity in Semantic Space
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Semantic similarity between:
+# "Apples are tasty" and "Bananas are delicious" = 0.8661
+
+import spacy
+from sklearn.metrics.pairwise import cosine_similarity
+
+# Medium model contains pretrained vectors
+nlp = spacy.load("en_core_web_md")
+
+text1 = "Apples are tasty"
+text2 = "Bananas are delicious"
+
+doc1 = nlp(text1)
+doc2 = nlp(text2)
+
+vec1 = doc1.vector.reshape(1, -1)
+vec2 = doc2.vector.reshape(1, -1)
+
+similarity = cosine_similarity(vec1, vec2)[0][0]
+
+print(f'Semantic similarity between:')
+print(f'"{text1}" and "{text2}" = {similarity:.4f}')

nlpkit14-0.3.0/src/nlpkit14/programs/q14.py

@@ -0,0 +1,40 @@
+# Question 14 (C7): Identify Context of Ambiguous Words using BERT
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Sentence 1: He sat on the river bank and watched the water flow.
+# Sentence 2: She went to the bank to deposit some money.
+# Contextual similarity of 'bank': 0.4972
+
+from transformers import BertTokenizer, BertModel
+import torch
+from sklearn.metrics.pairwise import cosine_similarity
+
+# Load pretrained BERT
+tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', local_files_only=True)
+model = BertModel.from_pretrained('bert-base-uncased', local_files_only=True)
+model.eval()
+
+sent1 = "He sat on the river bank and watched the water flow."
+sent2 = "She went to the bank to deposit some money."
+
+def get_word_embedding(sentence, target_word):
+    inputs = tokenizer(sentence, return_tensors='pt')
+
+    with torch.no_grad():
+        outputs = model(**inputs)
+
+    tokens = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])
+    embeddings = outputs.last_hidden_state.squeeze(0)
+
+    # Find token position
+    idx = tokens.index(target_word)
+    return embeddings[idx].numpy()
+
+embedding1 = get_word_embedding(sent1, "bank")
+embedding2 = get_word_embedding(sent2, "bank")
+
+similarity = cosine_similarity([embedding1], [embedding2])[0][0]
+
+print("Sentence 1:", sent1)
+print("Sentence 2:", sent2)
+print("Contextual similarity of 'bank':", round(similarity, 4))

nlpkit14-0.3.0/src/nlpkit14/programs/q2.py

@@ -0,0 +1,33 @@
+# Question 2 (B2): Tokenization and Stop-word Removal using spaCy
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Sentences: ['This is an example sentence, demonstrating tokenization and stop word removal.']
+# Tokens: ['This', 'is', 'an', 'example', 'sentence', ',', 'demonstrating', 'tokenization', 'and', 'stop', 'word', 'removal', '.']
+# Filtered Tokens: ['example', 'sentence', 'demonstrating', 'tokenization', 'stop', 'word', 'removal']
+
+import spacy
+
+# Load spaCy English model
+nlp = spacy.load("en_core_web_sm")
+
+# Sample text
+text = "This is an example sentence, demonstrating tokenization and stop word removal."
+
+# Process text
+Doc = nlp(text)
+
+# Sentence tokenization
+sentences = [sent.text for sent in Doc.sents]
+
+# Word tokenization
+tokens = [token.text for token in Doc]
+
+# Remove stop words and punctuation
+filtered_tokens = [
+    token.text for token in Doc
+    if not token.is_stop and not token.is_punct
+]
+
+print("Sentences:", sentences)
+print("Tokens:", tokens)
+print("Filtered Tokens:", filtered_tokens)

nlpkit14-0.3.0/src/nlpkit14/programs/q3.py

@@ -0,0 +1,29 @@
+# Question 3 (B3): Stemming and Lemmatization using spaCy
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Original Words   : ['Studies', 'studying', 'cries', 'cried', 'flying', 'flies']
+# Lemmatized Words : ['study', 'study', 'cry', 'cry', 'fly', 'fly']
+# Stemmed Words    : ['studi', 'studi', 'cri', 'cri', 'fli', 'fli']
+
+import spacy
+from nltk.stem import PorterStemmer
+
+# Load spaCy model and create stemmer
+nlp = spacy.load("en_core_web_sm")
+stemmer = PorterStemmer()
+
+sentence = "Studies studying cries cried flying flies"
+doc = nlp(sentence)
+
+# Original tokens
+original_words = [token.text for token in doc]
+
+# Lemmatization using spaCy
+lemmatized_words = [token.lemma_ for token in doc]
+
+# Stemming on spaCy tokens
+stemmed_words = [stemmer.stem(token.text) for token in doc]
+
+print("Original Words   :", original_words)
+print("Lemmatized Words :", lemmatized_words)
+print("Stemmed Words    :", stemmed_words)

nlpkit14-0.3.0/src/nlpkit14/programs/q4.py

@@ -0,0 +1,44 @@
+# Question 4 (B4): Stemming and Lemmatization using NLTK
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Original Words: ['Studies', 'studying', 'cries', 'cried', 'flying', 'flies']
+# Stemmed Words : ['studi', 'studi', 'cri', 'cri', 'fli', 'fli']
+# Lemmatized Words: ['Studies', 'study', 'cry', 'cry', 'fly', 'fly']
+
+import nltk
+from nltk.tokenize import word_tokenize
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+from nltk.corpus import wordnet
+
+# Resource download moved to n.setup().
+# Resource download moved to n.setup().
+# Resource download moved to n.setup().
+# Resource download moved to n.setup().
+stemmer = PorterStemmer()
+lemmatizer = WordNetLemmatizer()
+
+# Map Treebank POS tag to WordNet POS
+def get_wordnet_pos(tag):
+    if tag.startswith('J'):
+        return wordnet.ADJ
+    elif tag.startswith('V'):
+        return wordnet.VERB
+    elif tag.startswith('N'):
+        return wordnet.NOUN
+    elif tag.startswith('R'):
+        return wordnet.ADV
+    return wordnet.NOUN
+
+text = "Studies studying cries cried flying flies"
+tokens = word_tokenize(text)
+tagged = nltk.pos_tag(tokens)
+
+stems = [stemmer.stem(word) for word in tokens]
+lemmas = [
+    lemmatizer.lemmatize(word, get_wordnet_pos(tag))
+    for word, tag in tagged
+]
+
+print("Original Words:", tokens)
+print("Stemmed Words :", stems)
+print("Lemmatized Words:", lemmas)

nlpkit14-0.3.0/src/nlpkit14/programs/q5.py

@@ -0,0 +1,33 @@
+# Question 5 (B5): Design a Python Program to Illustrate Corpus
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# --- Document 1 ---
+# Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']
+# POS Tags: [('The', 'DET'), ('quick', 'ADJ'), ('brown', 'ADJ'), ('fox', 'NOUN'), ('jumps', 'VERB'), ('over', 'ADP'), ('the', 'DET'), ('lazy', 'ADJ'), ('dog', 'NOUN'), ('.', 'PUNCT')]
+# 
+# --- Document 2 ---
+# Tokens: ['SpaCy', 'is', 'a', 'popular', 'NLP', 'library', 'in', 'Python', '.']
+# 
+# --- Document 3 ---
+# Tokens: ['Natural', 'Language', 'Processing', 'enables', 'machines', 'to', 'understand', 'human', 'language', '.']
+
+import spacy
+
+nlp = spacy.load("en_core_web_sm")
+
+# A small corpus: collection of documents
+corpus = [
+    "The quick brown fox jumps over the lazy dog.",
+    "SpaCy is a popular NLP library in Python.",
+    "Natural Language Processing enables machines to understand human language."
+]
+
+for i, text in enumerate(corpus, start=1):
+    print(f"\n--- Document {i} ---")
+    doc = nlp(text)
+
+    tokens = [token.text for token in doc]
+    pos_tags = [(token.text, token.pos_) for token in doc]
+
+    print("Tokens:", tokens)
+    print("POS Tags:", pos_tags)

nlpkit14-0.3.0/src/nlpkit14/programs/q6.py

@@ -0,0 +1,43 @@
+# Question 6 (B6): Generate a Constituency Parse Tree
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Parse String:
+# (S (NP (DT The) (JJ quick) (JJ brown) (NN fox))
+#    (VP (VBZ jumps)
+#        (PP (IN over) (NP (DT the) (JJ lazy) (NN dog))))
+#    (. .))
+# 
+# Pretty Tree:
+#                       S
+#        _______________|__________________________
+#       |                         VP               |
+#       |                _________|___             |
+#       NP              |             PP           |
+#   ____|__________     |     ________|___         |
+#  DT   JJ    JJ   NN  VBZ   IN   DT      JJ   NN  .
+#  The quick brown fox jumps over the     lazy dog  .
+
+# Run once if required:
+# !pip install spacy benepar
+# !python -m spacy download en_core_web_sm
+
+import spacy
+import benepar
+import nltk
+
+# Model download moved to n.setup(models=True).
+nlp = spacy.load("en_core_web_sm")
+
+if not nlp.has_pipe("benepar"):
+    nlp.add_pipe("benepar", config={"model": "benepar_en3"})
+
+sentence = "The quick brown fox jumps over the lazy dog."
+doc = nlp(sentence)
+
+for sent in doc.sents:
+    parse_string = sent._.parse_string
+    print("Parse String:\n", parse_string)
+
+    tree = nltk.Tree.fromstring(parse_string)
+    print("\nPretty Tree:")
+    tree.pretty_print()

nlpkit14-0.3.0/src/nlpkit14/programs/q7.py

@@ -0,0 +1,31 @@
+# Question 7 (B7): Generate a Dependency Parse Tree
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# Word: The      POS: DET    Head: team     Dep: det
+# Word: cricket  POS: NOUN   Head: team     Dep: compound
+# Word: team     POS: NOUN   Head: won      Dep: nsubj
+# Word: won      POS: VERB   Head: won      Dep: ROOT
+# Word: the      POS: DET    Head: match    Dep: det
+# Word: match    POS: NOUN   Head: won      Dep: dobj
+# Word: in       POS: ADP    Head: won      Dep: prep
+# Word: Mumbai   POS: PROPN  Head: in       Dep: pobj
+# Word: .        POS: PUNCT  Head: won      Dep: punct
+
+import spacy
+from spacy import displacy
+
+nlp = spacy.load("en_core_web_sm")
+sentence = "The cricket team won the match in Mumbai."
+doc = nlp(sentence)
+
+# Print dependency information
+for token in doc:
+    print(
+        f"Word: {token.text:8} "
+        f"POS: {token.pos_:6} "
+        f"Head: {token.head.text:8} "
+        f"Dep: {token.dep_}"
+    )
+
+# In Jupyter / Colab
+# displacy.render(doc, style="dep", jupyter=True)

nlpkit14-0.3.0/src/nlpkit14/programs/q8.py

@@ -0,0 +1,39 @@
+# Question 8 (C1): POS Tagging on Text Scraped from a Website
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# POS Tags (sample):
+# [('B.N.M', 'NNP'), ('Institute', 'NNP'), ('of', 'IN'),
+#  ('Technology', 'NNP'), ('is', 'VBZ'), ('one', 'CD'),
+#  ('of', 'IN'), ('the', 'DT'), ('top', 'JJ'),
+#  ('engineering', 'NN'), ('colleges', 'NNS'), ...]
+
+import requests
+from bs4 import BeautifulSoup
+import nltk
+
+# Resource download moved to n.setup().
+# Resource download moved to n.setup().
+def scrape_text(url):
+    response = requests.get(url, timeout=10)
+    response.raise_for_status()
+    soup = BeautifulSoup(response.content, 'html.parser')
+
+    # Remove scripts and styles
+    for item in soup(['script', 'style']):
+        item.extract()
+
+    return soup.get_text(separator=' ')
+
+def pos_tag_text(text):
+    tokens = nltk.word_tokenize(text)
+    return nltk.pos_tag(tokens)
+
+url = "https://www.bnmit.org/"
+scraped_text = scrape_text(url)
+pos_tags = pos_tag_text(scraped_text)
+
+print("Scraped Text (first 300 characters):")
+print(scraped_text[:300])
+
+print("\nPOS Tags (first 30 tokens):")
+print(pos_tags[:30])

nlpkit14-0.3.0/src/nlpkit14/programs/q9.py

@@ -0,0 +1,36 @@
+# Question 9 (C2): Implement a Bigram Model and Calculate Test-Sentence Probability
+# Output below is the document example; actual model and website results may differ.
+# Expected output:
+# P(i|<s>) = 2/3 = 0.666667
+# P(like|i) = 2/2 = 1.000000
+# P(apples|like) = 1/2 = 0.500000
+# P(</s>|apples) = 2/2 = 1.000000
+# Sentence Probability = 0.33333333
+
+from collections import Counter
+
+# Training corpus
+corpus = [
+    ['i', 'like', 'apples'],
+    ['i', 'like', 'bananas'],
+    ['you', 'eat', 'apples']
+]
+
+unigrams = Counter()
+bigrams = Counter()
+
+for sentence in corpus:
+    tokens = ['<s>'] + sentence + ['</s>']
+    unigrams.update(tokens)
+    bigrams.update(zip(tokens[:-1], tokens[1:]))
+
+# Test sentence
+test = ['<s>', 'i', 'like', 'apples', '</s>']
+probability = 1.0
+
+for w1, w2 in zip(test[:-1], test[1:]):
+    p = bigrams[(w1, w2)] / unigrams[w1]
+    probability *= p
+    print(f"P({w2}|{w1}) = {bigrams[(w1,w2)]}/{unigrams[w1]} = {p:.6f}")
+
+print("Sentence Probability =", round(probability, 8))

nlpkit14-0.3.0/src/nlpkit14/questions.json

@@ -0,0 +1,72 @@
+[
+  {
+    "number": 1,
+    "label": "B1",
+    "question": "Tokenization and Stop-word Removal using NLTK"
+  },
+  {
+    "number": 2,
+    "label": "B2",
+    "question": "Tokenization and Stop-word Removal using spaCy"
+  },
+  {
+    "number": 3,
+    "label": "B3",
+    "question": "Stemming and Lemmatization using spaCy"
+  },
+  {
+    "number": 4,
+    "label": "B4",
+    "question": "Stemming and Lemmatization using NLTK"
+  },
+  {
+    "number": 5,
+    "label": "B5",
+    "question": "Design a Python Program to Illustrate Corpus"
+  },
+  {
+    "number": 6,
+    "label": "B6",
+    "question": "Generate a Constituency Parse Tree"
+  },
+  {
+    "number": 7,
+    "label": "B7",
+    "question": "Generate a Dependency Parse Tree"
+  },
+  {
+    "number": 8,
+    "label": "C1",
+    "question": "POS Tagging on Text Scraped from a Website"
+  },
+  {
+    "number": 9,
+    "label": "C2",
+    "question": "Implement a Bigram Model and Calculate Test-Sentence Probability"
+  },
+  {
+    "number": 10,
+    "label": "C3",
+    "question": "Add-One (Laplace) Smoothing for an N-gram Model"
+  },
+  {
+    "number": 11,
+    "label": "C4",
+    "question": "Calculate Good-Turing Frequency"
+  },
+  {
+    "number": 12,
+    "label": "C5",
+    "question": "Text Classification"
+  },
+  {
+    "number": 13,
+    "label": "C6",
+    "question": "Calculate Vector Similarity in Semantic Space"
+  },
+  {
+    "number": 14,
+    "label": "C7",
+    "question": "Identify Context of Ambiguous Words using BERT"
+  }
+]

nlpkit14-0.3.0/src/nlpkit14.egg-info/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.4
+Name: nlpkit14
+Version: 0.3.0
+Summary: Execute fourteen NLP practical programs with commented source and expected outputs
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: nltk<4,>=3.9
+Requires-Dist: spacy<4,>=3.8
+Requires-Dist: scikit-learn<2,>=1.5
+Requires-Dist: requests<3,>=2.32
+Requires-Dist: beautifulsoup4<5,>=4.12
+Provides-Extra: models
+Requires-Dist: torch<3,>=2.4; extra == "models"
+Requires-Dist: transformers<5,>=4.45; extra == "models"
+Requires-Dist: benepar<0.3,>=0.2.0; extra == "models"
+Requires-Dist: sentencepiece<1,>=0.2; extra == "models"
+
+# nlpkit14 0.3.0
+
+Run the fourteen programs from the supplied IA1 question/code/output document.
+
+```python
+import nlpkit14 as n
+n.q1()       # Actually runs question 1 and prints computed output
+n.source(1)  # Shows the code, question and expected output as comments
+n.questions()
+```
+
+Question numbers 1–7 are B1–B7. Numbers 8–14 are C1–C7. The question, expected output, and extra notes are comments in each bundled standalone Python program. Exercise calls print only the program's output. No expected-output text is substituted for model computation.
+
+## Install and prepare online once
+
+Install the shared wheel, then prepare the basic data and spaCy model:
+
+```sh
+python -m pip install ./nlpkit14-0.3.0-py3-none-any.whl
+python -c "import nlpkit14 as n; n.setup()"
+```
+
+For all exercises, install the model extra and run the complete setup (large downloads):
+
+```sh
+python -m pip install './nlpkit14-0.3.0-py3-none-any.whl[models]'
+python -c "import nlpkit14 as n; n.setup(models=True)"
+```
+
+This is a local release of the new nlpkit14 project, not yet uploaded to PyPI. Do not use pip install nlpkit14 until publication succeeds.
+
+## Offline installation
+
+On an internet-connected machine with the SAME OS, architecture and Python version as the offline machine, download the wheel and dependencies:
+
+```sh
+python -m pip download --dest wheelhouse './nlpkit14-0.3.0-py3-none-any.whl[models]'
+```
+
+Install and run setup online on that matching machine. Copy the wheelhouse, both installed spaCy model packages (or their installation wheels), and the full data/cache folder to the offline machine. Model data defaults to ~/.nlpkit14; set NLPKIT14_DATA before setup/import to choose a portable folder. Copying the entire prepared Python environment is platform specific; do not copy a Mac environment to Windows.
+
+On the offline computer:
+
+```sh
+python -m pip install --no-index --find-links=wheelhouse nlpkit14==0.3.0
+```
+
+Install the spaCy model wheels from local storage too. Keep the copied data folder at ~/.nlpkit14 or point NLPKIT14_DATA to it. Model and dependency archives are not included in this small library wheel.
+
+q8 fetches https://www.bnmit.org/ and requires internet even after setup. Other exercises can run without internet after all their requirements are installed. No downloads run implicitly from q1–q14. Missing packages/data raise their real installation errors rather than producing fake results.
+
+## Output fidelity
+
+The original algorithms and sample input are preserved. Downloads are moved to explicit setup. HTTP errors are checked. BERT uses evaluation mode and local cached model files. The raw Good–Turing demonstration is not a normalized probability estimator. The document's classifier fits its vocabulary before splitting and is an educational example, not an unbiased evaluation pipeline. Model outputs, parse trees, POS tags and similarity values can differ from the commented reference output. The document's B5 reference omits POS lines that the code actually prints. Read TEST_RESULTS.md for exactly which programs were executed.

nlpkit14-0.3.0/src/nlpkit14.egg-info/SOURCES.txt

@@ -0,0 +1,26 @@
+MANIFEST.in
+README.md
+TEST_RESULTS.md
+pyproject.toml
+src/nlpkit14/__init__.py
+src/nlpkit14/questions.json
+src/nlpkit14.egg-info/PKG-INFO
+src/nlpkit14.egg-info/SOURCES.txt
+src/nlpkit14.egg-info/dependency_links.txt
+src/nlpkit14.egg-info/requires.txt
+src/nlpkit14.egg-info/top_level.txt
+src/nlpkit14/programs/q1.py
+src/nlpkit14/programs/q10.py
+src/nlpkit14/programs/q11.py
+src/nlpkit14/programs/q12.py
+src/nlpkit14/programs/q13.py
+src/nlpkit14/programs/q14.py
+src/nlpkit14/programs/q2.py
+src/nlpkit14/programs/q3.py
+src/nlpkit14/programs/q4.py
+src/nlpkit14/programs/q5.py
+src/nlpkit14/programs/q6.py
+src/nlpkit14/programs/q7.py
+src/nlpkit14/programs/q8.py
+src/nlpkit14/programs/q9.py
+tests/test_programs.py

nlpkit14-0.3.0/src/nlpkit14.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

nlpkit14-0.3.0/src/nlpkit14.egg-info/requires.txt

@@ -0,0 +1,11 @@
+nltk<4,>=3.9
+spacy<4,>=3.8
+scikit-learn<2,>=1.5
+requests<3,>=2.32
+beautifulsoup4<5,>=4.12
+
+[models]
+torch<3,>=2.4
+transformers<5,>=4.45
+benepar<0.3,>=0.2.0
+sentencepiece<1,>=0.2

nlpkit14-0.3.0/src/nlpkit14.egg-info/top_level.txt

@@ -0,0 +1 @@
+nlpkit14

nlpkit14-0.3.0/tests/test_programs.py

@@ -0,0 +1,87 @@
+import contextlib
+import io
+import unittest
+from zipfile import BadZipFile
+from importlib.resources import files
+from unittest.mock import patch
+
+import nlpkit14 as n
+
+
+class ProgramTests(unittest.TestCase):
+    def require_nltk(self, *resources):
+        import nltk
+        n._configure()
+        for resource in resources:
+            try:
+                nltk.data.find(resource)
+            except (LookupError, BadZipFile):
+                self.skipTest(f'Install test data: {resource}')
+
+    def capture(self, number):
+        output = io.StringIO()
+        with contextlib.redirect_stdout(output):
+            getattr(n, f'q{number}')()
+        return output.getvalue()
+
+    def test_all_sources_compile_and_have_comments(self):
+        for n in range(1, 15):
+            source = files('nlpkit14').joinpath('programs', f'q{n}.py').read_text()
+            self.assertTrue(source.startswith(f'# Question {n} '))
+            self.assertIn('# Expected output:', source)
+            compile(source, f'q{n}.py', 'exec')
+
+    def test_q9_executes_bigram(self):
+        self.assertIn('Sentence Probability = 0.33333333', self.capture(9))
+
+    def test_q1_executes_tokenization(self):
+        self.require_nltk('tokenizers/punkt_tab/english/', 'corpora/stopwords')
+        output = self.capture(1)
+        self.assertIn("Filtered Tokens: ['simple', 'example', 'demonstrating'", output)
+        self.assertNotIn('Expected output', output)
+
+    def test_q4_executes_stemming(self):
+        self.require_nltk('tokenizers/punkt_tab/english/', 'taggers/averaged_perceptron_tagger_eng/', 'corpora/wordnet.zip')
+        self.assertIn("['studi', 'studi', 'cri', 'cri', 'fli', 'fli']", self.capture(4))
+
+    def test_q10_executes_laplace(self):
+        output = self.capture(10)
+        self.assertIn('Vocabulary Size = 8', output)
+        self.assertIn('Add-One Sentence Probability = 0.00490909', output)
+
+    def test_q11_executes_raw_good_turing(self):
+        self.assertIn('1.3333', self.capture(11))
+
+    def test_q12_trains_classifier(self):
+        self.assertIn('accuracy', self.capture(12))
+
+    def test_source_does_not_execute(self):
+        output = io.StringIO()
+        with contextlib.redirect_stdout(output):
+            n.source(14)
+        self.assertIn('# Question 14 ', output.getvalue())
+
+    def test_all_questions_listed(self):
+        output = io.StringIO()
+        with contextlib.redirect_stdout(output):
+            n.questions()
+        self.assertEqual(len(output.getvalue().splitlines()), 14)
+
+    def test_offline_numeric_exercises(self):
+        with patch('socket.socket.connect', side_effect=AssertionError('Network forbidden')):
+            for n in (9, 10, 11, 12):
+                self.capture(n)
+
+    def test_q8_tags_supplied_html_without_live_website(self):
+        self.require_nltk('tokenizers/punkt_tab/english/', 'taggers/averaged_perceptron_tagger_eng/')
+        import requests
+        class Response:
+            content = b'<p>Cats chase mice.</p>'
+            def raise_for_status(self):
+                pass
+        with patch.object(requests, 'get', return_value=Response()):
+            self.assertIn('POS Tags', self.capture(8))
+
+
+if __name__ == '__main__':
+    unittest.main()