nltkor 1.2.18__tar.gz → 1.2.20__tar.gz

{nltkor-1.2.18 → nltkor-1.2.20}/PKG-INFO

@@ -1,11 +1,8 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: nltkor
-Version: 1.2.18
-Summary: UNKNOWN
+Version: 1.2.20
 Home-page: https://modi.changwon.ac.kr/air_cwnu/nlp_tool/nltk_ko.git
-License: UNKNOWN
 Keywords: string matching,pattern matching,edit distance,string to string correction,string to string matching,Levenshtein edit distance,Hamming distance,Damerau-Levenshtein distance,Jaro-Winkler distance,longest common subsequence,longest common substring,dynamic programming,approximate string matching,semantic similarity,natural language processing,NLP,information retrieval,rouge,sacrebleu,bertscore,bartscore,fasttext,glove,cosine similarity,Smith-Waterman,Needleman-Wunsch,Hirschberg,Karp-Rabin,Knuth-Morris-Pratt,Boyer-Moore
-Platform: UNKNOWN
 Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
@@ -15,6 +12,31 @@ Classifier: Operating System :: OS Independent
 Classifier: Typing :: Typed
 Requires-Python: >=3.7
 License-File: LICENSE.txt
-
-UNKNOWN
-
+Requires-Dist: Cython
+Requires-Dist: numpy<=1.26.4,>=1.23.5
+Requires-Dist: regex
+Requires-Dist: tqdm>=4.40.0
+Requires-Dist: joblib
+Requires-Dist: requests
+Requires-Dist: nltk>3.0
+Requires-Dist: pyarrow
+Requires-Dist: beautifulSoup4
+Requires-Dist: faiss-cpu==1.7.3
+Requires-Dist: datasets
+Requires-Dist: torch
+Requires-Dist: dill<0.3.9
+Requires-Dist: scikit-learn>=0.22.1
+Requires-Dist: transformers==4.42.2
+Requires-Dist: protobuf
+Requires-Dist: sentencepiece
+Requires-Dist: pandas
+Requires-Dist: bert_score
+Requires-Dist: chardet
+Requires-Dist: GPUtil
+Requires-Dist: fasttext
+Dynamic: classifier
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python

{nltkor-1.2.18 → nltkor-1.2.20}/README.md

@@ -110,18 +110,13 @@ NLTKor는 한국어를 위한 NLTK이며 기존의 영어에서 사용하는 Wor
 
 ## 2. 사용 환경
 
-- 운영체제 : ubuntu 18.04, ubuntu 22.04, MacOS
+- 운영체제 : ubuntu 18.04, ubuntu 22.04, MacOS, Windows
 - 언어 : `python3.8`, `python3.9`, `python3.10`, `python3.11`
 - 라이브러리 : nltk>=1.1.3, numpy==1.23, faiss-cpu=1.7.3 **※ 해당 NLTKor는 영어 NLTK를 별도로 인스톨해야 함.**
 
 **주의사항**
 
-- Espresso5의 EspressoTagger의 사용가능 환경은 다음과 같다.
-
-| OS     | python                                    | 아키텍처      |
-| ------ | ----------------------------------------- | ------------- |
-| Mac    | python3.8                                 | arm64         |
-| ubuntu | python3.8 python3.9 python3.10 python3.11 | arm64, x86_64 |
+- Windows 환경에서 python 3.9~3.11을 사용할 경우 fasttext 라이브러리가 지원되지 않습니다. 대신 fasttext-wheel  라이브러리를 사용해야 합니다(pip install fasttext-wheel).
 
 ### 2.1 라이브러리 설치
 
@@ -1533,21 +1528,33 @@ P to Q2 : 0.1981
 ```
 
 #### 12.5 Faiss-Semantic 검색
-
-- **init**(model_name_or_path: str = 'facebook/bart-large', tokenizer_name_or_path: str = 'facebook/bart-large', device: str = 'cpu')→ None : FaissSearh를 초기화 합니다.
-- add_faiss_index(column_name: str = 'embeddings', metric_type: int | None = None, batch_size: int = 8, \*\*kwargs)→ None : FAISS index를 dataset에 추가합니다.
-- get_embeddings(text: str | List[str], embedding_type: str = 'last_hidden_state', batch_size: int = 8, num_workers: int = 4)→ Tensor : 텍스트를 임베딩합니다.
-- get_last_hidden_state(embeddings: Tensor)→ Tensor : 임베딩된 텍스트의 last hidden state를 반환합니다.
-- get_mean_pooling(embeddings: Tensor)→ Tensor : 입력 임베딩의 mean pooling을 반환합니다.
-- initialize_corpus(corpus: Dict[str, List[str]] | DataFrame | Dataset, section: str = 'text', index_column_name: str = 'embeddings', embedding_type: str = 'last_hidden_state', batch_size: int | None = None, num_workers: int | None = None, save_path: str | None = None)→ Dataset : 데이터셋을 초기화합니다.
-- load_dataset_from_json(json_path: str)→ Dataset : json 파일에서 데이터셋을 로드합니다.
-- load_faiss_index(index_name: str, file_path: str, device: str = 'cpu')→ None : FAISS index를 로드합니다.
-- save_faiss_index(index_name: str, file_path: str)→ None : 특정한 파일 경로로 FAISS index를 저장합니다.
-- search(query: str, k: int = 1, index_column_name: str = 'embeddings')→ DataFrame : 데이터셋에서 쿼리를 검색합니다.
+- class FaissSearch
+	- **__new__**(mode = None, model_name_or_path: str = 'klue/bert-base', tokneizer_name_or_path: str = 'klue/bert-base', device: str = 'cpu') -> None : mode에 따라 이용할 class로 이동합니다.
+		- mode = dense(dense | sparse)
+
+- class FaissSearch_Dense : 기존 Faiss를 이용한 검색
+	- **init**(model_name_or_path: str = 'klue/bert-base', tokenizer_name_or_path: str = 'klue/bert-base', device: str = 'cpu')→ None : FaissSearh_Dense를 초기화 합니다.
+	- add_faiss_index(column_name: str = 'embeddings', metric_type: int | None = None, batch_size: int = 8, \*\*kwargs)→ None : FAISS index를 dataset에 추가합니다.
+	- get_embeddings(text: str | List[str], embedding_type: str = 'last_hidden_state', batch_size: int = 8, num_workers: int = 4)→ Tensor : 텍스트를 임베딩합니다.
+	- get_last_hidden_state(embeddings: Tensor)→ Tensor : 임베딩된 텍스트의 last hidden state를 반환합니다.
+	- get_mean_pooling(embeddings: Tensor)→ Tensor : 입력 임베딩의 mean pooling을 반환합니다.
+	- initialize_corpus(corpus: Dict[str, List[str]] | DataFrame | Dataset, section: str = 'text', index_column_name: str = 'embeddings', embedding_type: str = 'last_hidden_state', batch_size: int | None = None, num_workers: int | None = None, save_path: str | None = None)→ Dataset : 데이터셋을 초기화합니다.
+	- load_dataset_from_json(json_path: str)→ Dataset : json 파일에서 데이터셋을 로드합니다.
+	- load_faiss_index(index_name: str, file_path: str, device: str = 'cpu')→ None : FAISS index를 로드합니다.
+	- save_faiss_index(index_name: str, file_path: str)→ None : 특정한 파일 경로로 FAISS index를 저장합니다.
+	- search(query: str, k: int = 1, index_column_name: str = 'embeddings')→ DataFrame : 데이터셋에서 쿼리를 검색합니다.
+
+- class FaissSearch_Sparse(FaissSearch_Dense) : 학습가능한 sparse representation을 이용하는 모델을 위한 faiss
+	- **init**(model_name_or_path: str = 'klue/bert-base', tokenizer_name_or_path: str = 'klue/bert-base', device: str = 'cpu') -> None : FaissSearch_Sparse를 초기화합니다.
+	- get_embeddings(text: str | List[str], embedding_type: str = 'last_hidden_state', batch_size: int = 8, num_workers: int = 4) -> Tensor : 텍스트를 임베딩합니다.
+
+<br>
+
+- mode = 'dense' : 원본 faiss를 실행합니다.
 
 ```python
->>> from nltkor.search import FaissSearch
->>> faiss = FaissSearch(model_name_or_path = 'facebook/bart-large')
+>>> from nltkor.search.faiss_search import FaissSearch
+>>> faiss = FaissSearch(model_name_or_path = 'klue/bert-base', mode='dense')
 >>> corpus = {
         'text': [
                 "오늘은 날씨가 매우 덥습니다.",
@@ -1583,29 +1590,93 @@ P to Q2 : 0.1981
         ],
     }
 >>> faiss.initialize_corpus(corpus=corpus, section='text', embedding_type='mean_pooling')
->>> query = "오늘은 날시가 매우 춥다."
+>>> query = "오늘은 날씨가 매우 춥다."
 >>> top_k = 5
 >>> result = faiss.search(query, top_k)
 >>> print(result)
-                       text                                         embeddings      score
-0          오늘은 날씨가 매우 덥습니다.  [-0.2576247453689575, 0.47791656851768494, -1....  14.051050
-1  한국 음식 중에서 떡볶이가 제일 맛있습니다.  [-0.2623925805091858, 0.46345704793930054, -1....  28.752083
-2      요즘 드라마를 많이 시청하고 있어요.  [-0.2683958113193512, 0.6801461577415466, -1.1...  29.339230
-3    다음 주에 시험이 있어서 공부해야 해요.  [-0.20012563467025757, 0.5758355855941772, -1....  31.358824
-4     피아노 연주는 나를 편안하게 해줍니다.  [-0.24231986701488495, 0.6492734551429749, -1....  34.069862
+
+
+text                                         embeddings       score
+0           오늘은 날씨가 매우 덥습니다.  [-0.06737425178289413, -0.6356450319290161, -0...   52.453941
+1  휴대폰 없이 하루를 보내는 것이 쉽지 않아요.  [0.09126424789428711, -0.011225797235965729, -...  168.310577
+2      내일은 친구와 영화를 보러 갈 거예요.  [-0.21793286502361298, -0.2237573117017746, 0....  181.051544
+3        요리를 만들면 집안이 좋아보입니다.  [0.7215852737426758, -0.426792711019516, -0.07...  203.423340
+4          스포츠를 하면 건강에 좋습니다.  [0.1290944665670395, -0.6169838905334473, -0.2...  205.527954
+
+```
+
+<br>
+
+- mode = 'sparse' : 학습가능한 sparse representation을 이용한 모델을 위한 faiss 코드를 실행합니다.
+
+```python
+>>> from nltkor.search.faiss_search import FaissSearch
+>>> model_name_or_path = 'klue/bert-base'
+>>> faiss = FaissSearch(model_name_or_path=model_name_or_path, mode='sparse')
+>>> corpus = {
+        'text': [
+                "오늘은 날씨가 매우 덥습니다.",
+                "저는 음악을 듣는 것을 좋아합니다.",
+                "한국 음식 중에서 떡볶이가 제일 맛있습니다.",
+                "도서관에서 책을 읽는 건 좋은 취미입니다.",
+                "내일은 친구와 영화를 보러 갈 거예요.",
+                "여름 휴가 때 해변에 가서 수영하고 싶어요.",
+                "한국의 문화는 다양하고 흥미로워요.",
+                "피아노 연주는 나를 편안하게 해줍니다.",
+                "공원에서 산책하면 스트레스가 풀립니다.",
+                "요즘 드라마를 많이 시청하고 있어요.",
+                "커피가 일상에서 필수입니다.",
+                "새로운 언어를 배우는 것은 어려운 일이에요.",
+                "가을에 단풍 구경을 가고 싶어요.",
+                "요리를 만들면 집안이 좋아보입니다.",
+                "휴대폰 없이 하루를 보내는 것이 쉽지 않아요.",
+                "스포츠를 하면 건강에 좋습니다.",
+                "고양이와 개 중에 어떤 동물을 좋아하세요?"
+                "천천히 걸어가면서 풍경을 감상하는 것이 좋아요.",
+                "일주일에 한 번은 가족과 모임을 가요.",
+                "공부할 때 집중력을 높이는 방법이 있을까요?",
+                "봄에 꽃들이 피어날 때가 기대되요.",
+                "여행 가방을 챙기고 싶어서 설레여요.",
+                "사진 찍는 걸 좋아하는데, 카메라가 필요해요.",
+                "다음 주에 시험이 있어서 공부해야 해요.",
+                "운동을 하면 몸이 가벼워집니다.",
+                "좋은 책을 읽으면 마음이 풍요로워져요.",
+                "새로운 음악을 발견하면 기분이 좋아져요.",
+                "미술 전시회에 가면 예술을 감상할 수 있어요.",
+                "친구들과 함께 시간을 보내는 건 즐거워요.",
+                "자전거 타면 바람을 맞으면서 즐거워집니다."
+        ],
+    }
+>>> faiss.initialize_corpus(corpus=corpus, section='text', embedding_type='last_hidden_state')
+>>> query = "오늘은 날씨가 매우 춥다."
+>>> top_k = 5
+>>> result = faiss.search(query=query, k=top_k)
+>>> print(result)
+
+text                                         embeddings     score
+0           오늘은 날씨가 매우 덥습니다.  [0.0, 0.055695388466119766, 0.0, 0.0, 0.0, 0.0...  0.130759
+1  휴대폰 없이 하루를 보내는 것이 쉽지 않아요.  [0.0, 0.06064636632800102, 0.0, 0.0, 0.0, 0.03...  0.418491
+2      내일은 친구와 영화를 보러 갈 거예요.  [0.0, 0.0474698506295681, 0.0, 0.0, 0.0, 0.039...  0.435895
+3         가을에 단풍 구경을 가고 싶어요.  [0.0, 0.05392831563949585, 0.0, 0.0, 0.0, 0.05...  0.488796
+4          스포츠를 하면 건강에 좋습니다.  [0.0, 0.05404529720544815, 0.0, 0.0, 0.0, 0.04...  0.496646
+
 ```
 
+<br>
+
+
 - faiss 검색을 매번 initialize 하지 않고, 미리 initialize 해놓은 후 검색을 수행할 수 있습니다.
 
+
 **사용법 & 결과**
 
 ```python
->>> from nltkor.search import FaissSearch
+>>> from nltkor.search.faiss_search import FaissSearch
 
 # if you use model and tokenizer in local
 # faiss = FaissSearch(model_name_or_path = '~/test_model/trained_model/', tokenizer_name_or_path = '~/test_model/trained_model/')
 
->>> faiss = FaissSearch(model_name_or_path = 'facebook/bart-large')
+>>> faiss = FaissSearch(model_name_or_path = 'klue/bert-base', mode='dense')
 >>> corpus = {
         'text': [
                 "오늘은 날씨가 매우 덥습니다.",
@@ -1646,12 +1717,12 @@ P to Q2 : 0.1981
 - `initialize_corpus()` 메소드 실행시 `save_path`를 지정하면, 해당 경로에 임베딩된 Dataset이 json형식으로 저장됩니다.
 
 ```python
->>> from nltkor.search import FaissSearch
+>>> from nltkor.search.faiss_search import FaissSearch
 
->>> faiss = FaissSearch(model_name_or_path = 'facebook/bart-large')
+>>> faiss = FaissSearch(model_name_or_path = 'klue/bert-base', mode='dense')
 >>> faiss.load_dataset_from_json('./test.json')
 >>> faiss.embedding_type = 'mean_pooling' # initalize_corpus() 메소드 실행시 지정한 embedding_type과 동일하게 지정해야 합니다.
->>> faiss,add_faiss_index(colum_name = 'embeddings')
+>>> faiss.add_faiss_index(colum_name = 'embeddings')
 >>> query = '오늘은 날씨가 매우 춥다.'
 >>> top_k = 5
 >>> result = faiss.search(query=query, top_k=top_k)
@@ -1674,27 +1745,27 @@ Adding FAISS index...
 
 ```python
 root = {}
-dict_file = '텍스트 파일 경로'
-sc = SearchDic(root)
+dict_file = '텍스트파일 경로'
+sc = TRIESearch(root)
 with open(dict_file, 'r') as f:
     for line in f:
         if ';;' in line[:2]: continue
         k, v = line.strip().split('\t')
-        sc.build_search_dict(k, v)
+        sc.build_trie_search(k, v)
 # print(root)
 word = '고용 노동부'
-values, value_data = sc.search_dict(word, True)
+values, value_data = sc.trie_search(word, True)
 print(values, value_data)
 
 word = '2시뉴스외전'
-values, value_data = sc.search_dict(word, True)
+values, value_data = sc.trie_search( word, True)
 print(values, value_data)
 word = '2시 뉴스외전'
-values, value_data = sc.search_dict(word, True)
+values, value_data = sc.trie_search( word, True)
 print(values, value_data)
 
 word = 'gbc'
-values, value_data = sc.search_dict(word, True)
+values, value_data = sc.trie_search( word, True)
 print(values, value_data)
 ```
 **결과**

{nltkor-1.2.18 → nltkor-1.2.20}/nltkor/__init__.py

@@ -13,4 +13,4 @@ from nltkor import trans
 from nltkor import Kor_char
 from nltkor import etc
 
-__version__ = '1.2.18'
+__version__ = '1.2.20'

{nltkor-1.2.18 → nltkor-1.2.20}/nltkor/search/__init__.py

@@ -8,4 +8,4 @@ from .classical import (
 )
 from .faiss_search import FaissSearch
 from .kobert_tokenizer import KoBERTTokenizer
-from .search_dict import SearchDic
+from .trie_search import TRIESearch

{nltkor-1.2.18 → nltkor-1.2.20}/nltkor/search/faiss_search.py

@@ -68,24 +68,22 @@ except ImportError:
 
 class FaissSearch:
     def __new__(cls,
-            mode = None,
+            mode = 'dense',
             model_name_or_path: str = 'klue/bert-base',
             tokenizer_name_or_path: str = 'klue/bert-base',
             embedding_type: str = 'last_hidden_state',
             device: str = 'cpu'
             ) -> None:
-        if mode == 'sentence':
-            return FaissSearch_SenEmbed(model_name_or_path=model_name_or_path, embedding_type=embedding_type)
-        elif mode == 'word':
-            return FaissSearch_WordEmbed(model_name_or_path=model_name_or_path, embedding_type=embedding_type)
+        if mode == 'dense':
+            return FaissSearch_Dense(model_name_or_path=model_name_or_path, embedding_type=embedding_type)
         elif mode == 'sparse':
             return FaissSearch_Sparse(model_name_or_path=model_name_or_path, embedding_type=embedding_type)
         else:
-            raise ValueError("choice 'sentence' or 'word' or 'sparse'")
+            raise ValueError("choice 'dense' or 'sparse'.")
 
 
 
-class FaissSearch_SenEmbed:
+class FaissSearch_Dense:
     def __init__(self,
         model_name_or_path: str = 'klue/bert-base',
         tokenizer_name_or_path: str = 'klue/bert-base',
@@ -474,7 +472,7 @@ class FaissSearch_SenEmbed:
 
 
 
-class FaissSearch_Sparse(FaissSearch_SenEmbed):
+class FaissSearch_Sparse(FaissSearch_Dense):
     def __init__(self,
         model_name_or_path: str = 'klue/bert-base',
         tokenizer_name_or_path: str = 'klue/bert-base',
@@ -586,312 +584,7 @@ class FaissSearch_Sparse(FaissSearch_SenEmbed):
         embeddings = embeddings['logits']
         
         embeddings = torch.sum(torch.log(1+torch.relu(embeddings)) * encoded_text['attention_mask'].unsqueeze(-1), dim=1)
-        e_norm = torch.nn.functional.normalize(embeddings, p=2, dim=1, eps=1e-8)
         
         # Return the embeddings
-        return e_norm
-
-
-
-# FAISS word embedding library wrapper class
-class FaissSearch_WordEmbed(FaissSearch_SenEmbed):
-    def __init__(self,
-        model_name_or_path: str = 'klue/bert-base',
-        tokenizer_name_or_path: str = 'klue/bert-base',
-        embedding_type: str = 'last_hidden_state',
-        device: str = 'cpu',
-        ) -> None:
-        r"""
-        This function initializes the wrapper for the FAISS library, which is used to perform semantic search.
-
-
-        .. attention::
-
-            * If you use this class, please make sure to cite the following paper:
-
-                .. code-block:: latex
-
-                    @article{johnson2019billion,
-                        title={Billion-scale similarity search with {GPUs}},
-                        author={Johnson, Jeff and Douze, Matthijs and J{\'e}gou, Herv{\'e}},
-                        journal={IEEE Transactions on Big Data},
-                        volume={7},
-                        number={3},
-                        pages={535--547},
-                        year={2019},
-                        publisher={IEEE}
-                    }
-
-            * The code is based on the following GitHub repository:
-                https://github.com/facebookresearch/faiss
-
-        Arguments:
-            model_name_or_path (str, optional): The name or path of the model to use. Defaults to 'facebook/bart-large'.
-            tokenizer_name_or_path (str, optional): The name or path of the tokenizer to use. Defaults to 'facebook/bart-large'.
-            device (str, optional): The device to use. Defaults to 'cpu'.
-
-        Returns:
-            None
-        """
-
-        # Set the device
-        self.device = device
-
-        # If the tokenizer is not specified, use the model name or path
-        if tokenizer_name_or_path is None:
-            tokenizer_name_or_path = model_name_or_path
-        
-        # Load the tokenizer
-        if tokenizer_name_or_path == 'skt/kobert-base-v1':
-            # self.tokenizer = KoBERTTokenizer.from_pretrained(tokenizer_name_or_path)
-            self.tokenizer = XLNetTokenizer.from_pretrained(tokenizer_name_or_path)
-        else:
-            self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
-
-        # Load the model
-        self.model = AutoModel.from_pretrained(model_name_or_path).to(self.device)
-
-
-        # Set the model to evaluation mode (since we do not need the gradients)
-        self.model.eval()
-
-        # Initialize the dataset
-        self.dataset = None
-
-
-    # Get the embeddings (new code)
-    def get_doc_embeddings(self,
-        #text: Union[str, List[str]],
-        text=None,
-        embedding_type: str = 'last_hidden_state',
-        batch_size: int = 8,
-        num_workers: int = 4,
-    ) -> torch.Tensor:
-        """
-        This function returns the embeddings of the input text.
-
-        Arguments:
-            text (Union[str, List[str]]): The input text.
-            embedding_type (str, optional): The type of embedding to use. Defaults to 'last_hidden_state'.
-            batch_size (int, optional): The batch size to use. Defaults to 8.
-            num_workers (int, optional): The number of workers to use. Defaults to 4.
-
-        Returns:
-            torch.Tensor: The embeddings.
-
-        Raises:
-            ValueError: If the embedding type is invalid.
-        """
-        
-        # Check if the embedding type is valid
-        if embedding_type not in ['last_hidden_state', 'mean_pooling']:
-            raise ValueError(f'Invalid embedding type: {embedding_type}. Only "last_hidden_state" and "mean_pooling" are supported.')
-
-        ids_dict = {}
-        # Tokenize the input text
-        for sentence in text['text']:
-            encoded_text = self.tokenizer(
-                sentence,
-                padding=False,
-                truncation=True,
-                return_tensors='pt',
-                add_special_tokens=False
-            )
-            # Move the input text to the device
-            encoded_text = encoded_text.to(self.device)
-            token_ids_list = encoded_text['input_ids'].tolist()
-            token_ids_list = token_ids_list[0]
-            for ids in token_ids_list:
-                if ids not in ids_dict.keys():
-                    ids_dict[ids] = [sentence]
-                else:
-                    if text not in ids_dict[ids]:
-                        ids_dict[ids].append(sentence)
-        # Get the embeddings
-        embedding_dict = {}
-        self.model.eval()
-        for key, value in ids_dict.items():
-            embed = self.model(torch.tensor([[key]]), output_hidden_states=True).hidden_states[-1][:,0,:].detach()
-            embedding_dict[embed] = value
-        
-        # Return the embeddings
-        return embedding_dict
-
-
-    # Get the embeddings (new code)
-    def get_query_embeddings(self,
-        text: Union[str, List[str]],
-        embedding_type: str = 'last_hidden_state',
-        batch_size: int = 8,
-        num_workers: int = 4,
-    ) -> torch.Tensor:
-        """
-        This function returns the embeddings of the input text.
-
-        Arguments:
-            text (Union[str, List[str]]): The input text.
-            embedding_type (str, optional): The type of embedding to use. Defaults to 'last_hidden_state'.
-            batch_size (int, optional): The batch size to use. Defaults to 8.
-            num_workers (int, optional): The number of workers to use. Defaults to 4.
-
-        Returns:
-            torch.Tensor: The embeddings.
-
-        Raises:
-            ValueError: If the embedding type is invalid.
-        """
-
-        # Check if the embedding type is valid
-        if embedding_type not in ['last_hidden_state', 'mean_pooling']:
-            raise ValueError(f'Invalid embedding type: {embedding_type}. Only "last_hidden_state" and "mean_pooling" are supported.')
-
-        # Tokenize the input text
-        encoded_text = self.tokenizer(
-            text,
-            padding=False,
-            truncation=True,
-            return_tensors='pt',
-            add_special_tokens=False,
-        )
-        
-        # Move the input text to the device
-        encoded_text = encoded_text.to(self.device)
-
-        token_ids_list = encoded_text['input_ids'].tolist()
-        token_ids_list = token_ids_list[0]
-        tensor_list = [torch.tensor([[value]]) for value in token_ids_list]
-        
-        # Get the embeddings
-        embeds = []
-        self.model.eval()
-        for index, tensor in enumerate(tensor_list):
-            embed = self.model(tensor, output_hidden_states=True).hidden_states[-1][:,0,:].detach().cpu().numpy()
-            embeds.append(embed)
-
-        # Return the embeddings
-        return embeds
-
-    
-    # Initialize the corpus using a dictionary or pandas DataFrame or HuggingFace Datasets object
-    def initialize_corpus(self,
-        corpus: Union[Dict[str, List[str]], pd.DataFrame, Dataset],
-        section: str = 'text',
-        index_column_name: str = 'embeddings',
-        embedding_type: str = 'last_hidden_state',
-        batch_size: Optional[int] = None,
-        num_workers: Optional[int] = None,
-        save_path: Optional[str] = None,
-    ) -> Dataset:
-        """
-        This function initializes a dataset using a dictionary or pandas DataFrame or HuggingFace Datasets object.
-
-        Arguments:
-            dataset_dict (Dict[str, List[str]]): The dataset dictionary.
-            section (str): The section of the dataset to use whose embeddings will be used for semantic search (e.g., 'text', 'title', etc.) (default: 'text').
-            index_column_name (str): The name of the column containing the embeddings (default: 'embeddings')
-            embedding_type (str): The type of embedding to use (default: 'last_hidden_state').
-            batch_size (int, optional): The batch size to use (default: 8).
-            max_length (int, optional): The maximum length of the input sequences.
-            num_workers (int, optional): The number of workers to use.
-            save_path (Optional[str], optional): The path to save the dataset (default: None).
-
-        Returns:
-            Dataset: The dataset object (HuggingFace Datasets).
-
-        Raises:
-            ValueError: If the dataset is not a dictionary or pandas DataFrame or HuggingFace Datasets object.
-        """
-
-        # corpus = { 'text': [...] } -> form_dict
-        
-        # Set the embedding_type
-        self.embedding_type = embedding_type
-        
-        # get embedding dict
-        embedding_dict = self.get_doc_embeddings(text=corpus, embedding_type=self.embedding_type)
-
-        data = {
-                'text' : embedding_dict.values(),
-                'embeddings': []
-                }
-
-        for embed in embedding_dict.keys():
-            embed_list = embed.tolist()
-            data['embeddings'].append(embed_list[0])
-
-        
-        if isinstance(data, dict):
-            self.dataset = Dataset.from_dict(data)
-        elif isinstance(data, pd.DataFrame):
-            self.dataset = Dataset.from_pandas(data)
-        elif isinstance(data, Dataset):
-            self.dataset = corpus
-        else:
-            raise ValueError('The dataset must be a dictionary or pandas DataFrame.')
-        
-        # Save the dataset
-        if save_path is not None:
-            self.dataset.to_json(save_path)
-        
-        # Add FAISS index
-        self.add_faiss_index(
-            column_name=index_column_name,
-        )
-
-        # Return the dataset
-        return self.dataset
-
-
-    # Search for the most similar elements in the dataset, given a query
-    def search(self,
-        query: str,
-        k: int = 1,
-        index_column_name: str = 'embeddings',
-    ) -> pd.DataFrame:
-        """
-        This function searches for the most similar elements in the dataset, given a query.
-
-        Arguments:
-            query (str): The query.
-            k (int, optional): The number of elements to return  (default: 1).
-            index_column_name (str, optional): The name of the column containing the embeddings (default: 'embeddings')
-
-        Returns:
-            pd.DataFrame: The most similar elements in the dataset (text, score, etc.), sorted by score.
-
-        Remarks:
-            The returned elements are dictionaries containing the text and the score.
-        """
-
-        # Get the embeddings of the query
-        query_embeddings = self.get_query_embeddings([query], embedding_type=self.embedding_type)
-
-        # query_embedding이랑 self.dataset['embeddings'] 값 비교
-        scores = []
-        similar_elts = []
-        for query in query_embeddings:
-            # Search for the most similar elements in the dataset
-            score, similar_elt = self.dataset.get_nearest_examples(
-                index_name=index_column_name,
-                query=query,
-                k=k,
-            )
-            scores.append(score)
-            similar_elts.append(similar_elt)
-        
+        return embeddings
 
-        text_list = []
-        for item in similar_elts:
-            for text in item['text']:
-                text_list.append(text)
-        
-        flat_list = [sentence for sublist in text_list for sentence in sublist]
-        count = Counter(flat_list)
-        count = dict(count.most_common(5))
-         
-        sorted_dict = dict(sorted(count.items(), key=lambda x: x[1], reverse=True))
-        # Convert the results to a pandas DataFrame
-        results_df = pd.DataFrame({'text': sorted_dict.keys() , 'freq': sorted_dict.values()})
-        
-        # Return the most similar elements
-        return results_df