tomoto 0.2.2 → 0.3.0

data/vendor/eigen/unsupported/README.txt

@@ -20,7 +20,7 @@ However, it:
  - must rely on Eigen,
  - must be highly related to math,
  - should have some general purpose in the sense that it could
-   potentially become an offical Eigen module (or be merged into another one).
+   potentially become an official Eigen module (or be merged into another one).
 
 In doubt feel free to contact us. For instance, if your addons is very too specific
 but it shows an interesting way of using Eigen, then it could be a nice demo.

data/vendor/tomotopy/README.kr.rst

@@ -198,6 +198,57 @@ add_doc은 `tomotopy.LDAModel.train`을 시작하기 전까지만 사용할 수
 infer 메소드는 `tomotopy.Document` 인스턴스 하나를 추론하거나 `tomotopy.Document` 인스턴스의 `list`를 추론하는데 사용할 수 있습니다. 
 자세한 것은 `tomotopy.LDAModel.infer`을 참조하길 바랍니다.
 
+Corpus와 transform
+--------------------
+`tomotopy`의 모든 토픽 모델들은 각자 별도의 내부적인 문헌 타입을 가지고 있습니다.
+그리고 이 문헌 타입들에 맞는 문헌들은 각 모델의 `add_doc` 메소드를 통해 생성될 수 있습니다.
+하지만 이 때문에 동일한 목록의 문헌들을 서로 다른 토픽 모델에 입력해야 하는 경우
+매 모델에 각 문헌을 추가할때마다 `add_doc`을 호출해야하기 때문에 비효율이 발생합니다.
+따라서 `tomotopy`에서는 여러 문헌을 묶어서 관리해주는 `tomotopy.utils.Corpus` 클래스를 제공합니다.
+토픽 모델 객체를 생성할때 `tomotopy.utils.Corpus`를 `__init__` 메소드의 `corpus` 인자로 넘겨줌으로써 
+어떤 모델에든 쉽게 문헌들을 삽입할 수 있게 해줍니다.
+`tomotopy.utils.Corpus`를 토픽 모델에 삽입하면 corpus 객체가 가지고 있는 문헌들 전부가 모델에 자동으로 삽입됩니다.
+
+그런데 일부 토픽 모델의 경우 문헌을 생성하기 위해 서로 다른 데이터를 요구합니다.
+예를 들어 `tomotopy.DMRModel`는 `metadata`라는 `str` 타입의 데이터를 요구하고, 
+`tomotopy.PLDAModel`는 `labels`라는 `List[str]` 타입의 데이터를 요구합니다. 
+그러나 `tomotopy.utils.Corpus`는 토픽 모델에 종속되지 않은 독립적인 문헌 데이터를 보관하기 때문에,
+corpus가 가지고 있는 문헌 데이터가 실제 토픽 모델이 요구하는 데이터와 일치하지 않을 가능성이 있습니다.
+이 경우 `transform`라는 인자를 통해 corpus 내의 데이터를 변형시켜 토픽 모델이 요구하는 실제 데이터와 일치시킬 수 있습니다.
+자세한 내용은 아래의 코드를 확인해주세요:
+
+::
+
+    from tomotopy import DMRModel
+    from tomotopy.utils import Corpus
+
+    corpus = Corpus()
+    corpus.add_doc("a b c d e".split(), a_data=1)
+    corpus.add_doc("e f g h i".split(), a_data=2)
+    corpus.add_doc("i j k l m".split(), a_data=3)
+
+    model = DMRModel(k=10)
+    model.add_corpus(corpus) 
+    # `corpus`에 있던 `a_data`는 사라지고
+    # `DMRModel`이 요구하는 `metadata`에는 기본값인 빈 문자열이 채워집니다.
+
+    assert model.docs[0].metadata == ''
+    assert model.docs[1].metadata == ''
+    assert model.docs[2].metadata == ''
+
+    def transform_a_data_to_metadata(misc: dict):
+        return {'metadata': str(misc['a_data'])}
+    # 이 함수는 `a_data`를 `metadata`로 변환합니다.
+
+    model = DMRModel(k=10)
+    model.add_corpus(corpus, transform=transform_a_data_to_metadata)
+    # 이제 `model`에는 기본값이 아닌 `metadata`가 입력됩니다. 이들은 `transform`에 의해 `a_data`로부터 생성됩니다.
+
+    assert model.docs[0].metadata == '1'
+    assert model.docs[1].metadata == '2'
+    assert model.docs[2].metadata == '3'
+
+
 병렬 샘플링 알고리즘
 ----------------------------
 `tomotopy`는 0.5.0버전부터 병렬 알고리즘을 고를 수 있는 선택지를 제공합니다.
@@ -254,6 +305,33 @@ tomotopy의 Python3 예제 코드는 https://github.com/bab2min/tomotopy/blob/ma
 
 역사
 -------
+* 0.12.3 (2022-07-19)
+    * 기능 개선
+        * 이제 `tomotopy.LDAModel.add_doc()`로 빈 문서를 삽입할 경우 예외를 발생시키는 대신 그냥 무시합니다. 새로 추가된 인자인 `ignore_empty_words`를 False로 설정할 경우 이전처럼 예외를 발생시킵니다.
+        * 유효하지 않은 토픽들을 모델에서 제거하는 `tomotopy.HDPModel.purge_dead_topics()` 메소드가 추가되었습니다.
+    * 버그 수정
+        * `tomotopy.SLDAModel`에서 `nuSq` 값을 지정할 때 발생하는 문제를 해결했습니다. (by @jucendrero)
+        * `tomotopy.utils.Coherence`가 `tomotopy.DTModel`에 대해서 작동하지 않는 문제를 해결했습니다.
+        * `train()` 호출 전에 `make_dic()`을 호출할 때 종종 크래시가 발생하는 문제를 해결했습니다.
+        * seed가 고정된 상태에서도 `tomotopy.DMRModel`, `tomotopy.GDMRModel`의 결과가 다르게 나오는 문제를 해결했습니다.
+        * `tomotopy.DMRModel`, `tomotopy.GDMRModel`의 파라미터 최적화 과정이 부정확하던 문제를 해결했습니다.
+        * `tomotopy.PTModel.copy()` 호출 시 종종 크래시가 발생하는 문제를 해결했습니다.
+        * 싱글스레드에서의 작동 효율을 개선했습니다.
+
+* 0.12.2 (2021-09-06)
+    * `min_cf > 0`, `min_df > 0`나 `rm_top > 0`로 설정된 `tomotopy.HDPModel`에서 `convert_to_lda`를 호출할때 크래시가 발생하는 문제를 해결했습니다.
+    * `tomotopy.Document.get_topics`와 `tomotopy.Document.get_topic_dist`에 `from_pseudo_doc` 인자가 추가되었습니다. 
+      이 인자는 `PTModel`에 대해서만 유효하며, 이를 통해 토픽 분포를 구할 때 가상 문헌을 사용할지 여부를 선택할 수 있습니다.
+    * `tomotopy.PTModel` 생성시 기본 인자값이 변경되었습니다. `p`를 생략시 `k * 10`으로 설정됩니다.
+    * `make_doc`으로 생성한 문헌을 `infer` 없이 사용할 경우 발생하는 크래시를 해결하고 경고 메세지를 추가했습니다.
+    * 내부 C++코드가 clang c++17 환경에서 컴파일에 실패하는 문제를 해결했습니다.
+
+* 0.12.1 (2021-06-20)
+    * `tomotopy.LDAModel.set_word_prior()`가 크래시를 발생시키던 문제를 해결했습니다.
+    * 이제 `tomotopy.LDAModel.perplexity`와 `tomotopy.LDAModel.ll_per_word`가 TermWeight가 ONE이 아닌 경우에도 정확한 값을 반환합니다.
+    * 용어가중치가 적용된 빈도수를 반환하는 `tomotopy.LDAModel.used_vocab_weighted_freq`가 추가되었습니다.
+    * 이제 `tomotopy.LDAModel.summary()`가 단어의 엔트로피뿐만 아니라, 용어 가중치가 적용된 단어의 엔트로피도 함께 보여줍니다.
+
 * 0.12.0 (2021-04-26)
     * 이제 `tomotopy.DMRModel`와 `tomotopy.GDMRModel`가 다중 메타데이터를 지원합니다. (https://github.com/bab2min/tomotopy/blob/main/examples/dmr_multi_label.py 참조)
     * `tomotopy.GDMRModel`의 성능이 개선되었습니다.

data/vendor/tomotopy/README.rst

@@ -202,6 +202,55 @@ Inference for unseen document should be performed using `tomotopy.LDAModel.infer
 The `infer` method can infer only one instance of `tomotopy.Document` or a `list` of instances of `tomotopy.Document`. 
 See more at `tomotopy.LDAModel.infer`.
 
+Corpus and transform
+--------------------
+Every topic model in `tomotopy` has its own internal document type.
+A document can be created and added into suitable for each model through each model's `add_doc` method. 
+However, trying to add the same list of documents to different models becomes quite inconvenient, 
+because `add_doc` should be called for the same list of documents to each different model.
+Thus, `tomotopy` provides `tomotopy.utils.Corpus` class that holds a list of documents. 
+`tomotopy.utils.Corpus` can be inserted into any model by passing as argument `corpus` to `__init__` or `add_corpus` method of each model. 
+So, inserting `tomotopy.utils.Corpus` just has the same effect to inserting documents the corpus holds.
+
+Some topic models requires different data for its documents. 
+For example, `tomotopy.DMRModel` requires argument `metadata` in `str` type, 
+but `tomotopy.PLDAModel` requires argument `labels` in `List[str]` type. 
+Since `tomotopy.utils.Corpus` holds an independent set of documents rather than being tied to a specific topic model, 
+data types required by a topic model may be inconsistent when a corpus is added into that topic model. 
+In this case, miscellaneous data can be transformed to be fitted target topic model using argument `transform`. 
+See more details in the following code:
+
+::
+
+    from tomotopy import DMRModel
+    from tomotopy.utils import Corpus
+
+    corpus = Corpus()
+    corpus.add_doc("a b c d e".split(), a_data=1)
+    corpus.add_doc("e f g h i".split(), a_data=2)
+    corpus.add_doc("i j k l m".split(), a_data=3)
+
+    model = DMRModel(k=10)
+    model.add_corpus(corpus) 
+    # You lose `a_data` field in `corpus`, 
+    # and `metadata` that `DMRModel` requires is filled with the default value, empty str.
+
+    assert model.docs[0].metadata == ''
+    assert model.docs[1].metadata == ''
+    assert model.docs[2].metadata == ''
+
+    def transform_a_data_to_metadata(misc: dict):
+        return {'metadata': str(misc['a_data'])}
+    # this function transforms `a_data` to `metadata`
+
+    model = DMRModel(k=10)
+    model.add_corpus(corpus, transform=transform_a_data_to_metadata)
+    # Now docs in `model` has non-default `metadata`, that generated from `a_data` field.
+
+    assert model.docs[0].metadata == '1'
+    assert model.docs[1].metadata == '2'
+    assert model.docs[2].metadata == '3'
+
 Parallel Sampling Algorithms
 ----------------------------
 Since version 0.5.0, `tomotopy` allows you to choose a parallelism algorithm. 
@@ -260,6 +309,32 @@ meaning you can use it for any reasonable purpose and remain in complete ownersh
 
 History
 -------
+* 0.12.3 (2022-07-19)
+    * New features
+        * Now, inserting an empty document using `tomotopy.LDAModel.add_doc()` just ignores it instead of raising an exception. If the newly added argument `ignore_empty_words` is set to False, an exception is raised as before.
+        * `tomotopy.HDPModel.purge_dead_topics()` method is added to remove non-live topics from the model.
+    * Bug fixes
+        * Fixed an issue that prevents setting user defined values for nuSq in `tomotopy.SLDAModel` (by @jucendrero).
+        * Fixed an issue where `tomotopy.utils.Coherence` did not work for `tomotopy.DTModel`.
+        * Fixed an issue that often crashed when calling `make_dic()` before calling `train()`.
+        * Resolved the problem that the results of `tomotopy.DMRModel` and `tomotopy.GDMRModel` are different even when the seed is fixed.
+        * The parameter optimization process of `tomotopy.DMRModel` and `tomotopy.GDMRModel` has been improved.
+        * Fixed an issue that sometimes crashed when calling `tomotopy.PTModel.copy()`.
+
+* 0.12.2 (2021-09-06)
+    * An issue where calling `convert_to_lda` of `tomotopy.HDPModel` with `min_cf > 0`, `min_df > 0` or `rm_top > 0` causes a crash has been fixed.
+    * A new argument `from_pseudo_doc` is added to `tomotopy.Document.get_topics` and `tomotopy.Document.get_topic_dist`.
+      This argument is only valid for documents of `PTModel`, it enables to control a source for computing topic distribution.
+    * A default value for argument `p` of `tomotopy.PTModel` has been changed. The new default value is `k * 10`.
+    * Using documents generated by `make_doc` without calling `infer` doesn't cause a crash anymore, but just print warning messages.
+    * An issue where the internal C++ code isn't compiled at clang c++17 environment has been fixed.
+
+* 0.12.1 (2021-06-20)
+    * An issue where `tomotopy.LDAModel.set_word_prior()` causes a crash has been fixed.
+    * Now `tomotopy.LDAModel.perplexity` and `tomotopy.LDAModel.ll_per_word` return the accurate value when `TermWeight` is not `ONE`.
+    * `tomotopy.LDAModel.used_vocab_weighted_freq` was added, which returns term-weighted frequencies of words.
+    * Now `tomotopy.LDAModel.summary()` shows not only the entropy of words, but also the entropy of term-weighted words.
+
 * 0.12.0 (2021-04-26)
     * Now `tomotopy.DMRModel` and `tomotopy.GDMRModel` support multiple values of metadata (see https://github.com/bab2min/tomotopy/blob/main/examples/dmr_multi_label.py )
     * The performance of `tomotopy.GDMRModel` was improved.

data/vendor/tomotopy/src/Labeling/FoRelevance.cpp

@@ -238,7 +238,7 @@ void FoRelevance::estimateContexts()
 
 	Eigen::ArrayXi df = Eigen::ArrayXi::Zero(tm->getV());
 
-	if (pool)
+	if (pool && pool->getNumWorkers() > 1)
 	{
 		const size_t groups = pool->getNumWorkers() * 4;
 		std::vector<std::future<Eigen::ArrayXi>> futures;
@@ -379,7 +379,7 @@ void FoRelevance::estimateContexts()
 			((wcPMI.array() + smoothing) * totDocCnt / docCnt / df.cast<Float>()).log().matrix();
 	};
 
-	if (pool)
+	if (pool && pool->getNumWorkers() > 1)
 	{
 		const size_t groups = pool->getNumWorkers() * 4;
 		std::vector<std::future<void>> futures;

data/vendor/tomotopy/src/Labeling/Phraser.hpp

@@ -197,7 +197,7 @@ namespace tomoto
 		template<typename _LocalData, typename _ReduceFn>
 		_LocalData parallelReduce(std::vector<_LocalData>&& data, _ReduceFn&& fn, ThreadPool* pool = nullptr)
 		{
-			if (pool)
+			if (pool && pool->getNumWorkers() > 1)
 			{
 				for (size_t s = data.size(); s > 1; s = (s + 1) / 2)
 				{
@@ -316,7 +316,7 @@ namespace tomoto
 				}
 			}
 
-			float totN = std::accumulate(vocabFreqs.begin(), vocabFreqs.end(), (size_t)0);
+			float totN = (float)std::accumulate(vocabFreqs.begin(), vocabFreqs.end(), (size_t)0);
 			const float logTotN = std::log(totN);
 
 			// calculating PMIs
@@ -489,7 +489,7 @@ namespace tomoto
 
 				float rbe = branchingEntropy(trieNodes[0].getNext(bigram.first)->getNext(bigram.second), candMinCnt);
 				float lbe = branchingEntropy(trieNodesBw[0].getNext(bigram.second)->getNext(bigram.first), candMinCnt);
-				float nbe = std::sqrt(rbe * lbe) / std::log(p.second);
+				float nbe = std::sqrt(rbe * lbe) / (float)std::log(p.second);
 				if (nbe < minNBE) continue;
 				candidates.emplace_back(npmi * nbe, bigram.first, bigram.second);
 				candidates.back().cf = p.second;
@@ -512,7 +512,7 @@ namespace tomoto
 
 					float rbe = branchingEntropy(node, candMinCnt);
 					float lbe = branchingEntropy(trieNodesBw[0].findNode(rkeys.rbegin(), rkeys.rend()), candMinCnt);
-					float nbe = std::sqrt(rbe * lbe) / std::log(node->val);
+					float nbe = std::sqrt(rbe * lbe) / (float)std::log(node->val);
 					if (nbe < minNBE) return;
 					candidates.emplace_back(npmi * nbe, rkeys);
 					candidates.back().cf = node->val;

data/vendor/tomotopy/src/TopicModel/CTModel.hpp

@@ -33,7 +33,10 @@ namespace tomoto
 		friend typename BaseClass::BaseClass;
 		using WeightType = typename BaseClass::WeightType;
 
-		static constexpr char TMID[] = "CTM\0";
+		static constexpr auto tmid()
+		{
+			return serializer::to_key("CTM\0");
+		}
 
 		uint64_t numBetaSample = 10;
 		uint64_t numTMNSample = 5;
@@ -128,7 +131,7 @@ namespace tomoto
 		{
 			if (this->globalStep < this->burnIn || !this->optimInterval || (this->globalStep + 1) % this->optimInterval != 0) return;
 
-			if (pool)
+			if (pool && pool->getNumWorkers() > 1)
 			{
 				std::vector<std::future<void>> res;
 				const size_t chStride = pool->getNumWorkers() * 8;
@@ -247,8 +250,9 @@ namespace tomoto
 			this->optimInterval = 2;
 		}
 
-		std::vector<Float> getTopicsByDoc(const _DocType& doc, bool normalize) const
+		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
 			if (normalize)

data/vendor/tomotopy/src/TopicModel/DMRModel.hpp

@@ -47,7 +47,10 @@ namespace tomoto
 		friend typename BaseClass::BaseClass;
 		using WeightType = typename BaseClass::WeightType;
 
-		static constexpr char TMID[] = "DMR\0";
+		static constexpr auto tmid()
+		{
+			return serializer::to_key("DMR\0");
+		}
 
 		Matrix lambda;
 		mutable std::unordered_map<std::pair<uint64_t, Vector>, size_t, MdHash> mdHashMap;
@@ -83,7 +86,7 @@ namespace tomoto
 			const size_t chStride = pool.getNumWorkers() * 8;
 			for (size_t ch = 0; ch < chStride; ++ch)
 			{
-				res.emplace_back(pool.enqueue([&](size_t threadId)
+				res.emplace_back(pool.enqueue([&, ch](size_t threadId)
 				{
 					auto& tmpK = localData[threadId].tmpK;
 					if (!tmpK.size()) tmpK.resize(this->K);
@@ -449,8 +452,9 @@ namespace tomoto
 			optimRepeat = _optimRepeat;
 		}
 
-		std::vector<Float> getTopicsByDoc(const _DocType& doc, bool normalize) const
+		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			auto alphaDoc = getCachedAlpha(doc);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };

data/vendor/tomotopy/src/TopicModel/DTModel.hpp

@@ -41,7 +41,10 @@ namespace tomoto
 		friend typename BaseClass::BaseClass;
 		using WeightType = typename BaseClass::WeightType;
 
-		static constexpr char TMID[] = "DTM\0";
+		static constexpr auto tmid()
+		{
+			return serializer::to_key("DTM\0");
+		}
 
 		uint64_t T;
 		Float shapeA = 0.03f, shapeB = 0.1f, shapeC = 0.55f;
@@ -54,7 +57,7 @@ namespace tomoto
 		std::vector<sample::AliasMethod<>> wordAliasTables; // Dim: (Word * Time)
 
 		template<int _inc>
-		inline void addWordTo(_ModelState& ld, _DocType& doc, uint32_t pid, Vid vid, Tid tid) const
+		inline void addWordTo(_ModelState& ld, _DocType& doc, size_t pid, Vid vid, Tid tid) const
 		{
 			assert(tid < this->K);
 			assert(vid < this->realV);
@@ -233,7 +236,7 @@ namespace tomoto
 			for (size_t t = 0; t < T; ++t)
 			{
 				// update alias tables for word proposal
-				if (pool)
+				if (pool && pool->getNumWorkers() > 1)
 				{
 					const size_t chStride = pool->getNumWorkers() * 8;
 					std::vector<std::future<void>> futures;

data/vendor/tomotopy/src/TopicModel/GDMRModel.hpp

@@ -85,8 +85,8 @@ namespace tomoto
 
 		Float getNegativeLambdaLL(Eigen::Ref<Vector> x, Vector& g) const
 		{
-			auto mappedX = Eigen::Map<Matrix>(x.data(), this->K, this->F);
-			auto mappedG = Eigen::Map<Matrix>(g.data(), this->K, this->F);
+			auto mappedX = Eigen::Map<Matrix>(x.data(), this->K, this->F * this->fCont);
+			auto mappedG = Eigen::Map<Matrix>(g.data(), this->K, this->F * this->fCont);
 
 			Float fx = 0;
 			for (size_t k = 0; k < this->K; ++k)

data/vendor/tomotopy/src/TopicModel/HDP.h

@@ -79,5 +79,6 @@ namespace tomoto
 		virtual bool isLiveTopic(Tid tid) const = 0;
 
 		virtual std::unique_ptr<ILDAModel> convertToLDA(float topicThreshold, std::vector<Tid>& newK) const = 0;
+		virtual std::vector<Tid> purgeDeadTopics() = 0;
 	};
 }

data/vendor/tomotopy/src/TopicModel/HDPModel.hpp

@@ -168,7 +168,7 @@ namespace tomoto
 		}
 
 		template<int _inc> 
-		inline void addWordTo(_ModelState& ld, _DocType& doc, uint32_t pid, Vid vid, size_t tableId, Tid tid) const
+		inline void addWordTo(_ModelState& ld, _DocType& doc, size_t pid, Vid vid, size_t tableId, Tid tid) const
 		{
 			addOnlyWordTo<_inc>(ld, doc, pid, vid, tid);
 			constexpr bool _dec = _inc < 0 && _tw != TermWeight::one;
@@ -490,8 +490,9 @@ namespace tomoto
 			THROW_ERROR_WITH_INFO(exc::Unimplemented, "HDPModel doesn't provide setWordPrior function.");
 		}
 
-		std::vector<Float> getTopicsByDoc(const _DocType& doc, bool normalize) const
+		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
 			if (normalize)
@@ -522,7 +523,7 @@ namespace tomoto
 			for (size_t i = 0; i < cntIdx.size(); ++i)
 			{
 				if (i && cntIdx[i].first / sum <= topicThreshold) break;
-				newK[cntIdx[i].second] = i;
+				newK[cntIdx[i].second] = (Tid)i;
 				liveK++;
 			}
 
@@ -538,8 +539,11 @@ namespace tomoto
 				auto d = lda->_makeFromRawDoc(doc);
 				lda->_addDoc(d);
 			}
-			
-			lda->prepare(true, this->minWordCf, this->minWordDf, this->removeTopN);
+
+			lda->realV = this->realV;
+			lda->realN = this->realN;
+			lda->weightedN = this->weightedN;
+			lda->prepare(true, 0, 0, 0, false);
 
 			auto selectFirst = [&](const std::pair<size_t, size_t>& p) { return std::max(p.first / sum - topicThreshold, 0.f); };
 			std::discrete_distribution<size_t> randomTopic{
@@ -558,7 +562,7 @@ namespace tomoto
 						lda->docs[i].Zs[j] = non_topic_id;
 						continue;
 					}
-					size_t newTopic = newK[this->docs[i].numTopicByTable[this->docs[i].Zs[j]].topic];
+					Tid newTopic = newK[this->docs[i].numTopicByTable[this->docs[i].Zs[j]].topic];
 					while (newTopic == (Tid)-1) newTopic = newK[randomTopic(rng)];
 					lda->docs[i].Zs[j] = newTopic;
 				}
@@ -569,6 +573,53 @@ namespace tomoto
 
 			return lda;
 		}
+
+		std::vector<Tid> purgeDeadTopics() override
+		{
+			std::vector<Tid> relocation(this->K, -1);
+			Tid numLiveTopics = 0;
+			for (size_t i = 0; i < this->K; ++i)
+			{
+				if (this->globalState.numTableByTopic[i])
+				{
+					relocation[i] = numLiveTopics++;
+				}
+			}
+
+			for (auto& doc : this->docs)
+			{
+				for (auto& nt : doc.numTopicByTable)
+				{
+					nt.topic = (relocation[nt.topic] == (Tid)-1) ? 0 : relocation[nt.topic];
+				}
+
+				for (size_t i = 0; i < relocation.size(); ++i)
+				{
+					if (relocation[i] == (Tid)-1) continue;
+					doc.numByTopic[relocation[i]] = doc.numByTopic[i];
+				}
+				doc.numByTopic.conservativeResize(numLiveTopics, 1);
+			}
+
+			for (auto tt : { &this->globalState, &this->tState })
+			{
+				auto& numTableByTopic = tt->numTableByTopic;
+				auto& numByTopic = tt->numByTopic;
+				auto& numByTopicWord = tt->numByTopicWord;
+				for (size_t i = 0; i < relocation.size(); ++i)
+				{
+					if (relocation[i] == (Tid)-1) continue;
+					numTableByTopic[relocation[i]] = numTableByTopic[i];
+					numByTopic[relocation[i]] = numByTopic[i];
+					numByTopicWord.row(relocation[i]) = numByTopicWord.row(i);
+				}
+				numTableByTopic.conservativeResize(numLiveTopics);
+				numByTopic.conservativeResize(numLiveTopics);
+				numByTopicWord.conservativeResize(numLiveTopics, numByTopicWord.cols());
+			}
+			this->K = numLiveTopics;
+			return relocation;
+		}
 	};
 
 	template<TermWeight _tw>

data/vendor/tomotopy/src/TopicModel/HLDAModel.hpp

@@ -229,7 +229,7 @@ namespace tomoto
 				};
 
 				// we elide the likelihood for root node because its weight applied to all path and can be seen as constant.
-				if (pool)
+				if (pool && pool->getNumWorkers() > 1)
 				{
 					const size_t chStride = pool->getNumWorkers() * 8;
 					for (size_t ch = 0; ch < chStride; ++ch)
@@ -335,7 +335,10 @@ namespace tomoto
 		friend typename BaseClass::BaseClass;
 		using WeightType = typename BaseClass::WeightType;
 
-		static constexpr char TMID[] = "hLDA";
+		static constexpr auto tmid()
+		{
+			return serializer::to_key("hLDA");
+		}
 
 		Float gamma;
 
@@ -422,7 +425,7 @@ namespace tomoto
 		}
 
 		template<int _inc>
-		inline void addWordTo(_ModelState& ld, _DocType& doc, uint32_t pid, Vid vid, Tid level) const
+		inline void addWordTo(_ModelState& ld, _DocType& doc, size_t pid, Vid vid, Tid level) const
 		{
 			assert(vid < this->realV);
 			constexpr bool _dec = _inc < 0 && _tw != TermWeight::one;

data/vendor/tomotopy/src/TopicModel/HPAModel.hpp

@@ -143,7 +143,7 @@ namespace tomoto
 		}
 
 		template<int _inc>
-		inline void addWordTo(_ModelState& ld, _DocType& doc, uint32_t pid, Vid vid, Tid z1, Tid z2) const
+		inline void addWordTo(_ModelState& ld, _DocType& doc, size_t pid, Vid vid, Tid z1, Tid z2) const
 		{
 			assert(vid < this->realV);
 			constexpr bool _dec = _inc < 0 && _tw != TermWeight::one;
@@ -540,8 +540,9 @@ namespace tomoto
 			return ret;
 		}
 
-		std::vector<Float> getTopicsByDoc(const _DocType& doc, bool normalize) const
+		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(1 + this->K + K2);
 			Float sum = doc.getSumWordWeight() + this->alphas.sum();
 			if (!normalize) sum = 1;

data/vendor/tomotopy/src/TopicModel/LDA.h

@@ -121,7 +121,7 @@ namespace tomoto
 
 		void updateSumWordWeight(size_t realV)
 		{
-			sumWordWeight = std::count_if(static_cast<_Base*>(this)->words.begin(), static_cast<_Base*>(this)->words.end(), [realV](Vid w)
+			sumWordWeight = (int32_t)std::count_if(static_cast<_Base*>(this)->words.begin(), static_cast<_Base*>(this)->words.end(), [realV](Vid w)
 			{
 				return w < realV;
 			});
@@ -164,8 +164,8 @@ namespace tomoto
 	struct LDAArgs
 	{
 		size_t k = 1;
-		std::vector<Float> alpha = { 0.1 };
-		Float eta = 0.01;
+		std::vector<Float> alpha = { (Float)0.1 };
+		Float eta = (Float)0.01;
 		size_t seed = std::random_device{}();
 	};
 

data/vendor/tomotopy/src/TopicModel/LDACVB0Model.hpp

@@ -82,7 +82,7 @@ namespace tomoto
 		friend BaseClass;
 
 		static constexpr const char TWID[] = "one\0";
-		static constexpr static constexpr char TMID[] = "LDA\0";
+		static constexpr const char TMID[] = "LDA\0";
 
 		Float alpha;
 		Vector alphas;
@@ -125,7 +125,7 @@ namespace tomoto
 		}
 
 		template<int _Inc, typename _Vec>
-		inline void addWordTo(_ModelState& ld, _DocType& doc, uint32_t pid, Vid vid, _Vec tDist) const
+		inline void addWordTo(_ModelState& ld, _DocType& doc, size_t pid, Vid vid, _Vec tDist) const
 		{
 			assert(vid < this->realV);
 			constexpr bool _dec = _Inc < 0;
@@ -366,9 +366,9 @@ namespace tomoto
 			}
 		}
 
-		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0) override
+		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0, bool updateStopwords = true) override
 		{
-			if (initDocs) this->removeStopwords(minWordCnt, minWordDf, removeTopN);
+			if (initDocs) this->removeStopwords(minWordCnt, minWordDf, removeTopN, updateStopwords);
 			static_cast<DerivedClass*>(this)->updateWeakArray();
 			static_cast<DerivedClass*>(this)->initGlobalState(initDocs);
 
@@ -392,7 +392,7 @@ namespace tomoto
 			return static_cast<const DerivedClass*>(this)->_getTopicsCount();
 		}
 
-		std::vector<Float> getTopicsByDoc(const _DocType& doc) const
+		std::vector<Float> _getTopicsByDoc(const _DocType& doc) const
 		{
 			std::vector<Float> ret(K);
 			Float sum = doc.getSumWordWeight() + K * alpha;