tomoto 0.2.3 → 0.3.0

data/vendor/eigen/unsupported/README.txt

@@ -20,7 +20,7 @@ However, it:
  - must rely on Eigen,
  - must be highly related to math,
  - should have some general purpose in the sense that it could
-   potentially become an offical Eigen module (or be merged into another one).
+   potentially become an official Eigen module (or be merged into another one).
 
 In doubt feel free to contact us. For instance, if your addons is very too specific
 but it shows an interesting way of using Eigen, then it could be a nice demo.

data/vendor/tomotopy/README.kr.rst

@@ -305,6 +305,27 @@ tomotopy의 Python3 예제 코드는 https://github.com/bab2min/tomotopy/blob/ma
 
 역사
 -------
+* 0.12.3 (2022-07-19)
+    * 기능 개선
+        * 이제 `tomotopy.LDAModel.add_doc()`로 빈 문서를 삽입할 경우 예외를 발생시키는 대신 그냥 무시합니다. 새로 추가된 인자인 `ignore_empty_words`를 False로 설정할 경우 이전처럼 예외를 발생시킵니다.
+        * 유효하지 않은 토픽들을 모델에서 제거하는 `tomotopy.HDPModel.purge_dead_topics()` 메소드가 추가되었습니다.
+    * 버그 수정
+        * `tomotopy.SLDAModel`에서 `nuSq` 값을 지정할 때 발생하는 문제를 해결했습니다. (by @jucendrero)
+        * `tomotopy.utils.Coherence`가 `tomotopy.DTModel`에 대해서 작동하지 않는 문제를 해결했습니다.
+        * `train()` 호출 전에 `make_dic()`을 호출할 때 종종 크래시가 발생하는 문제를 해결했습니다.
+        * seed가 고정된 상태에서도 `tomotopy.DMRModel`, `tomotopy.GDMRModel`의 결과가 다르게 나오는 문제를 해결했습니다.
+        * `tomotopy.DMRModel`, `tomotopy.GDMRModel`의 파라미터 최적화 과정이 부정확하던 문제를 해결했습니다.
+        * `tomotopy.PTModel.copy()` 호출 시 종종 크래시가 발생하는 문제를 해결했습니다.
+        * 싱글스레드에서의 작동 효율을 개선했습니다.
+
+* 0.12.2 (2021-09-06)
+    * `min_cf > 0`, `min_df > 0`나 `rm_top > 0`로 설정된 `tomotopy.HDPModel`에서 `convert_to_lda`를 호출할때 크래시가 발생하는 문제를 해결했습니다.
+    * `tomotopy.Document.get_topics`와 `tomotopy.Document.get_topic_dist`에 `from_pseudo_doc` 인자가 추가되었습니다. 
+      이 인자는 `PTModel`에 대해서만 유효하며, 이를 통해 토픽 분포를 구할 때 가상 문헌을 사용할지 여부를 선택할 수 있습니다.
+    * `tomotopy.PTModel` 생성시 기본 인자값이 변경되었습니다. `p`를 생략시 `k * 10`으로 설정됩니다.
+    * `make_doc`으로 생성한 문헌을 `infer` 없이 사용할 경우 발생하는 크래시를 해결하고 경고 메세지를 추가했습니다.
+    * 내부 C++코드가 clang c++17 환경에서 컴파일에 실패하는 문제를 해결했습니다.
+
 * 0.12.1 (2021-06-20)
     * `tomotopy.LDAModel.set_word_prior()`가 크래시를 발생시키던 문제를 해결했습니다.
     * 이제 `tomotopy.LDAModel.perplexity`와 `tomotopy.LDAModel.ll_per_word`가 TermWeight가 ONE이 아닌 경우에도 정확한 값을 반환합니다.

data/vendor/tomotopy/README.rst

@@ -309,6 +309,26 @@ meaning you can use it for any reasonable purpose and remain in complete ownersh
 
 History
 -------
+* 0.12.3 (2022-07-19)
+    * New features
+        * Now, inserting an empty document using `tomotopy.LDAModel.add_doc()` just ignores it instead of raising an exception. If the newly added argument `ignore_empty_words` is set to False, an exception is raised as before.
+        * `tomotopy.HDPModel.purge_dead_topics()` method is added to remove non-live topics from the model.
+    * Bug fixes
+        * Fixed an issue that prevents setting user defined values for nuSq in `tomotopy.SLDAModel` (by @jucendrero).
+        * Fixed an issue where `tomotopy.utils.Coherence` did not work for `tomotopy.DTModel`.
+        * Fixed an issue that often crashed when calling `make_dic()` before calling `train()`.
+        * Resolved the problem that the results of `tomotopy.DMRModel` and `tomotopy.GDMRModel` are different even when the seed is fixed.
+        * The parameter optimization process of `tomotopy.DMRModel` and `tomotopy.GDMRModel` has been improved.
+        * Fixed an issue that sometimes crashed when calling `tomotopy.PTModel.copy()`.
+
+* 0.12.2 (2021-09-06)
+    * An issue where calling `convert_to_lda` of `tomotopy.HDPModel` with `min_cf > 0`, `min_df > 0` or `rm_top > 0` causes a crash has been fixed.
+    * A new argument `from_pseudo_doc` is added to `tomotopy.Document.get_topics` and `tomotopy.Document.get_topic_dist`.
+      This argument is only valid for documents of `PTModel`, it enables to control a source for computing topic distribution.
+    * A default value for argument `p` of `tomotopy.PTModel` has been changed. The new default value is `k * 10`.
+    * Using documents generated by `make_doc` without calling `infer` doesn't cause a crash anymore, but just print warning messages.
+    * An issue where the internal C++ code isn't compiled at clang c++17 environment has been fixed.
+
 * 0.12.1 (2021-06-20)
     * An issue where `tomotopy.LDAModel.set_word_prior()` causes a crash has been fixed.
     * Now `tomotopy.LDAModel.perplexity` and `tomotopy.LDAModel.ll_per_word` return the accurate value when `TermWeight` is not `ONE`.

data/vendor/tomotopy/src/Labeling/FoRelevance.cpp

@@ -238,7 +238,7 @@ void FoRelevance::estimateContexts()
 
 	Eigen::ArrayXi df = Eigen::ArrayXi::Zero(tm->getV());
 
-	if (pool)
+	if (pool && pool->getNumWorkers() > 1)
 	{
 		const size_t groups = pool->getNumWorkers() * 4;
 		std::vector<std::future<Eigen::ArrayXi>> futures;
@@ -379,7 +379,7 @@ void FoRelevance::estimateContexts()
 			((wcPMI.array() + smoothing) * totDocCnt / docCnt / df.cast<Float>()).log().matrix();
 	};
 
-	if (pool)
+	if (pool && pool->getNumWorkers() > 1)
 	{
 		const size_t groups = pool->getNumWorkers() * 4;
 		std::vector<std::future<void>> futures;

data/vendor/tomotopy/src/Labeling/Phraser.hpp

@@ -197,7 +197,7 @@ namespace tomoto
 		template<typename _LocalData, typename _ReduceFn>
 		_LocalData parallelReduce(std::vector<_LocalData>&& data, _ReduceFn&& fn, ThreadPool* pool = nullptr)
 		{
-			if (pool)
+			if (pool && pool->getNumWorkers() > 1)
 			{
 				for (size_t s = data.size(); s > 1; s = (s + 1) / 2)
 				{

data/vendor/tomotopy/src/TopicModel/CTModel.hpp

@@ -131,7 +131,7 @@ namespace tomoto
 		{
 			if (this->globalStep < this->burnIn || !this->optimInterval || (this->globalStep + 1) % this->optimInterval != 0) return;
 
-			if (pool)
+			if (pool && pool->getNumWorkers() > 1)
 			{
 				std::vector<std::future<void>> res;
 				const size_t chStride = pool->getNumWorkers() * 8;
@@ -252,6 +252,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
 			if (normalize)

data/vendor/tomotopy/src/TopicModel/DMRModel.hpp

@@ -86,7 +86,7 @@ namespace tomoto
 			const size_t chStride = pool.getNumWorkers() * 8;
 			for (size_t ch = 0; ch < chStride; ++ch)
 			{
-				res.emplace_back(pool.enqueue([&](size_t threadId)
+				res.emplace_back(pool.enqueue([&, ch](size_t threadId)
 				{
 					auto& tmpK = localData[threadId].tmpK;
 					if (!tmpK.size()) tmpK.resize(this->K);
@@ -454,6 +454,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			auto alphaDoc = getCachedAlpha(doc);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };

data/vendor/tomotopy/src/TopicModel/DTModel.hpp

@@ -236,7 +236,7 @@ namespace tomoto
 			for (size_t t = 0; t < T; ++t)
 			{
 				// update alias tables for word proposal
-				if (pool)
+				if (pool && pool->getNumWorkers() > 1)
 				{
 					const size_t chStride = pool->getNumWorkers() * 8;
 					std::vector<std::future<void>> futures;

data/vendor/tomotopy/src/TopicModel/GDMRModel.hpp

@@ -85,8 +85,8 @@ namespace tomoto
 
 		Float getNegativeLambdaLL(Eigen::Ref<Vector> x, Vector& g) const
 		{
-			auto mappedX = Eigen::Map<Matrix>(x.data(), this->K, this->F);
-			auto mappedG = Eigen::Map<Matrix>(g.data(), this->K, this->F);
+			auto mappedX = Eigen::Map<Matrix>(x.data(), this->K, this->F * this->fCont);
+			auto mappedG = Eigen::Map<Matrix>(g.data(), this->K, this->F * this->fCont);
 
 			Float fx = 0;
 			for (size_t k = 0; k < this->K; ++k)

data/vendor/tomotopy/src/TopicModel/HDP.h

@@ -79,5 +79,6 @@ namespace tomoto
 		virtual bool isLiveTopic(Tid tid) const = 0;
 
 		virtual std::unique_ptr<ILDAModel> convertToLDA(float topicThreshold, std::vector<Tid>& newK) const = 0;
+		virtual std::vector<Tid> purgeDeadTopics() = 0;
 	};
 }

data/vendor/tomotopy/src/TopicModel/HDPModel.hpp

@@ -492,6 +492,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
 			if (normalize)
@@ -538,8 +539,11 @@ namespace tomoto
 				auto d = lda->_makeFromRawDoc(doc);
 				lda->_addDoc(d);
 			}
-			
-			lda->prepare(true, this->minWordCf, this->minWordDf, this->removeTopN);
+
+			lda->realV = this->realV;
+			lda->realN = this->realN;
+			lda->weightedN = this->weightedN;
+			lda->prepare(true, 0, 0, 0, false);
 
 			auto selectFirst = [&](const std::pair<size_t, size_t>& p) { return std::max(p.first / sum - topicThreshold, 0.f); };
 			std::discrete_distribution<size_t> randomTopic{
@@ -569,6 +573,53 @@ namespace tomoto
 
 			return lda;
 		}
+
+		std::vector<Tid> purgeDeadTopics() override
+		{
+			std::vector<Tid> relocation(this->K, -1);
+			Tid numLiveTopics = 0;
+			for (size_t i = 0; i < this->K; ++i)
+			{
+				if (this->globalState.numTableByTopic[i])
+				{
+					relocation[i] = numLiveTopics++;
+				}
+			}
+
+			for (auto& doc : this->docs)
+			{
+				for (auto& nt : doc.numTopicByTable)
+				{
+					nt.topic = (relocation[nt.topic] == (Tid)-1) ? 0 : relocation[nt.topic];
+				}
+
+				for (size_t i = 0; i < relocation.size(); ++i)
+				{
+					if (relocation[i] == (Tid)-1) continue;
+					doc.numByTopic[relocation[i]] = doc.numByTopic[i];
+				}
+				doc.numByTopic.conservativeResize(numLiveTopics, 1);
+			}
+
+			for (auto tt : { &this->globalState, &this->tState })
+			{
+				auto& numTableByTopic = tt->numTableByTopic;
+				auto& numByTopic = tt->numByTopic;
+				auto& numByTopicWord = tt->numByTopicWord;
+				for (size_t i = 0; i < relocation.size(); ++i)
+				{
+					if (relocation[i] == (Tid)-1) continue;
+					numTableByTopic[relocation[i]] = numTableByTopic[i];
+					numByTopic[relocation[i]] = numByTopic[i];
+					numByTopicWord.row(relocation[i]) = numByTopicWord.row(i);
+				}
+				numTableByTopic.conservativeResize(numLiveTopics);
+				numByTopic.conservativeResize(numLiveTopics);
+				numByTopicWord.conservativeResize(numLiveTopics, numByTopicWord.cols());
+			}
+			this->K = numLiveTopics;
+			return relocation;
+		}
 	};
 
 	template<TermWeight _tw>

data/vendor/tomotopy/src/TopicModel/HLDAModel.hpp

@@ -229,7 +229,7 @@ namespace tomoto
 				};
 
 				// we elide the likelihood for root node because its weight applied to all path and can be seen as constant.
-				if (pool)
+				if (pool && pool->getNumWorkers() > 1)
 				{
 					const size_t chStride = pool->getNumWorkers() * 8;
 					for (size_t ch = 0; ch < chStride; ++ch)

data/vendor/tomotopy/src/TopicModel/HPAModel.hpp

@@ -542,6 +542,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(1 + this->K + K2);
 			Float sum = doc.getSumWordWeight() + this->alphas.sum();
 			if (!normalize) sum = 1;

data/vendor/tomotopy/src/TopicModel/LDACVB0Model.hpp

@@ -366,9 +366,9 @@ namespace tomoto
 			}
 		}
 
-		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0) override
+		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0, bool updateStopwords = true) override
 		{
-			if (initDocs) this->removeStopwords(minWordCnt, minWordDf, removeTopN);
+			if (initDocs) this->removeStopwords(minWordCnt, minWordDf, removeTopN, updateStopwords);
 			static_cast<DerivedClass*>(this)->updateWeakArray();
 			static_cast<DerivedClass*>(this)->initGlobalState(initDocs);
 

data/vendor/tomotopy/src/TopicModel/LDAModel.hpp

@@ -141,6 +141,12 @@ namespace tomoto
 		}
 	};
 
+	inline Float floorBit(Float x, int bitsUnderPoint = 8)
+	{
+		Float s = (1 << bitsUnderPoint);
+		return floor(x * s) / s;
+	}
+
 	// to make HDP friend of LDA for HDPModel::converToLDA
 	template<TermWeight _tw,
 		typename _RandGen,
@@ -482,11 +488,11 @@ namespace tomoto
 		{
 			if (_ps == ParallelScheme::partition)
 			{
-				return (this->realV + 3) / 4;
+				return std::max(((size_t)this->realV + 3) / 4, (size_t)1);
 			}
 			if (_ps == ParallelScheme::copy_merge)
 			{
-				return (this->docs.size() + 1) / 2;
+				return std::max((this->docs.size() + 1) / 2, (size_t)1);
 			}
 			return (size_t)-1;
 		}
@@ -1057,14 +1063,18 @@ namespace tomoto
 			}
 		}
 
-		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0) override
+		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0, bool updateStopwords = true) override
 		{
-			if (initDocs) this->removeStopwords(minWordCnt, minWordDf, removeTopN);
+			if (initDocs && updateStopwords) this->removeStopwords(minWordCnt, minWordDf, removeTopN);
 			static_cast<DerivedClass*>(this)->updateWeakArray();
 			static_cast<DerivedClass*>(this)->initGlobalState(initDocs);
 			static_cast<DerivedClass*>(this)->prepareWordPriors();
 
 			const size_t V = this->realV;
+			if (V == 0) 
+			{
+				std::cerr << "[warn] No valid vocabs in the model!" << std::endl;
+			}
 
 			if (initDocs)
 			{
@@ -1116,7 +1126,7 @@ namespace tomoto
 				for (auto& doc : this->docs) doc.updateSumWordWeight(this->realV);
 			}
 			static_cast<DerivedClass*>(this)->prepareShared();
-			BaseClass::prepare(initDocs, minWordCnt, minWordDf, removeTopN);
+			BaseClass::prepare(initDocs, minWordCnt, minWordDf, removeTopN, updateStopwords);
 		}
 
 		std::vector<uint64_t> getCountByTopic() const override
@@ -1126,6 +1136,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), K };
 			if (normalize)

data/vendor/tomotopy/src/TopicModel/LLDAModel.hpp

@@ -176,6 +176,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			auto maskedAlphas = this->alphas.array() * doc.labelMask.template cast<Float>().array();
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };

data/vendor/tomotopy/src/TopicModel/MGLDAModel.hpp

@@ -529,6 +529,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K + KL);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K + KL };
 			if (normalize)

data/vendor/tomotopy/src/TopicModel/PLDAModel.hpp

@@ -183,6 +183,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (!doc.numByTopic.size()) return {};
 			std::vector<Float> ret(this->K);
 			auto maskedAlphas = this->alphas.array() * doc.labelMask.template cast<Float>().array();
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };

data/vendor/tomotopy/src/TopicModel/PT.h

@@ -18,7 +18,7 @@ namespace tomoto
 
 	struct PTArgs : public LDAArgs
 	{
-		size_t p = 100;
+		size_t p = 0;
 		Float lambda = 0.01;
 	};
 
@@ -30,5 +30,7 @@ namespace tomoto
 			bool scalarRng = false);
 
 		virtual size_t getP() const = 0;
+		virtual std::vector<Float> getTopicsFromPseudoDoc(const DocumentBase* doc, bool normalize = true) const = 0;
+		virtual std::vector<std::pair<Tid, Float>> getTopicsFromPseudoDocSorted(const DocumentBase* doc, size_t topN) const = 0;
 	};
 }

data/vendor/tomotopy/src/TopicModel/PTModel.hpp

@@ -75,7 +75,7 @@ namespace tomoto
 			}
 			--ld.numDocsByPDoc[doc.pseudoDoc];
 			
-			if (pool)
+			if (pool && pool->getNumWorkers() > 1)
 			{
 				std::vector<std::future<void>> futures;
 				for (size_t w = 0; w < pool->getNumWorkers(); ++w)
@@ -253,6 +253,16 @@ namespace tomoto
 			}
 		}
 
+		void updateForCopy()
+		{
+			BaseClass::updateForCopy();
+			size_t offset = 0;
+			for (auto& doc : this->docs)
+			{
+				doc.template update<>(this->globalState.numByTopicPDoc.col(doc.pseudoDoc).data(), *static_cast<DerivedClass*>(this));
+			}
+		}
+
 	public:
 		DEFINE_SERIALIZER_AFTER_BASE_WITH_VERSION(BaseClass, 0, numPDocs, lambda);
 		DEFINE_TAGGED_SERIALIZER_AFTER_BASE_WITH_VERSION(BaseClass, 1, 0x00010001, numPDocs, lambda);
@@ -266,6 +276,7 @@ namespace tomoto
 
 		std::vector<Float> _getTopicsByDoc(const _DocType& doc, bool normalize) const
 		{
+			if (doc.Zs.empty()) return {};
 			std::vector<Float> ret(this->K);
 			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
 			m = this->alphas.array();
@@ -280,6 +291,25 @@ namespace tomoto
 			return ret;
 		}
 
+		std::vector<Float> getTopicsFromPseudoDoc(const DocumentBase* _doc, bool normalize) const override
+		{
+			auto& doc = *static_cast<const _DocType*>(_doc);
+			if (!doc.numByTopic.size()) return {};
+			std::vector<Float> ret(this->K);
+			Eigen::Map<Eigen::Array<Float, -1, 1>> m{ ret.data(), this->K };
+			m = doc.numByTopic.array().template cast<Float>() + this->alphas.array();
+			if (normalize)
+			{
+				m /= m.sum();
+			}
+			return ret;
+		}
+
+		std::vector<std::pair<Tid, Float>> getTopicsFromPseudoDocSorted(const DocumentBase* doc, size_t topN) const override
+		{
+			return extractTopN<Tid>(getTopicsFromPseudoDoc(doc, true), topN);
+		}
+
 		void updateDocs()
 		{
 			for (auto& doc : this->docs)

data/vendor/tomotopy/src/TopicModel/SLDAModel.hpp

@@ -379,11 +379,11 @@ namespace tomoto
 			{
 				nuSq = Vector::Ones(F);
 			}
-			else if (args.mu.size() == 1)
+			else if (args.nuSq.size() == 1)
 			{
 				nuSq = Vector::Constant(F, args.nuSq[0]);
 			}
-			else if (args.mu.size() == F)
+			else if (args.nuSq.size() == F)
 			{
 				nuSq = Eigen::Map<const Vector>(args.nuSq.data(), args.nuSq.size());
 			}

data/vendor/tomotopy/src/TopicModel/TopicModel.hpp

@@ -13,7 +13,7 @@
 
 namespace tomoto
 {
-	using RandGen = Eigen::Rand::P8_mt19937_64<uint32_t>;
+	using RandGen = Eigen::Rand::P8_mt19937_64_32;
 	using ScalarRandGen = Eigen::Rand::UniversalRandomEngine<uint32_t, std::mt19937_64>;
 
 	using Vector = Eigen::Matrix<Float, -1, 1>;
@@ -254,7 +254,7 @@ namespace tomoto
 
 		virtual int train(size_t iteration, size_t numWorkers, ParallelScheme ps = ParallelScheme::default_, bool freeze_topics = false) = 0;
 		virtual size_t getGlobalStep() const = 0;
-		virtual void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0) = 0;
+		virtual void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0, bool updateStopwords = true) = 0;
 		
 		virtual size_t getK() const = 0;
 		virtual std::vector<Float> getWidsByTopic(size_t tid, bool normalize = true) const = 0;
@@ -437,7 +437,7 @@ namespace tomoto
 			}
 			else
 			{
-				throw exc::InvalidArgument{ "Either `words` or `rawWords` must be filled." };
+				throw exc::EmptyWordArgument{ "Either `words` or `rawWords` must be filled." };
 			}
 			return doc;
 		}
@@ -605,7 +605,7 @@ namespace tomoto
 			return empty;
 		}
 
-		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0) override
+		void prepare(bool initDocs = true, size_t minWordCnt = 0, size_t minWordDf = 0, size_t removeTopN = 0, bool updateStopwords = true) override
 		{
 			auto p = countRealN();
 			realN = p.first;
@@ -674,7 +674,8 @@ namespace tomoto
 			auto state = ps == ParallelScheme::none ? &globalState : localData.data();
 			for (size_t i = 0; i < iteration; ++i)
 			{
-				while (1)
+				size_t retry;
+				for (retry = 0; retry < 10; ++retry)
 				{
 					try
 					{
@@ -703,6 +704,7 @@ namespace tomoto
 						if(ret < 0) return ret;
 					}
 				}
+				if (retry >= 10) return -1;
 				++globalStep;
 			}
 			return 0;

data/vendor/tomotopy/src/Utils/EigenAddonOps.hpp

@@ -56,7 +56,7 @@ namespace Eigen
 	}
 }
 #endif
-#if defined(EIGEN_VECTORIZE_SSE2)
+#ifdef EIGEN_VECTORIZE_SSE2
 #include <xmmintrin.h>
 #include "sse_gamma.h"
 
@@ -91,6 +91,41 @@ namespace Eigen
 	}
 }
 #endif
+#ifdef EIGEN_VECTORIZE_NEON
+#include <arm_neon.h>
+
+
+namespace Eigen
+{
+	namespace internal
+	{
+		template<> struct to_int_packet<Packet4f>
+		{
+			typedef Packet4i type;
+		};
+
+		template<> struct to_float_packet<Packet4i>
+		{
+			typedef Packet4f type;
+		};
+
+		EIGEN_STRONG_INLINE Packet4f p_to_f32(const Packet4i& a)
+		{
+			return vcvtq_f32_s32(a);
+		}
+
+		EIGEN_STRONG_INLINE Packet4f p_bool2float(const Packet4f& a)
+		{
+			return vcvtq_f32_s32(vandq_s32(a, vdupq_n_s32(1)));
+		}
+
+		EIGEN_STRONG_INLINE Packet4f p_bool2float(const Packet4i& a)
+		{
+			return p_bool2float((Packet4f)vreinterpretq_f32_s32((int32x4_t)a));
+		}
+	}
+}
+#endif
 
 namespace Eigen
 {

data/vendor/tomotopy/src/Utils/exception.h

@@ -22,6 +22,12 @@ namespace tomoto
 		public:
 			using std::invalid_argument::invalid_argument;
 		};
+
+		class EmptyWordArgument : public InvalidArgument
+		{
+		public:
+			using InvalidArgument::InvalidArgument;
+		};
 	}
 }
 

data/vendor/tomotopy/src/Utils/sample.hpp

@@ -1,10 +1,8 @@
 #pragma once
 
 #include <random>
-#ifdef __AVX__
+#if defined(__AVX__) || defined(__SSE2__)
 #include <immintrin.h>
-#elif defined(__SSE2__) || defined(_WIN64)
-#include <xmmintrin.h>
 #else
 
 #endif
@@ -23,17 +21,21 @@ namespace tomoto
 			return __popcnt(i);
 		}
 
-#ifdef _WIN64
+	#ifdef _WIN64
 		inline uint64_t log2_ceil(uint64_t i)
 		{
-			return 64 - __lzcnt64(i) - ((i & (i - 1)) == 0 ? 1 : 0);
+			unsigned long idx;
+			if (!_BitScanReverse64(&idx, i)) return 0;
+			return idx + 1 - ((i & (i - 1)) == 0 ? 1 : 0);
 		}
-#else
+	#else
 		inline uint32_t log2_ceil(uint32_t i)
 		{
-			return 32 - __lzcnt(i) - ((i & (i - 1)) == 0 ? 1 : 0);
+			unsigned long idx;
+			if (!_BitScanReverse(&idx, i)) return 0;
+			return idx + 1 - ((i & (i - 1)) == 0 ? 1 : 0);
 		}
-#endif
+	#endif
 
 #else
 		inline uint32_t popcnt(uint32_t i)
@@ -41,22 +43,22 @@ namespace tomoto
 			return __builtin_popcount(i);
 		}
 
-#ifdef __x86_64
+	#ifdef __x86_64
 		inline uint64_t log2_ceil(uint64_t i)
 		{
 			return 64 - __builtin_clzll(i) - ((i & (i - 1)) == 0 ? 1 : 0);
 		}
-#else
+	#else
 		inline uint32_t log2_ceil(uint32_t i)
 		{
 			return 32 - __builtin_clz(i) - ((i & (i - 1)) == 0 ? 1 : 0);
 		}
-#endif
+	#endif
 
 #endif
 
 
-#if defined(__SSE2__) || defined(_WIN64)
+#if defined(__SSE2__)
 		inline __m128 scan_SSE(__m128 x)
 		{
 			x = _mm_add_ps(x, _mm_castsi128_ps(_mm_slli_si128(_mm_castps_si128(x), 4)));

data/vendor/tomotopy/src/Utils/sse_gamma.h

@@ -1,7 +1,4 @@
 #pragma once
-#if _WIN64 || __x86_64__
-#define __SSE2__
-#endif
 #include "sse_mathfun.h"
 
 // approximation : lgamma(z) ~= (z+2.5)ln(z+3) - z - 3 + 0.5 ln (2pi) + 1/12/(z + 3) - ln (z(z+1)(z+2))