fasttext 0.1.3 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fb6649e0a3992c6e12572d672e8ba768220662efc37b982c278a8d0713716029
-  data.tar.gz: 12a0441cf1030bfbfe99d26824fe757d5b83c2f47de899fecf13a73aa657bd76
+  metadata.gz: 1bb25234ba16e6af2a30087522d498984a933274d24719be58cd348f6f456c11
+  data.tar.gz: 2ec68073ce633dae21077be74264f686153abff5431d850c5cf87aa26304b299
 SHA512:
-  metadata.gz: dadcbc9a0d39468b4d1070cbf74abece85e687fe2d5de9dfb03c8e79f630b47a0ce37bf73f1383bccf917190bf4f1a2acb12e4741b4aaf45d0c688991baf7893
-  data.tar.gz: b9615f2edf557b7a2bb9c90457581489dd86be7539e7f4e9b75ee72cb3fe26ca9a216c5ecda4ff6e82ef348fbcc0482cd4e727e9c6652f2b38195ad3edba1752
+  metadata.gz: '058233f6d6dcfc698b4e00f940d927962c28bef1aa51048ffdfb969accadcaf0616fbc3353c56c35e6609f1324c5d2ba469c4ba242b8a87daf60fd2b1fa7a35b'
+  data.tar.gz: e2bf6042ddc218398584b399acdde2ad89ece74e3009ac5348981e4ce958057233b10b5a5a504cb2e316c4a03fb4595fea524e3af0f30d787a1fb689885f6d5d

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.2.0 (2021-05-17)
+
+- Updated to Rice 4
+- Dropped support for Ruby < 2.6
+
 ## 0.1.3 (2020-04-28)
 
 - Updated fastText to 0.9.2

data/LICENSE.txt

@@ -1,22 +1,22 @@
-Copyright (c) 2019-2020 Andrew Kane
-
 MIT License
 
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
+Copyright (c) 2016-present, Facebook, Inc.
+Copyright (c) 2019-2021 Andrew Kane
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -2,7 +2,7 @@
 
 [fastText](https://fasttext.cc) - efficient text classification and representation learning - for Ruby
 
-[![Build Status](https://travis-ci.org/ankane/fasttext.svg?branch=master)](https://travis-ci.org/ankane/fasttext) [![Build status](https://ci.appveyor.com/api/projects/status/67yby3w6mth766y9/branch/master?svg=true)](https://ci.appveyor.com/project/ankane/fasttext/branch/master)
+[![Build Status](https://github.com/ankane/fastText/workflows/build/badge.svg?branch=master)](https://github.com/ankane/fastText/actions)
 
 ## Installation
 
@@ -127,6 +127,12 @@ Get a word vector
 model.word_vector("carrot")
 ```
 
+Get a sentence vector
+
+```ruby
+model.sentence_vector("sentence text")
+```
+
 Get words
 
 ```ruby
@@ -172,7 +178,7 @@ FastText::Classifier.new(
   thread: 3,                  # number of threads
   lr_update_rate: 100,        # change the rate of updates for the learning rate
   t: 0.0001,                  # sampling threshold
-  label_prefix: "__label__"   # label prefix
+  label_prefix: "__label__",  # label prefix
   verbose: 2,                 # verbose
   pretrained_vectors: nil,    # pretrained word vectors (.vec file)
   autotune_metric: "f1",      # autotune optimization metric
@@ -232,7 +238,7 @@ __label__spam text from document three
 
 ## Pretrained Models
 
-There are a number of [pretrained models](https://fasttext.cc/docs/en/english-vectors.html) you can download
+There are a number of [pretrained models](https://fasttext.cc/docs/en/supervised-models.html) you can download
 
 ### Language Identification
 
@@ -248,15 +254,6 @@ Get language predictions
 model.predict("bon appétit")
 ```
 
-## rbenv
-
-This library uses [Rice](https://github.com/jasonroelofs/rice) to interface with the fastText C++ library. Rice and earlier versions of rbenv don’t play nicely together. If you encounter an error during installation, upgrade ruby-build and reinstall your Ruby version.
-
-```sh
-brew upgrade ruby-build
-rbenv install [version]
-```
-
 ## History
 
 View the [changelog](https://github.com/ankane/fasttext/blob/master/CHANGELOG.md)
@@ -273,7 +270,7 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 To get started with development:
 
 ```sh
-git clone https://github.com/ankane/fastText.git
+git clone --recursive https://github.com/ankane/fastText.git
 cd fastText
 bundle install
 bundle exec rake compile

data/ext/fasttext/ext.cpp

@@ -13,10 +13,8 @@
 #include <vector.h>
 
 // rice
-#include <rice/Array.hpp>
-#include <rice/Constructor.hpp>
-#include <rice/Data_Type.hpp>
-#include <rice/Hash.hpp>
+#include <rice/rice.hpp>
+#include <rice/stl.hpp>
 
 using fasttext::FastText;
 
@@ -29,18 +27,24 @@ using Rice::define_class_under;
 using Rice::define_module;
 using Rice::define_module_under;
 
-template<>
-inline
-Object to_ruby<std::vector<std::pair<fasttext::real, std::string>>>(std::vector<std::pair<fasttext::real, std::string>> const & x)
+namespace Rice::detail
 {
-  Array ret;
-  for (const auto& v : x) {
-    Array a;
-    a.push(v.first);
-    a.push(v.second);
-    ret.push(a);
-  }
-  return ret;
+  template<>
+  class To_Ruby<std::vector<std::pair<fasttext::real, std::string>>>
+  {
+  public:
+    VALUE convert(std::vector<std::pair<fasttext::real, std::string>> const & x)
+    {
+      Array ret;
+      for (const auto& v : x) {
+        Array a;
+        a.push(v.first);
+        a.push(v.second);
+        ret.push(a);
+      }
+      return ret;
+    }
+  };
 }
 
 fasttext::Args buildArgs(Hash h) {
@@ -52,33 +56,33 @@ fasttext::Args buildArgs(Hash h) {
 
   for(; it != end; ++it)
   {
-    std::string name = from_ruby<std::string>(it->key.to_s());
-    Object value = it->value;
+    std::string name = it->key.to_s().str();
+    VALUE value = (it->value).value();
 
     if (name == "input") {
-      a.input = from_ruby<std::string>(value);
+      a.input = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "output") {
-      a.output = from_ruby<std::string>(value);
+      a.output = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "lr") {
-      a.lr = from_ruby<double>(value);
+      a.lr = Rice::detail::From_Ruby<double>().convert(value);
     } else if (name == "lr_update_rate") {
-      a.lrUpdateRate = from_ruby<int>(value);
+      a.lrUpdateRate = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "dim") {
-      a.dim = from_ruby<int>(value);
+      a.dim = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "ws") {
-      a.ws = from_ruby<int>(value);
+      a.ws = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "epoch") {
-      a.epoch = from_ruby<int>(value);
+      a.epoch = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "min_count") {
-      a.minCount = from_ruby<int>(value);
+      a.minCount = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "min_count_label") {
-      a.minCountLabel = from_ruby<int>(value);
+      a.minCountLabel = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "neg") {
-      a.neg = from_ruby<int>(value);
+      a.neg = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "word_ngrams") {
-      a.wordNgrams = from_ruby<int>(value);
+      a.wordNgrams = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "loss") {
-      std::string str = from_ruby<std::string>(value);
+      std::string str = Rice::detail::From_Ruby<std::string>().convert(value);
       if (str == "softmax") {
         a.loss = fasttext::loss_name::softmax;
       } else if (str == "ns") {
@@ -91,7 +95,7 @@ fasttext::Args buildArgs(Hash h) {
         throw std::invalid_argument("Unknown loss: " + str);
       }
     } else if (name == "model") {
-      std::string str = from_ruby<std::string>(value);
+      std::string str = Rice::detail::From_Ruby<std::string>().convert(value);
       if (str == "supervised") {
         a.model = fasttext::model_name::sup;
       } else if (str == "skipgram") {
@@ -102,35 +106,35 @@ fasttext::Args buildArgs(Hash h) {
         throw std::invalid_argument("Unknown model: " + str);
       }
     } else if (name == "bucket") {
-      a.bucket = from_ruby<int>(value);
+      a.bucket = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "minn") {
-      a.minn = from_ruby<int>(value);
+      a.minn = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "maxn") {
-      a.maxn = from_ruby<int>(value);
+      a.maxn = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "thread") {
-      a.thread = from_ruby<int>(value);
+      a.thread = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "t") {
-      a.t = from_ruby<double>(value);
+      a.t = Rice::detail::From_Ruby<double>().convert(value);
     } else if (name == "label_prefix") {
-      a.label = from_ruby<std::string>(value);
+      a.label = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "verbose") {
-      a.verbose = from_ruby<int>(value);
+      a.verbose = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "pretrained_vectors") {
-      a.pretrainedVectors = from_ruby<std::string>(value);
+      a.pretrainedVectors = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "save_output") {
-      a.saveOutput = from_ruby<bool>(value);
+      a.saveOutput = Rice::detail::From_Ruby<bool>().convert(value);
     } else if (name == "seed") {
-      a.seed = from_ruby<int>(value);
+      a.seed = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "autotune_validation_file") {
-      a.autotuneValidationFile = from_ruby<std::string>(value);
+      a.autotuneValidationFile = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "autotune_metric") {
-      a.autotuneMetric = from_ruby<std::string>(value);
+      a.autotuneMetric = Rice::detail::From_Ruby<std::string>().convert(value);
     } else if (name == "autotune_predictions") {
-      a.autotunePredictions = from_ruby<int>(value);
+      a.autotunePredictions = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "autotune_duration") {
-      a.autotuneDuration = from_ruby<int>(value);
+      a.autotuneDuration = Rice::detail::From_Ruby<int>().convert(value);
     } else if (name == "autotune_model_size") {
-      a.autotuneModelSize = from_ruby<std::string>(value);
+      a.autotuneModelSize = Rice::detail::From_Ruby<std::string>().convert(value);
     } else {
       throw std::invalid_argument("Unknown argument: " + name);
     }
@@ -148,7 +152,7 @@ void Init_ext()
     .define_constructor(Constructor<FastText>())
     .define_method(
       "words",
-      *[](FastText& m) {
+      [](FastText& m) {
         std::shared_ptr<const fasttext::Dictionary> d = m.getDictionary();
         std::vector<int64_t> freq = d->getCounts(fasttext::entry_type::word);
 
@@ -166,7 +170,7 @@ void Init_ext()
       })
     .define_method(
       "labels",
-      *[](FastText& m) {
+      [](FastText& m) {
         std::shared_ptr<const fasttext::Dictionary> d = m.getDictionary();
         std::vector<int64_t> freq = d->getCounts(fasttext::entry_type::label);
 
@@ -184,7 +188,7 @@ void Init_ext()
       })
     .define_method(
       "test",
-      *[](FastText& m, const std::string filename, int32_t k) {
+      [](FastText& m, const std::string& filename, int32_t k) {
         std::ifstream ifs(filename);
         if (!ifs.is_open()) {
           throw std::invalid_argument("Test file cannot be opened!");
@@ -201,17 +205,21 @@ void Init_ext()
       })
     .define_method(
       "load_model",
-      *[](FastText& m, std::string s) { m.loadModel(s); })
+      [](FastText& m, const std::string& s) {
+        m.loadModel(s);
+      })
     .define_method(
       "save_model",
-      *[](FastText& m, std::string s) { m.saveModel(s); })
+      [](FastText& m, const std::string& s) {
+        m.saveModel(s);
+      })
     .define_method("dimension", &FastText::getDimension)
     .define_method("quantized?", &FastText::isQuant)
     .define_method("word_id", &FastText::getWordId)
     .define_method("subword_id", &FastText::getSubwordId)
     .define_method(
       "predict",
-      *[](FastText& m, const std::string text, int32_t k, float threshold) {
+      [](FastText& m, const std::string& text, int32_t k, float threshold) {
         std::stringstream ioss(text);
         std::vector<std::pair<fasttext::real, std::string>> predictions;
         m.predictLine(ioss, predictions, k, threshold);
@@ -219,14 +227,14 @@ void Init_ext()
       })
     .define_method(
       "nearest_neighbors",
-      *[](FastText& m, const std::string& word, int32_t k) {
+      [](FastText& m, const std::string& word, int32_t k) {
         return m.getNN(word, k);
       })
     .define_method("analogies", &FastText::getAnalogies)
-    .define_method("ngram_vectors", &FastText::getNgramVectors)
+    // .define_method("ngram_vectors", &FastText::getNgramVectors)
     .define_method(
       "word_vector",
-      *[](FastText& m, const std::string word) {
+      [](FastText& m, const std::string& word) {
         int dimension = m.getDimension();
         fasttext::Vector vec = fasttext::Vector(dimension);
         m.getWordVector(vec, word);
@@ -239,7 +247,7 @@ void Init_ext()
       })
     .define_method(
       "subwords",
-      *[](FastText& m, const std::string word) {
+      [](FastText& m, const std::string& word) {
         std::vector<std::string> subwords;
         std::vector<int32_t> ngrams;
         std::shared_ptr<const fasttext::Dictionary> d = m.getDictionary();
@@ -253,7 +261,7 @@ void Init_ext()
       })
     .define_method(
       "sentence_vector",
-      *[](FastText& m, const std::string text) {
+      [](FastText& m, const std::string& text) {
         std::istringstream in(text);
         int dimension = m.getDimension();
         fasttext::Vector vec = fasttext::Vector(dimension);
@@ -267,7 +275,7 @@ void Init_ext()
       })
     .define_method(
       "train",
-      *[](FastText& m, Hash h) {
+      [](FastText& m, Hash h) {
         auto a = buildArgs(h);
         if (a.hasAutotune()) {
           fasttext::Autotune autotune(std::shared_ptr<fasttext::FastText>(&m, [](fasttext::FastText*) {}));
@@ -278,17 +286,17 @@ void Init_ext()
       })
     .define_method(
       "quantize",
-      *[](FastText& m, Hash h) {
+      [](FastText& m, Hash h) {
         m.quantize(buildArgs(h));
       })
     .define_method(
       "supervised?",
-      *[](FastText& m) {
+      [](FastText& m) {
         return m.getArgs().model == fasttext::model_name::sup;
       })
     .define_method(
       "label_prefix",
-      *[](FastText& m) {
+      [](FastText& m) {
         return m.getArgs().label;
       });
 }

data/ext/fasttext/extconf.rb

@@ -1,8 +1,7 @@
 require "mkmf-rice"
 
-# TODO use -std=c++14 when available
 # -pthread and -O3 set by default
-$CXXFLAGS << " -std=c++11 -funroll-loops " << with_config("optflags", "-march=native")
+$CXXFLAGS << " -std=c++17 -funroll-loops " << with_config("optflags", "-march=native")
 
 ext = File.expand_path(".", __dir__)
 fasttext = File.expand_path("../../vendor/fastText/src", __dir__)

data/lib/fasttext/version.rb

@@ -1,3 +1,3 @@
 module FastText
-  VERSION = "0.1.3"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fasttext
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.2.0
 platform: ruby
 authors:
 - Andrew Kane
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-04-28 00:00:00.000000000 Z
+date: 2021-05-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rice
@@ -16,72 +16,16 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2.2'
+        version: 4.0.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2.2'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake-compiler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: minitest
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '5'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '5'
-description: 
-email: andrew@chartkick.com
+        version: 4.0.2
+description:
+email: andrew@ankane.org
 executables: []
 extensions:
 - ext/fasttext/extconf.rb
@@ -131,7 +75,7 @@ homepage: https://github.com/ankane/fastText
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -139,15 +83,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.4'
+      version: '2.6'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.2
-signing_key: 
+rubygems_version: 3.2.3
+signing_key:
 specification_version: 4
 summary: fastText - efficient text classification and representation learning - for
   Ruby