fasttext-modern 0.10.0__tar.gz

fasttext_modern-0.10.0/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to fastText Modern are documented here.
+
+## [0.10.0] - 2026-09-02
+
+### Added
+
+- Prebuilt CPython 3.10 through 3.14 wheels for Linux x86-64, Windows x86-64,
+  macOS x86-64, and macOS Apple Silicon.
+- Cross-platform native CMake validation and packaging tests.
+- Trusted Publishing release automation for PyPI.
+
+### Changed
+
+- Renamed the Python distribution to `fasttext-modern`; the import remains
+  `fasttext`.
+- Raised the native compiler baseline to C++17 and CMake 3.16.
+- Made host-specific CPU optimization opt-in for portable binaries.
+- Declared Python 3.10 through 3.14 support and pybind11 3.x builds.
+
+### Fixed
+
+- NumPy 2 compatibility for prediction and evaluation arrays.
+- Windows shared/static library and Python binding portability issues.
+- Default thread selection on single-core systems.
+
+[0.10.0]: https://github.com/Naviden/fastText/releases/tag/v0.10.0

fasttext_modern-0.10.0/CONTRIBUTING.md

@@ -0,0 +1,28 @@
+# Contributing to fastText Modern
+We want to make contributing to this project as easy and transparent as possible.
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is clear and has sufficient instructions to be able to reproduce the issue.
+
+### Reproducing issues
+Please make sure that the issue you mention is not a result of one of the existing third-party libraries. For example, please do not post an issue if you encountered an error within a third-party Python library. We can only help you with errors which can be directly reproduced either with our C++ code or the corresponding Python bindings. If you do find an error, please post detailed steps to reproduce it. If we can't reproduce your error, we can't help you fix it.
+
+## Pull Requests
+For substantial changes, please open an issue before submitting a pull request.
+Small, well-scoped fixes may be submitted directly.
+
+To create a pull request:
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code is formatted consistently with the surrounding code.
+
+## Tests
+First, you will need to make sure you have the required data. For that, please have a look at the fetch_test_data.sh script under tests. Next run the tests using the runtests.py script passing a path to the directory containing the datasets.
+
+## License
+By contributing to fastText Modern, you agree that your contributions will be
+licensed under the repository's MIT License. No separate contributor license
+agreement is required by this fork.

fasttext_modern-0.10.0/FORK.md

@@ -0,0 +1,34 @@
+# About this fork
+
+fastText Modern is an independently maintained fork of
+[facebookresearch/fastText](https://github.com/facebookresearch/fastText). It
+focuses on keeping the native library and Python bindings buildable and easy to
+install on current platforms.
+
+## Identity and compatibility
+
+- The Python distribution is named `fasttext-modern`.
+- The Python import remains `fasttext` for source compatibility.
+- This project is not affiliated with or endorsed by Meta.
+- fastText and Meta names and marks belong to their respective owners. Their
+  use here identifies the upstream project and compatibility target.
+
+## License and attribution
+
+The upstream source is licensed under the MIT License. The original
+[LICENSE](LICENSE), copyright notices, and Git history are preserved. New
+contributions to this repository are made under the same MIT License.
+
+## Models and datasets
+
+This repository and its Python distributions contain source code only. Linked
+pretrained models and datasets are provided by third parties or the upstream
+project and may have separate terms. They are not relicensed or redistributed
+by fastText Modern.
+
+## Maintenance policy
+
+The fork follows upstream bug fixes when they remain compatible with the goals
+of this project. Fork releases use their own version numbers and changelog, and
+changes are tested on the supported Python and native-platform matrix before
+release.

fasttext_modern-0.10.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2016-present, Facebook, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fasttext_modern-0.10.0/MANIFEST.in

@@ -0,0 +1,8 @@
+include LICENSE
+include CHANGELOG.md
+include CONTRIBUTING.md
+include FORK.md
+include SECURITY.md
+
+recursive-include python *.md
+recursive-include src *.cc *.h

fasttext_modern-0.10.0/PKG-INFO

@@ -0,0 +1,379 @@
+Metadata-Version: 2.4
+Name: fasttext-modern
+Version: 0.10.0
+Summary: Community-maintained fastText Python bindings
+Home-page: https://github.com/Naviden/fastText
+Author: Onur Celebi and fastText contributors
+Maintainer: Naviden
+License: MIT
+Project-URL: Source, https://github.com/Naviden/fastText
+Project-URL: Issues, https://github.com/Naviden/fastText/issues
+Project-URL: Changelog, https://github.com/Naviden/fastText/blob/main/CHANGELOG.md
+Project-URL: Upstream, https://github.com/facebookresearch/fastText
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: C++
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Topic :: Software Development
+Classifier: Topic :: Scientific/Engineering
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX
+Classifier: Operating System :: Unix
+Classifier: Operating System :: MacOS
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: maintainer
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# fastText Modern Python bindings
+
+> This is the Python guide for the independently maintained fastText Modern
+> fork. It is not affiliated with or endorsed by Meta. The import remains
+> compatible with upstream fastText. See the
+> [fork policy](https://github.com/Naviden/fastText/blob/main/FORK.md) for
+> attribution and scope.
+
+[fastText](https://fasttext.cc/) is a library for efficient learning of word representations and sentence classification.
+
+In this document we present how to use fastText in python.
+
+## Table of contents
+
+* [Requirements](#requirements)
+* [Installation](#installation)
+* [Usage overview](#usage-overview)
+   * [Word representation model](#word-representation-model)
+   * [Text classification model](#text-classification-model)
+   * [IMPORTANT: Preprocessing data / encoding conventions](#important-preprocessing-data-encoding-conventions)
+   * [More examples](#more-examples)
+* [API](#api)
+   * [`train_unsupervised` parameters](#train_unsupervised-parameters)
+   * [`train_supervised` parameters](#train_supervised-parameters)
+   * [`model` object](#model-object)
+
+
+# Requirements
+
+[fastText](https://fasttext.cc/) builds on Linux, macOS, and Windows.
+It requires a compiler with C\++17 support, [Python](https://www.python.org/)
+3.10 or newer, [NumPy](https://numpy.org/), and
+[pybind11](https://github.com/pybind/pybind11).
+
+
+# Installation
+
+To install the latest release, you can do :
+```bash
+$ python -m pip install fasttext-modern
+```
+
+or, to get the latest development version of fasttext, you can install from our github repository :
+```bash
+$ git clone https://github.com/Naviden/fastText.git
+$ cd fastText
+$ python -m pip install .
+```
+
+# Usage overview
+
+
+## Word representation model
+
+In order to learn word vectors, as [described here](https://fasttext.cc/docs/en/references.html#enriching-word-vectors-with-subword-information), we can use `fasttext.train_unsupervised` function like this:
+
+
+```py
+import fasttext
+
+# Skipgram model :
+model = fasttext.train_unsupervised('data.txt', model='skipgram')
+
+# or, cbow model :
+model = fasttext.train_unsupervised('data.txt', model='cbow')
+
+```
+
+where `data.txt` is a training file containing utf-8 encoded text.
+
+
+The returned `model` object represents your learned model, and you can use it to retrieve information.
+
+```py
+print(model.words)   # list of words in dictionary
+print(model['king']) # get the vector of the word 'king'
+```
+
+
+### Saving and loading a model object
+
+You can save your trained model object by calling the function `save_model`.
+```py
+model.save_model("model_filename.bin")
+```
+
+and retrieve it later thanks to the function `load_model` :
+```py
+model = fasttext.load_model("model_filename.bin")
+```
+
+For more information about word representation usage of fasttext, you can refer to our [word representations tutorial](https://fasttext.cc/docs/en/unsupervised-tutorial.html).
+
+
+## Text classification model
+
+In order to train a text classifier using the method [described here](https://fasttext.cc/docs/en/references.html#bag-of-tricks-for-efficient-text-classification), we can use `fasttext.train_supervised` function like this:
+
+
+```py
+import fasttext
+
+model = fasttext.train_supervised('data.train.txt')
+```
+
+where `data.train.txt` is a text file containing a training sentence per line along with the labels. By default, we assume that labels are words that are prefixed by the string `__label__`
+
+Once the model is trained, we can retrieve the list of words and labels:
+
+```py
+print(model.words)
+print(model.labels)
+```
+
+To evaluate our model by computing the precision at 1 (P@1) and the recall on a test set, we use the `test` function:
+
+```py
+def print_results(N, p, r):
+    print("N\t" + str(N))
+    print("P@{}\t{:.3f}".format(1, p))
+    print("R@{}\t{:.3f}".format(1, r))
+
+print_results(*model.test('test.txt'))
+```
+
+We can also predict labels for a specific text :
+
+```py
+model.predict("Which baking dish is best to bake a banana bread ?")
+```
+
+By default, `predict` returns only one label : the one with the highest probability. You can also predict more than one label by specifying the parameter `k`:
+```py
+model.predict("Which baking dish is best to bake a banana bread ?", k=3)
+```
+
+If you want to predict more than one sentence you can pass an array of strings :
+
+```py
+model.predict(["Which baking dish is best to bake a banana bread ?", "Why not put knives in the dishwasher?"], k=3)
+```
+
+
+Of course, you can also save and load a model to/from a file as [in the word representation usage](#saving-and-loading-a-model-object).
+
+For more information about text classification usage of fasttext, you can refer to our [text classification tutorial](https://fasttext.cc/docs/en/supervised-tutorial.html).
+
+
+
+
+### Compress model files with quantization
+
+When you want to save a supervised model file, fastText can compress it in order to have a much smaller model file by sacrificing only a little bit performance.
+
+```py
+# with the previously trained `model` object, call :
+model.quantize(input='data.train.txt', retrain=True)
+
+# then display results and save the new model :
+print_results(*model.test(valid_data))
+model.save_model("model_filename.ftz")
+```
+
+`model_filename.ftz` will have a much smaller size than `model_filename.bin`.
+
+For further reading on quantization, you can refer to [this paragraph from our blog post](https://fasttext.cc/blog/2017/10/02/blog-post.html#model-compression).
+
+
+## IMPORTANT: Preprocessing data / encoding conventions
+
+In general it is important to properly preprocess your data. In particular our example scripts in the [root folder](https://github.com/Naviden/fastText) do this.
+
+fastText assumes UTF-8 encoded text. All text must be Python `str` values. The
+passed text is [encoded as UTF-8 by pybind11](https://pybind11.readthedocs.io/en/stable/advanced/cast/strings.html)
+before it is passed to the fastText C++ library. This means it is important to
+use UTF-8 encoded text when building a model. On Unix-like systems you can
+convert text using [iconv](https://en.wikipedia.org/wiki/Iconv).
+
+fastText will tokenize (split text into pieces) based on the following ASCII characters (bytes). In particular, it is not aware of UTF-8 whitespace. We advice the user to convert UTF-8 whitespace / word boundaries into one of the following symbols as appropiate.
+
+* space
+* tab
+* vertical tab
+* carriage return
+* formfeed
+* the null character
+
+The newline character is used to delimit lines of text. In particular, the EOS token is appended to a line of text if a newline character is encountered. The only exception is if the number of tokens exceeds the MAX\_LINE\_SIZE constant as defined in the [Dictionary header](https://github.com/Naviden/fastText/blob/main/src/dictionary.h). This means if you have text that is not separate by newlines, such as the [fil9 dataset](http://mattmahoney.net/dc/textdata), it will be broken into chunks with MAX\_LINE\_SIZE of tokens and the EOS token is not appended.
+
+The length of a token is the number of UTF-8 characters by considering the [leading two bits of a byte](https://en.wikipedia.org/wiki/UTF-8#Description) to identify [subsequent bytes of a multi-byte sequence](https://github.com/Naviden/fastText/blob/main/src/dictionary.cc). Knowing this is especially important when choosing the minimum and maximum length of subwords. Further, the EOS token (as specified in the [Dictionary header](https://github.com/Naviden/fastText/blob/main/src/dictionary.h)) is considered a character and will not be broken into subwords.
+
+## More examples
+
+In order to have a better knowledge of fastText models, please consider the main [README](https://github.com/Naviden/fastText/blob/main/README.md) and the [upstream tutorials](https://fasttext.cc/docs/en/supervised-tutorial.html).
+
+You can find further python examples in [the doc folder](https://github.com/Naviden/fastText/tree/main/python/doc/examples).
+
+As with any package you can get help on any Python function using the help function.
+
+For example
+
+```
++>>> import fasttext
++>>> help(fasttext.FastText)
+
+Help on module fasttext.FastText in fasttext:
+
+NAME
+    fasttext.FastText
+
+DESCRIPTION
+    # Copyright (c) 2017-present, Facebook, Inc.
+    # All rights reserved.
+    #
+    # This source code is licensed under the MIT license found in the
+    # LICENSE file in the root directory of this source tree.
+
+FUNCTIONS
+    load_model(path)
+        Load a model given a filepath and return a model object.
+
+    tokenize(text)
+        Given a string of text, tokenize it and return a list of tokens
+[...]
+```
+
+
+# API
+
+
+## `train_unsupervised` parameters
+
+```python
+    input             # training file path (required)
+    model             # unsupervised fasttext model {cbow, skipgram} [skipgram]
+    lr                # learning rate [0.05]
+    dim               # size of word vectors [100]
+    ws                # size of the context window [5]
+    epoch             # number of epochs [5]
+    minCount          # minimal number of word occurences [5]
+    minn              # min length of char ngram [3]
+    maxn              # max length of char ngram [6]
+    neg               # number of negatives sampled [5]
+    wordNgrams        # max length of word ngram [1]
+    loss              # loss function {ns, hs, softmax, ova} [ns]
+    bucket            # number of buckets [2000000]
+    thread            # number of threads [number of cpus]
+    lrUpdateRate      # change the rate of updates for the learning rate [100]
+    t                 # sampling threshold [0.0001]
+    verbose           # verbose [2]
+```
+
+## `train_supervised` parameters
+
+```python
+    input             # training file path (required)
+    lr                # learning rate [0.1]
+    dim               # size of word vectors [100]
+    ws                # size of the context window [5]
+    epoch             # number of epochs [5]
+    minCount          # minimal number of word occurences [1]
+    minCountLabel     # minimal number of label occurences [1]
+    minn              # min length of char ngram [0]
+    maxn              # max length of char ngram [0]
+    neg               # number of negatives sampled [5]
+    wordNgrams        # max length of word ngram [1]
+    loss              # loss function {ns, hs, softmax, ova} [softmax]
+    bucket            # number of buckets [2000000]
+    thread            # number of threads [number of cpus]
+    lrUpdateRate      # change the rate of updates for the learning rate [100]
+    t                 # sampling threshold [0.0001]
+    label             # label prefix ['__label__']
+    verbose           # verbose [2]
+    pretrainedVectors # pretrained word vectors (.vec file) for supervised learning []
+```
+
+## `model` object
+
+`train_supervised`, `train_unsupervised` and `load_model` functions return an instance of `_FastText` class, that we generaly name `model` object.
+
+This object exposes those training arguments as properties : `lr`, `dim`, `ws`, `epoch`, `minCount`, `minCountLabel`, `minn`, `maxn`, `neg`, `wordNgrams`, `loss`, `bucket`, `thread`, `lrUpdateRate`, `t`, `label`, `verbose`, `pretrainedVectors`. So `model.wordNgrams` will give you the max length of word ngram used for training this model.
+
+In addition, the object exposes several functions :
+
+```python
+    get_dimension           # Get the dimension (size) of a lookup vector (hidden layer).
+                            # This is equivalent to `dim` property.
+    get_input_vector        # Given an index, get the corresponding vector of the Input Matrix.
+    get_input_matrix        # Get a copy of the full input matrix of a Model.
+    get_labels              # Get the entire list of labels of the dictionary
+                            # This is equivalent to `labels` property.
+    get_line                # Split a line of text into words and labels.
+    get_output_matrix       # Get a copy of the full output matrix of a Model.
+    get_sentence_vector     # Given a string, get a single vector represenation. This function
+                            # assumes to be given a single line of text. We split words on
+                            # whitespace (space, newline, tab, vertical tab) and the control
+                            # characters carriage return, formfeed and the null character.
+    get_subword_id          # Given a subword, return the index (within input matrix) it hashes to.
+    get_subwords            # Given a word, get the subwords and their indicies.
+    get_word_id             # Given a word, get the word id within the dictionary.
+    get_word_vector         # Get the vector representation of word.
+    get_words               # Get the entire list of words of the dictionary
+                            # This is equivalent to `words` property.
+    is_quantized            # whether the model has been quantized
+    predict                 # Given a string, get a list of labels and a list of corresponding probabilities.
+    quantize                # Quantize the model reducing the size of the model and it's memory footprint.
+    save_model              # Save the model to the given path
+    test                    # Evaluate supervised model using file given by path
+    test_label              # Return the precision and recall score for each label.    
+```
+
+The properties `words`, `labels` return the words and labels from the dictionary :
+```py
+model.words         # equivalent to model.get_words()
+model.labels        # equivalent to model.get_labels()
+```
+
+The object overrides `__getitem__` and `__contains__` functions in order to return the representation of a word and to check if a word is in the vocabulary.
+
+```py
+model['king']       # equivalent to model.get_word_vector('king')
+'king' in model     # equivalent to `'king' in model.get_words()`
+```
+
+
+Support and community
+---------------------
+
+- [fastText Modern issues](https://github.com/Naviden/fastText/issues)
+- [Upstream fastText](https://github.com/facebookresearch/fastText)
+- [Stack Overflow](https://stackoverflow.com/questions/tagged/fasttext)
+- [Upstream Google group](https://groups.google.com/forum/#!forum/fasttext-library)