rababa 0.1.0 → 0.1.1

data/models-data/README.md

@@ -1,6 +0,0 @@
-### model data dir
-
-contains:
-	ONNX data
-	Pickle sample data
-

data/python/README.md

@@ -1,163 +0,0 @@
-# Diacritization Model
-
-## Try out Rababa
-* download the torch model under /Assets at [releases](https://github.com/secryst/rababa-models/releases)
-* Put the model under python/log_dir/CA_MSA.base.cbhg/models/2000000-snapshot.pt
-* single sentences and text can now be diacritized as below:
-```bash
-python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text 'قطر'
-python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text_file relative_path_to_text_file
-```
-The maximal string length is set in configs at 600.
-Larger line will need to be breaken down, for instance using the library introduced in the Ruby Try out section: ../lib/README.md
-
-## Core: Python Deep Learning models for recovering Arabic language diacritics
-
-We are referring here to the [code](https://github.com/almodhfer/Arabic_Diacritization) and
-[Effective Deep Learning Models for Automatic Diacritization of Arabic Text](https://ieeexplore.ieee.org/document/9274427)
-that we have selected for this project from a list of alternatives listed in the
-docs readme.
-
-Out of the four models that [almodhfer](https://github.com/almodhfer) has
-implemented, we selected the simplest and most performant ones:
-
-- The baseline model (`baseline`): consists of 3 bidirectional LSTM layers with
-  optional batch norm layers.
-
-- The CBHG model (`cbhg`): uses only the encoder of the Tacotron-based model with
-  optional post LSTM, and batch norm layers.
-
-### Python Version & Dependencies
-
-- version: 3.6
-- dependencies:
-```bash
-pip install -r requirement.txt
-```
-
-### Datasets
-
-- We have chosen the Tashkeela corpus ~2800000 sentences:
-    * [sourceforge](https://sourceforge.net/projects/tashkeela-processed/)
-
-Other datasets are discussed in the reviewed literature or in the article referenced above.
-
-```bash
-    mkdir data
-    mkdir data/CA_MSA
-```
-
-For training, data need to be in format:
-
-```bash
-    > ls data/CA_MSA/*
-        --> data/CA_MSA/eval.csv  data/CA_MSA/train.csv  data/CA_MSA/test.csv
-```
-
-For instance:
-
-```bash
-unzip data.zip
-for d in `ls tashkeela_val/*`; do; cat $d >> data/CA_MSA/eval.csv; done
-for d in `ls tashkeela_train/*`; do; cat $d >> data/CA_MSA/train.csv; done
-for d in `ls tashkeela_test/*`; do; cat $d >> data/CA_MSA/test.csv; done
-```
-
-### Load Model
-
-Alternatively, trained CBHG models are available under
-[releases](https://github.com/secryst/rababa-models).
-Models are to be copied as specified in the link just above under:
-> log_dir/CA_MSA.base.cbhg/models/2000000-snapshot.pt
-
-
-### Config Files
-
-One can adjust the model configurations in the `/config` repository.
-
-The model configurations are about the layers but also the dataset to be used
-and various other options.
-
-The configuration files are called explicitly in the below applications.
-
-### Data Preprocessing
-
-The original work cited above allow for both raw and preprocessed.
-
-We go for the simplest raw version here:
-- As mentioned above, corpus must have test.csv, train.csv, and valid.csv.
-
-- Specify that the data is not preprocessed in the config.
-  In that case, each batch will be processed and the text and diacritics
-  will be extracted from the original text.
-
-- You also have to specify the text encoder and the cleaner functions.
-  Two text encoders were included: BasicArabicEncoder, ArabicEncoderWithStartSymbol.
-
-  Moreover, we have one cleaning function: valid_arabic_cleaners, which clean
-  all characters except valid Arabic characters, which include Arabic letters,
-  punctuations, and diacritics.
-
-### Training
-
-All models config are placed in the config directory.
-
-```bash
-python train.py --model model_name --config config/config_name.yml
-```
-
-The model will report the WER and DER while training using the
-diacritization_evaluation package. The frequency of calculating WER and
-DER can be specified in the config file.
-
-### Testing
-
-The testing is done in the same way as the training,
-For instance, with the CBHG model on the data in `/data/CA_MSA/test.csv`:
-
-```bash
-python test.py --model 'cbhg' --config config/cbhg.yml
-```
-
-The model will load the last saved model unless you specified it in the config:
-`test_data_path`. The test file is expected to have the correct diacritization!
-
-If the test file name is different than `test.csv`, you
-can add it to the `config: test_file_name`.
-
-### "Diacritize" Text or Files
-
-Single sentences or files can be processed. The code outputs is the diacritized
-text or lines.
-
-```bash
-python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text 'قطر'
-python diacritize.py --model_kind "cbhg" --config config/cbhg.yml --text_file relative_path_to_text_file
-```
-
-
-### Convert CBHG, python model to ONNX
-
-The last model stored during training is automatically chosen and the ONNX model
-is saved into a hardcoded location: `../models-data/diacritization_model.onnx`
-
-#### Run
-
-```bash
-python diacritization_model_to_onnx.py
-```
-
-#### Important parameters
-
-They are hardcoded in the beginning of the script:
-
-* `max_len`:
-	* match string length, initial model value is given in config.
-	* this param allows tuning the model speed and size!
-	* the Ruby ../lib/README.md points to resources for preprocessing
-
-* batch_size:
-	* the value is given by the original model and its training.
-	* this constrain how the ONNX model can be put in production:
-		1. if > 1, single lines involve redundant computations
-		2. if > 1, files are processed in batches.

data/python/log_dir/CA_MSA.base.cbhg/models/Readme.md

@@ -1,2 +0,0 @@
-#### Put model trained with CA_MSA here:
-2000000-snapshot.pt 

data/python/log_dir/README.md

@@ -1 +0,0 @@
-### Model storage directory for training and inference