BatchalignHK 0.7.17.post13__tar.gz

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/PKG-INFO

@@ -0,0 +1,311 @@
+Metadata-Version: 2.2
+Name: BatchalignHK
+Version: 0.7.17.post13
+Summary: Python Speech Language Sample Analysis
+Author: Brian MacWhinney, Houjun Liu
+Author-email: macw@cmu.edu, houjun@cmu.edu
+Classifier: Development Status :: 3 - Alpha
+Classifier: Topic :: Utilities
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.4
+Requires-Dist: nltk>=3.8
+Requires-Dist: praatio<6.1.0,>=6.0.0
+Requires-Dist: torch>=2.6.0
+Requires-Dist: torchaudio
+Requires-Dist: hmmlearn==0.3.0
+Requires-Dist: eyed3
+Requires-Dist: pydub
+Requires-Dist: imblearn
+Requires-Dist: plotly>=5.3.0
+Requires-Dist: transformers>=4.37
+Requires-Dist: tokenizers>=0.14.1
+Requires-Dist: pycountry>=22.3
+Requires-Dist: stanza[transformers]>=1.10.1
+Requires-Dist: scipy~=1.11
+Requires-Dist: rev_ai>=2.18.0
+Requires-Dist: rich~=13.6
+Requires-Dist: click~=8.1
+Requires-Dist: matplotlib<4.0.0,>=3.8.0
+Requires-Dist: pyfiglet==1.0.2
+Requires-Dist: setuptools
+Requires-Dist: soundfile~=0.12.0
+Requires-Dist: rich-click>=1.7.0
+Requires-Dist: typing-extensions
+Requires-Dist: num2words
+Requires-Dist: tiktoken
+Requires-Dist: blobfile
+Requires-Dist: sentencepiece
+Requires-Dist: tencentcloud-sdk-python-common
+Requires-Dist: tencentcloud-sdk-python-asr
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Provides-Extra: train
+Requires-Dist: accelerate~=0.27; extra == "train"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: summary
+
+# TalkBank | Batchalign2
+
+Welcome! **Batchalign2** is a Python suite of language sample analysis (LSA) software from the TalkBank project. It is used to interact with conversation audio files and their transcripts, and provides a whole host of analyses within this space.
+
+The TalkBank Project, of which Batchalign is a part, is supported by NIH grant HD082736.
+
+----
+
+## Quick Start
+
+The following instructions provide a quick start to installing Batchalign. For most users aiming to process CHAT and audio with Batchalign, we recommend more detailed usage instructions: for [usage](https://talkbank.org/info/BA2-usage.pdf) and [human transcript cleanup](https://talkbank.org/info/BA2-cleanup.pdf). The following provides a quick start guide for the program.
+
+### Get Python
+- We support Python versions 3.9, 3.10, 3.11 and 3.12.  
+- First, check to see if you have Python by running `python`. If it reports any of the versions above, skip the following step.
+- To install Python, follow the instructions...
+    -  for macOS
+        1. Install Brew: [visit this link](https://brew.sh/)
+        2. Install Python and Pipx: execute `brew install python@3.11 pipx`
+        3. Copy the pipx path to yours: execute `pipx ensurepath` 
+    -  for Windows
+        1.  Install Python 3.11: [via this link](https://www.python.org/ftp/python/3.11.7/python-3.11.7-amd64.exe) 
+        2.  If later commands report `pip module not found`, [this page may help](https://github.com/TalkBank/batchalign2/wiki/Troubleshooting-Tips#get-pip-on-windows)
+    -  your distribution's instructions for Linux
+    
+For first-time users of Python, note that if you didn't install Python 3.11 (as we recommended above), it may be complex to change Python versions downstream and may cause additional problems. We recommend explicitly installing Python 3.11 by installing it explicitly via specifying a version number as we show above.
+
+### Install and Update the Package
+You can get Batchalign from PyPi, and you can update the package in the same way:
+
+#### macOS
+
+Since Batchalign is a command line program, we recommend using `pipx`, the newly standard environment-based Python program installer, on macOS:
+
+```
+pipx install batchalign --python $(brew --prefix)/bin/python3.11
+```
+
+#### Linux
+
+```
+pip install -U batchalign
+```
+
+#### Windows
+
+```
+py -m pip install -U batchalign
+```
+
+Note that if your system reports `pip: command not found`, replace every use of `pip` in the instructions with `pip3` and try again.
+
+### Rock and Roll
+There are two main ways of interacting with Batchalign. Batchalign can be used as a program to batch-process CHAT (hence the name), or as a Python LSA library.
+
+- to get started with the Batchalign program, [tap here](#quick-start-command-line)
+- to get started on the Batchalign Library (assumes familiarity with Python), [tap here](#quick-start-python)
+
+## Quick Start: Command Line
+
+### Basic Usage 
+
+Once installed, you can invoke the Batchalign program by typing `batchalign` into the Terminal (MacOS) or Command Prompt (Windows).
+
+It is used in the following basic way:
+
+```
+batchalign [verb] [input_dir] [output_dir]
+```
+
+Where `verb` includes:
+
+1. `transcribe` - by placing only an audio of video file (`.mp3/.mp4/.wav`) in the input directory, this function performs ASR on the audio, diarizes utterances, identifies some basic conversational features like retracing and filled pauses, and generates word-level alignments. You must supply a language code flag: `--lang=[three letter ISO language code]` for the ASR system to know what language the transcript is in. You can choose the flags `--rev` to use Rev.AI, a commercial ASR service, or `--whisper`, to use a local copy of OpenAI Whisper.
+2. `align` - by placing both an audio of video file (`.mp3/.mp4/.wav`) and an *utterance-aligned* CHAT file in the input directory, this function recovers utterance-level time alignments (if they are not already annotated) and generates word-level alignments. The @Languages header in the CHAT file tells the program which language is in the transcript.
+3. `morphotag` - by placing a CHAT file in the input directory, this function uses Stanford NLP Stanza to generate morphological and dependency analyses. The @Languages header in the CHAT file tells the program which language is in the transcript. You must supply a language code flag: `--lang=[three letter ISO language code]` for the alignment system to know what language the transcript is in. 
+<!-- 4. `bulletize` - placing both an audio of video file (`.mp3/.mp4/.wav`) and an *unlinked* CHAT file in the input directory, generate utterance-level alignments through ASR -->
+
+You can get a CHAT transcript to experiment with [at the TalkBank website](https://talkbank.org/), under any of the "Banks" that are available. You can also generate and parse a CHAT transcript via [the Python program](https://github.com/TalkBank/batchalign2?tab=readme-ov-file#chat).
+
+### Sample Commands
+For input files (CHAT and audio for `align`, CHAT only for `morphotag`, and audio only for `transcribe`), located in `~/ba_input` dumping the output to `~/ba_output`, one could write:
+
+#### ASR + Segmentation
+
+```
+batchalign transcribe --lang=eng ~/ba_input ~/ba_output
+```
+
+#### morphosyntactic analysis
+
+```
+batchalign morphotag ~/ba_input ~/ba_output
+```
+
+#### forced alignment
+
+```
+batchalign align ~/ba_input ~/ba_output
+```
+
+
+-----
+
+Follow instructions from
+
+```
+batchalign --help
+```
+
+and 
+
+```
+batchalign [verb] --help
+```
+
+to learn more about other options.
+
+### Verbosity
+
+Placing one or multiple `-v` *behind the word `batchalign`* (i.e. behind the `[verb]` will not work) increases the verbosity of Batchalign. The default mode and one `-v` will use the normal Batchalign interface, whereas Batchalign with more than 1 `-v` will switch to the text-based "logging" interface.
+
+For instance, here is the instruction for running Batchalign to perform forced-alignment:
+
+```
+batchalign align input output
+```
+
+With one `-v`, you can get stack trace information about any files that crashes: 
+
+```
+batchalign -v align input output
+```
+
+and with two `-vv`, we will ditch the loading bar user interface and instead switch to a logging-based interface that has more information about what Batchalign is doing under the hood:
+
+```
+batchalign -vv align input output
+```
+
+## Quick Start: Python
+
+Let's begin!
+
+```python
+import batchalign as ba
+```
+
+### Document
+The `Document` is the most basic object in Bachalign. All processing pipelines expect `Document` as input, and will spit out `Document` as output.
+
+```python
+doc = ba.Document.new("Hello, this is a transcript! I have two utterances.", 
+                      media_path="audio.mp3", lang="eng")
+
+# navigating the document
+first_utterance = doc[0]
+first_form = doc[0][0]
+the_comma = doc[0][1]
+
+assert the_comma.text == ','
+assert the_comma.type == ba.TokenType.PUNCT
+
+# taking a transcript
+sentences = doc.transcript(include_tiers=False, strip=True)
+```
+
+Notably, if you have a Document that you haven't transcribed yet, you still can make a Document!
+
+```python
+doc = ba.Document.new(media_path="audio.mp3", lang="eng")
+```
+
+### Pipelines
+<!-- You can process the language samples you got (perform ASR, forced alignment, utterance segmentation, and more!) via `BatchalignPipeline`. There are two levels of access to this API: you can either create a pipeline and use our default settings, or create and customize the underlying `BatchalignEngine`s yourself to perform processing. -->
+
+#### Quick Pipeline
+Say you wanted to perform ASR, and then tag morphology of the resulting output.
+
+```python
+nlp = ba.BatchalignPipeline.new("asr,morphosyntax", lang="eng", num_speakers=2)
+doc = ba.Document.new(media_path="audio.mp3", lang="eng")
+doc = nlp(doc) # this is equivalent to nlp("audio.mp3"), we will make the initial doc for you
+
+first_word_pos = doc[0][0].morphology
+first_word_time = doc[0][0].time
+first_utterance_time = doc[0].alignment
+```
+
+The quick API (right now) has support for the following tasks, which you can pass in a comma-separated list in the first argument:
+
+- `asr`: ASR!
+- `morphosyntax`: PoS and dependency analysis
+- `fa`: Forced Alignment (require utterance-level timings already)
+
+We will support many, many, many more tasks soon with this API. For now, to gain access to the whole suite of tools, use the second pipeline API discussed below.
+
+#### Manual Pipeline
+Batchalign ships with a plurality of engines which preform the actual processing. For instance, to recreate the demo we had above using the Engines API, we would write
+
+```python
+# ASR
+whisper = ba.WhisperEngine(lang="eng")
+# retracing and disfluency analysis
+retrace = ba.NgramRetraceEngine()
+disfluency = ba.DisfluencyReplacementEngine()
+# morphosyntax
+morphosyntax = ba.StanzaEngine()
+
+# create a pipeline
+nlp = ba.BatchalignPipeline(whisper, retrace, disfluency, morphosyntax)
+                             
+# and run it!                             
+doc = nlp("audio.mp3") 
+```
+
+[Here's a list](https://github.com/TalkBank/batchalign2/blob/master/batchalign/pipelines/__init__.py) of available engines.
+
+### Formats
+We currently support reading and writing two transcript formats: TalkBank CHAT, and Praat TextGrid.
+
+#### CHAT
+
+Here's how to read and write a CHAT file to parse a TalkBank transcript!
+
+```python
+# reading
+chat = ba.CHATFile(path="chat.cha")
+doc = chat.doc
+
+# writing
+chat = ba.CHATFile(doc=doc)
+chat.write("chat.cha")
+```
+
+We will automatically detect audio files located within the same directory as the CHAT file, and associate it with the Batchalign Document.
+
+#### TextGrid
+
+Importantly, there are two ways a TextGrid could be written: we can either place each **utterance** in an individual `IntervalTier`, or each **word** in its own `IntervalTier`; we leave that decision up to you. To learn more about TextGrid, [visit this page](https://github.com/timmahrt/praatIO).
+
+```python
+# reading; recall we can either interpret each IntervalTier as a word or utterance
+tg_utterance = ba.TextGridFile("utterance", path="tg_ut.TextGrid", lang="eng")
+tg_word = ba.TextGridFile("word", path="tg_w.TextGrid", lang="eng")
+
+doc1 = tg_utterance.doc
+doc2 = tg_word.doc
+
+# writing
+tg_utterance = ba.TextGridFile("utterance", doc=doc1)
+tg_word = ba.TextGridFile("word", doc=doc2)
+
+tg_utterance.write("tg_ut.TextGrid")
+tg_word.write("tg_w.TextGrid")
+```
+## Questions?
+If you have any questions or concerns, please reach out! If something isn't working right, [open an issue on GitHub](https://github.com/TalkBank/batchalign2/issues); if you need support, please feel free to email `houjun@cmu.edu` and `macw@cmu.edu`.
+

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/SOURCES.txt

@@ -0,0 +1,117 @@
+LICENSE
+MANIFEST.in
+README.md
+setup.py
+BatchalignHK.egg-info/PKG-INFO
+BatchalignHK.egg-info/SOURCES.txt
+BatchalignHK.egg-info/dependency_links.txt
+BatchalignHK.egg-info/entry_points.txt
+BatchalignHK.egg-info/requires.txt
+BatchalignHK.egg-info/top_level.txt
+batchalign/__init__.py
+batchalign/__main__.py
+batchalign/constants.py
+batchalign/document.py
+batchalign/errors.py
+batchalign/version
+batchalign/cli/__init__.py
+batchalign/cli/cli.py
+batchalign/cli/dispatch.py
+batchalign/formats/__init__.py
+batchalign/formats/base.py
+batchalign/formats/chat/__init__.py
+batchalign/formats/chat/file.py
+batchalign/formats/chat/generator.py
+batchalign/formats/chat/lexer.py
+batchalign/formats/chat/parser.py
+batchalign/formats/chat/utils.py
+batchalign/formats/textgrid/__init__.py
+batchalign/formats/textgrid/file.py
+batchalign/formats/textgrid/generator.py
+batchalign/formats/textgrid/parser.py
+batchalign/models/__init__.py
+batchalign/models/resolve.py
+batchalign/models/utils.py
+batchalign/models/speaker/__init__.py
+batchalign/models/speaker/config.yaml
+batchalign/models/speaker/infer.py
+batchalign/models/speaker/utils.py
+batchalign/models/training/__init__.py
+batchalign/models/training/run.py
+batchalign/models/training/utils.py
+batchalign/models/utterance/__init__.py
+batchalign/models/utterance/cantonese_infer.py
+batchalign/models/utterance/dataset.py
+batchalign/models/utterance/execute.py
+batchalign/models/utterance/infer.py
+batchalign/models/utterance/prep.py
+batchalign/models/utterance/train.py
+batchalign/models/wave2vec/__init__.py
+batchalign/models/wave2vec/infer_fa.py
+batchalign/models/whisper/__init__.py
+batchalign/models/whisper/infer_asr.py
+batchalign/models/whisper/infer_fa.py
+batchalign/pipelines/__init__.py
+batchalign/pipelines/base.py
+batchalign/pipelines/dispatch.py
+batchalign/pipelines/pipeline.py
+batchalign/pipelines/analysis/__init__.py
+batchalign/pipelines/analysis/eval.py
+batchalign/pipelines/asr/__init__.py
+batchalign/pipelines/asr/num2chinese.py
+batchalign/pipelines/asr/rev.py
+batchalign/pipelines/asr/tencent.py
+batchalign/pipelines/asr/utils.py
+batchalign/pipelines/asr/whisper.py
+batchalign/pipelines/asr/whisperx.py
+batchalign/pipelines/cleanup/__init__.py
+batchalign/pipelines/cleanup/cleanup.py
+batchalign/pipelines/cleanup/disfluencies.py
+batchalign/pipelines/cleanup/parse_support.py
+batchalign/pipelines/cleanup/retrace.py
+batchalign/pipelines/cleanup/support/filled_pauses.eng
+batchalign/pipelines/cleanup/support/replacements.eng
+batchalign/pipelines/cleanup/support/test.test
+batchalign/pipelines/fa/__init__.py
+batchalign/pipelines/fa/wave2vec_fa.py
+batchalign/pipelines/fa/whisper_fa.py
+batchalign/pipelines/morphosyntax/__init__.py
+batchalign/pipelines/morphosyntax/coref.py
+batchalign/pipelines/morphosyntax/ud.py
+batchalign/pipelines/morphosyntax/en/irr.py
+batchalign/pipelines/morphosyntax/fr/apm.py
+batchalign/pipelines/morphosyntax/fr/apmn.py
+batchalign/pipelines/morphosyntax/fr/case.py
+batchalign/pipelines/morphosyntax/ja/verbforms.py
+batchalign/pipelines/speaker/__init__.py
+batchalign/pipelines/speaker/nemo_speaker.py
+batchalign/pipelines/translate/__init__.py
+batchalign/pipelines/translate/seamless.py
+batchalign/pipelines/utr/__init__.py
+batchalign/pipelines/utr/rev_utr.py
+batchalign/pipelines/utr/utils.py
+batchalign/pipelines/utr/whisper_utr.py
+batchalign/pipelines/utterance/__init__.py
+batchalign/pipelines/utterance/ud_utterance.py
+batchalign/tests/__init__.py
+batchalign/tests/conftest.py
+batchalign/tests/test_document.py
+batchalign/tests/formats/chat/test_chat_file.py
+batchalign/tests/formats/chat/test_chat_generator.py
+batchalign/tests/formats/chat/test_chat_lexer.py
+batchalign/tests/formats/chat/test_chat_parser.py
+batchalign/tests/formats/chat/test_chat_utils.py
+batchalign/tests/formats/textgrid/test_textgrid.py
+batchalign/tests/pipelines/fixures.py
+batchalign/tests/pipelines/test_pipeline.py
+batchalign/tests/pipelines/test_pipeline_models.py
+batchalign/tests/pipelines/analysis/test_eval.py
+batchalign/tests/pipelines/asr/test_asr_pipeline.py
+batchalign/tests/pipelines/asr/test_asr_utils.py
+batchalign/tests/pipelines/cleanup/test_disfluency.py
+batchalign/tests/pipelines/cleanup/test_parse_support.py
+batchalign/tests/pipelines/fa/test_fa_pipeline.py
+batchalign/utils/__init__.py
+batchalign/utils/config.py
+batchalign/utils/dp.py
+batchalign/utils/utils.py

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+batchalign = batchalign.cli.cli:batchalign

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/requires.txt

@@ -0,0 +1,36 @@
+pydantic>=2.4
+nltk>=3.8
+praatio<6.1.0,>=6.0.0
+torch>=2.6.0
+torchaudio
+hmmlearn==0.3.0
+eyed3
+pydub
+imblearn
+plotly>=5.3.0
+transformers>=4.37
+tokenizers>=0.14.1
+pycountry>=22.3
+stanza[transformers]>=1.10.1
+scipy~=1.11
+rev_ai>=2.18.0
+rich~=13.6
+click~=8.1
+matplotlib<4.0.0,>=3.8.0
+pyfiglet==1.0.2
+setuptools
+soundfile~=0.12.0
+rich-click>=1.7.0
+typing-extensions
+num2words
+tiktoken
+blobfile
+sentencepiece
+tencentcloud-sdk-python-common
+tencentcloud-sdk-python-asr
+
+[dev]
+pytest
+
+[train]
+accelerate~=0.27

batchalignhk-0.7.17.post13/BatchalignHK.egg-info/top_level.txt

@@ -0,0 +1 @@
+batchalign

batchalignhk-0.7.17.post13/LICENSE

@@ -0,0 +1,11 @@
+Copyright 2024 Brian MacWhinney
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

batchalignhk-0.7.17.post13/MANIFEST.in

@@ -0,0 +1,4 @@
+recursive-include batchalign *.py
+include batchalign/version
+include batchalign/pipelines/cleanup/support/*
+include batchalign/models/speaker/*.yaml