opusfilter 3.3.1__tar.gz → 3.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {opusfilter-3.3.1 → opusfilter-3.4.0}/.github/workflows/ci.yml +24 -11
- {opusfilter-3.3.1 → opusfilter-3.4.0}/.gitignore +1 -0
- opusfilter-3.4.0/AGENTS.md +177 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/PKG-INFO +52 -25
- {opusfilter-3.3.1 → opusfilter-3.4.0}/README.md +11 -1
- opusfilter-3.4.0/bin/opusfilter +14 -0
- opusfilter-3.4.0/bin/opusfilter-autogen +14 -0
- opusfilter-3.4.0/bin/opusfilter-cmd +14 -0
- opusfilter-3.4.0/bin/opusfilter-diagram +14 -0
- opusfilter-3.4.0/bin/opusfilter-duplicates +14 -0
- opusfilter-3.4.0/bin/opusfilter-scores +14 -0
- opusfilter-3.4.0/bin/opusfilter-slurm +15 -0
- opusfilter-3.4.0/bin/opusfilter-test +14 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/CHANGELOG.md +27 -1
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/CONTRIBUTING.md +3 -3
- opusfilter-3.4.0/docs/filters/document_scoring_filters.md +103 -0
- opusfilter-3.4.0/docs/functions/downloading_and_selecting_data.md +317 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/filtering_and_scoring.md +18 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/training_language_and_alignment_models.md +1 -1
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/index.rst +1 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/installation.md +37 -3
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/morfessor_segmentation.md +1 -1
- opusfilter-3.4.0/docs/slurm_integration.md +382 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/usage.md +102 -3
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/qed_lm_langid.yaml +18 -0
- opusfilter-3.4.0/example_configs/slurm_example.yaml +155 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/__init__.py +5 -0
- opusfilter-3.4.0/opusfilter/_version.py +24 -0
- opusfilter-3.4.0/opusfilter/cli/__init__.py +3 -0
- opusfilter-3.4.0/opusfilter/cli/autogen.py +101 -0
- opusfilter-3.4.0/opusfilter/cli/cmd.py +94 -0
- opusfilter-3.4.0/opusfilter/cli/diagram.py +83 -0
- opusfilter-3.4.0/opusfilter/cli/duplicates.py +92 -0
- opusfilter-3.4.0/opusfilter/cli/main.py +70 -0
- opusfilter-3.3.1/bin/opusfilter-scores → opusfilter-3.4.0/opusfilter/cli/scores.py +68 -38
- opusfilter-3.4.0/opusfilter/cli/slurm.py +65 -0
- opusfilter-3.4.0/opusfilter/cli/slurm_status.py +168 -0
- opusfilter-3.4.0/opusfilter/cli/slurm_submit.py +67 -0
- opusfilter-3.4.0/opusfilter/cli/test.py +85 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/filters.py +11 -4
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/lid.py +16 -4
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/opusfilter.py +451 -71
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/pipeline.py +50 -9
- opusfilter-3.4.0/opusfilter/slurm.py +607 -0
- opusfilter-3.4.0/opusfilter/slurm_utils.py +411 -0
- opusfilter-3.4.0/opusfilter/util.py +648 -0
- opusfilter-3.4.0/opusfilter/validate.py +281 -0
- opusfilter-3.4.0/opusfilter/wds.py +257 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/PKG-INFO +52 -25
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/SOURCES.txt +40 -2
- opusfilter-3.4.0/opusfilter.egg-info/entry_points.txt +11 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/requires.txt +17 -14
- opusfilter-3.4.0/opusfilter.egg-info/scm_file_list.json +125 -0
- opusfilter-3.4.0/opusfilter.egg-info/scm_version.json +8 -0
- opusfilter-3.4.0/pyproject.toml +184 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/requirements.txt +3 -2
- opusfilter-3.4.0/templates/slurm/step_template.sbatch +30 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_autogen.py +2 -1
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_classifier.py +1 -1
- opusfilter-3.4.0/tests/test_cli/__init__.py +1 -0
- opusfilter-3.4.0/tests/test_cli/test_autogen.py +38 -0
- opusfilter-3.4.0/tests/test_cli/test_cmd.py +38 -0
- opusfilter-3.4.0/tests/test_cli/test_diagram.py +75 -0
- opusfilter-3.4.0/tests/test_cli/test_duplicates.py +59 -0
- opusfilter-3.4.0/tests/test_cli/test_main.py +40 -0
- opusfilter-3.4.0/tests/test_cli/test_scores.py +60 -0
- opusfilter-3.4.0/tests/test_cli/test_test.py +52 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_filter_pipeline.py +101 -5
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_lid.py +1 -1
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_opusfilter.py +1100 -17
- opusfilter-3.4.0/tests/test_slurm.py +129 -0
- opusfilter-3.4.0/tests/test_slurm_e2e.py +70 -0
- opusfilter-3.4.0/tests/test_slurm_simple.py +88 -0
- opusfilter-3.4.0/tests/test_slurm_utils.py +207 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_util.py +91 -0
- opusfilter-3.4.0/tests/test_validate.py +316 -0
- opusfilter-3.4.0/tests/test_wds.py +232 -0
- opusfilter-3.3.1/bin/opusfilter +0 -33
- opusfilter-3.3.1/bin/opusfilter-autogen +0 -91
- opusfilter-3.3.1/bin/opusfilter-cmd +0 -86
- opusfilter-3.3.1/bin/opusfilter-diagram +0 -108
- opusfilter-3.3.1/bin/opusfilter-duplicates +0 -85
- opusfilter-3.3.1/bin/opusfilter-test +0 -76
- opusfilter-3.3.1/docs/functions/downloading_and_selecting_data.md +0 -178
- opusfilter-3.3.1/opusfilter/util.py +0 -274
- opusfilter-3.3.1/pyproject.toml +0 -12
- opusfilter-3.3.1/setup.py +0 -105
- {opusfilter-3.3.1 → opusfilter-3.4.0}/.pylintrc +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/LICENSE +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/automatic_configuration.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/command_line_tools.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/conf.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/alignment_model_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/custom_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/language_model_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/length_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/script_and_language_identification_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/sentence_embedding_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/special_character_and_similarity_filters.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/preprocessing_text.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/training_and_using_classifiers.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/using_score_files.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/bpe_segmentation.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/custom_preprocessors.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/detokenizer.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/monolingual_sentence_splitter.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/reg_exp_sub.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/tokenizer.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/whitespaceNormalizer.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/references.bib +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/references.rst +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/requirements.txt +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/README.md +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_ce_sets.yaml +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_domain_sets.yaml +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_roc_auc_sets.yaml +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_1_labels.jsonl +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_2_labels.jsonl +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/prepare_data.yaml +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/autogen.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/autogen_cluster.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/classifier.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/embeddings.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/lm.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/preprocessors.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/segment_hash.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/subwords.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/tokenization.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/word_alignment.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/dependency_links.txt +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/top_level.txt +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/setup.cfg +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_embeddings.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_filters.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_lm_filter.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_preprocessors.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_segment_hash.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_subwords.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_tokenization.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_wordalign_filter.py +0 -0
- {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_yaml.py +0 -0
|
@@ -21,13 +21,17 @@ jobs:
|
|
|
21
21
|
fail-fast: false
|
|
22
22
|
matrix:
|
|
23
23
|
os: [ubuntu-latest]
|
|
24
|
-
python-version: ["3.
|
|
24
|
+
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
|
|
25
25
|
requirements-file: ["requirements.txt"]
|
|
26
|
+
include:
|
|
27
|
+
- os: ubuntu-22.04
|
|
28
|
+
python-version: "3.9"
|
|
29
|
+
requirements-file: "requirements.txt"
|
|
26
30
|
runs-on: ${{ matrix.os }}
|
|
27
31
|
steps:
|
|
28
|
-
- uses: actions/checkout@
|
|
32
|
+
- uses: actions/checkout@v6
|
|
29
33
|
- name: Set up Python ${{ matrix.python-version }}
|
|
30
|
-
uses: actions/setup-python@
|
|
34
|
+
uses: actions/setup-python@v6
|
|
31
35
|
with:
|
|
32
36
|
python-version: ${{ matrix.python-version }}
|
|
33
37
|
- name: Install dependencies
|
|
@@ -46,12 +50,21 @@ jobs:
|
|
|
46
50
|
flake8 opusfilter --count --exit-zero --statistics
|
|
47
51
|
- name: Run tests with pytest
|
|
48
52
|
run: pytest
|
|
53
|
+
- name: Test CLI entry points
|
|
54
|
+
run: |
|
|
55
|
+
opusfilter --help
|
|
56
|
+
opusfilter-cmd --help
|
|
57
|
+
opusfilter-scores list --help
|
|
58
|
+
opusfilter-autogen --help
|
|
59
|
+
opusfilter-test --help
|
|
60
|
+
opusfilter-diagram --help
|
|
61
|
+
opusfilter-duplicates --help
|
|
49
62
|
build-n-publish:
|
|
50
63
|
name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
|
|
51
64
|
needs: build-n-test
|
|
52
65
|
runs-on: ubuntu-latest
|
|
53
66
|
steps:
|
|
54
|
-
- uses: actions/checkout@
|
|
67
|
+
- uses: actions/checkout@v6
|
|
55
68
|
- name: Check for main release tag
|
|
56
69
|
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
|
|
57
70
|
id: check-tag
|
|
@@ -59,11 +72,11 @@ jobs:
|
|
|
59
72
|
if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+(rc[0-9]*)?$ ]]; then
|
|
60
73
|
echo "match=true" >> $GITHUB_OUTPUT
|
|
61
74
|
fi
|
|
62
|
-
- name: Set up Python 3.
|
|
75
|
+
- name: Set up Python 3.11
|
|
63
76
|
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
|
|
64
|
-
uses: actions/setup-python@
|
|
77
|
+
uses: actions/setup-python@v6
|
|
65
78
|
with:
|
|
66
|
-
python-version: 3.
|
|
79
|
+
python-version: "3.11"
|
|
67
80
|
- name: Install pypa/build
|
|
68
81
|
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
|
|
69
82
|
run: >-
|
|
@@ -96,11 +109,11 @@ jobs:
|
|
|
96
109
|
runs-on: ubuntu-latest
|
|
97
110
|
needs: [build-n-test]
|
|
98
111
|
steps:
|
|
99
|
-
- uses: actions/checkout@
|
|
100
|
-
- name: Set up Python 3.
|
|
101
|
-
uses: actions/setup-python@
|
|
112
|
+
- uses: actions/checkout@v6
|
|
113
|
+
- name: Set up Python 3.11
|
|
114
|
+
uses: actions/setup-python@v6
|
|
102
115
|
with:
|
|
103
|
-
python-version: 3.
|
|
116
|
+
python-version: "3.11"
|
|
104
117
|
- name: Install dependencies to build docs
|
|
105
118
|
run: |
|
|
106
119
|
python -m pip install --upgrade pip
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# OpusFilter Development Guide
|
|
2
|
+
|
|
3
|
+
This guide contains essential information for agents working on the OpusFilter codebase.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
OpusFilter is a tool for filtering and processing parallel corpora. It's written in Python and supports filtering through various methods including language identification, language models, word alignment, and custom filters.
|
|
8
|
+
|
|
9
|
+
## Development Commands
|
|
10
|
+
|
|
11
|
+
### Testing
|
|
12
|
+
- **Run all tests**: `pytest tests/`
|
|
13
|
+
- **Run single test file**: `pytest tests/test_filename.py`
|
|
14
|
+
- **Run specific test**: `pytest tests/test_filename.py::ClassName::test_method`
|
|
15
|
+
- **Run with verbose output**: `pytest -v tests/`
|
|
16
|
+
|
|
17
|
+
### Linting and Code Quality
|
|
18
|
+
- **Check style**: `flake8 opusfilter/`
|
|
19
|
+
- **Run flake8 with statistics**: `flake8 opusfilter --count --exit-zero --statistics`
|
|
20
|
+
- **Run full linting (recommended for contributions)**: `pylint opusfilter/`
|
|
21
|
+
- **Format code**: `black opusfilter/ opusfilter/cli/`
|
|
22
|
+
- **Sort imports**: `isort opusfilter/ opusfilter/cli/`
|
|
23
|
+
- **Type checking**: `mypy opusfilter/ opusfilter/cli/`
|
|
24
|
+
|
|
25
|
+
### Environment Setup
|
|
26
|
+
- **Install with test dependencies**: `pip install .[test]`
|
|
27
|
+
- **Install optional dependencies for full functionality**: `pip install .[test,eflomal,jieba,mecab,varikn]`
|
|
28
|
+
|
|
29
|
+
## Code Style Guidelines
|
|
30
|
+
|
|
31
|
+
### General Rules
|
|
32
|
+
- Follow PEP 8 with a maximum line length of 127 characters (not 79)
|
|
33
|
+
- Support Python 3.9 to 3.14
|
|
34
|
+
- Use type hints where appropriate (see existing code for patterns)
|
|
35
|
+
- Maintain backward compatibility when possible
|
|
36
|
+
- Do not add trailing whitespace
|
|
37
|
+
|
|
38
|
+
### Import Organization
|
|
39
|
+
```python
|
|
40
|
+
# Standard library imports
|
|
41
|
+
import os
|
|
42
|
+
import logging
|
|
43
|
+
from typing import Iterator, List, Tuple
|
|
44
|
+
|
|
45
|
+
# Third-party imports
|
|
46
|
+
import numpy as np
|
|
47
|
+
import pandas as pd
|
|
48
|
+
import regex
|
|
49
|
+
|
|
50
|
+
# Local imports
|
|
51
|
+
from . import FilterABC, ConfigurationError
|
|
52
|
+
from .util import file_open, count_lines
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Naming Conventions
|
|
56
|
+
- **Classes**: PascalCase (e.g., `LengthFilter`, `CrossEntropyFilter`)
|
|
57
|
+
- **Functions/Methods**: snake_case (e.g., `get_length`, `score_pairs`)
|
|
58
|
+
- **Variables**: snake_case (e.g., `min_length`, `accept_threshold`)
|
|
59
|
+
- **Constants**: UPPER_SNAKE_CASE (e.g., `CLEAN_LOW`, `CLEAN_HIGH`)
|
|
60
|
+
- **Private members**: Leading underscore (e.g., `_internal_method`)
|
|
61
|
+
|
|
62
|
+
### Error Handling
|
|
63
|
+
- Use custom exception classes from `opusfilter.__init__`:
|
|
64
|
+
- `OpusFilterError`: Base exception
|
|
65
|
+
- `ConfigurationError`: For configuration-related errors
|
|
66
|
+
- `OpusFilterRuntimeError`: For runtime errors
|
|
67
|
+
- Always include descriptive error messages
|
|
68
|
+
- Use logging for warnings and debug information
|
|
69
|
+
|
|
70
|
+
### Filter Implementation Pattern
|
|
71
|
+
When creating new filters, follow this pattern:
|
|
72
|
+
```python
|
|
73
|
+
class NewFilter(FilterABC):
|
|
74
|
+
"""Brief description of the filter"""
|
|
75
|
+
|
|
76
|
+
score_direction = CLEAN_LOW # or CLEAN_HIGH, CLEAN_BETWEEN, etc.
|
|
77
|
+
accept_threshold = <value>
|
|
78
|
+
reject_threshold = <value>
|
|
79
|
+
|
|
80
|
+
def __init__(self, required_param, optional_param=default, **kwargs):
|
|
81
|
+
# Validate parameters
|
|
82
|
+
self.required_param = required_param
|
|
83
|
+
self.optional_param = optional_param
|
|
84
|
+
super().__init__(**kwargs)
|
|
85
|
+
|
|
86
|
+
def score(self, pairs):
|
|
87
|
+
"""Yield scores for each sentence pair"""
|
|
88
|
+
for pair in pairs:
|
|
89
|
+
# Calculate score
|
|
90
|
+
yield score
|
|
91
|
+
|
|
92
|
+
def accept(self, score):
|
|
93
|
+
"""Return True if score passes the filter"""
|
|
94
|
+
return <condition>
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### Testing Guidelines
|
|
98
|
+
- Write unit tests for all new functionality
|
|
99
|
+
- Use descriptive test method names
|
|
100
|
+
- Test both success and failure cases
|
|
101
|
+
- Use `unittest.TestCase` as base class
|
|
102
|
+
- Mark tests that require optional dependencies with appropriate skips
|
|
103
|
+
|
|
104
|
+
### Logging
|
|
105
|
+
- Use module-level logger: `logger = logging.getLogger(__name__)`
|
|
106
|
+
- Log important operations and warnings
|
|
107
|
+
- Avoid excessive debug logging in production code
|
|
108
|
+
|
|
109
|
+
### File I/O
|
|
110
|
+
- Use `file_open()` from `opusfilter.util` for handling compressed files
|
|
111
|
+
- Support common formats: plain text, gzip, bzip2, lzma
|
|
112
|
+
- Use context managers for file operations
|
|
113
|
+
|
|
114
|
+
### Configuration
|
|
115
|
+
- Filters receive configuration through `__init__` parameters
|
|
116
|
+
- Use `**kwargs` to capture extra parameters and warn about them
|
|
117
|
+
- Validate parameters and raise `ConfigurationError` for invalid values
|
|
118
|
+
|
|
119
|
+
### Type Hints
|
|
120
|
+
- Use type hints for public APIs
|
|
121
|
+
- Import common types from `typing` module
|
|
122
|
+
- Follow existing patterns for complex types
|
|
123
|
+
|
|
124
|
+
### Documentation
|
|
125
|
+
- Add docstrings to all public classes and methods
|
|
126
|
+
- Use Google-style or NumPy-style docstrings
|
|
127
|
+
- Include examples in docstrings when helpful
|
|
128
|
+
|
|
129
|
+
## Project Structure
|
|
130
|
+
|
|
131
|
+
- `opusfilter/`: Main package directory
|
|
132
|
+
- `filters.py`: Core filter implementations
|
|
133
|
+
- `preprocessors.py`: Text preprocessing functions
|
|
134
|
+
- `lm.py`: Language model filters
|
|
135
|
+
- `lid.py`: Language identification filters
|
|
136
|
+
- `embeddings.py`: Sentence embedding filters
|
|
137
|
+
- `word_alignment.py`: Word alignment filters
|
|
138
|
+
- `util.py`: Utility functions and helpers
|
|
139
|
+
- `pipeline.py`: Processing pipeline implementation
|
|
140
|
+
- `opusfilter.py`: Main application entry point
|
|
141
|
+
- `tests/`: Unit tests
|
|
142
|
+
- `docs/`: Documentation source files
|
|
143
|
+
- `work/`: Example configurations and working files
|
|
144
|
+
|
|
145
|
+
## Common Patterns
|
|
146
|
+
|
|
147
|
+
### Iterating over sentence pairs
|
|
148
|
+
```python
|
|
149
|
+
for pair in pairs:
|
|
150
|
+
# pair is a list/tuple of strings
|
|
151
|
+
# pair[0] is source, pair[1] is target, etc.
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
### Yielding scores
|
|
155
|
+
```python
|
|
156
|
+
def score(self, pairs):
|
|
157
|
+
for pair in pairs:
|
|
158
|
+
# Calculate score(s)
|
|
159
|
+
yield score_value # or yield [score1, score2, ...]
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
### Checking filter acceptance
|
|
163
|
+
```python
|
|
164
|
+
def accept(self, score):
|
|
165
|
+
if isinstance(score, list):
|
|
166
|
+
# Multiple scores
|
|
167
|
+
return all(condition(s) for s in score)
|
|
168
|
+
else:
|
|
169
|
+
# Single score
|
|
170
|
+
return condition(score)
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## Git Workflow
|
|
174
|
+
- Make pull requests to the `develop` branch (not `master`)
|
|
175
|
+
- Include tests for new features
|
|
176
|
+
- Run linting and tests before submitting
|
|
177
|
+
- Follow conventional commit messages if possible
|
|
@@ -1,18 +1,35 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: opusfilter
|
|
3
|
-
Version: 3.
|
|
4
|
-
Summary: Toolbox for filtering parallel corpora
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
Version: 3.4.0
|
|
4
|
+
Summary: Toolbox for filtering and processing parallel corpora
|
|
5
|
+
Author-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
|
|
6
|
+
Maintainer-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
|
|
7
|
+
License-Expression: MIT
|
|
8
|
+
Project-URL: Homepage, https://github.com/Helsinki-NLP/OpusFilter
|
|
9
|
+
Project-URL: Documentation, https://helsinki-nlp.github.io/OpusFilter/
|
|
10
|
+
Project-URL: Repository, https://github.com/Helsinki-NLP/OpusFilter.git
|
|
11
|
+
Project-URL: Bug Tracker, https://github.com/Helsinki-NLP/OpusFilter/issues
|
|
12
|
+
Project-URL: Changelog, https://github.com/Helsinki-NLP/OpusFilter/blob/main/docs/CHANGELOG.md
|
|
13
|
+
Project-URL: Paper (ACL 2020), https://www.aclweb.org/anthology/2020.acl-demos.20
|
|
14
|
+
Keywords: parallel-corpora,corpus-filtering,natural-language-processing,machine-translation,data-cleaning,nlp,text-processing
|
|
15
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
16
|
+
Classifier: Intended Audience :: Developers
|
|
17
|
+
Classifier: Intended Audience :: Science/Research
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
19
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
8
20
|
Classifier: Programming Language :: Python :: 3
|
|
9
|
-
Classifier:
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
24
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
25
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
26
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
10
27
|
Classifier: Operating System :: OS Independent
|
|
11
|
-
Requires-Python: >=3.
|
|
28
|
+
Requires-Python: >=3.9
|
|
12
29
|
Description-Content-Type: text/markdown
|
|
13
30
|
License-File: LICENSE
|
|
14
31
|
Requires-Dist: setuptools
|
|
15
|
-
Requires-Dist: opustools>=1.
|
|
32
|
+
Requires-Dist: opustools>=1.10.0
|
|
16
33
|
Requires-Dist: beautifulsoup4>=4.8.0
|
|
17
34
|
Requires-Dist: graphviz
|
|
18
35
|
Requires-Dist: py3langid>=0.2.2
|
|
@@ -29,16 +46,14 @@ Requires-Dist: requests
|
|
|
29
46
|
Requires-Dist: scikit-learn
|
|
30
47
|
Requires-Dist: subword_nmt
|
|
31
48
|
Requires-Dist: tqdm
|
|
49
|
+
Requires-Dist: lingua-language-detector>=2.2.0; python_version >= "3.12"
|
|
50
|
+
Requires-Dist: lingua-language-detector<2.2,>=1.3.0; python_version >= "3.10" and python_version < "3.12"
|
|
32
51
|
Requires-Dist: lingua-language-detector<2.1,>=1.3.0; python_version < "3.10"
|
|
33
|
-
Requires-Dist: lingua-language-detector>=2.1.1; python_version >= "3.10"
|
|
34
52
|
Requires-Dist: iso639-lang
|
|
35
|
-
Provides-Extra: test
|
|
36
|
-
Requires-Dist: pytest; extra == "test"
|
|
37
53
|
Provides-Extra: pycld2
|
|
38
54
|
Requires-Dist: pycld2; extra == "pycld2"
|
|
39
55
|
Provides-Extra: fasttext
|
|
40
56
|
Requires-Dist: py3langid<0.3.0; extra == "fasttext"
|
|
41
|
-
Requires-Dist: numpy<2.0.0; extra == "fasttext"
|
|
42
57
|
Requires-Dist: fasttext; extra == "fasttext"
|
|
43
58
|
Provides-Extra: eflomal
|
|
44
59
|
Requires-Dist: eflomal>=2.0.0; extra == "eflomal"
|
|
@@ -53,23 +68,25 @@ Provides-Extra: varikn
|
|
|
53
68
|
Requires-Dist: varikn; extra == "varikn"
|
|
54
69
|
Provides-Extra: heliport
|
|
55
70
|
Requires-Dist: heliport>=0.10.0; extra == "heliport"
|
|
71
|
+
Provides-Extra: huggingface
|
|
72
|
+
Requires-Dist: datasets>=2.14.0; extra == "huggingface"
|
|
73
|
+
Provides-Extra: test
|
|
74
|
+
Requires-Dist: pytest; extra == "test"
|
|
56
75
|
Provides-Extra: docs
|
|
57
76
|
Requires-Dist: myst-parser; extra == "docs"
|
|
58
77
|
Requires-Dist: sphinx; extra == "docs"
|
|
59
78
|
Requires-Dist: sphinx-rtd-theme; extra == "docs"
|
|
60
79
|
Requires-Dist: sphinxcontrib-bibtex; extra == "docs"
|
|
61
80
|
Provides-Extra: all
|
|
62
|
-
Requires-Dist: eflomal
|
|
63
|
-
|
|
64
|
-
Requires-Dist:
|
|
65
|
-
Requires-Dist:
|
|
66
|
-
Requires-Dist:
|
|
67
|
-
Requires-Dist:
|
|
68
|
-
Requires-Dist:
|
|
69
|
-
Requires-Dist:
|
|
70
|
-
|
|
71
|
-
Requires-Dist: sphinx-rtd-theme; extra == "all"
|
|
72
|
-
Requires-Dist: sphinxcontrib-bibtex; extra == "all"
|
|
81
|
+
Requires-Dist: opusfilter[docs,eflomal,fasttext,heliport,jieba,laser,mecab,pycld2,test,varikn]; extra == "all"
|
|
82
|
+
Provides-Extra: dev
|
|
83
|
+
Requires-Dist: pytest; extra == "dev"
|
|
84
|
+
Requires-Dist: pytest-cov; extra == "dev"
|
|
85
|
+
Requires-Dist: black; extra == "dev"
|
|
86
|
+
Requires-Dist: isort; extra == "dev"
|
|
87
|
+
Requires-Dist: pylint; extra == "dev"
|
|
88
|
+
Requires-Dist: mypy; extra == "dev"
|
|
89
|
+
Dynamic: license-file
|
|
73
90
|
|
|
74
91
|
# OpusFilter
|
|
75
92
|
|
|
@@ -98,7 +115,7 @@ Install from source:
|
|
|
98
115
|
|
|
99
116
|
### Troubleshooting
|
|
100
117
|
|
|
101
|
-
OpusFilter should generally work fine on Python 3.
|
|
118
|
+
OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
|
|
102
119
|
|
|
103
120
|
* `pip install -r requirements.txt`
|
|
104
121
|
|
|
@@ -138,3 +155,13 @@ A full bibliography of papers cited in the documentation and code can be found f
|
|
|
138
155
|
## Contributing
|
|
139
156
|
|
|
140
157
|
See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
|
|
158
|
+
|
|
159
|
+
|
|
160
|
+
## Acknowledgements
|
|
161
|
+
|
|
162
|
+
OpusFilter has partially been supported by various projects such as:
|
|
163
|
+
|
|
164
|
+
* [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
|
|
165
|
+
* [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
|
|
166
|
+
* [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
|
|
167
|
+
* [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
|
|
@@ -25,7 +25,7 @@ Install from source:
|
|
|
25
25
|
|
|
26
26
|
### Troubleshooting
|
|
27
27
|
|
|
28
|
-
OpusFilter should generally work fine on Python 3.
|
|
28
|
+
OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
|
|
29
29
|
|
|
30
30
|
* `pip install -r requirements.txt`
|
|
31
31
|
|
|
@@ -65,3 +65,13 @@ A full bibliography of papers cited in the documentation and code can be found f
|
|
|
65
65
|
## Contributing
|
|
66
66
|
|
|
67
67
|
See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
## Acknowledgements
|
|
71
|
+
|
|
72
|
+
OpusFilter has partially been supported by various projects such as:
|
|
73
|
+
|
|
74
|
+
* [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
|
|
75
|
+
* [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
|
|
76
|
+
* [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
|
|
77
|
+
* [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import sys
|
|
4
|
+
import warnings
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'opusfilter' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.main import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-autogen is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'autogen' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.autogen import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-cmd is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'cmd' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.cmd import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-diagram is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'diagram' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.diagram import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-duplicates is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'opusfilter-duplicates' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.duplicates import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-scores is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'scores' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.scores import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import sys
|
|
4
|
+
import warnings
|
|
5
|
+
|
|
6
|
+
from opusfilter.cli.slurm import main
|
|
7
|
+
|
|
8
|
+
warnings.warn(
|
|
9
|
+
"opusfilter-slurm is experimental. Please report issues at "
|
|
10
|
+
"https://github.com/Helsinki-NLP/OpusFilter/issues",
|
|
11
|
+
UserWarning,
|
|
12
|
+
stacklevel=2
|
|
13
|
+
)
|
|
14
|
+
|
|
15
|
+
sys.exit(main())
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
|
|
3
|
+
import warnings
|
|
4
|
+
import sys
|
|
5
|
+
|
|
6
|
+
warnings.warn(
|
|
7
|
+
"Direct execution of bin/opusfilter-test is deprecated and will be removed in a future release. "
|
|
8
|
+
"Please use the 'opusfilter-test' command instead.",
|
|
9
|
+
DeprecationWarning,
|
|
10
|
+
stacklevel=2
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
from opusfilter.cli.test import main
|
|
14
|
+
sys.exit(main())
|
|
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [3.4.0] - 2026-11-18
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- support for Python 3.14
|
|
15
|
+
- `opusfilter-slurm-*` commands for running OpusFilter workflows in SLURM clusters
|
|
16
|
+
- improved validation for configuration files
|
|
17
|
+
- support for encoding text in JSONL files to preserve newlines
|
|
18
|
+
- `with_decision` option for the `score` function
|
|
19
|
+
- `columns` option for the `unzip` function
|
|
20
|
+
- `hf_read` function for reading data from Hugging Face datasets
|
|
21
|
+
- inline filtering for `hf_read` and `opus_read`
|
|
22
|
+
- inline preprocessing for `hf_read` and `opus_read`
|
|
23
|
+
- `doc_level` and `limit` options for `opus_read`
|
|
24
|
+
- `WDSFilter` based on Web Docs Scorer
|
|
25
|
+
- `score_output` option for saving filter scores from `filter`, `opus_read`, and `hf_read`
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- modernized packaging and CLI scripts
|
|
30
|
+
|
|
31
|
+
### Removed
|
|
32
|
+
|
|
33
|
+
- support for Python 3.8
|
|
34
|
+
|
|
10
35
|
## [3.3.1] - 2025-12-10
|
|
11
36
|
|
|
12
37
|
### Changed
|
|
@@ -245,7 +270,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
245
270
|
First tagged version.
|
|
246
271
|
|
|
247
272
|
|
|
248
|
-
[Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.
|
|
273
|
+
[Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.4.0...develop
|
|
274
|
+
[3.4.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.1...3.4.0
|
|
249
275
|
[3.3.1]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.0...3.3.1
|
|
250
276
|
[3.3.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.2.0...3.3.0
|
|
251
277
|
[3.2.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.1.0...3.2.0
|
|
@@ -5,7 +5,7 @@ issues page. We are also happy to consider pull requests. There are a
|
|
|
5
5
|
few rules for pull requests:
|
|
6
6
|
|
|
7
7
|
* Make a pull request to the `develop` branch instead of `master`.
|
|
8
|
-
* The code should support at least Python versions from 3.
|
|
8
|
+
* The code should support at least Python versions from 3.9 to 3.14.
|
|
9
9
|
* Please follow [PEP 8](https://www.python.org/dev/peps/pep-0008/). Exception: The maximum line length is 127 characters instead of 79.
|
|
10
10
|
* Especially for new features, please include test cases for unit testing.
|
|
11
11
|
|
|
@@ -19,8 +19,8 @@ work, if you have VariKN and eflomal set up as instructed - `pytest`
|
|
|
19
19
|
skips the respective tests if not.)
|
|
20
20
|
|
|
21
21
|
GitHub workflows defined in the project run automatically `flake8`
|
|
22
|
-
checks and unit testing with `pytest` using Python 3.
|
|
23
|
-
3.11, 3.12, and 3.
|
|
22
|
+
checks and unit testing with `pytest` using Python 3.9, 3.10,
|
|
23
|
+
3.11, 3.12, and 3.14.
|
|
24
24
|
|
|
25
25
|
Especially for larger contributions, consider using a code analysis
|
|
26
26
|
tool like [Pylint](https://github.com/PyCQA/pylint). Install it
|