opusfilter 3.3.0__tar.gz → 3.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (141) hide show
  1. {opusfilter-3.3.0 → opusfilter-3.4.0}/.github/workflows/ci.yml +24 -11
  2. {opusfilter-3.3.0 → opusfilter-3.4.0}/.gitignore +1 -0
  3. opusfilter-3.4.0/AGENTS.md +177 -0
  4. {opusfilter-3.3.0/opusfilter.egg-info → opusfilter-3.4.0}/PKG-INFO +53 -27
  5. {opusfilter-3.3.0 → opusfilter-3.4.0}/README.md +11 -1
  6. opusfilter-3.4.0/bin/opusfilter +14 -0
  7. opusfilter-3.4.0/bin/opusfilter-autogen +14 -0
  8. opusfilter-3.4.0/bin/opusfilter-cmd +14 -0
  9. opusfilter-3.4.0/bin/opusfilter-diagram +14 -0
  10. opusfilter-3.4.0/bin/opusfilter-duplicates +14 -0
  11. opusfilter-3.4.0/bin/opusfilter-scores +14 -0
  12. opusfilter-3.4.0/bin/opusfilter-slurm +15 -0
  13. opusfilter-3.4.0/bin/opusfilter-test +14 -0
  14. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/CHANGELOG.md +39 -1
  15. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/CONTRIBUTING.md +3 -3
  16. opusfilter-3.4.0/docs/filters/document_scoring_filters.md +103 -0
  17. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/script_and_language_identification_filters.md +20 -20
  18. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/special_character_and_similarity_filters.md +6 -0
  19. opusfilter-3.4.0/docs/functions/downloading_and_selecting_data.md +317 -0
  20. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/functions/filtering_and_scoring.md +20 -1
  21. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/functions/training_language_and_alignment_models.md +1 -1
  22. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/index.rst +1 -0
  23. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/installation.md +46 -6
  24. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/morfessor_segmentation.md +1 -1
  25. opusfilter-3.4.0/docs/slurm_integration.md +382 -0
  26. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/usage.md +102 -3
  27. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/qed_lm_langid.yaml +18 -0
  28. opusfilter-3.4.0/example_configs/slurm_example.yaml +155 -0
  29. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/__init__.py +5 -0
  30. opusfilter-3.4.0/opusfilter/_version.py +24 -0
  31. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/autogen.py +11 -2
  32. opusfilter-3.4.0/opusfilter/cli/__init__.py +3 -0
  33. opusfilter-3.4.0/opusfilter/cli/autogen.py +101 -0
  34. opusfilter-3.4.0/opusfilter/cli/cmd.py +94 -0
  35. opusfilter-3.4.0/opusfilter/cli/diagram.py +83 -0
  36. opusfilter-3.4.0/opusfilter/cli/duplicates.py +92 -0
  37. opusfilter-3.4.0/opusfilter/cli/main.py +70 -0
  38. opusfilter-3.3.0/bin/opusfilter-scores → opusfilter-3.4.0/opusfilter/cli/scores.py +68 -38
  39. opusfilter-3.4.0/opusfilter/cli/slurm.py +65 -0
  40. opusfilter-3.4.0/opusfilter/cli/slurm_status.py +168 -0
  41. opusfilter-3.4.0/opusfilter/cli/slurm_submit.py +67 -0
  42. opusfilter-3.4.0/opusfilter/cli/test.py +85 -0
  43. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/filters.py +12 -5
  44. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/lid.py +16 -4
  45. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/opusfilter.py +451 -71
  46. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/pipeline.py +50 -9
  47. opusfilter-3.4.0/opusfilter/slurm.py +607 -0
  48. opusfilter-3.4.0/opusfilter/slurm_utils.py +411 -0
  49. opusfilter-3.4.0/opusfilter/util.py +648 -0
  50. opusfilter-3.4.0/opusfilter/validate.py +281 -0
  51. opusfilter-3.4.0/opusfilter/wds.py +257 -0
  52. {opusfilter-3.3.0 → opusfilter-3.4.0/opusfilter.egg-info}/PKG-INFO +53 -27
  53. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter.egg-info/SOURCES.txt +40 -2
  54. opusfilter-3.4.0/opusfilter.egg-info/entry_points.txt +11 -0
  55. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter.egg-info/requires.txt +17 -19
  56. opusfilter-3.4.0/opusfilter.egg-info/scm_file_list.json +125 -0
  57. opusfilter-3.4.0/opusfilter.egg-info/scm_version.json +8 -0
  58. opusfilter-3.4.0/pyproject.toml +184 -0
  59. {opusfilter-3.3.0 → opusfilter-3.4.0}/requirements.txt +3 -2
  60. opusfilter-3.4.0/templates/slurm/step_template.sbatch +30 -0
  61. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_autogen.py +2 -1
  62. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_classifier.py +1 -1
  63. opusfilter-3.4.0/tests/test_cli/__init__.py +1 -0
  64. opusfilter-3.4.0/tests/test_cli/test_autogen.py +38 -0
  65. opusfilter-3.4.0/tests/test_cli/test_cmd.py +38 -0
  66. opusfilter-3.4.0/tests/test_cli/test_diagram.py +75 -0
  67. opusfilter-3.4.0/tests/test_cli/test_duplicates.py +59 -0
  68. opusfilter-3.4.0/tests/test_cli/test_main.py +40 -0
  69. opusfilter-3.4.0/tests/test_cli/test_scores.py +60 -0
  70. opusfilter-3.4.0/tests/test_cli/test_test.py +52 -0
  71. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_filter_pipeline.py +101 -5
  72. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_filters.py +2 -12
  73. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_lid.py +5 -2
  74. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_opusfilter.py +1100 -17
  75. opusfilter-3.4.0/tests/test_slurm.py +129 -0
  76. opusfilter-3.4.0/tests/test_slurm_e2e.py +70 -0
  77. opusfilter-3.4.0/tests/test_slurm_simple.py +88 -0
  78. opusfilter-3.4.0/tests/test_slurm_utils.py +207 -0
  79. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_util.py +91 -0
  80. opusfilter-3.4.0/tests/test_validate.py +316 -0
  81. opusfilter-3.4.0/tests/test_wds.py +232 -0
  82. opusfilter-3.3.0/bin/opusfilter +0 -33
  83. opusfilter-3.3.0/bin/opusfilter-autogen +0 -91
  84. opusfilter-3.3.0/bin/opusfilter-cmd +0 -86
  85. opusfilter-3.3.0/bin/opusfilter-diagram +0 -108
  86. opusfilter-3.3.0/bin/opusfilter-duplicates +0 -85
  87. opusfilter-3.3.0/bin/opusfilter-test +0 -76
  88. opusfilter-3.3.0/docs/functions/downloading_and_selecting_data.md +0 -178
  89. opusfilter-3.3.0/opusfilter/util.py +0 -274
  90. opusfilter-3.3.0/pyproject.toml +0 -12
  91. opusfilter-3.3.0/setup.py +0 -106
  92. {opusfilter-3.3.0 → opusfilter-3.4.0}/.pylintrc +0 -0
  93. {opusfilter-3.3.0 → opusfilter-3.4.0}/LICENSE +0 -0
  94. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/automatic_configuration.md +0 -0
  95. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/command_line_tools.md +0 -0
  96. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/conf.py +0 -0
  97. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/alignment_model_filters.md +0 -0
  98. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/custom_filters.md +0 -0
  99. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/language_model_filters.md +0 -0
  100. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/length_filters.md +0 -0
  101. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/filters/sentence_embedding_filters.md +0 -0
  102. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/functions/preprocessing_text.md +0 -0
  103. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/functions/training_and_using_classifiers.md +0 -0
  104. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/functions/using_score_files.md +0 -0
  105. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/bpe_segmentation.md +0 -0
  106. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/custom_preprocessors.md +0 -0
  107. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/detokenizer.md +0 -0
  108. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/monolingual_sentence_splitter.md +0 -0
  109. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/reg_exp_sub.md +0 -0
  110. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/tokenizer.md +0 -0
  111. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/preprocessors/whitespaceNormalizer.md +0 -0
  112. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/references.bib +0 -0
  113. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/references.rst +0 -0
  114. {opusfilter-3.3.0 → opusfilter-3.4.0}/docs/requirements.txt +0 -0
  115. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/README.md +0 -0
  116. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_ce_sets.yaml +0 -0
  117. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_domain_sets.yaml +0 -0
  118. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_roc_auc_sets.yaml +0 -0
  119. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_1_labels.jsonl +0 -0
  120. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_2_labels.jsonl +0 -0
  121. {opusfilter-3.3.0 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/prepare_data.yaml +0 -0
  122. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/autogen_cluster.py +0 -0
  123. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/classifier.py +0 -0
  124. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/embeddings.py +0 -0
  125. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/lm.py +0 -0
  126. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/preprocessors.py +0 -0
  127. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/segment_hash.py +0 -0
  128. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/subwords.py +0 -0
  129. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/tokenization.py +0 -0
  130. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter/word_alignment.py +0 -0
  131. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter.egg-info/dependency_links.txt +0 -0
  132. {opusfilter-3.3.0 → opusfilter-3.4.0}/opusfilter.egg-info/top_level.txt +0 -0
  133. {opusfilter-3.3.0 → opusfilter-3.4.0}/setup.cfg +0 -0
  134. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_embeddings.py +0 -0
  135. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_lm_filter.py +0 -0
  136. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_preprocessors.py +0 -0
  137. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_segment_hash.py +0 -0
  138. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_subwords.py +0 -0
  139. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_tokenization.py +0 -0
  140. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_wordalign_filter.py +0 -0
  141. {opusfilter-3.3.0 → opusfilter-3.4.0}/tests/test_yaml.py +0 -0
@@ -21,13 +21,17 @@ jobs:
21
21
  fail-fast: false
22
22
  matrix:
23
23
  os: [ubuntu-latest]
24
- python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
24
+ python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
25
25
  requirements-file: ["requirements.txt"]
26
+ include:
27
+ - os: ubuntu-22.04
28
+ python-version: "3.9"
29
+ requirements-file: "requirements.txt"
26
30
  runs-on: ${{ matrix.os }}
27
31
  steps:
28
- - uses: actions/checkout@v4
32
+ - uses: actions/checkout@v6
29
33
  - name: Set up Python ${{ matrix.python-version }}
30
- uses: actions/setup-python@v5
34
+ uses: actions/setup-python@v6
31
35
  with:
32
36
  python-version: ${{ matrix.python-version }}
33
37
  - name: Install dependencies
@@ -46,12 +50,21 @@ jobs:
46
50
  flake8 opusfilter --count --exit-zero --statistics
47
51
  - name: Run tests with pytest
48
52
  run: pytest
53
+ - name: Test CLI entry points
54
+ run: |
55
+ opusfilter --help
56
+ opusfilter-cmd --help
57
+ opusfilter-scores list --help
58
+ opusfilter-autogen --help
59
+ opusfilter-test --help
60
+ opusfilter-diagram --help
61
+ opusfilter-duplicates --help
49
62
  build-n-publish:
50
63
  name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
51
64
  needs: build-n-test
52
65
  runs-on: ubuntu-latest
53
66
  steps:
54
- - uses: actions/checkout@v4
67
+ - uses: actions/checkout@v6
55
68
  - name: Check for main release tag
56
69
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
57
70
  id: check-tag
@@ -59,11 +72,11 @@ jobs:
59
72
  if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+(rc[0-9]*)?$ ]]; then
60
73
  echo "match=true" >> $GITHUB_OUTPUT
61
74
  fi
62
- - name: Set up Python 3.8
75
+ - name: Set up Python 3.11
63
76
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
64
- uses: actions/setup-python@v5
77
+ uses: actions/setup-python@v6
65
78
  with:
66
- python-version: 3.8
79
+ python-version: "3.11"
67
80
  - name: Install pypa/build
68
81
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
69
82
  run: >-
@@ -96,11 +109,11 @@ jobs:
96
109
  runs-on: ubuntu-latest
97
110
  needs: [build-n-test]
98
111
  steps:
99
- - uses: actions/checkout@v4
100
- - name: Set up Python 3.8
101
- uses: actions/setup-python@v5
112
+ - uses: actions/checkout@v6
113
+ - name: Set up Python 3.11
114
+ uses: actions/setup-python@v6
102
115
  with:
103
- python-version: 3.8
116
+ python-version: "3.11"
104
117
  - name: Install dependencies to build docs
105
118
  run: |
106
119
  python -m pip install --upgrade pip
@@ -13,3 +13,4 @@ htmlcov
13
13
  *.zip
14
14
  *.gz
15
15
  venv
16
+ opusfilter/_version.py
@@ -0,0 +1,177 @@
1
+ # OpusFilter Development Guide
2
+
3
+ This guide contains essential information for agents working on the OpusFilter codebase.
4
+
5
+ ## Project Overview
6
+
7
+ OpusFilter is a tool for filtering and processing parallel corpora. It's written in Python and supports filtering through various methods including language identification, language models, word alignment, and custom filters.
8
+
9
+ ## Development Commands
10
+
11
+ ### Testing
12
+ - **Run all tests**: `pytest tests/`
13
+ - **Run single test file**: `pytest tests/test_filename.py`
14
+ - **Run specific test**: `pytest tests/test_filename.py::ClassName::test_method`
15
+ - **Run with verbose output**: `pytest -v tests/`
16
+
17
+ ### Linting and Code Quality
18
+ - **Check style**: `flake8 opusfilter/`
19
+ - **Run flake8 with statistics**: `flake8 opusfilter --count --exit-zero --statistics`
20
+ - **Run full linting (recommended for contributions)**: `pylint opusfilter/`
21
+ - **Format code**: `black opusfilter/ opusfilter/cli/`
22
+ - **Sort imports**: `isort opusfilter/ opusfilter/cli/`
23
+ - **Type checking**: `mypy opusfilter/ opusfilter/cli/`
24
+
25
+ ### Environment Setup
26
+ - **Install with test dependencies**: `pip install .[test]`
27
+ - **Install optional dependencies for full functionality**: `pip install .[test,eflomal,jieba,mecab,varikn]`
28
+
29
+ ## Code Style Guidelines
30
+
31
+ ### General Rules
32
+ - Follow PEP 8 with a maximum line length of 127 characters (not 79)
33
+ - Support Python 3.9 to 3.14
34
+ - Use type hints where appropriate (see existing code for patterns)
35
+ - Maintain backward compatibility when possible
36
+ - Do not add trailing whitespace
37
+
38
+ ### Import Organization
39
+ ```python
40
+ # Standard library imports
41
+ import os
42
+ import logging
43
+ from typing import Iterator, List, Tuple
44
+
45
+ # Third-party imports
46
+ import numpy as np
47
+ import pandas as pd
48
+ import regex
49
+
50
+ # Local imports
51
+ from . import FilterABC, ConfigurationError
52
+ from .util import file_open, count_lines
53
+ ```
54
+
55
+ ### Naming Conventions
56
+ - **Classes**: PascalCase (e.g., `LengthFilter`, `CrossEntropyFilter`)
57
+ - **Functions/Methods**: snake_case (e.g., `get_length`, `score_pairs`)
58
+ - **Variables**: snake_case (e.g., `min_length`, `accept_threshold`)
59
+ - **Constants**: UPPER_SNAKE_CASE (e.g., `CLEAN_LOW`, `CLEAN_HIGH`)
60
+ - **Private members**: Leading underscore (e.g., `_internal_method`)
61
+
62
+ ### Error Handling
63
+ - Use custom exception classes from `opusfilter.__init__`:
64
+ - `OpusFilterError`: Base exception
65
+ - `ConfigurationError`: For configuration-related errors
66
+ - `OpusFilterRuntimeError`: For runtime errors
67
+ - Always include descriptive error messages
68
+ - Use logging for warnings and debug information
69
+
70
+ ### Filter Implementation Pattern
71
+ When creating new filters, follow this pattern:
72
+ ```python
73
+ class NewFilter(FilterABC):
74
+ """Brief description of the filter"""
75
+
76
+ score_direction = CLEAN_LOW # or CLEAN_HIGH, CLEAN_BETWEEN, etc.
77
+ accept_threshold = <value>
78
+ reject_threshold = <value>
79
+
80
+ def __init__(self, required_param, optional_param=default, **kwargs):
81
+ # Validate parameters
82
+ self.required_param = required_param
83
+ self.optional_param = optional_param
84
+ super().__init__(**kwargs)
85
+
86
+ def score(self, pairs):
87
+ """Yield scores for each sentence pair"""
88
+ for pair in pairs:
89
+ # Calculate score
90
+ yield score
91
+
92
+ def accept(self, score):
93
+ """Return True if score passes the filter"""
94
+ return <condition>
95
+ ```
96
+
97
+ ### Testing Guidelines
98
+ - Write unit tests for all new functionality
99
+ - Use descriptive test method names
100
+ - Test both success and failure cases
101
+ - Use `unittest.TestCase` as base class
102
+ - Mark tests that require optional dependencies with appropriate skips
103
+
104
+ ### Logging
105
+ - Use module-level logger: `logger = logging.getLogger(__name__)`
106
+ - Log important operations and warnings
107
+ - Avoid excessive debug logging in production code
108
+
109
+ ### File I/O
110
+ - Use `file_open()` from `opusfilter.util` for handling compressed files
111
+ - Support common formats: plain text, gzip, bzip2, lzma
112
+ - Use context managers for file operations
113
+
114
+ ### Configuration
115
+ - Filters receive configuration through `__init__` parameters
116
+ - Use `**kwargs` to capture extra parameters and warn about them
117
+ - Validate parameters and raise `ConfigurationError` for invalid values
118
+
119
+ ### Type Hints
120
+ - Use type hints for public APIs
121
+ - Import common types from `typing` module
122
+ - Follow existing patterns for complex types
123
+
124
+ ### Documentation
125
+ - Add docstrings to all public classes and methods
126
+ - Use Google-style or NumPy-style docstrings
127
+ - Include examples in docstrings when helpful
128
+
129
+ ## Project Structure
130
+
131
+ - `opusfilter/`: Main package directory
132
+ - `filters.py`: Core filter implementations
133
+ - `preprocessors.py`: Text preprocessing functions
134
+ - `lm.py`: Language model filters
135
+ - `lid.py`: Language identification filters
136
+ - `embeddings.py`: Sentence embedding filters
137
+ - `word_alignment.py`: Word alignment filters
138
+ - `util.py`: Utility functions and helpers
139
+ - `pipeline.py`: Processing pipeline implementation
140
+ - `opusfilter.py`: Main application entry point
141
+ - `tests/`: Unit tests
142
+ - `docs/`: Documentation source files
143
+ - `work/`: Example configurations and working files
144
+
145
+ ## Common Patterns
146
+
147
+ ### Iterating over sentence pairs
148
+ ```python
149
+ for pair in pairs:
150
+ # pair is a list/tuple of strings
151
+ # pair[0] is source, pair[1] is target, etc.
152
+ ```
153
+
154
+ ### Yielding scores
155
+ ```python
156
+ def score(self, pairs):
157
+ for pair in pairs:
158
+ # Calculate score(s)
159
+ yield score_value # or yield [score1, score2, ...]
160
+ ```
161
+
162
+ ### Checking filter acceptance
163
+ ```python
164
+ def accept(self, score):
165
+ if isinstance(score, list):
166
+ # Multiple scores
167
+ return all(condition(s) for s in score)
168
+ else:
169
+ # Single score
170
+ return condition(score)
171
+ ```
172
+
173
+ ## Git Workflow
174
+ - Make pull requests to the `develop` branch (not `master`)
175
+ - Include tests for new features
176
+ - Run linting and tests before submitting
177
+ - Follow conventional commit messages if possible
@@ -1,25 +1,41 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: opusfilter
3
- Version: 3.3.0
4
- Summary: Toolbox for filtering parallel corpora
5
- Home-page: https://github.com/Helsinki-NLP/OpusFilter
6
- Author: Mikko Aulamo, Sami Virpioja
7
- Author-email: mikko.aulamo@helsinki.fi
3
+ Version: 3.4.0
4
+ Summary: Toolbox for filtering and processing parallel corpora
5
+ Author-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
6
+ Maintainer-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
7
+ License-Expression: MIT
8
+ Project-URL: Homepage, https://github.com/Helsinki-NLP/OpusFilter
9
+ Project-URL: Documentation, https://helsinki-nlp.github.io/OpusFilter/
10
+ Project-URL: Repository, https://github.com/Helsinki-NLP/OpusFilter.git
11
+ Project-URL: Bug Tracker, https://github.com/Helsinki-NLP/OpusFilter/issues
12
+ Project-URL: Changelog, https://github.com/Helsinki-NLP/OpusFilter/blob/main/docs/CHANGELOG.md
13
+ Project-URL: Paper (ACL 2020), https://www.aclweb.org/anthology/2020.acl-demos.20
14
+ Keywords: parallel-corpora,corpus-filtering,natural-language-processing,machine-translation,data-cleaning,nlp,text-processing
15
+ Classifier: Development Status :: 5 - Production/Stable
16
+ Classifier: Intended Audience :: Developers
17
+ Classifier: Intended Audience :: Science/Research
18
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
19
+ Classifier: Topic :: Text Processing :: Linguistic
8
20
  Classifier: Programming Language :: Python :: 3
9
- Classifier: License :: OSI Approved :: MIT License
21
+ Classifier: Programming Language :: Python :: 3.9
22
+ Classifier: Programming Language :: Python :: 3.10
23
+ Classifier: Programming Language :: Python :: 3.11
24
+ Classifier: Programming Language :: Python :: 3.12
25
+ Classifier: Programming Language :: Python :: 3.13
26
+ Classifier: Programming Language :: Python :: 3.14
10
27
  Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.8
28
+ Requires-Python: >=3.9
12
29
  Description-Content-Type: text/markdown
13
30
  License-File: LICENSE
14
31
  Requires-Dist: setuptools
15
- Requires-Dist: opustools>=1.6.2
32
+ Requires-Dist: opustools>=1.10.0
16
33
  Requires-Dist: beautifulsoup4>=4.8.0
17
34
  Requires-Dist: graphviz
18
35
  Requires-Dist: py3langid>=0.2.2
19
36
  Requires-Dist: matplotlib
20
37
  Requires-Dist: morfessor
21
- Requires-Dist: opus-fast-mosestokenizer==0.0.8.7; python_version < "3.9"
22
- Requires-Dist: opus-fast-mosestokenizer>=0.0.8.7; python_version >= "3.9"
38
+ Requires-Dist: opus-fast-mosestokenizer>=0.0.8.11
23
39
  Requires-Dist: pandas>=1.0.0
24
40
  Requires-Dist: xxhash>=3.2.0
25
41
  Requires-Dist: sentence-splitter
@@ -30,16 +46,14 @@ Requires-Dist: requests
30
46
  Requires-Dist: scikit-learn
31
47
  Requires-Dist: subword_nmt
32
48
  Requires-Dist: tqdm
49
+ Requires-Dist: lingua-language-detector>=2.2.0; python_version >= "3.12"
50
+ Requires-Dist: lingua-language-detector<2.2,>=1.3.0; python_version >= "3.10" and python_version < "3.12"
33
51
  Requires-Dist: lingua-language-detector<2.1,>=1.3.0; python_version < "3.10"
34
- Requires-Dist: lingua-language-detector>=2.1.1; python_version >= "3.10"
35
52
  Requires-Dist: iso639-lang
36
- Provides-Extra: test
37
- Requires-Dist: pytest; extra == "test"
38
53
  Provides-Extra: pycld2
39
54
  Requires-Dist: pycld2; extra == "pycld2"
40
55
  Provides-Extra: fasttext
41
56
  Requires-Dist: py3langid<0.3.0; extra == "fasttext"
42
- Requires-Dist: numpy<2.0.0; extra == "fasttext"
43
57
  Requires-Dist: fasttext; extra == "fasttext"
44
58
  Provides-Extra: eflomal
45
59
  Requires-Dist: eflomal>=2.0.0; extra == "eflomal"
@@ -54,23 +68,25 @@ Provides-Extra: varikn
54
68
  Requires-Dist: varikn; extra == "varikn"
55
69
  Provides-Extra: heliport
56
70
  Requires-Dist: heliport>=0.10.0; extra == "heliport"
71
+ Provides-Extra: huggingface
72
+ Requires-Dist: datasets>=2.14.0; extra == "huggingface"
73
+ Provides-Extra: test
74
+ Requires-Dist: pytest; extra == "test"
57
75
  Provides-Extra: docs
58
76
  Requires-Dist: myst-parser; extra == "docs"
59
77
  Requires-Dist: sphinx; extra == "docs"
60
78
  Requires-Dist: sphinx-rtd-theme; extra == "docs"
61
79
  Requires-Dist: sphinxcontrib-bibtex; extra == "docs"
62
80
  Provides-Extra: all
63
- Requires-Dist: eflomal>=2.0.0; extra == "all"
64
- Requires-Dist: jieba>=0.42; extra == "all"
65
- Requires-Dist: mecab-python3>=1.0.8; extra == "all"
66
- Requires-Dist: unidic-lite; extra == "all"
67
- Requires-Dist: laserembeddings; extra == "all"
68
- Requires-Dist: varikn; extra == "all"
69
- Requires-Dist: pytest; extra == "all"
70
- Requires-Dist: myst-parser; extra == "all"
71
- Requires-Dist: sphinx; extra == "all"
72
- Requires-Dist: sphinx-rtd-theme; extra == "all"
73
- Requires-Dist: sphinxcontrib-bibtex; extra == "all"
81
+ Requires-Dist: opusfilter[docs,eflomal,fasttext,heliport,jieba,laser,mecab,pycld2,test,varikn]; extra == "all"
82
+ Provides-Extra: dev
83
+ Requires-Dist: pytest; extra == "dev"
84
+ Requires-Dist: pytest-cov; extra == "dev"
85
+ Requires-Dist: black; extra == "dev"
86
+ Requires-Dist: isort; extra == "dev"
87
+ Requires-Dist: pylint; extra == "dev"
88
+ Requires-Dist: mypy; extra == "dev"
89
+ Dynamic: license-file
74
90
 
75
91
  # OpusFilter
76
92
 
@@ -99,7 +115,7 @@ Install from source:
99
115
 
100
116
  ### Troubleshooting
101
117
 
102
- OpusFilter should generally work fine on Python 3.8 to 3.13. In the case of troubles, try installing the exact versions in `requirements.txt`:
118
+ OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
103
119
 
104
120
  * `pip install -r requirements.txt`
105
121
 
@@ -139,3 +155,13 @@ A full bibliography of papers cited in the documentation and code can be found f
139
155
  ## Contributing
140
156
 
141
157
  See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
158
+
159
+
160
+ ## Acknowledgements
161
+
162
+ OpusFilter has partially been supported by various projects such as:
163
+
164
+ * [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
165
+ * [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
166
+ * [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
167
+ * [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
@@ -25,7 +25,7 @@ Install from source:
25
25
 
26
26
  ### Troubleshooting
27
27
 
28
- OpusFilter should generally work fine on Python 3.8 to 3.13. In the case of troubles, try installing the exact versions in `requirements.txt`:
28
+ OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
29
29
 
30
30
  * `pip install -r requirements.txt`
31
31
 
@@ -65,3 +65,13 @@ A full bibliography of papers cited in the documentation and code can be found f
65
65
  ## Contributing
66
66
 
67
67
  See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
68
+
69
+
70
+ ## Acknowledgements
71
+
72
+ OpusFilter has partially been supported by various projects such as:
73
+
74
+ * [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
75
+ * [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
76
+ * [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
77
+ * [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import sys
4
+ import warnings
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.main import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-autogen is deprecated and will be removed in a future release. "
8
+ "Please use the 'autogen' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.autogen import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-cmd is deprecated and will be removed in a future release. "
8
+ "Please use the 'cmd' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.cmd import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-diagram is deprecated and will be removed in a future release. "
8
+ "Please use the 'diagram' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.diagram import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-duplicates is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter-duplicates' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.duplicates import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-scores is deprecated and will be removed in a future release. "
8
+ "Please use the 'scores' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.scores import main
14
+ sys.exit(main())
@@ -0,0 +1,15 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import sys
4
+ import warnings
5
+
6
+ from opusfilter.cli.slurm import main
7
+
8
+ warnings.warn(
9
+ "opusfilter-slurm is experimental. Please report issues at "
10
+ "https://github.com/Helsinki-NLP/OpusFilter/issues",
11
+ UserWarning,
12
+ stacklevel=2
13
+ )
14
+
15
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-test is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter-test' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.test import main
14
+ sys.exit(main())
@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [3.4.0] - 2026-11-18
11
+
12
+ ### Added
13
+
14
+ - support for Python 3.14
15
+ - `opusfilter-slurm-*` commands for running OpusFilter workflows in SLURM clusters
16
+ - improved validation for configuration files
17
+ - support for encoding text in JSONL files to preserve newlines
18
+ - `with_decision` option for the `score` function
19
+ - `columns` option for the `unzip` function
20
+ - `hf_read` function for reading data from Hugging Face datasets
21
+ - inline filtering for `hf_read` and `opus_read`
22
+ - inline preprocessing for `hf_read` and `opus_read`
23
+ - `doc_level` and `limit` options for `opus_read`
24
+ - `WDSFilter` based on Web Docs Scorer
25
+ - `score_output` option for saving filter scores from `filter`, `opus_read`, and `hf_read`
26
+
27
+ ### Changed
28
+
29
+ - modernized packaging and CLI scripts
30
+
31
+ ### Removed
32
+
33
+ - support for Python 3.8
34
+
35
+ ## [3.3.1] - 2025-12-10
36
+
37
+ ### Changed
38
+
39
+ - changed `HtmlTagFilter` to accept broken tags in order to avoid inconsistent results
40
+ - update required `opus-fast-mosestokenizer` version for better platform support
41
+
42
+ ### Fixed
43
+
44
+ - broken citations and links in the documentation
45
+
10
46
  ## [3.3.0] - 2025-07-02
11
47
 
12
48
  ### Added
@@ -234,7 +270,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
234
270
  First tagged version.
235
271
 
236
272
 
237
- [Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.0...develop
273
+ [Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.4.0...develop
274
+ [3.4.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.1...3.4.0
275
+ [3.3.1]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.0...3.3.1
238
276
  [3.3.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.2.0...3.3.0
239
277
  [3.2.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.1.0...3.2.0
240
278
  [3.1.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.0.0...3.1.0
@@ -5,7 +5,7 @@ issues page. We are also happy to consider pull requests. There are a
5
5
  few rules for pull requests:
6
6
 
7
7
  * Make a pull request to the `develop` branch instead of `master`.
8
- * The code should support at least Python versions from 3.8 to 3.13.
8
+ * The code should support at least Python versions from 3.9 to 3.14.
9
9
  * Please follow [PEP 8](https://www.python.org/dev/peps/pep-0008/). Exception: The maximum line length is 127 characters instead of 79.
10
10
  * Especially for new features, please include test cases for unit testing.
11
11
 
@@ -19,8 +19,8 @@ work, if you have VariKN and eflomal set up as instructed - `pytest`
19
19
  skips the respective tests if not.)
20
20
 
21
21
  GitHub workflows defined in the project run automatically `flake8`
22
- checks and unit testing with `pytest` using Python 3.8, 3.9, 3.10,
23
- 3.11, 3.12, and 3.13.
22
+ checks and unit testing with `pytest` using Python 3.9, 3.10,
23
+ 3.11, 3.12, and 3.14.
24
24
 
25
25
  Especially for larger contributions, consider using a code analysis
26
26
  tool like [Pylint](https://github.com/PyCQA/pylint). Install it