opusfilter 3.3.1__tar.gz → 3.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (141) hide show
  1. {opusfilter-3.3.1 → opusfilter-3.4.0}/.github/workflows/ci.yml +24 -11
  2. {opusfilter-3.3.1 → opusfilter-3.4.0}/.gitignore +1 -0
  3. opusfilter-3.4.0/AGENTS.md +177 -0
  4. {opusfilter-3.3.1 → opusfilter-3.4.0}/PKG-INFO +52 -25
  5. {opusfilter-3.3.1 → opusfilter-3.4.0}/README.md +11 -1
  6. opusfilter-3.4.0/bin/opusfilter +14 -0
  7. opusfilter-3.4.0/bin/opusfilter-autogen +14 -0
  8. opusfilter-3.4.0/bin/opusfilter-cmd +14 -0
  9. opusfilter-3.4.0/bin/opusfilter-diagram +14 -0
  10. opusfilter-3.4.0/bin/opusfilter-duplicates +14 -0
  11. opusfilter-3.4.0/bin/opusfilter-scores +14 -0
  12. opusfilter-3.4.0/bin/opusfilter-slurm +15 -0
  13. opusfilter-3.4.0/bin/opusfilter-test +14 -0
  14. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/CHANGELOG.md +27 -1
  15. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/CONTRIBUTING.md +3 -3
  16. opusfilter-3.4.0/docs/filters/document_scoring_filters.md +103 -0
  17. opusfilter-3.4.0/docs/functions/downloading_and_selecting_data.md +317 -0
  18. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/filtering_and_scoring.md +18 -0
  19. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/training_language_and_alignment_models.md +1 -1
  20. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/index.rst +1 -0
  21. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/installation.md +37 -3
  22. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/morfessor_segmentation.md +1 -1
  23. opusfilter-3.4.0/docs/slurm_integration.md +382 -0
  24. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/usage.md +102 -3
  25. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/qed_lm_langid.yaml +18 -0
  26. opusfilter-3.4.0/example_configs/slurm_example.yaml +155 -0
  27. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/__init__.py +5 -0
  28. opusfilter-3.4.0/opusfilter/_version.py +24 -0
  29. opusfilter-3.4.0/opusfilter/cli/__init__.py +3 -0
  30. opusfilter-3.4.0/opusfilter/cli/autogen.py +101 -0
  31. opusfilter-3.4.0/opusfilter/cli/cmd.py +94 -0
  32. opusfilter-3.4.0/opusfilter/cli/diagram.py +83 -0
  33. opusfilter-3.4.0/opusfilter/cli/duplicates.py +92 -0
  34. opusfilter-3.4.0/opusfilter/cli/main.py +70 -0
  35. opusfilter-3.3.1/bin/opusfilter-scores → opusfilter-3.4.0/opusfilter/cli/scores.py +68 -38
  36. opusfilter-3.4.0/opusfilter/cli/slurm.py +65 -0
  37. opusfilter-3.4.0/opusfilter/cli/slurm_status.py +168 -0
  38. opusfilter-3.4.0/opusfilter/cli/slurm_submit.py +67 -0
  39. opusfilter-3.4.0/opusfilter/cli/test.py +85 -0
  40. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/filters.py +11 -4
  41. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/lid.py +16 -4
  42. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/opusfilter.py +451 -71
  43. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/pipeline.py +50 -9
  44. opusfilter-3.4.0/opusfilter/slurm.py +607 -0
  45. opusfilter-3.4.0/opusfilter/slurm_utils.py +411 -0
  46. opusfilter-3.4.0/opusfilter/util.py +648 -0
  47. opusfilter-3.4.0/opusfilter/validate.py +281 -0
  48. opusfilter-3.4.0/opusfilter/wds.py +257 -0
  49. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/PKG-INFO +52 -25
  50. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/SOURCES.txt +40 -2
  51. opusfilter-3.4.0/opusfilter.egg-info/entry_points.txt +11 -0
  52. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/requires.txt +17 -14
  53. opusfilter-3.4.0/opusfilter.egg-info/scm_file_list.json +125 -0
  54. opusfilter-3.4.0/opusfilter.egg-info/scm_version.json +8 -0
  55. opusfilter-3.4.0/pyproject.toml +184 -0
  56. {opusfilter-3.3.1 → opusfilter-3.4.0}/requirements.txt +3 -2
  57. opusfilter-3.4.0/templates/slurm/step_template.sbatch +30 -0
  58. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_autogen.py +2 -1
  59. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_classifier.py +1 -1
  60. opusfilter-3.4.0/tests/test_cli/__init__.py +1 -0
  61. opusfilter-3.4.0/tests/test_cli/test_autogen.py +38 -0
  62. opusfilter-3.4.0/tests/test_cli/test_cmd.py +38 -0
  63. opusfilter-3.4.0/tests/test_cli/test_diagram.py +75 -0
  64. opusfilter-3.4.0/tests/test_cli/test_duplicates.py +59 -0
  65. opusfilter-3.4.0/tests/test_cli/test_main.py +40 -0
  66. opusfilter-3.4.0/tests/test_cli/test_scores.py +60 -0
  67. opusfilter-3.4.0/tests/test_cli/test_test.py +52 -0
  68. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_filter_pipeline.py +101 -5
  69. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_lid.py +1 -1
  70. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_opusfilter.py +1100 -17
  71. opusfilter-3.4.0/tests/test_slurm.py +129 -0
  72. opusfilter-3.4.0/tests/test_slurm_e2e.py +70 -0
  73. opusfilter-3.4.0/tests/test_slurm_simple.py +88 -0
  74. opusfilter-3.4.0/tests/test_slurm_utils.py +207 -0
  75. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_util.py +91 -0
  76. opusfilter-3.4.0/tests/test_validate.py +316 -0
  77. opusfilter-3.4.0/tests/test_wds.py +232 -0
  78. opusfilter-3.3.1/bin/opusfilter +0 -33
  79. opusfilter-3.3.1/bin/opusfilter-autogen +0 -91
  80. opusfilter-3.3.1/bin/opusfilter-cmd +0 -86
  81. opusfilter-3.3.1/bin/opusfilter-diagram +0 -108
  82. opusfilter-3.3.1/bin/opusfilter-duplicates +0 -85
  83. opusfilter-3.3.1/bin/opusfilter-test +0 -76
  84. opusfilter-3.3.1/docs/functions/downloading_and_selecting_data.md +0 -178
  85. opusfilter-3.3.1/opusfilter/util.py +0 -274
  86. opusfilter-3.3.1/pyproject.toml +0 -12
  87. opusfilter-3.3.1/setup.py +0 -105
  88. {opusfilter-3.3.1 → opusfilter-3.4.0}/.pylintrc +0 -0
  89. {opusfilter-3.3.1 → opusfilter-3.4.0}/LICENSE +0 -0
  90. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/automatic_configuration.md +0 -0
  91. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/command_line_tools.md +0 -0
  92. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/conf.py +0 -0
  93. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/alignment_model_filters.md +0 -0
  94. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/custom_filters.md +0 -0
  95. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/language_model_filters.md +0 -0
  96. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/length_filters.md +0 -0
  97. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/script_and_language_identification_filters.md +0 -0
  98. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/sentence_embedding_filters.md +0 -0
  99. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/filters/special_character_and_similarity_filters.md +0 -0
  100. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/preprocessing_text.md +0 -0
  101. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/training_and_using_classifiers.md +0 -0
  102. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/functions/using_score_files.md +0 -0
  103. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/bpe_segmentation.md +0 -0
  104. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/custom_preprocessors.md +0 -0
  105. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/detokenizer.md +0 -0
  106. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/monolingual_sentence_splitter.md +0 -0
  107. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/reg_exp_sub.md +0 -0
  108. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/tokenizer.md +0 -0
  109. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/preprocessors/whitespaceNormalizer.md +0 -0
  110. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/references.bib +0 -0
  111. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/references.rst +0 -0
  112. {opusfilter-3.3.1 → opusfilter-3.4.0}/docs/requirements.txt +0 -0
  113. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/README.md +0 -0
  114. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_ce_sets.yaml +0 -0
  115. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_domain_sets.yaml +0 -0
  116. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/create_roc_auc_sets.yaml +0 -0
  117. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_1_labels.jsonl +0 -0
  118. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/devset_100_2_labels.jsonl +0 -0
  119. {opusfilter-3.3.1 → opusfilter-3.4.0}/example_configs/paracrawl_fi-en/prepare_data.yaml +0 -0
  120. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/autogen.py +0 -0
  121. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/autogen_cluster.py +0 -0
  122. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/classifier.py +0 -0
  123. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/embeddings.py +0 -0
  124. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/lm.py +0 -0
  125. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/preprocessors.py +0 -0
  126. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/segment_hash.py +0 -0
  127. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/subwords.py +0 -0
  128. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/tokenization.py +0 -0
  129. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter/word_alignment.py +0 -0
  130. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/dependency_links.txt +0 -0
  131. {opusfilter-3.3.1 → opusfilter-3.4.0}/opusfilter.egg-info/top_level.txt +0 -0
  132. {opusfilter-3.3.1 → opusfilter-3.4.0}/setup.cfg +0 -0
  133. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_embeddings.py +0 -0
  134. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_filters.py +0 -0
  135. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_lm_filter.py +0 -0
  136. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_preprocessors.py +0 -0
  137. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_segment_hash.py +0 -0
  138. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_subwords.py +0 -0
  139. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_tokenization.py +0 -0
  140. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_wordalign_filter.py +0 -0
  141. {opusfilter-3.3.1 → opusfilter-3.4.0}/tests/test_yaml.py +0 -0
@@ -21,13 +21,17 @@ jobs:
21
21
  fail-fast: false
22
22
  matrix:
23
23
  os: [ubuntu-latest]
24
- python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
24
+ python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
25
25
  requirements-file: ["requirements.txt"]
26
+ include:
27
+ - os: ubuntu-22.04
28
+ python-version: "3.9"
29
+ requirements-file: "requirements.txt"
26
30
  runs-on: ${{ matrix.os }}
27
31
  steps:
28
- - uses: actions/checkout@v4
32
+ - uses: actions/checkout@v6
29
33
  - name: Set up Python ${{ matrix.python-version }}
30
- uses: actions/setup-python@v5
34
+ uses: actions/setup-python@v6
31
35
  with:
32
36
  python-version: ${{ matrix.python-version }}
33
37
  - name: Install dependencies
@@ -46,12 +50,21 @@ jobs:
46
50
  flake8 opusfilter --count --exit-zero --statistics
47
51
  - name: Run tests with pytest
48
52
  run: pytest
53
+ - name: Test CLI entry points
54
+ run: |
55
+ opusfilter --help
56
+ opusfilter-cmd --help
57
+ opusfilter-scores list --help
58
+ opusfilter-autogen --help
59
+ opusfilter-test --help
60
+ opusfilter-diagram --help
61
+ opusfilter-duplicates --help
49
62
  build-n-publish:
50
63
  name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
51
64
  needs: build-n-test
52
65
  runs-on: ubuntu-latest
53
66
  steps:
54
- - uses: actions/checkout@v4
67
+ - uses: actions/checkout@v6
55
68
  - name: Check for main release tag
56
69
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
57
70
  id: check-tag
@@ -59,11 +72,11 @@ jobs:
59
72
  if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+(rc[0-9]*)?$ ]]; then
60
73
  echo "match=true" >> $GITHUB_OUTPUT
61
74
  fi
62
- - name: Set up Python 3.8
75
+ - name: Set up Python 3.11
63
76
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
64
- uses: actions/setup-python@v5
77
+ uses: actions/setup-python@v6
65
78
  with:
66
- python-version: 3.8
79
+ python-version: "3.11"
67
80
  - name: Install pypa/build
68
81
  if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
69
82
  run: >-
@@ -96,11 +109,11 @@ jobs:
96
109
  runs-on: ubuntu-latest
97
110
  needs: [build-n-test]
98
111
  steps:
99
- - uses: actions/checkout@v4
100
- - name: Set up Python 3.8
101
- uses: actions/setup-python@v5
112
+ - uses: actions/checkout@v6
113
+ - name: Set up Python 3.11
114
+ uses: actions/setup-python@v6
102
115
  with:
103
- python-version: 3.8
116
+ python-version: "3.11"
104
117
  - name: Install dependencies to build docs
105
118
  run: |
106
119
  python -m pip install --upgrade pip
@@ -13,3 +13,4 @@ htmlcov
13
13
  *.zip
14
14
  *.gz
15
15
  venv
16
+ opusfilter/_version.py
@@ -0,0 +1,177 @@
1
+ # OpusFilter Development Guide
2
+
3
+ This guide contains essential information for agents working on the OpusFilter codebase.
4
+
5
+ ## Project Overview
6
+
7
+ OpusFilter is a tool for filtering and processing parallel corpora. It's written in Python and supports filtering through various methods including language identification, language models, word alignment, and custom filters.
8
+
9
+ ## Development Commands
10
+
11
+ ### Testing
12
+ - **Run all tests**: `pytest tests/`
13
+ - **Run single test file**: `pytest tests/test_filename.py`
14
+ - **Run specific test**: `pytest tests/test_filename.py::ClassName::test_method`
15
+ - **Run with verbose output**: `pytest -v tests/`
16
+
17
+ ### Linting and Code Quality
18
+ - **Check style**: `flake8 opusfilter/`
19
+ - **Run flake8 with statistics**: `flake8 opusfilter --count --exit-zero --statistics`
20
+ - **Run full linting (recommended for contributions)**: `pylint opusfilter/`
21
+ - **Format code**: `black opusfilter/ opusfilter/cli/`
22
+ - **Sort imports**: `isort opusfilter/ opusfilter/cli/`
23
+ - **Type checking**: `mypy opusfilter/ opusfilter/cli/`
24
+
25
+ ### Environment Setup
26
+ - **Install with test dependencies**: `pip install .[test]`
27
+ - **Install optional dependencies for full functionality**: `pip install .[test,eflomal,jieba,mecab,varikn]`
28
+
29
+ ## Code Style Guidelines
30
+
31
+ ### General Rules
32
+ - Follow PEP 8 with a maximum line length of 127 characters (not 79)
33
+ - Support Python 3.9 to 3.14
34
+ - Use type hints where appropriate (see existing code for patterns)
35
+ - Maintain backward compatibility when possible
36
+ - Do not add trailing whitespace
37
+
38
+ ### Import Organization
39
+ ```python
40
+ # Standard library imports
41
+ import os
42
+ import logging
43
+ from typing import Iterator, List, Tuple
44
+
45
+ # Third-party imports
46
+ import numpy as np
47
+ import pandas as pd
48
+ import regex
49
+
50
+ # Local imports
51
+ from . import FilterABC, ConfigurationError
52
+ from .util import file_open, count_lines
53
+ ```
54
+
55
+ ### Naming Conventions
56
+ - **Classes**: PascalCase (e.g., `LengthFilter`, `CrossEntropyFilter`)
57
+ - **Functions/Methods**: snake_case (e.g., `get_length`, `score_pairs`)
58
+ - **Variables**: snake_case (e.g., `min_length`, `accept_threshold`)
59
+ - **Constants**: UPPER_SNAKE_CASE (e.g., `CLEAN_LOW`, `CLEAN_HIGH`)
60
+ - **Private members**: Leading underscore (e.g., `_internal_method`)
61
+
62
+ ### Error Handling
63
+ - Use custom exception classes from `opusfilter.__init__`:
64
+ - `OpusFilterError`: Base exception
65
+ - `ConfigurationError`: For configuration-related errors
66
+ - `OpusFilterRuntimeError`: For runtime errors
67
+ - Always include descriptive error messages
68
+ - Use logging for warnings and debug information
69
+
70
+ ### Filter Implementation Pattern
71
+ When creating new filters, follow this pattern:
72
+ ```python
73
+ class NewFilter(FilterABC):
74
+ """Brief description of the filter"""
75
+
76
+ score_direction = CLEAN_LOW # or CLEAN_HIGH, CLEAN_BETWEEN, etc.
77
+ accept_threshold = <value>
78
+ reject_threshold = <value>
79
+
80
+ def __init__(self, required_param, optional_param=default, **kwargs):
81
+ # Validate parameters
82
+ self.required_param = required_param
83
+ self.optional_param = optional_param
84
+ super().__init__(**kwargs)
85
+
86
+ def score(self, pairs):
87
+ """Yield scores for each sentence pair"""
88
+ for pair in pairs:
89
+ # Calculate score
90
+ yield score
91
+
92
+ def accept(self, score):
93
+ """Return True if score passes the filter"""
94
+ return <condition>
95
+ ```
96
+
97
+ ### Testing Guidelines
98
+ - Write unit tests for all new functionality
99
+ - Use descriptive test method names
100
+ - Test both success and failure cases
101
+ - Use `unittest.TestCase` as base class
102
+ - Mark tests that require optional dependencies with appropriate skips
103
+
104
+ ### Logging
105
+ - Use module-level logger: `logger = logging.getLogger(__name__)`
106
+ - Log important operations and warnings
107
+ - Avoid excessive debug logging in production code
108
+
109
+ ### File I/O
110
+ - Use `file_open()` from `opusfilter.util` for handling compressed files
111
+ - Support common formats: plain text, gzip, bzip2, lzma
112
+ - Use context managers for file operations
113
+
114
+ ### Configuration
115
+ - Filters receive configuration through `__init__` parameters
116
+ - Use `**kwargs` to capture extra parameters and warn about them
117
+ - Validate parameters and raise `ConfigurationError` for invalid values
118
+
119
+ ### Type Hints
120
+ - Use type hints for public APIs
121
+ - Import common types from `typing` module
122
+ - Follow existing patterns for complex types
123
+
124
+ ### Documentation
125
+ - Add docstrings to all public classes and methods
126
+ - Use Google-style or NumPy-style docstrings
127
+ - Include examples in docstrings when helpful
128
+
129
+ ## Project Structure
130
+
131
+ - `opusfilter/`: Main package directory
132
+ - `filters.py`: Core filter implementations
133
+ - `preprocessors.py`: Text preprocessing functions
134
+ - `lm.py`: Language model filters
135
+ - `lid.py`: Language identification filters
136
+ - `embeddings.py`: Sentence embedding filters
137
+ - `word_alignment.py`: Word alignment filters
138
+ - `util.py`: Utility functions and helpers
139
+ - `pipeline.py`: Processing pipeline implementation
140
+ - `opusfilter.py`: Main application entry point
141
+ - `tests/`: Unit tests
142
+ - `docs/`: Documentation source files
143
+ - `work/`: Example configurations and working files
144
+
145
+ ## Common Patterns
146
+
147
+ ### Iterating over sentence pairs
148
+ ```python
149
+ for pair in pairs:
150
+ # pair is a list/tuple of strings
151
+ # pair[0] is source, pair[1] is target, etc.
152
+ ```
153
+
154
+ ### Yielding scores
155
+ ```python
156
+ def score(self, pairs):
157
+ for pair in pairs:
158
+ # Calculate score(s)
159
+ yield score_value # or yield [score1, score2, ...]
160
+ ```
161
+
162
+ ### Checking filter acceptance
163
+ ```python
164
+ def accept(self, score):
165
+ if isinstance(score, list):
166
+ # Multiple scores
167
+ return all(condition(s) for s in score)
168
+ else:
169
+ # Single score
170
+ return condition(score)
171
+ ```
172
+
173
+ ## Git Workflow
174
+ - Make pull requests to the `develop` branch (not `master`)
175
+ - Include tests for new features
176
+ - Run linting and tests before submitting
177
+ - Follow conventional commit messages if possible
@@ -1,18 +1,35 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: opusfilter
3
- Version: 3.3.1
4
- Summary: Toolbox for filtering parallel corpora
5
- Home-page: https://github.com/Helsinki-NLP/OpusFilter
6
- Author: Mikko Aulamo, Sami Virpioja
7
- Author-email: mikko.aulamo@helsinki.fi
3
+ Version: 3.4.0
4
+ Summary: Toolbox for filtering and processing parallel corpora
5
+ Author-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
6
+ Maintainer-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
7
+ License-Expression: MIT
8
+ Project-URL: Homepage, https://github.com/Helsinki-NLP/OpusFilter
9
+ Project-URL: Documentation, https://helsinki-nlp.github.io/OpusFilter/
10
+ Project-URL: Repository, https://github.com/Helsinki-NLP/OpusFilter.git
11
+ Project-URL: Bug Tracker, https://github.com/Helsinki-NLP/OpusFilter/issues
12
+ Project-URL: Changelog, https://github.com/Helsinki-NLP/OpusFilter/blob/main/docs/CHANGELOG.md
13
+ Project-URL: Paper (ACL 2020), https://www.aclweb.org/anthology/2020.acl-demos.20
14
+ Keywords: parallel-corpora,corpus-filtering,natural-language-processing,machine-translation,data-cleaning,nlp,text-processing
15
+ Classifier: Development Status :: 5 - Production/Stable
16
+ Classifier: Intended Audience :: Developers
17
+ Classifier: Intended Audience :: Science/Research
18
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
19
+ Classifier: Topic :: Text Processing :: Linguistic
8
20
  Classifier: Programming Language :: Python :: 3
9
- Classifier: License :: OSI Approved :: MIT License
21
+ Classifier: Programming Language :: Python :: 3.9
22
+ Classifier: Programming Language :: Python :: 3.10
23
+ Classifier: Programming Language :: Python :: 3.11
24
+ Classifier: Programming Language :: Python :: 3.12
25
+ Classifier: Programming Language :: Python :: 3.13
26
+ Classifier: Programming Language :: Python :: 3.14
10
27
  Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.8
28
+ Requires-Python: >=3.9
12
29
  Description-Content-Type: text/markdown
13
30
  License-File: LICENSE
14
31
  Requires-Dist: setuptools
15
- Requires-Dist: opustools>=1.6.2
32
+ Requires-Dist: opustools>=1.10.0
16
33
  Requires-Dist: beautifulsoup4>=4.8.0
17
34
  Requires-Dist: graphviz
18
35
  Requires-Dist: py3langid>=0.2.2
@@ -29,16 +46,14 @@ Requires-Dist: requests
29
46
  Requires-Dist: scikit-learn
30
47
  Requires-Dist: subword_nmt
31
48
  Requires-Dist: tqdm
49
+ Requires-Dist: lingua-language-detector>=2.2.0; python_version >= "3.12"
50
+ Requires-Dist: lingua-language-detector<2.2,>=1.3.0; python_version >= "3.10" and python_version < "3.12"
32
51
  Requires-Dist: lingua-language-detector<2.1,>=1.3.0; python_version < "3.10"
33
- Requires-Dist: lingua-language-detector>=2.1.1; python_version >= "3.10"
34
52
  Requires-Dist: iso639-lang
35
- Provides-Extra: test
36
- Requires-Dist: pytest; extra == "test"
37
53
  Provides-Extra: pycld2
38
54
  Requires-Dist: pycld2; extra == "pycld2"
39
55
  Provides-Extra: fasttext
40
56
  Requires-Dist: py3langid<0.3.0; extra == "fasttext"
41
- Requires-Dist: numpy<2.0.0; extra == "fasttext"
42
57
  Requires-Dist: fasttext; extra == "fasttext"
43
58
  Provides-Extra: eflomal
44
59
  Requires-Dist: eflomal>=2.0.0; extra == "eflomal"
@@ -53,23 +68,25 @@ Provides-Extra: varikn
53
68
  Requires-Dist: varikn; extra == "varikn"
54
69
  Provides-Extra: heliport
55
70
  Requires-Dist: heliport>=0.10.0; extra == "heliport"
71
+ Provides-Extra: huggingface
72
+ Requires-Dist: datasets>=2.14.0; extra == "huggingface"
73
+ Provides-Extra: test
74
+ Requires-Dist: pytest; extra == "test"
56
75
  Provides-Extra: docs
57
76
  Requires-Dist: myst-parser; extra == "docs"
58
77
  Requires-Dist: sphinx; extra == "docs"
59
78
  Requires-Dist: sphinx-rtd-theme; extra == "docs"
60
79
  Requires-Dist: sphinxcontrib-bibtex; extra == "docs"
61
80
  Provides-Extra: all
62
- Requires-Dist: eflomal>=2.0.0; extra == "all"
63
- Requires-Dist: jieba>=0.42; extra == "all"
64
- Requires-Dist: mecab-python3>=1.0.8; extra == "all"
65
- Requires-Dist: unidic-lite; extra == "all"
66
- Requires-Dist: laserembeddings; extra == "all"
67
- Requires-Dist: varikn; extra == "all"
68
- Requires-Dist: pytest; extra == "all"
69
- Requires-Dist: myst-parser; extra == "all"
70
- Requires-Dist: sphinx; extra == "all"
71
- Requires-Dist: sphinx-rtd-theme; extra == "all"
72
- Requires-Dist: sphinxcontrib-bibtex; extra == "all"
81
+ Requires-Dist: opusfilter[docs,eflomal,fasttext,heliport,jieba,laser,mecab,pycld2,test,varikn]; extra == "all"
82
+ Provides-Extra: dev
83
+ Requires-Dist: pytest; extra == "dev"
84
+ Requires-Dist: pytest-cov; extra == "dev"
85
+ Requires-Dist: black; extra == "dev"
86
+ Requires-Dist: isort; extra == "dev"
87
+ Requires-Dist: pylint; extra == "dev"
88
+ Requires-Dist: mypy; extra == "dev"
89
+ Dynamic: license-file
73
90
 
74
91
  # OpusFilter
75
92
 
@@ -98,7 +115,7 @@ Install from source:
98
115
 
99
116
  ### Troubleshooting
100
117
 
101
- OpusFilter should generally work fine on Python 3.8 to 3.13. In the case of troubles, try installing the exact versions in `requirements.txt`:
118
+ OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
102
119
 
103
120
  * `pip install -r requirements.txt`
104
121
 
@@ -138,3 +155,13 @@ A full bibliography of papers cited in the documentation and code can be found f
138
155
  ## Contributing
139
156
 
140
157
  See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
158
+
159
+
160
+ ## Acknowledgements
161
+
162
+ OpusFilter has partially been supported by various projects such as:
163
+
164
+ * [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
165
+ * [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
166
+ * [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
167
+ * [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
@@ -25,7 +25,7 @@ Install from source:
25
25
 
26
26
  ### Troubleshooting
27
27
 
28
- OpusFilter should generally work fine on Python 3.8 to 3.13. In the case of troubles, try installing the exact versions in `requirements.txt`:
28
+ OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:
29
29
 
30
30
  * `pip install -r requirements.txt`
31
31
 
@@ -65,3 +65,13 @@ A full bibliography of papers cited in the documentation and code can be found f
65
65
  ## Contributing
66
66
 
67
67
  See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).
68
+
69
+
70
+ ## Acknowledgements
71
+
72
+ OpusFilter has partially been supported by various projects such as:
73
+
74
+ * [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
75
+ * [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
76
+ * [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
77
+ * [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import sys
4
+ import warnings
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.main import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-autogen is deprecated and will be removed in a future release. "
8
+ "Please use the 'autogen' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.autogen import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-cmd is deprecated and will be removed in a future release. "
8
+ "Please use the 'cmd' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.cmd import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-diagram is deprecated and will be removed in a future release. "
8
+ "Please use the 'diagram' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.diagram import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-duplicates is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter-duplicates' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.duplicates import main
14
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-scores is deprecated and will be removed in a future release. "
8
+ "Please use the 'scores' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.scores import main
14
+ sys.exit(main())
@@ -0,0 +1,15 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import sys
4
+ import warnings
5
+
6
+ from opusfilter.cli.slurm import main
7
+
8
+ warnings.warn(
9
+ "opusfilter-slurm is experimental. Please report issues at "
10
+ "https://github.com/Helsinki-NLP/OpusFilter/issues",
11
+ UserWarning,
12
+ stacklevel=2
13
+ )
14
+
15
+ sys.exit(main())
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env python3
2
+
3
+ import warnings
4
+ import sys
5
+
6
+ warnings.warn(
7
+ "Direct execution of bin/opusfilter-test is deprecated and will be removed in a future release. "
8
+ "Please use the 'opusfilter-test' command instead.",
9
+ DeprecationWarning,
10
+ stacklevel=2
11
+ )
12
+
13
+ from opusfilter.cli.test import main
14
+ sys.exit(main())
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [3.4.0] - 2026-11-18
11
+
12
+ ### Added
13
+
14
+ - support for Python 3.14
15
+ - `opusfilter-slurm-*` commands for running OpusFilter workflows in SLURM clusters
16
+ - improved validation for configuration files
17
+ - support for encoding text in JSONL files to preserve newlines
18
+ - `with_decision` option for the `score` function
19
+ - `columns` option for the `unzip` function
20
+ - `hf_read` function for reading data from Hugging Face datasets
21
+ - inline filtering for `hf_read` and `opus_read`
22
+ - inline preprocessing for `hf_read` and `opus_read`
23
+ - `doc_level` and `limit` options for `opus_read`
24
+ - `WDSFilter` based on Web Docs Scorer
25
+ - `score_output` option for saving filter scores from `filter`, `opus_read`, and `hf_read`
26
+
27
+ ### Changed
28
+
29
+ - modernized packaging and CLI scripts
30
+
31
+ ### Removed
32
+
33
+ - support for Python 3.8
34
+
10
35
  ## [3.3.1] - 2025-12-10
11
36
 
12
37
  ### Changed
@@ -245,7 +270,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
245
270
  First tagged version.
246
271
 
247
272
 
248
- [Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.1...develop
273
+ [Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.4.0...develop
274
+ [3.4.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.1...3.4.0
249
275
  [3.3.1]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.3.0...3.3.1
250
276
  [3.3.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.2.0...3.3.0
251
277
  [3.2.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.1.0...3.2.0
@@ -5,7 +5,7 @@ issues page. We are also happy to consider pull requests. There are a
5
5
  few rules for pull requests:
6
6
 
7
7
  * Make a pull request to the `develop` branch instead of `master`.
8
- * The code should support at least Python versions from 3.8 to 3.13.
8
+ * The code should support at least Python versions from 3.9 to 3.14.
9
9
  * Please follow [PEP 8](https://www.python.org/dev/peps/pep-0008/). Exception: The maximum line length is 127 characters instead of 79.
10
10
  * Especially for new features, please include test cases for unit testing.
11
11
 
@@ -19,8 +19,8 @@ work, if you have VariKN and eflomal set up as instructed - `pytest`
19
19
  skips the respective tests if not.)
20
20
 
21
21
  GitHub workflows defined in the project run automatically `flake8`
22
- checks and unit testing with `pytest` using Python 3.8, 3.9, 3.10,
23
- 3.11, 3.12, and 3.13.
22
+ checks and unit testing with `pytest` using Python 3.9, 3.10,
23
+ 3.11, 3.12, and 3.14.
24
24
 
25
25
  Especially for larger contributions, consider using a code analysis
26
26
  tool like [Pylint](https://github.com/PyCQA/pylint). Install it