mistral_translator 0.2.1 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.env.example +3 -0
- data/CHANGELOG.md +39 -0
- data/CONTRIBUTING.md +70 -0
- data/LICENSE.txt +6 -6
- data/README.md +48 -19
- data/docs/advanced-usage/concurrent-async.md +270 -0
- data/docs/api-reference/configuration.md +1 -1
- data/docs/api-reference/methods.md +1 -1
- data/docs/installation.md +1 -1
- data/lib/mistral_translator/client_helpers.rb +35 -5
- data/lib/mistral_translator/configuration.rb +56 -35
- data/lib/mistral_translator/logger.rb +11 -4
- data/lib/mistral_translator/version.rb +2 -2
- metadata +20 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 278978acb1df0d447293200f13ff58745640a62f5bf43a41ebc7b31db2d50b98
|
|
4
|
+
data.tar.gz: c86fc021d49a554fd4d44883e1383a5ea4ba82251074b4b358b3b5cf97f27fd2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ca6e0c3b2e2346ab8432dfc24080adcb994f75df453111851d6f8441465d09107d59216fc84efc1f84acf823a4a4ec284c5d1b1212c8ceda9a3d8861e95a2208
|
|
7
|
+
data.tar.gz: 39b55072eb24870acd861b8c7ec7c8aa279a9f7b03b4a84f0cdabd4982c2d9f04da1b39b2eea05024d6e3a2c1bc7737ee95557f0eaa588c61ec3fbc5d04af540
|
data/.env.example
ADDED
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,44 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.3.1] - 2026-10-27
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
|
|
7
|
+
- Default model is now `mistral-small-latest`. Mistral retired the `mistral-small` name, so every call without an explicit `config.model` failed with a 404. The alias follows the current Mistral Small version; set `config.model` to pin one.
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- `bin/check_api`: checks the API key and the default model, then runs the examples against the real API when the plan allows generation. The key is read from `MISTRAL_TEST_API_KEY`, `.env`, or a hidden prompt, and never printed.
|
|
12
|
+
- Weekly workflow running `bin/check_api`, which opens an issue on failure.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- CI: a single workflow, RuboCop plus the test suite on Ruby 3.2, 3.3, 3.4 and 4.0.
|
|
17
|
+
- The test suite no longer prints part of the API key, and no longer deletes VCR cassettes after each run.
|
|
18
|
+
|
|
19
|
+
## [0.3.0] - 2025-12-09
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
|
|
23
|
+
- Thread-safe metrics with Mutex protection for concurrent usage
|
|
24
|
+
- Connection pooling with net-http-persistent (automatic connection reuse)
|
|
25
|
+
- SSL/TLS configuration options: `ssl_verify_mode`, `ssl_ca_file`, `ssl_ca_path`, `ssl_timeout`
|
|
26
|
+
- Thread-safe logger cache for `warn_once` method
|
|
27
|
+
- Comprehensive concurrent/async documentation with SolidQueue, Sidekiq, Concurrent Ruby examples
|
|
28
|
+
- 39 new tests for thread-safety, SSL configuration, and connection pooling
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
|
|
32
|
+
- HTTP client now uses Net::HTTP::Persistent for better performance
|
|
33
|
+
- Configuration metrics now thread-safe across concurrent requests
|
|
34
|
+
- Documentation reorganized with SolidQueue as recommended background job backend
|
|
35
|
+
|
|
36
|
+
### Performance
|
|
37
|
+
|
|
38
|
+
- Connection pooling reduces TCP handshake overhead
|
|
39
|
+
- Thread-safe implementation enables safe concurrent translations
|
|
40
|
+
- Configurable SSL timeout for production environments
|
|
41
|
+
|
|
3
42
|
## [0.2.0] - 2025-09-09
|
|
4
43
|
|
|
5
44
|
### Added
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Contributing to MistralTranslator
|
|
2
|
+
|
|
3
|
+
Thank you for your interest in contributing to MistralTranslator! We welcome contributions of all kinds.
|
|
4
|
+
|
|
5
|
+
## Development Setup
|
|
6
|
+
|
|
7
|
+
### Prerequisites
|
|
8
|
+
|
|
9
|
+
- Ruby 3.2+
|
|
10
|
+
- Bundler
|
|
11
|
+
|
|
12
|
+
### Installation
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
git clone https://github.com/your-username/mistral_translator.git
|
|
16
|
+
cd mistral_translator
|
|
17
|
+
bundle install
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Running Tests
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
# Unit tests (no API key required)
|
|
24
|
+
bundle exec rspec
|
|
25
|
+
|
|
26
|
+
# Integration tests (requires real API key)
|
|
27
|
+
export MISTRAL_API_KEY=your_key && bundle exec rspec spec/integration/mistral_api_integration_spec.rb
|
|
28
|
+
|
|
29
|
+
# All tests with coverage
|
|
30
|
+
bundle exec rspec --format documentation
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Contributing Guidelines
|
|
34
|
+
|
|
35
|
+
### Reporting Issues
|
|
36
|
+
|
|
37
|
+
- Use GitHub Issues for bug reports and feature requests
|
|
38
|
+
- Include Ruby version, gem version, and relevant code snippets
|
|
39
|
+
- For API-related issues, include sanitized request/response examples
|
|
40
|
+
|
|
41
|
+
### Pull Requests
|
|
42
|
+
|
|
43
|
+
1. Fork the repository
|
|
44
|
+
2. Create a feature branch: `git checkout -b feature/your-feature`
|
|
45
|
+
3. Make your changes with tests
|
|
46
|
+
4. Ensure all tests pass: `bundle exec rspec`
|
|
47
|
+
5. Follow Ruby style conventions (we use RuboCop - no error) : `bundle exec rubocop`
|
|
48
|
+
6. Submit a pull request with a clear description
|
|
49
|
+
|
|
50
|
+
### Code Standards
|
|
51
|
+
|
|
52
|
+
- Write tests for new features and bug fixes
|
|
53
|
+
- Follow existing code patterns and architecture
|
|
54
|
+
- Update documentation for public API changes
|
|
55
|
+
- Keep commits atomic and well-described
|
|
56
|
+
|
|
57
|
+
### Testing
|
|
58
|
+
|
|
59
|
+
- Unit tests should not require API calls (use mocks/stubs)
|
|
60
|
+
- Integration tests should use VCR cassettes when possible
|
|
61
|
+
- Test edge cases and error conditions
|
|
62
|
+
- Test coverage must be at 100% (for now)
|
|
63
|
+
|
|
64
|
+
## Release Process
|
|
65
|
+
|
|
66
|
+
Releases are handled by maintainers following semantic versioning.
|
|
67
|
+
|
|
68
|
+
## Questions?
|
|
69
|
+
|
|
70
|
+
Feel free to open an issue for any questions about contributing.
|
data/LICENSE.txt
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
|
|
1
|
+
MIT License
|
|
2
2
|
|
|
3
|
-
Copyright (c)
|
|
3
|
+
Copyright (c) 2024 Pierre Cherbero (peyochanchan)
|
|
4
4
|
|
|
5
5
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
6
|
of this software and associated documentation files (the "Software"), to deal
|
|
@@ -9,13 +9,13 @@ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
|
9
9
|
copies of the Software, and to permit persons to whom the Software is
|
|
10
10
|
furnished to do so, subject to the following conditions:
|
|
11
11
|
|
|
12
|
-
The above copyright notice and this permission notice shall be included in
|
|
13
|
-
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
14
|
|
|
15
15
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
16
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
17
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
18
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
19
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
-
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
CHANGED
|
@@ -2,9 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
Ruby gem for AI-powered translation and text summarization using Mistral AI API, with advanced Rails support.
|
|
4
4
|
|
|
5
|
-
[](https://rubygems.org/gems/mistral_translator)
|
|
6
|
+
[](https://rubygems.org/gems/mistral_translator)
|
|
7
|
+
[](https://github.com/Peyochanchan/mistral_translator/actions/workflows/ci.yml)
|
|
8
|
+
[](https://github.com/Peyochanchan/mistral_translator/actions/workflows/weekly-api-check.yml)
|
|
9
|
+
[](LICENSE.txt)
|
|
8
10
|
|
|
9
11
|
## Quick Start
|
|
10
12
|
|
|
@@ -30,7 +32,7 @@ MistralTranslator.translate("Bonjour le monde", from: "fr", to: "en")
|
|
|
30
32
|
- **Multi-level summarization** with translation
|
|
31
33
|
- **Robust error handling** with automatic retry and fallback
|
|
32
34
|
- **Complete monitoring** with metrics and callbacks
|
|
33
|
-
- **
|
|
35
|
+
- **Thread-safe & concurrent** with connection pooling ([docs](https://peyochanchan.github.io/mistral_translator/#/advanced-usage/concurrent-async))
|
|
34
36
|
- **Built-in rate limiting** and security
|
|
35
37
|
|
|
36
38
|
## Supported Languages
|
|
@@ -157,7 +159,7 @@ MistralTranslator.configure do |config|
|
|
|
157
159
|
config.api_key = ENV['MISTRAL_API_KEY']
|
|
158
160
|
|
|
159
161
|
# Optional
|
|
160
|
-
config.model = "mistral-small"
|
|
162
|
+
config.model = "mistral-small-latest" # AI model to use (an alias that follows the current Mistral Small)
|
|
161
163
|
config.retry_delays = [1, 2, 4, 8, 16] # Retry delays in seconds
|
|
162
164
|
config.enable_metrics = true # Enable performance metrics
|
|
163
165
|
|
|
@@ -222,29 +224,47 @@ The complete documentation includes:
|
|
|
222
224
|
|
|
223
225
|
### Quick Links
|
|
224
226
|
|
|
225
|
-
- [Installation & Setup](https://peyochanchan.github.io/mistral_translator
|
|
226
|
-
- [Rails Integration Guide](https://peyochanchan.github.io/mistral_translator
|
|
227
|
-
- [API Methods Reference](https://peyochanchan.github.io/mistral_translator
|
|
228
|
-
- [Error Handling Guide](https://peyochanchan.github.io/mistral_translator
|
|
229
|
-
- [Monitoring Setup](https://peyochanchan.github.io/mistral_translator
|
|
227
|
+
- [Installation & Setup](https://peyochanchan.github.io/mistral_translator/#/installation)
|
|
228
|
+
- [Rails Integration Guide](https://peyochanchan.github.io/mistral_translator/#/rails-integration/setup)
|
|
229
|
+
- [API Methods Reference](https://peyochanchan.github.io/mistral_translator/#/api-reference/methods)
|
|
230
|
+
- [Error Handling Guide](https://peyochanchan.github.io/mistral_translator/#/api-reference/errors)
|
|
231
|
+
- [Monitoring Setup](https://peyochanchan.github.io/mistral_translator/#/advanced-usage/monitoring)
|
|
230
232
|
|
|
231
233
|
## Requirements
|
|
232
234
|
|
|
233
235
|
- Ruby 3.2+
|
|
234
|
-
- Mistral AI API key
|
|
236
|
+
- Mistral AI API key, on a plan with API access (Mistral's free plan cannot generate text)
|
|
235
237
|
- Rails 6.0+ (optional, for Rails integration features)
|
|
236
238
|
|
|
237
239
|
## Testing
|
|
238
240
|
|
|
239
241
|
```bash
|
|
240
|
-
#
|
|
241
|
-
bundle
|
|
242
|
+
# Install dependencies
|
|
243
|
+
bundle install
|
|
244
|
+
|
|
245
|
+
# Run all unit tests (no API key required)
|
|
246
|
+
bundle exec rspec spec/mistral_translator/
|
|
247
|
+
|
|
248
|
+
# Run all tests with documentation format
|
|
249
|
+
bundle exec rspec --format documentation
|
|
250
|
+
|
|
251
|
+
# Check the API key and default model, then run the examples against the real
|
|
252
|
+
# API (reads MISTRAL_TEST_API_KEY, .env or asks for the key, never prints it)
|
|
253
|
+
bin/check_api
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
### Integration Tests
|
|
257
|
+
|
|
258
|
+
Integration tests require a real Mistral API key. **Never commit API keys to version control.**
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
# Set API key via environment variable (recommended)
|
|
262
|
+
export MISTRAL_API_KEY="your_api_key_here"
|
|
242
263
|
|
|
243
|
-
# Run
|
|
244
|
-
|
|
264
|
+
# Run integration tests
|
|
265
|
+
bundle exec rspec spec/integration/
|
|
245
266
|
|
|
246
|
-
|
|
247
|
-
MISTRAL_API_KEY=your_key bundle exec rspec --tag integration
|
|
267
|
+
**Security Note:** Use a dedicated test API key with limited quotas for integration testing.
|
|
248
268
|
```
|
|
249
269
|
|
|
250
270
|
## Contributing
|
|
@@ -259,15 +279,24 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
|
|
|
259
279
|
|
|
260
280
|
## License
|
|
261
281
|
|
|
262
|
-
MIT License. See [LICENSE](LICENSE) for details.
|
|
282
|
+
MIT License. See [LICENSE](LICENSE.txt) for details.
|
|
263
283
|
|
|
264
284
|
## Support
|
|
265
285
|
|
|
266
286
|
- 📖 [Documentation](https://peyochanchan.github.io/mistral_translator/)
|
|
267
287
|
- 🐛 [Issues](../../issues)
|
|
268
|
-
- 💬 [Discussions](../../discussions)
|
|
269
288
|
- 📧 Support: Create an issue for help
|
|
270
289
|
|
|
290
|
+
### ☕ Support the Project
|
|
291
|
+
|
|
292
|
+
If this gem helps you, consider supporting its development:
|
|
293
|
+
|
|
294
|
+
[](https://buymeacoffee.com/peyochanchan)
|
|
295
|
+
|
|
271
296
|
---
|
|
272
297
|
|
|
273
298
|
Built with Ruby ❤️ by [@peyochanchan](https://github.com/peyochanchan)
|
|
299
|
+
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
```
|
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# Concurrent & Asynchronous Processing
|
|
2
|
+
|
|
3
|
+
MistralTranslator is thread-safe and optimized for concurrent usage with protected metrics, connection pooling, and built-in rate limiting.
|
|
4
|
+
|
|
5
|
+
## Thread Safety
|
|
6
|
+
|
|
7
|
+
**Protected components:**
|
|
8
|
+
- Metrics tracking with Mutex
|
|
9
|
+
- Logger cache with concurrent access control
|
|
10
|
+
- Rate limiter with thread synchronization
|
|
11
|
+
- HTTP connection pooling (Net::HTTP::Persistent)
|
|
12
|
+
|
|
13
|
+
## Ruby Threads
|
|
14
|
+
|
|
15
|
+
### Basic Usage
|
|
16
|
+
|
|
17
|
+
```ruby
|
|
18
|
+
languages = ['fr', 'es', 'de', 'it']
|
|
19
|
+
threads = languages.map do |lang|
|
|
20
|
+
Thread.new do
|
|
21
|
+
MistralTranslator.translate("Hello", from: 'en', to: lang)
|
|
22
|
+
rescue MistralTranslator::Error => e
|
|
23
|
+
{ error: e.message }
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
results = threads.map(&:value)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
### Thread Pool Pattern
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
# Safe: Limited threads
|
|
34
|
+
texts.each_slice(5) do |batch|
|
|
35
|
+
threads = batch.map { |t| Thread.new { translate(t) } }
|
|
36
|
+
threads.each(&:join)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Dangerous: Unlimited threads can exhaust memory
|
|
40
|
+
texts.map { |t| Thread.new { translate(t) } } # Avoid this
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Concurrent Ruby (Recommended)
|
|
44
|
+
|
|
45
|
+
### Installation
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
# Gemfile
|
|
49
|
+
gem 'concurrent-ruby', '~> 1.2'
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
### Fixed Thread Pool
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
require 'concurrent'
|
|
56
|
+
|
|
57
|
+
pool = Concurrent::FixedThreadPool.new(5)
|
|
58
|
+
|
|
59
|
+
futures = languages.map do |lang|
|
|
60
|
+
Concurrent::Future.execute(executor: pool) do
|
|
61
|
+
MistralTranslator.translate(text, from: 'en', to: lang)
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
results = futures.map(&:value)
|
|
66
|
+
|
|
67
|
+
pool.shutdown
|
|
68
|
+
pool.wait_for_termination
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
### Promise Chains
|
|
72
|
+
|
|
73
|
+
```ruby
|
|
74
|
+
promise = Concurrent::Promise.execute do
|
|
75
|
+
MistralTranslator.translate_auto(text, to: 'en')
|
|
76
|
+
end.then do |english|
|
|
77
|
+
target_langs.map do |lang|
|
|
78
|
+
MistralTranslator.translate(english, from: 'en', to: lang)
|
|
79
|
+
end
|
|
80
|
+
end.rescue do |error|
|
|
81
|
+
Rails.logger.error "Pipeline failed: #{error}"
|
|
82
|
+
[]
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
results = promise.value
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Background Jobs
|
|
89
|
+
|
|
90
|
+
### SolidQueue (Rails 8+, Recommended)
|
|
91
|
+
|
|
92
|
+
SolidQueue is the default ActiveJob backend in Rails 8, database-backed and Redis-free.
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
# config/database.yml - SolidQueue uses your existing database
|
|
96
|
+
production:
|
|
97
|
+
primary:
|
|
98
|
+
<<: *default
|
|
99
|
+
queue: # Separate DB for jobs (optional)
|
|
100
|
+
<<: *default
|
|
101
|
+
database: app_queue
|
|
102
|
+
migrations_paths: db/queue_migrate
|
|
103
|
+
|
|
104
|
+
# app/jobs/translation_job.rb
|
|
105
|
+
class TranslationJob < ApplicationJob
|
|
106
|
+
queue_as :translations
|
|
107
|
+
|
|
108
|
+
retry_on MistralTranslator::RateLimitError, wait: :polynomially_longer
|
|
109
|
+
retry_on MistralTranslator::ApiError, wait: 5.seconds, attempts: 3
|
|
110
|
+
|
|
111
|
+
def perform(text, from_locale, to_locale)
|
|
112
|
+
MistralTranslator.translate(text, from: from_locale, to: to_locale)
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
# Usage
|
|
117
|
+
languages.each { |lang| TranslationJob.perform_later(text, 'en', lang) }
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
**Configuration:**
|
|
121
|
+
```ruby
|
|
122
|
+
# config/recurring.yml - Schedule periodic translations
|
|
123
|
+
production:
|
|
124
|
+
sync_translations:
|
|
125
|
+
class: TranslationSyncJob
|
|
126
|
+
schedule: every day at 3am
|
|
127
|
+
queue: translations
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
### Sidekiq / Other Backends
|
|
131
|
+
|
|
132
|
+
Compatible with any ActiveJob backend:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
class TranslationJob < ApplicationJob
|
|
136
|
+
queue_as :translations
|
|
137
|
+
retry_on MistralTranslator::RateLimitError, wait: :exponentially_longer
|
|
138
|
+
|
|
139
|
+
def perform(text, from, to)
|
|
140
|
+
MistralTranslator.translate(text, from: from, to: to)
|
|
141
|
+
end
|
|
142
|
+
end
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
## Performance Best Practices
|
|
146
|
+
|
|
147
|
+
### Configuration for Concurrent Use
|
|
148
|
+
|
|
149
|
+
```ruby
|
|
150
|
+
MistralTranslator.configure do |config|
|
|
151
|
+
config.enable_metrics = true
|
|
152
|
+
config.retry_delays = [1, 2, 4, 8] # Shorter for concurrent use
|
|
153
|
+
|
|
154
|
+
config.on_rate_limit = ->(from, to, wait, attempt, ts) {
|
|
155
|
+
Rails.logger.warn "Rate limit: #{wait}s wait"
|
|
156
|
+
}
|
|
157
|
+
end
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
### Recommendations
|
|
161
|
+
|
|
162
|
+
**Do:**
|
|
163
|
+
- Use fixed thread pools (5-10 threads)
|
|
164
|
+
- Enable metrics for monitoring
|
|
165
|
+
- Use background jobs for non-critical tasks
|
|
166
|
+
- Batch similar requests
|
|
167
|
+
|
|
168
|
+
**Avoid:**
|
|
169
|
+
- Unlimited thread creation
|
|
170
|
+
- Blocking user requests with translations
|
|
171
|
+
- Sharing API keys across environments
|
|
172
|
+
|
|
173
|
+
## Rate Limiting
|
|
174
|
+
|
|
175
|
+
The built-in rate limiter (50 requests/60s) is thread-safe but per-process.
|
|
176
|
+
|
|
177
|
+
**Multiple processes:** Each has its own limiter. With 4 Puma workers = 200 requests/min total.
|
|
178
|
+
|
|
179
|
+
### Custom Rate Limiter
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
class RedisRateLimiter
|
|
183
|
+
def wait_and_record!
|
|
184
|
+
key = "mistral:#{Time.now.to_i / 60}"
|
|
185
|
+
count = REDIS.incr(key)
|
|
186
|
+
REDIS.expire(key, 60) if count == 1
|
|
187
|
+
|
|
188
|
+
sleep(60 - Time.now.to_i % 60) if count > 50
|
|
189
|
+
end
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
client = MistralTranslator::Client.new(
|
|
193
|
+
rate_limiter: RedisRateLimiter.new
|
|
194
|
+
)
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
## Monitoring
|
|
198
|
+
|
|
199
|
+
```ruby
|
|
200
|
+
MistralTranslator.configure do |config|
|
|
201
|
+
config.enable_metrics = true
|
|
202
|
+
|
|
203
|
+
config.on_translation_complete = ->(from, to, orig, trans, duration) {
|
|
204
|
+
Rails.logger.info "[#{Thread.current.object_id}] #{from}→#{to}: #{duration.round(2)}s"
|
|
205
|
+
}
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
# Thread-safe metrics
|
|
209
|
+
metrics = MistralTranslator.metrics
|
|
210
|
+
puts "Total: #{metrics[:total_translations]}"
|
|
211
|
+
puts "Avg time: #{metrics[:average_translation_time]}s"
|
|
212
|
+
puts "Error rate: #{metrics[:error_rate]}%"
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
## Example: High-Performance Service
|
|
216
|
+
|
|
217
|
+
```ruby
|
|
218
|
+
class TranslationService
|
|
219
|
+
def initialize(max_threads: 5)
|
|
220
|
+
@pool = Concurrent::FixedThreadPool.new(max_threads)
|
|
221
|
+
end
|
|
222
|
+
|
|
223
|
+
def translate_all(texts, from:, to_languages:)
|
|
224
|
+
futures = texts.flat_map do |text|
|
|
225
|
+
to_languages.map do |lang|
|
|
226
|
+
Concurrent::Future.execute(executor: @pool) do
|
|
227
|
+
MistralTranslator.translate(text, from: from, to: lang)
|
|
228
|
+
end
|
|
229
|
+
end
|
|
230
|
+
end
|
|
231
|
+
futures.map(&:value)
|
|
232
|
+
ensure
|
|
233
|
+
@pool.shutdown
|
|
234
|
+
@pool.wait_for_termination(30)
|
|
235
|
+
end
|
|
236
|
+
end
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
## Troubleshooting
|
|
240
|
+
|
|
241
|
+
**"Too many open files"**
|
|
242
|
+
```bash
|
|
243
|
+
ulimit -n 4096
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Rate limits despite low volume**
|
|
247
|
+
Check if multiple processes share the API key. Consider Redis-based rate limiting.
|
|
248
|
+
|
|
249
|
+
**Memory growth**
|
|
250
|
+
```ruby
|
|
251
|
+
# Join threads periodically
|
|
252
|
+
threads = []
|
|
253
|
+
texts.each do |text|
|
|
254
|
+
threads << Thread.new { translate(text) }
|
|
255
|
+
|
|
256
|
+
if threads.size >= 10
|
|
257
|
+
threads.each(&:join)
|
|
258
|
+
threads.clear
|
|
259
|
+
end
|
|
260
|
+
end
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
## Summary
|
|
264
|
+
|
|
265
|
+
- Thread-safe with protected metrics and configuration
|
|
266
|
+
- Connection pooling automatic with net-http-persistent
|
|
267
|
+
- Use fixed-size thread pools (5-10 recommended)
|
|
268
|
+
- Built-in rate limiting works across threads
|
|
269
|
+
- Background jobs recommended for non-critical tasks
|
|
270
|
+
- Separate API keys per environment
|
|
@@ -11,7 +11,7 @@ Référence complète des options de configuration.
|
|
|
11
11
|
```ruby
|
|
12
12
|
MistralTranslator.configure do |config|
|
|
13
13
|
config.api_key = ENV['MISTRAL_API_KEY'] # Obligatoire
|
|
14
|
-
config.model = "mistral-small" # Défaut
|
|
14
|
+
config.model = "mistral-small-latest" # Défaut
|
|
15
15
|
config.api_url = "https://api.mistral.ai" # URL API
|
|
16
16
|
config.default_max_tokens = 4000 # Limite tokens
|
|
17
17
|
config.default_temperature = 0.3 # Créativité (0-1)
|
|
@@ -268,7 +268,7 @@ Configure la gem avec un bloc.
|
|
|
268
268
|
```ruby
|
|
269
269
|
MistralTranslator.configure do |config|
|
|
270
270
|
config.api_key = ENV['MISTRAL_API_KEY']
|
|
271
|
-
config.model = "mistral-small"
|
|
271
|
+
config.model = "mistral-small-latest"
|
|
272
272
|
config.retry_delays = [1, 2, 4, 8]
|
|
273
273
|
config.enable_metrics = true
|
|
274
274
|
end
|
data/docs/installation.md
CHANGED
|
@@ -76,7 +76,7 @@ end
|
|
|
76
76
|
```ruby
|
|
77
77
|
MistralTranslator.configure do |config|
|
|
78
78
|
config.api_key = ENV['MISTRAL_API_KEY']
|
|
79
|
-
config.model = "mistral-small" # Modèle par défaut
|
|
79
|
+
config.model = "mistral-small-latest" # Modèle par défaut
|
|
80
80
|
config.retry_delays = [2, 4, 8, 16] # Délais retry
|
|
81
81
|
config.enable_metrics = true # Suivi des stats
|
|
82
82
|
config.default_max_tokens = 4000 # Limite tokens
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "net/http/persistent"
|
|
4
|
+
|
|
3
5
|
module MistralTranslator
|
|
4
6
|
module ClientHelpers
|
|
5
7
|
# Helper pour la gestion des requêtes HTTP
|
|
@@ -9,14 +11,10 @@ module MistralTranslator
|
|
|
9
11
|
|
|
10
12
|
request_body = build_request_body(prompt, max_tokens, temperature)
|
|
11
13
|
|
|
12
|
-
http = Net::HTTP.new(uri.host, uri.port)
|
|
13
|
-
http.use_ssl = (uri.scheme == "https")
|
|
14
|
-
http.read_timeout = 60 # 60 secondes de timeout
|
|
15
|
-
|
|
16
14
|
request = Net::HTTP::Post.new(uri.path, headers)
|
|
17
15
|
request.body = request_body.to_json
|
|
18
16
|
|
|
19
|
-
response =
|
|
17
|
+
response = http_pool.request(uri, request)
|
|
20
18
|
log_request_response(request_body, response)
|
|
21
19
|
|
|
22
20
|
response
|
|
@@ -26,6 +24,38 @@ module MistralTranslator
|
|
|
26
24
|
raise ApiError, "HTTP error: #{e.message}"
|
|
27
25
|
end
|
|
28
26
|
|
|
27
|
+
def http_pool
|
|
28
|
+
@http_pool ||= begin
|
|
29
|
+
pool = Net::HTTP::Persistent.new(name: "mistral_translator")
|
|
30
|
+
pool.read_timeout = 60
|
|
31
|
+
pool.idle_timeout = 30
|
|
32
|
+
pool.open_timeout = MistralTranslator.configuration.ssl_timeout
|
|
33
|
+
pool.max_requests = 100
|
|
34
|
+
|
|
35
|
+
# SSL/TLS configuration
|
|
36
|
+
configure_ssl(pool)
|
|
37
|
+
|
|
38
|
+
pool
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def configure_ssl(pool)
|
|
43
|
+
config = MistralTranslator.configuration
|
|
44
|
+
|
|
45
|
+
# SSL verify mode
|
|
46
|
+
pool.verify_mode = case config.ssl_verify_mode
|
|
47
|
+
when :peer then OpenSSL::SSL::VERIFY_PEER
|
|
48
|
+
when :none then OpenSSL::SSL::VERIFY_NONE
|
|
49
|
+
else config.ssl_verify_mode
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Custom CA certificate file
|
|
53
|
+
pool.ca_file = config.ssl_ca_file if config.ssl_ca_file
|
|
54
|
+
|
|
55
|
+
# Custom CA certificates directory
|
|
56
|
+
pool.ca_path = config.ssl_ca_path if config.ssl_ca_path
|
|
57
|
+
end
|
|
58
|
+
|
|
29
59
|
def build_request_body(prompt, max_tokens, temperature)
|
|
30
60
|
body = {
|
|
31
61
|
model: @model,
|
|
@@ -4,13 +4,14 @@ module MistralTranslator
|
|
|
4
4
|
class Configuration
|
|
5
5
|
attr_accessor :api_key, :model, :default_max_tokens, :default_temperature, :retry_delays,
|
|
6
6
|
:on_translation_start, :on_translation_complete, :on_translation_error,
|
|
7
|
-
:on_rate_limit, :on_batch_complete, :enable_metrics
|
|
7
|
+
:on_rate_limit, :on_batch_complete, :enable_metrics,
|
|
8
|
+
:ssl_verify_mode, :ssl_ca_file, :ssl_ca_path, :ssl_timeout
|
|
8
9
|
attr_reader :api_url
|
|
9
10
|
|
|
10
11
|
def initialize
|
|
11
12
|
@api_key = nil
|
|
12
13
|
@api_url = "https://api.mistral.ai"
|
|
13
|
-
@model = "mistral-small"
|
|
14
|
+
@model = "mistral-small-latest"
|
|
14
15
|
@default_max_tokens = nil
|
|
15
16
|
@default_temperature = nil
|
|
16
17
|
@retry_delays = [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
|
|
@@ -23,7 +24,14 @@ module MistralTranslator
|
|
|
23
24
|
@on_batch_complete = nil
|
|
24
25
|
@enable_metrics = false
|
|
25
26
|
|
|
26
|
-
#
|
|
27
|
+
# SSL/TLS configuration
|
|
28
|
+
@ssl_verify_mode = :peer # :peer, :none, or OpenSSL constant
|
|
29
|
+
@ssl_ca_file = nil # Path to CA certificate file
|
|
30
|
+
@ssl_ca_path = nil # Path to CA certificates directory
|
|
31
|
+
@ssl_timeout = 60 # SSL handshake timeout in seconds
|
|
32
|
+
|
|
33
|
+
# Métriques intégrées (thread-safe avec mutex)
|
|
34
|
+
@metrics_mutex = Mutex.new
|
|
27
35
|
@metrics = {
|
|
28
36
|
total_translations: 0,
|
|
29
37
|
total_characters: 0,
|
|
@@ -54,9 +62,11 @@ module MistralTranslator
|
|
|
54
62
|
|
|
55
63
|
return unless @enable_metrics
|
|
56
64
|
|
|
57
|
-
@
|
|
58
|
-
|
|
59
|
-
|
|
65
|
+
@metrics_mutex.synchronize do
|
|
66
|
+
@metrics[:total_translations] += 1
|
|
67
|
+
@metrics[:total_characters] += text_length
|
|
68
|
+
@metrics[:translations_by_language]["#{from_locale}->#{to_locale}"] += 1
|
|
69
|
+
end
|
|
60
70
|
end
|
|
61
71
|
|
|
62
72
|
def trigger_translation_complete(from_locale, to_locale, original_length, translated_length, duration)
|
|
@@ -64,7 +74,9 @@ module MistralTranslator
|
|
|
64
74
|
|
|
65
75
|
return unless @enable_metrics
|
|
66
76
|
|
|
67
|
-
@
|
|
77
|
+
@metrics_mutex.synchronize do
|
|
78
|
+
@metrics[:total_duration] += duration
|
|
79
|
+
end
|
|
68
80
|
end
|
|
69
81
|
|
|
70
82
|
def trigger_translation_error(from_locale, to_locale, error, attempt)
|
|
@@ -72,7 +84,9 @@ module MistralTranslator
|
|
|
72
84
|
|
|
73
85
|
return unless @enable_metrics
|
|
74
86
|
|
|
75
|
-
@
|
|
87
|
+
@metrics_mutex.synchronize do
|
|
88
|
+
@metrics[:errors_count] += 1
|
|
89
|
+
end
|
|
76
90
|
end
|
|
77
91
|
|
|
78
92
|
def trigger_rate_limit(from_locale, to_locale, wait_time, attempt)
|
|
@@ -80,7 +94,9 @@ module MistralTranslator
|
|
|
80
94
|
|
|
81
95
|
return unless @enable_metrics
|
|
82
96
|
|
|
83
|
-
@
|
|
97
|
+
@metrics_mutex.synchronize do
|
|
98
|
+
@metrics[:rate_limits_hit] += 1
|
|
99
|
+
end
|
|
84
100
|
end
|
|
85
101
|
|
|
86
102
|
def trigger_batch_complete(batch_size, total_duration, success_count, error_count)
|
|
@@ -91,35 +107,40 @@ module MistralTranslator
|
|
|
91
107
|
def metrics
|
|
92
108
|
return {} unless @enable_metrics
|
|
93
109
|
|
|
94
|
-
@
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
110
|
+
@metrics_mutex.synchronize do
|
|
111
|
+
@metrics.merge({
|
|
112
|
+
average_translation_time: if @metrics[:total_translations].positive?
|
|
113
|
+
(@metrics[:total_duration] /
|
|
114
|
+
@metrics[:total_translations]).round(3)
|
|
115
|
+
else
|
|
116
|
+
0
|
|
117
|
+
end,
|
|
118
|
+
average_characters_per_translation: if @metrics[:total_translations].positive?
|
|
119
|
+
(@metrics[:total_characters] /
|
|
120
|
+
@metrics[:total_translations]).round(0)
|
|
121
|
+
else
|
|
122
|
+
0
|
|
123
|
+
end,
|
|
124
|
+
error_rate: if @metrics[:total_translations].positive?
|
|
125
|
+
((@metrics[:errors_count].to_f / @metrics[:total_translations]) * 100).round(2)
|
|
126
|
+
else
|
|
127
|
+
0
|
|
128
|
+
end
|
|
129
|
+
})
|
|
130
|
+
end
|
|
112
131
|
end
|
|
113
132
|
|
|
114
133
|
def reset_metrics!
|
|
115
|
-
@
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
134
|
+
@metrics_mutex.synchronize do
|
|
135
|
+
@metrics = {
|
|
136
|
+
total_translations: 0,
|
|
137
|
+
total_characters: 0,
|
|
138
|
+
total_duration: 0.0,
|
|
139
|
+
rate_limits_hit: 0,
|
|
140
|
+
errors_count: 0,
|
|
141
|
+
translations_by_language: Hash.new(0)
|
|
142
|
+
}
|
|
143
|
+
end
|
|
123
144
|
end
|
|
124
145
|
|
|
125
146
|
# Configuration helpers pour les callbacks les plus communs
|
|
@@ -17,12 +17,15 @@ module MistralTranslator
|
|
|
17
17
|
|
|
18
18
|
# Log seulement si pas déjà loggé récemment (évite la spam)
|
|
19
19
|
def warn_once(message, key: nil, sensitive: false, ttl: 300)
|
|
20
|
-
|
|
21
|
-
|
|
20
|
+
cache_mutex.synchronize do
|
|
21
|
+
@warn_cache ||= {}
|
|
22
|
+
cache_key = key || message
|
|
22
23
|
|
|
23
|
-
|
|
24
|
+
return unless should_log_warning?(cache_key, ttl)
|
|
25
|
+
|
|
26
|
+
@warn_cache[cache_key] = Time.now
|
|
27
|
+
end
|
|
24
28
|
|
|
25
|
-
@warn_cache[cache_key] = Time.now
|
|
26
29
|
log(:warn, message, sensitive)
|
|
27
30
|
end
|
|
28
31
|
|
|
@@ -74,6 +77,10 @@ module MistralTranslator
|
|
|
74
77
|
|
|
75
78
|
Time.now - @warn_cache[key] > ttl
|
|
76
79
|
end
|
|
80
|
+
|
|
81
|
+
def cache_mutex
|
|
82
|
+
@cache_mutex ||= Mutex.new
|
|
83
|
+
end
|
|
77
84
|
end
|
|
78
85
|
end
|
|
79
86
|
end
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module MistralTranslator
|
|
4
|
-
VERSION = "0.
|
|
4
|
+
VERSION = "0.3.1"
|
|
5
5
|
|
|
6
6
|
# Informations additionnelles sur la gem
|
|
7
7
|
API_VERSION = "v1"
|
|
8
|
-
SUPPORTED_MODEL = "mistral-small"
|
|
8
|
+
SUPPORTED_MODEL = "mistral-small-latest"
|
|
9
9
|
|
|
10
10
|
# Méthode pour obtenir des informations complètes sur la version
|
|
11
11
|
def self.version_info
|
metadata
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: mistral_translator
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
|
-
-
|
|
7
|
+
- Pierre Cherbero
|
|
8
8
|
bindir: exe
|
|
9
9
|
cert_chain: []
|
|
10
10
|
date: 1980-01-02 00:00:00.000000000 Z
|
|
@@ -23,6 +23,20 @@ dependencies:
|
|
|
23
23
|
- - "~>"
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
25
|
version: '0.4'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: net-http-persistent
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - "~>"
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '4.0'
|
|
33
|
+
type: :runtime
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - "~>"
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: '4.0'
|
|
26
40
|
- !ruby/object:Gem::Dependency
|
|
27
41
|
name: rake
|
|
28
42
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -103,11 +117,13 @@ executables: []
|
|
|
103
117
|
extensions: []
|
|
104
118
|
extra_rdoc_files: []
|
|
105
119
|
files:
|
|
120
|
+
- ".env.example"
|
|
106
121
|
- ".rspec"
|
|
107
122
|
- ".rubocop.yml"
|
|
108
123
|
- ".ruby-version"
|
|
109
124
|
- CHANGELOG.md
|
|
110
125
|
- CODE_OF_CONDUCT.md
|
|
126
|
+
- CONTRIBUTING.md
|
|
111
127
|
- LICENSE.txt
|
|
112
128
|
- README.md
|
|
113
129
|
- README_TESTING.md
|
|
@@ -117,6 +133,7 @@ files:
|
|
|
117
133
|
- docs/404.html
|
|
118
134
|
- docs/README.md
|
|
119
135
|
- docs/advanced-usage/batch-processing.md
|
|
136
|
+
- docs/advanced-usage/concurrent-async.md
|
|
120
137
|
- docs/advanced-usage/error-handling.md
|
|
121
138
|
- docs/advanced-usage/monitoring.md
|
|
122
139
|
- docs/advanced-usage/summarization.md
|
|
@@ -180,7 +197,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
180
197
|
- !ruby/object:Gem::Version
|
|
181
198
|
version: '0'
|
|
182
199
|
requirements: []
|
|
183
|
-
rubygems_version:
|
|
200
|
+
rubygems_version: 4.0.5
|
|
184
201
|
specification_version: 4
|
|
185
202
|
summary: Gem to translate and summarize text with Mistral API
|
|
186
203
|
test_files: []
|