i18nema 0.0.4 → 0.0.5
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +109 -7
- data/ext/i18nema/i18nema.c +178 -62
- data/ext/i18nema/{syck.h → vendor/syck.h} +0 -0
- data/ext/i18nema/{uthash.h → vendor/uthash.h} +0 -0
- data/lib/i18nema.rb +40 -1
- data/test/i18nema_test.rb +14 -4
- metadata +4 -7
- data/ext/i18nema/extconf.h +0 -3
- data/ext/i18nema/gram.h +0 -79
- data/ext/i18nema/yamlbyte.h +0 -171
data/README.md
CHANGED
@@ -3,9 +3,9 @@
|
|
3
3
|
Fast I18n backend to keep things running smoothly.
|
4
4
|
|
5
5
|
I18nema is a drop-in replacement for I18n::Backend::Simple, for faster
|
6
|
-
lookups and quicker
|
7
|
-
heap, and lookups happen in C (rather than the usual
|
8
|
-
ruby hashes).
|
6
|
+
lookups (2x!) and quicker GC runs (ymmv). Translations are stored
|
7
|
+
outside of the ruby heap, and lookups happen in C (rather than the usual
|
8
|
+
inject on nested ruby hashes).
|
9
9
|
|
10
10
|
## How do I use it?
|
11
11
|
|
@@ -15,11 +15,113 @@ and then do something like this in an initializer:
|
|
15
15
|
|
16
16
|
I18n.backend = I18nema::Backend.new
|
17
17
|
|
18
|
-
|
18
|
+
You can pull in additional features, e.g.
|
19
19
|
|
20
20
|
I18nema::Backend.send(:include, I18n::Backend::Fallbacks)
|
21
21
|
|
22
|
-
|
22
|
+
As with regular I18n, you should probably load translations before you
|
23
|
+
fork, so that all processes can use the same translations in memory. In
|
24
|
+
an initializer, just do `I18n.backend.init_translations`.
|
25
|
+
|
26
|
+
## What sort of improvements will I see?
|
27
|
+
|
28
|
+
### Faster Startup
|
29
|
+
|
30
|
+
Loading all the translations into memory is dramtically faster with
|
31
|
+
I18nema (about 4x). While this is just a one time hit, it's pretty
|
32
|
+
noticeable when you're waiting on it (e.g. console, specs). In
|
33
|
+
[canvas-lms](https://github/com/instructure/canvas-lms), I18nema brings
|
34
|
+
it down to just over half a second (from almost 2.5).
|
35
|
+
|
36
|
+
### Faster GC Runs
|
37
|
+
|
38
|
+
Because there are fewer ruby objects, the periodic GC runs will be
|
39
|
+
proportionally faster. How much faster is a question of how many
|
40
|
+
translations you have versus how many other ruby objects. Applications
|
41
|
+
that are localized in more languages should see a bigger boost (since
|
42
|
+
the translations make up a bigger share of the original ObjectSpace).
|
43
|
+
|
44
|
+
For example, [canvas-lms](https://github/com/instructure/canvas-lms) is
|
45
|
+
translated into seven other languages, and I18nema reduces (startup)
|
46
|
+
ObjectSpace by about 18% and GC runtime by about 11%.
|
47
|
+
|
48
|
+
I18nema also moves I18n's normalized_key_cache into C structs. This key
|
49
|
+
cache grows over time (it eventually holds a key/value for every
|
50
|
+
translation key used in the app), so that's another area where I18nema
|
51
|
+
is nicer on ObjectSpace than vanilla I18n.
|
52
|
+
|
53
|
+
### Faster Translation Lookups
|
54
|
+
|
55
|
+
Simple lookups (i.e. no options or interpolation) take about 55% less
|
56
|
+
time, and that holds true no matter how deeply scoped your keys are
|
57
|
+
(e.g. `foo.bar.baz.lol`).
|
58
|
+
|
59
|
+
Lookups with options are also faster (about 35% less time). The gains
|
60
|
+
aren't quite as big there yet, since proportionally more of the time is
|
61
|
+
spent in unoptimized places like `I18n#interpolate`.
|
62
|
+
|
63
|
+
## Show me the benchmarks
|
64
|
+
|
65
|
+
Here are some basic ones done with `Benchmark.bmbm` (edited for brevity)
|
66
|
+
We run `I18n.translate` 100000 times on 4 different translation keys.
|
67
|
+
The `n` in `translate(n)` denotes how many parts there are in the key,
|
68
|
+
e.g. `I18n.t('foo') -> 1`, `I18n.t('foo.bar') -> 2`
|
69
|
+
|
70
|
+
### simple `translate` (no options)
|
71
|
+
|
72
|
+
#### I18nema
|
73
|
+
|
74
|
+
user system total real
|
75
|
+
translate(1): 0.960000 0.010000 0.970000 ( 0.963384)
|
76
|
+
translate(2): 1.070000 0.010000 1.080000 ( 1.068789)
|
77
|
+
translate(3): 1.080000 0.000000 1.080000 ( 1.083826)
|
78
|
+
translate(4): 1.260000 0.010000 1.270000 ( 1.263967)
|
79
|
+
|
80
|
+
#### I18n
|
81
|
+
|
82
|
+
user system total real
|
83
|
+
translate(1): 2.140000 0.000000 2.140000 ( 2.137579)
|
84
|
+
translate(2): 2.380000 0.000000 2.380000 ( 2.382720)
|
85
|
+
translate(3): 2.420000 0.000000 2.420000 ( 2.428245)
|
86
|
+
translate(4): 2.510000 0.000000 2.510000 ( 2.514133)
|
87
|
+
|
88
|
+
### `translate` with options (locale, interpolation)
|
89
|
+
|
90
|
+
#### I18nema
|
91
|
+
|
92
|
+
user system total real
|
93
|
+
translate(1): 1.770000 0.000000 1.770000 ( 1.771934)
|
94
|
+
translate(2): 1.870000 0.010000 1.880000 ( 1.880586)
|
95
|
+
translate(3): 1.890000 0.000000 1.890000 ( 1.893065)
|
96
|
+
translate(4): 2.080000 0.000000 2.080000 ( 2.083093)
|
97
|
+
|
98
|
+
#### I18n
|
99
|
+
|
100
|
+
user system total real
|
101
|
+
translate(1): 2.720000 0.000000 2.720000 ( 2.720541)
|
102
|
+
translate(2): 2.950000 0.010000 2.960000 ( 2.953175)
|
103
|
+
translate(3): 3.010000 0.010000 3.020000 ( 3.028288)
|
104
|
+
translate(4): 3.120000 0.010000 3.130000 ( 3.117135)
|
105
|
+
|
106
|
+
|
107
|
+
## OK, so what's the catch?
|
108
|
+
|
109
|
+
I18nema is still a work in progress, so there are some compatibility
|
110
|
+
notes you should be aware of:
|
111
|
+
|
112
|
+
I18nema requires ruby 1.9.3 or later.
|
113
|
+
|
114
|
+
I18nema only supports `.yml` translation files (no `.rb`).
|
115
|
+
|
116
|
+
I18nema requires UTF-8 `.yml` files. That means that your translations
|
117
|
+
should actually be in their UTF-8 form (e.g. "Contraseña"), not some
|
118
|
+
escaped representation. I18nema uses a simplified syck implementation
|
119
|
+
and does not support many optional yml types (e.g. `binary`).
|
120
|
+
|
121
|
+
I18nema doesn't yet support symbols as translation *values* (note that
|
122
|
+
symbol [keys](http://guides.rubyonrails.org/i18n.html#basic-lookup-scopes-and-nested-keys)
|
123
|
+
and [defaults](http://guides.rubyonrails.org/i18n.html#defaults) work
|
124
|
+
just fine). Symbol values in your `.yml` file can be used in the same
|
125
|
+
way that symbol defaults can, i.e. they tell I18n to find the
|
126
|
+
translation under some other key.
|
23
127
|
|
24
|
-
You should probably make sure translations are loaded before you fork. In
|
25
|
-
an initializer, just do `I18n.backend.init_translations`
|
data/ext/i18nema/i18nema.c
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
#include <ruby.h>
|
2
2
|
#include <ruby/encoding.h>
|
3
|
-
#include
|
4
|
-
#include "uthash.h"
|
3
|
+
#include "vendor/syck.h"
|
4
|
+
#include "vendor/uthash.h"
|
5
5
|
|
6
6
|
VALUE I18nema = Qnil,
|
7
7
|
I18nemaBackend = Qnil,
|
@@ -12,8 +12,10 @@ struct i_key_value;
|
|
12
12
|
static VALUE array_to_rarray(struct i_object *array);
|
13
13
|
static VALUE hash_to_rhash(struct i_object *hash);
|
14
14
|
static void merge_hash(struct i_object *hash, struct i_object *other_hash);
|
15
|
-
static void delete_hash(struct i_key_value **hash, int
|
16
|
-
static void delete_object(struct i_object *object, int
|
15
|
+
static void delete_hash(struct i_key_value **hash, int recurse);
|
16
|
+
static void delete_object(struct i_object *object, int recurse);
|
17
|
+
static void delete_object_r(struct i_object *object);
|
18
|
+
static VALUE normalize_key(VALUE self, VALUE key, VALUE separator);
|
17
19
|
|
18
20
|
enum i_object_type {
|
19
21
|
i_type_none,
|
@@ -105,13 +107,37 @@ hash_to_rhash(i_object_t *hash)
|
|
105
107
|
}
|
106
108
|
|
107
109
|
static i_object_t*
|
108
|
-
|
110
|
+
i_object_get(VALUE self, const char *iv)
|
109
111
|
{
|
110
|
-
i_object_t *
|
111
|
-
VALUE
|
112
|
-
|
113
|
-
Data_Get_Struct(
|
114
|
-
return
|
112
|
+
i_object_t *object;
|
113
|
+
VALUE wrapped;
|
114
|
+
wrapped = rb_iv_get(self, iv);
|
115
|
+
Data_Get_Struct(wrapped, i_object_t, object);
|
116
|
+
return object;
|
117
|
+
}
|
118
|
+
|
119
|
+
static i_object_t*
|
120
|
+
translations_get(VALUE self)
|
121
|
+
{
|
122
|
+
return i_object_get(self, "@translations");
|
123
|
+
}
|
124
|
+
|
125
|
+
static i_object_t*
|
126
|
+
normalized_key_cache_get(VALUE self)
|
127
|
+
{
|
128
|
+
return i_object_get(self, "@normalized_key_cache");
|
129
|
+
}
|
130
|
+
|
131
|
+
static i_object_t*
|
132
|
+
hash_get(i_object_t *current, VALUE *keys, int num_keys)
|
133
|
+
{
|
134
|
+
i_key_value_t *kv = NULL;
|
135
|
+
for (int i = 0; i < num_keys && current != NULL && current->type == i_type_hash; i++) {
|
136
|
+
Check_Type(keys[i], T_STRING);
|
137
|
+
HASH_FIND_STR(current->data.hash, StringValueCStr(keys[i]), kv);
|
138
|
+
current = kv == NULL ? NULL : kv->value;
|
139
|
+
}
|
140
|
+
return current;
|
115
141
|
}
|
116
142
|
|
117
143
|
/*
|
@@ -127,37 +153,25 @@ root_object_get(VALUE self)
|
|
127
153
|
static VALUE
|
128
154
|
direct_lookup(int argc, VALUE *argv, VALUE self)
|
129
155
|
{
|
130
|
-
i_object_t *
|
131
|
-
|
132
|
-
VALUE rs;
|
133
|
-
char *s;
|
134
|
-
|
135
|
-
for (int i = 0; i < argc && result != NULL && result->type == i_type_hash; i++) {
|
136
|
-
rs = rb_funcall(argv[i], s_to_s, 0);
|
137
|
-
s = StringValueCStr(rs);
|
138
|
-
HASH_FIND_STR(result->data.hash, s, kv);
|
139
|
-
result = kv == NULL ? NULL : kv->value;
|
140
|
-
}
|
141
|
-
|
142
|
-
return i_object_to_robject(result);
|
156
|
+
i_object_t *translations = translations_get(self);
|
157
|
+
return i_object_to_robject(hash_get(translations, argv, argc));
|
143
158
|
}
|
144
159
|
|
145
160
|
static void
|
146
|
-
empty_object(i_object_t *object, int
|
161
|
+
empty_object(i_object_t *object, int recurse)
|
147
162
|
{
|
148
163
|
if (object == NULL)
|
149
164
|
return;
|
150
165
|
|
151
166
|
switch (object->type) {
|
152
167
|
case i_type_array:
|
153
|
-
if (
|
168
|
+
if (recurse)
|
154
169
|
for (unsigned long i = 0; i < object->size; i++)
|
155
|
-
|
156
|
-
}
|
170
|
+
delete_object_r(object->data.array[i]);
|
157
171
|
xfree(object->data.array);
|
158
172
|
break;
|
159
173
|
case i_type_hash:
|
160
|
-
delete_hash(&object->data.hash,
|
174
|
+
delete_hash(&object->data.hash, recurse);
|
161
175
|
break;
|
162
176
|
case i_type_none:
|
163
177
|
break;
|
@@ -168,35 +182,35 @@ empty_object(i_object_t *object, int delete_child_objects)
|
|
168
182
|
}
|
169
183
|
|
170
184
|
static void
|
171
|
-
|
185
|
+
delete_object(i_object_t *object, int recurse)
|
172
186
|
{
|
173
|
-
|
187
|
+
empty_object(object, recurse);
|
188
|
+
if (object->type != i_type_null && object->type != i_type_true && object->type != i_type_false)
|
189
|
+
xfree(object);
|
174
190
|
}
|
175
191
|
|
176
192
|
static void
|
177
|
-
|
193
|
+
delete_object_r(i_object_t *object)
|
178
194
|
{
|
179
|
-
|
180
|
-
if (object->type != i_type_null && object->type != i_type_true && object->type != i_type_false)
|
181
|
-
xfree(object);
|
195
|
+
delete_object(object, 1);
|
182
196
|
}
|
183
197
|
|
184
198
|
static void
|
185
199
|
delete_key_value(i_key_value_t *kv, int delete_value)
|
186
200
|
{
|
187
201
|
if (delete_value)
|
188
|
-
|
202
|
+
delete_object_r(kv->value);
|
189
203
|
xfree(kv->key);
|
190
204
|
xfree(kv);
|
191
205
|
}
|
192
206
|
|
193
207
|
static void
|
194
|
-
delete_hash(i_key_value_t **hash, int
|
208
|
+
delete_hash(i_key_value_t **hash, int recurse)
|
195
209
|
{
|
196
210
|
i_key_value_t *kv, *tmp;
|
197
211
|
HASH_ITER(hh, *hash, kv, tmp) {
|
198
212
|
HASH_DEL(*hash, kv);
|
199
|
-
delete_key_value(kv,
|
213
|
+
delete_key_value(kv, recurse);
|
200
214
|
}
|
201
215
|
}
|
202
216
|
|
@@ -227,7 +241,7 @@ merge_hash(i_object_t *hash, i_object_t *other_hash)
|
|
227
241
|
HASH_DEL(other_hash->data.hash, kv);
|
228
242
|
add_key_value(&hash->data.hash, kv);
|
229
243
|
}
|
230
|
-
|
244
|
+
delete_object_r(other_hash);
|
231
245
|
}
|
232
246
|
|
233
247
|
static int
|
@@ -236,7 +250,7 @@ delete_syck_st_entry(char *key, char *value, char *arg)
|
|
236
250
|
i_object_t *object = (i_object_t *)value;
|
237
251
|
// key object whose string we have yoinked into a kv
|
238
252
|
if (object->type == i_type_none)
|
239
|
-
|
253
|
+
delete_object_r(object);
|
240
254
|
return ST_DELETE;
|
241
255
|
}
|
242
256
|
|
@@ -270,18 +284,53 @@ handle_syck_badanchor(SyckParser *parser, char *anchor)
|
|
270
284
|
return NULL;
|
271
285
|
}
|
272
286
|
|
287
|
+
static char*
|
288
|
+
new_string(char *orig, long len)
|
289
|
+
{
|
290
|
+
char *str = xmalloc(len + 1);
|
291
|
+
strncpy(str, orig, len);
|
292
|
+
str[len] = '\0';
|
293
|
+
return str;
|
294
|
+
}
|
295
|
+
|
273
296
|
static i_object_t*
|
274
297
|
new_string_object(char *str, long len)
|
275
298
|
{
|
276
299
|
i_object_t *object = ALLOC(i_object_t);
|
277
300
|
object->type = i_type_string;
|
278
301
|
object->size = len;
|
279
|
-
object->data.string =
|
280
|
-
|
281
|
-
|
302
|
+
object->data.string = new_string(str, len);
|
303
|
+
return object;
|
304
|
+
}
|
305
|
+
|
306
|
+
static i_object_t*
|
307
|
+
new_array_object(long size)
|
308
|
+
{
|
309
|
+
i_object_t *object = ALLOC(i_object_t);
|
310
|
+
object->type = i_type_array;
|
311
|
+
object->size = size;
|
312
|
+
object->data.array = ALLOC_N(i_object_t*, size);
|
282
313
|
return object;
|
283
314
|
}
|
284
315
|
|
316
|
+
static i_object_t*
|
317
|
+
new_hash_object()
|
318
|
+
{
|
319
|
+
i_object_t *object = ALLOC(i_object_t);
|
320
|
+
object->type = i_type_hash;
|
321
|
+
object->data.hash = NULL;
|
322
|
+
return object;
|
323
|
+
}
|
324
|
+
|
325
|
+
static i_key_value_t*
|
326
|
+
new_key_value(char *key, i_object_t *value)
|
327
|
+
{
|
328
|
+
i_key_value_t *kv = ALLOC(i_key_value_t);
|
329
|
+
kv->key = key;
|
330
|
+
kv->value = value;
|
331
|
+
return kv;
|
332
|
+
}
|
333
|
+
|
285
334
|
static SYMID
|
286
335
|
handle_syck_node(SyckParser *parser, SyckNode *node)
|
287
336
|
{
|
@@ -315,10 +364,7 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
|
|
315
364
|
}
|
316
365
|
break;
|
317
366
|
case syck_seq_kind:
|
318
|
-
result =
|
319
|
-
result->type = i_type_array;
|
320
|
-
result->size = node->data.list->idx;
|
321
|
-
result->data.array = ALLOC_N(i_object_t*, node->data.list->idx);
|
367
|
+
result = new_array_object(node->data.list->idx);
|
322
368
|
for (long i = 0; i < node->data.list->idx; i++) {
|
323
369
|
i_object_t *item = NULL;
|
324
370
|
|
@@ -330,9 +376,7 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
|
|
330
376
|
}
|
331
377
|
break;
|
332
378
|
case syck_map_kind:
|
333
|
-
result =
|
334
|
-
result->type = i_type_hash;
|
335
|
-
result->data.hash = NULL;
|
379
|
+
result = new_hash_object();
|
336
380
|
for (long i = 0; i < node->data.pairs->idx; i++) {
|
337
381
|
i_object_t *key = NULL, *value = NULL;
|
338
382
|
|
@@ -341,11 +385,8 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
|
|
341
385
|
oid = syck_map_read(node, map_value, i);
|
342
386
|
syck_lookup_sym(parser, oid, (void **)&value);
|
343
387
|
|
344
|
-
i_key_value_t *kv;
|
345
|
-
kv = ALLOC(i_key_value_t);
|
346
|
-
kv->key = key->data.string;
|
388
|
+
i_key_value_t *kv = new_key_value(key->data.string, value);
|
347
389
|
key->type = i_type_none; // so we know to free this node in delete_syck_st_entry
|
348
|
-
kv->value = value;
|
349
390
|
if (value->type == i_type_string)
|
350
391
|
current_translation_count++;
|
351
392
|
add_key_value(&result->data.hash, kv);
|
@@ -371,7 +412,7 @@ static VALUE
|
|
371
412
|
load_yml_string(VALUE self, VALUE yml)
|
372
413
|
{
|
373
414
|
SYMID oid;
|
374
|
-
i_object_t *root_object =
|
415
|
+
i_object_t *root_object = translations_get(self);
|
375
416
|
i_object_t *new_root_object = NULL;
|
376
417
|
current_translation_count = 0;
|
377
418
|
SyckParser* parser = syck_new_parser();
|
@@ -387,7 +428,7 @@ load_yml_string(VALUE self, VALUE yml)
|
|
387
428
|
st_foreach(parser->syms, delete_syck_st_entry, 0);
|
388
429
|
syck_free_parser(parser);
|
389
430
|
if (new_root_object == NULL || new_root_object->type != i_type_hash) {
|
390
|
-
|
431
|
+
delete_object_r(new_root_object);
|
391
432
|
rb_raise(I18nemaBackendLoadError, "root yml node is not a hash");
|
392
433
|
}
|
393
434
|
merge_hash(root_object, new_root_object);
|
@@ -409,7 +450,7 @@ available_locales(VALUE self)
|
|
409
450
|
{
|
410
451
|
if (!RTEST(rb_iv_get(self, "@initialized")))
|
411
452
|
rb_funcall(self, s_init_translations, 0);
|
412
|
-
i_object_t *root_object =
|
453
|
+
i_object_t *root_object = translations_get(self);
|
413
454
|
i_key_value_t *current = root_object->data.hash;
|
414
455
|
VALUE ary = rb_ary_new2(0);
|
415
456
|
|
@@ -431,22 +472,96 @@ available_locales(VALUE self)
|
|
431
472
|
static VALUE
|
432
473
|
reload(VALUE self)
|
433
474
|
{
|
434
|
-
i_object_t *root_object =
|
475
|
+
i_object_t *root_object = translations_get(self);
|
435
476
|
empty_object(root_object, 1);
|
436
477
|
rb_iv_set(self, "@initialized", Qfalse);
|
437
|
-
root_object = NULL;
|
438
478
|
return Qtrue;
|
439
479
|
}
|
440
480
|
|
481
|
+
static VALUE
|
482
|
+
join_array_key(VALUE self, VALUE key, VALUE separator)
|
483
|
+
{
|
484
|
+
long len = RARRAY_LEN(key);
|
485
|
+
if (len == 0)
|
486
|
+
return rb_str_new("", 0);
|
487
|
+
|
488
|
+
VALUE ret = rb_ary_join(normalize_key(self, RARRAY_PTR(key)[0], separator), separator);
|
489
|
+
for (long i = 1; i < len; i++) {
|
490
|
+
rb_str_concat(ret, separator);
|
491
|
+
rb_str_concat(ret, rb_ary_join(normalize_key(self, RARRAY_PTR(key)[i], separator), separator));
|
492
|
+
}
|
493
|
+
return ret;
|
494
|
+
}
|
495
|
+
|
496
|
+
/*
|
497
|
+
* call-seq:
|
498
|
+
* backend.normalize_key(key, separator) -> key
|
499
|
+
*
|
500
|
+
* Normalizes and splits a key based on the separator.
|
501
|
+
*
|
502
|
+
* backend.normalize_key "asdf", "." #=> ["asdf"]
|
503
|
+
* backend.normalize_key "a.b.c", "." #=> ["a", "b", "c"]
|
504
|
+
* backend.normalize_key "a.b.c", ":" #=> ["a.b.c"]
|
505
|
+
* backend.normalize_key %{a b.c}, "." #=> ["a", "b", "c"]
|
506
|
+
*/
|
507
|
+
|
508
|
+
static VALUE
|
509
|
+
normalize_key(VALUE self, VALUE key, VALUE separator)
|
510
|
+
{
|
511
|
+
Check_Type(separator, T_STRING);
|
512
|
+
|
513
|
+
i_object_t *key_map = normalized_key_cache_get(self),
|
514
|
+
*sub_map = hash_get(key_map, &separator, 1);
|
515
|
+
if (sub_map == NULL) {
|
516
|
+
sub_map = new_hash_object();
|
517
|
+
char *key = new_string(RSTRING_PTR(separator), RSTRING_LEN(separator));
|
518
|
+
i_key_value_t *kv = new_key_value(key, sub_map);
|
519
|
+
add_key_value(&key_map->data.hash, kv);
|
520
|
+
}
|
521
|
+
|
522
|
+
if (TYPE(key) == T_ARRAY)
|
523
|
+
key = join_array_key(self, key, separator);
|
524
|
+
else if (TYPE(key) != T_STRING)
|
525
|
+
key = rb_funcall(key, s_to_s, 0);
|
526
|
+
|
527
|
+
i_object_t *key_frd = hash_get(sub_map, &key, 1);
|
528
|
+
|
529
|
+
if (key_frd == NULL) {
|
530
|
+
char *sep = StringValueCStr(separator);
|
531
|
+
VALUE parts = rb_str_split(key, sep);
|
532
|
+
long parts_len = RARRAY_LEN(parts),
|
533
|
+
skipped = 0;
|
534
|
+
key_frd = new_array_object(parts_len);
|
535
|
+
for (long i = 0; i < parts_len; i++) {
|
536
|
+
VALUE part = RARRAY_PTR(parts)[i];
|
537
|
+
// TODO: don't alloc for empty strings, since we discard them
|
538
|
+
if (RSTRING_LEN(part) == 0)
|
539
|
+
skipped++;
|
540
|
+
else
|
541
|
+
key_frd->data.array[i - skipped] = new_string_object(RSTRING_PTR(part), RSTRING_LEN(part));
|
542
|
+
}
|
543
|
+
key_frd->size -= skipped;
|
544
|
+
|
545
|
+
char *key_orig = new_string(RSTRING_PTR(key), RSTRING_LEN(key));
|
546
|
+
i_key_value_t *kv = new_key_value(key_orig, key_frd);
|
547
|
+
add_key_value(&sub_map->data.hash, kv);
|
548
|
+
}
|
549
|
+
return i_object_to_robject(key_frd);
|
550
|
+
}
|
551
|
+
|
441
552
|
static VALUE
|
442
553
|
initialize(VALUE self)
|
443
554
|
{
|
444
|
-
VALUE translations;
|
445
|
-
|
446
|
-
root_object
|
447
|
-
root_object->data.hash = NULL;
|
555
|
+
VALUE translations, key_cache;
|
556
|
+
|
557
|
+
i_object_t *root_object = new_hash_object();
|
448
558
|
translations = Data_Wrap_Struct(I18nemaBackend, 0, delete_object_r, root_object);
|
449
559
|
rb_iv_set(self, "@translations", translations);
|
560
|
+
|
561
|
+
i_object_t *key_map = new_hash_object();
|
562
|
+
key_cache = Data_Wrap_Struct(I18nemaBackend, 0, delete_object_r, key_map);
|
563
|
+
rb_iv_set(self, "@normalized_key_cache", key_cache);
|
564
|
+
|
450
565
|
return self;
|
451
566
|
}
|
452
567
|
|
@@ -471,4 +586,5 @@ Init_i18nema()
|
|
471
586
|
rb_define_method(I18nemaBackend, "available_locales", available_locales, 0);
|
472
587
|
rb_define_method(I18nemaBackend, "reload!", reload, 0);
|
473
588
|
rb_define_method(I18nemaBackend, "direct_lookup", direct_lookup, -1);
|
589
|
+
rb_define_method(I18nemaBackend, "normalize_key", normalize_key, 2);
|
474
590
|
}
|
File without changes
|
File without changes
|
data/lib/i18nema.rb
CHANGED
@@ -18,6 +18,36 @@ module I18nema
|
|
18
18
|
@initialized = true
|
19
19
|
end
|
20
20
|
|
21
|
+
RESERVED_KEY_MAP = Hash[I18n::RESERVED_KEYS.map{|k|[k,true]}]
|
22
|
+
|
23
|
+
def translate(locale, key, options = {})
|
24
|
+
raise I18n::InvalidLocale.new(locale) unless locale
|
25
|
+
entry = key && lookup(locale, key, options[:scope], options)
|
26
|
+
|
27
|
+
if options.empty?
|
28
|
+
entry = resolve(locale, key, entry, options)
|
29
|
+
else
|
30
|
+
count, default = options.values_at(:count, :default)
|
31
|
+
# significant speedup over Hash#except
|
32
|
+
values = options.reject { |key, value| RESERVED_KEY_MAP.key?(key) }
|
33
|
+
entry = entry.nil? && default ?
|
34
|
+
default(locale, key, default, options) : resolve(locale, key, entry, options)
|
35
|
+
end
|
36
|
+
|
37
|
+
throw(:exception, I18n::MissingTranslation.new(locale, key, options)) if entry.nil?
|
38
|
+
# no need to dup, since I18nema gives us a new string
|
39
|
+
|
40
|
+
entry = pluralize(locale, entry, count) if count
|
41
|
+
entry = interpolate(locale, entry, values) if values
|
42
|
+
entry
|
43
|
+
end
|
44
|
+
|
45
|
+
alias_method :reload_old, :reload!
|
46
|
+
def reload!
|
47
|
+
debugger
|
48
|
+
reload_old
|
49
|
+
end
|
50
|
+
|
21
51
|
protected
|
22
52
|
def load_file(filename)
|
23
53
|
type = File.extname(filename).tr('.', '').downcase
|
@@ -35,8 +65,17 @@ module I18nema
|
|
35
65
|
|
36
66
|
def lookup(locale, key, scope = [], options = {})
|
37
67
|
init_translations unless initialized?
|
38
|
-
keys =
|
68
|
+
keys = normalize_keys(locale, key, scope, options[:separator])
|
39
69
|
direct_lookup(*keys)
|
40
70
|
end
|
71
|
+
|
72
|
+
def normalize_keys(locale, key, scope, separator = nil)
|
73
|
+
separator ||= I18n.default_separator
|
74
|
+
|
75
|
+
keys = [locale.to_s]
|
76
|
+
keys.concat normalize_key(scope, separator) if scope && scope.size > 0
|
77
|
+
keys.concat normalize_key(key, separator)
|
78
|
+
keys
|
79
|
+
end
|
41
80
|
end
|
42
81
|
end
|
data/test/i18nema_test.rb
CHANGED
@@ -68,10 +68,6 @@ class I18nemaTest < Test::Unit::TestCase
|
|
68
68
|
assert_equal("root yml node is not a hash", exception.message)
|
69
69
|
assert_equal({}, backend.direct_lookup)
|
70
70
|
|
71
|
-
# FIXME ... ruby syck does this differently, and we get a non
|
72
|
-
# i_object_t as the root node, causing delete_object to asplode when
|
73
|
-
# it tries to free a garbage pointer
|
74
|
-
#
|
75
71
|
exception = assert_raise(I18nema::Backend::LoadError) {
|
76
72
|
backend.load_yml_string("en:\n foo: \"lol\"\n\tbar: notabs!")
|
77
73
|
}
|
@@ -84,4 +80,18 @@ class I18nemaTest < Test::Unit::TestCase
|
|
84
80
|
assert_match(/bad anchor `a'/, exception.message)
|
85
81
|
assert_equal({}, backend.direct_lookup)
|
86
82
|
end
|
83
|
+
|
84
|
+
def test_normalize_key
|
85
|
+
backend = I18nema::Backend.new
|
86
|
+
assert_equal %w{asdf},
|
87
|
+
backend.normalize_key("asdf", ".")
|
88
|
+
assert_equal %w{asdf asdf},
|
89
|
+
backend.normalize_key("asdf.asdf", ".")
|
90
|
+
assert_equal %w{asdf.asdf},
|
91
|
+
backend.normalize_key("asdf.asdf", ",")
|
92
|
+
assert_equal %w{asdf asdf},
|
93
|
+
backend.normalize_key("asdf...asdf", ".")
|
94
|
+
assert_equal %w{asdf asdf asdf},
|
95
|
+
backend.normalize_key(%w{asdf asdf.asdf}, ".")
|
96
|
+
end
|
87
97
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: i18nema
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.5
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-
|
12
|
+
date: 2013-06-01 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: i18n
|
@@ -55,11 +55,8 @@ files:
|
|
55
55
|
- Rakefile
|
56
56
|
- README.md
|
57
57
|
- ext/i18nema/i18nema.c
|
58
|
-
- ext/i18nema/
|
59
|
-
- ext/i18nema/
|
60
|
-
- ext/i18nema/syck.h
|
61
|
-
- ext/i18nema/uthash.h
|
62
|
-
- ext/i18nema/yamlbyte.h
|
58
|
+
- ext/i18nema/vendor/syck.h
|
59
|
+
- ext/i18nema/vendor/uthash.h
|
63
60
|
- ext/i18nema/extconf.rb
|
64
61
|
- ext/i18nema/mkrf_conf.rb
|
65
62
|
- lib/i18nema/core_ext/hash.rb
|
data/ext/i18nema/extconf.h
DELETED
data/ext/i18nema/gram.h
DELETED
@@ -1,79 +0,0 @@
|
|
1
|
-
/* A Bison parser, made by GNU Bison 1.875d. */
|
2
|
-
|
3
|
-
/* Skeleton parser for Yacc-like parsing with Bison,
|
4
|
-
Copyright (C) 1984, 1989, 1990, 2000, 2001, 2002, 2003, 2004 Free Software Foundation, Inc.
|
5
|
-
|
6
|
-
This program is free software; you can redistribute it and/or modify
|
7
|
-
it under the terms of the GNU General Public License as published by
|
8
|
-
the Free Software Foundation; either version 2, or (at your option)
|
9
|
-
any later version.
|
10
|
-
|
11
|
-
This program is distributed in the hope that it will be useful,
|
12
|
-
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
13
|
-
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
14
|
-
GNU General Public License for more details.
|
15
|
-
|
16
|
-
You should have received a copy of the GNU General Public License
|
17
|
-
along with this program; if not, write to the Free Software
|
18
|
-
Foundation, Inc., 59 Temple Place - Suite 330,
|
19
|
-
Boston, MA 02111-1307, USA. */
|
20
|
-
|
21
|
-
/* As a special exception, when this file is copied by Bison into a
|
22
|
-
Bison output file, you may use that output file without restriction.
|
23
|
-
This special exception was added by the Free Software Foundation
|
24
|
-
in version 1.24 of Bison. */
|
25
|
-
|
26
|
-
/* Tokens. */
|
27
|
-
#ifndef YYTOKENTYPE
|
28
|
-
# define YYTOKENTYPE
|
29
|
-
/* Put the tokens into the symbol table, so that GDB and other debuggers
|
30
|
-
know about them. */
|
31
|
-
enum yytokentype {
|
32
|
-
YAML_ANCHOR = 258,
|
33
|
-
YAML_ALIAS = 259,
|
34
|
-
YAML_TRANSFER = 260,
|
35
|
-
YAML_TAGURI = 261,
|
36
|
-
YAML_ITRANSFER = 262,
|
37
|
-
YAML_WORD = 263,
|
38
|
-
YAML_PLAIN = 264,
|
39
|
-
YAML_BLOCK = 265,
|
40
|
-
YAML_DOCSEP = 266,
|
41
|
-
YAML_IOPEN = 267,
|
42
|
-
YAML_INDENT = 268,
|
43
|
-
YAML_IEND = 269
|
44
|
-
};
|
45
|
-
#endif
|
46
|
-
#define YAML_ANCHOR 258
|
47
|
-
#define YAML_ALIAS 259
|
48
|
-
#define YAML_TRANSFER 260
|
49
|
-
#define YAML_TAGURI 261
|
50
|
-
#define YAML_ITRANSFER 262
|
51
|
-
#define YAML_WORD 263
|
52
|
-
#define YAML_PLAIN 264
|
53
|
-
#define YAML_BLOCK 265
|
54
|
-
#define YAML_DOCSEP 266
|
55
|
-
#define YAML_IOPEN 267
|
56
|
-
#define YAML_INDENT 268
|
57
|
-
#define YAML_IEND 269
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
#if ! defined (YYSTYPE) && ! defined (YYSTYPE_IS_DECLARED)
|
63
|
-
#line 35 "gram.y"
|
64
|
-
typedef union YYSTYPE {
|
65
|
-
SYMID nodeId;
|
66
|
-
SyckNode *nodeData;
|
67
|
-
char *name;
|
68
|
-
} YYSTYPE;
|
69
|
-
/* Line 1285 of yacc.c. */
|
70
|
-
#line 71 "gram.h"
|
71
|
-
# define yystype YYSTYPE /* obsolescent; will be withdrawn */
|
72
|
-
# define YYSTYPE_IS_DECLARED 1
|
73
|
-
# define YYSTYPE_IS_TRIVIAL 1
|
74
|
-
#endif
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
data/ext/i18nema/yamlbyte.h
DELETED
@@ -1,171 +0,0 @@
|
|
1
|
-
/* yamlbyte.h
|
2
|
-
*
|
3
|
-
* The YAML bytecode "C" interface header file. See the YAML bytecode
|
4
|
-
* reference for bytecode sequence rules and for the meaning of each
|
5
|
-
* bytecode.
|
6
|
-
*/
|
7
|
-
|
8
|
-
#ifndef YAMLBYTE_H
|
9
|
-
#define YAMLBYTE_H
|
10
|
-
#include <stddef.h>
|
11
|
-
|
12
|
-
/* define what a character is */
|
13
|
-
typedef unsigned char yamlbyte_utf8_t;
|
14
|
-
typedef unsigned short yamlbyte_utf16_t;
|
15
|
-
#ifdef YAMLBYTE_UTF8
|
16
|
-
#ifdef YAMLBYTE_UTF16
|
17
|
-
#error Must only define YAMLBYTE_UTF8 or YAMLBYTE_UTF16
|
18
|
-
#endif
|
19
|
-
typedef yamlbyte_utf8_t yamlbyte_char_t;
|
20
|
-
#else
|
21
|
-
#ifdef YAMLBYTE_UTF16
|
22
|
-
typedef yamlbyte_utf16_t yamlbyte_char_t;
|
23
|
-
#else
|
24
|
-
#error Must define YAMLBYTE_UTF8 or YAMLBYTE_UTF16
|
25
|
-
#endif
|
26
|
-
#endif
|
27
|
-
|
28
|
-
/* specify list of bytecodes */
|
29
|
-
#define YAMLBYTE_FINISH ((yamlbyte_char_t) 0)
|
30
|
-
#define YAMLBYTE_DOCUMENT ((yamlbyte_char_t)'D')
|
31
|
-
#define YAMLBYTE_DIRECTIVE ((yamlbyte_char_t)'V')
|
32
|
-
#define YAMLBYTE_PAUSE ((yamlbyte_char_t)'P')
|
33
|
-
#define YAMLBYTE_MAPPING ((yamlbyte_char_t)'M')
|
34
|
-
#define YAMLBYTE_SEQUENCE ((yamlbyte_char_t)'Q')
|
35
|
-
#define YAMLBYTE_END_BRANCH ((yamlbyte_char_t)'E')
|
36
|
-
#define YAMLBYTE_SCALAR ((yamlbyte_char_t)'S')
|
37
|
-
#define YAMLBYTE_CONTINUE ((yamlbyte_char_t)'C')
|
38
|
-
#define YAMLBYTE_NEWLINE ((yamlbyte_char_t)'N')
|
39
|
-
#define YAMLBYTE_NULLCHAR ((yamlbyte_char_t)'Z')
|
40
|
-
#define YAMLBYTE_ANCHOR ((yamlbyte_char_t)'A')
|
41
|
-
#define YAMLBYTE_ALIAS ((yamlbyte_char_t)'R')
|
42
|
-
#define YAMLBYTE_TRANSFER ((yamlbyte_char_t)'T')
|
43
|
-
/* formatting bytecodes */
|
44
|
-
#define YAMLBYTE_COMMENT ((yamlbyte_char_t)'c')
|
45
|
-
#define YAMLBYTE_INDENT ((yamlbyte_char_t)'i')
|
46
|
-
#define YAMLBYTE_STYLE ((yamlbyte_char_t)'s')
|
47
|
-
/* other bytecodes */
|
48
|
-
#define YAMLBYTE_LINE_NUMBER ((yamlbyte_char_t)'#')
|
49
|
-
#define YAMLBYTE_WHOLE_SCALAR ((yamlbyte_char_t)'<')
|
50
|
-
#define YAMLBYTE_NOTICE ((yamlbyte_char_t)'!')
|
51
|
-
#define YAMLBYTE_SPAN ((yamlbyte_char_t)')')
|
52
|
-
#define YAMLBYTE_ALLOC ((yamlbyte_char_t)'@')
|
53
|
-
|
54
|
-
/* second level style bytecodes, ie "s>" */
|
55
|
-
#define YAMLBYTE_FLOW ((yamlbyte_char_t)'>')
|
56
|
-
#define YAMLBYTE_LITERAL ((yamlbyte_char_t)'|')
|
57
|
-
#define YAMLBYTE_BLOCK ((yamlbyte_char_t)'b')
|
58
|
-
#define YAMLBYTE_PLAIN ((yamlbyte_char_t)'p')
|
59
|
-
#define YAMLBYTE_INLINE_MAPPING ((yamlbyte_char_t)'{')
|
60
|
-
#define YAMLBYTE_INLINE_SEQUENCE ((yamlbyte_char_t)'[')
|
61
|
-
#define YAMLBYTE_SINGLE_QUOTED ((yamlbyte_char_t)39)
|
62
|
-
#define YAMLBYTE_DOUBLE_QUOTED ((yamlbyte_char_t)'"')
|
63
|
-
|
64
|
-
/*
|
65
|
-
* The "C" API has two variants, one based on instructions,
|
66
|
-
* with events delivered via pointers; and the other one
|
67
|
-
* is character based where one or more instructions are
|
68
|
-
* serialized into a buffer.
|
69
|
-
*
|
70
|
-
* Note: In the instruction based API, WHOLE_SCALAR does
|
71
|
-
* not have the '<here' marshalling stuff.
|
72
|
-
*/
|
73
|
-
|
74
|
-
typedef void * yamlbyte_consumer_t;
|
75
|
-
typedef void * yamlbyte_producer_t;
|
76
|
-
|
77
|
-
/* push and pull APIs need a way to communicate results */
|
78
|
-
typedef enum {
|
79
|
-
YAMLBYTE_OK = 0, /* proceed */
|
80
|
-
YAMLBYTE_E_MEMORY = 'M', /* could not allocate memory */
|
81
|
-
YAMLBYTE_E_READ = 'R', /* input stream read error */
|
82
|
-
YAMLBYTE_E_WRITE = 'W', /* output stream write error */
|
83
|
-
YAMLBYTE_E_OTHER = '?', /* some other error condition */
|
84
|
-
YAMLBYTE_E_PARSE = 'P', /* parse error, check bytecodes */
|
85
|
-
YAMLBYTE_MAX
|
86
|
-
} yamlbyte_result_t;
|
87
|
-
|
88
|
-
typedef const yamlbyte_char_t *yamlbyte_buff_t;
|
89
|
-
|
90
|
-
/*
|
91
|
-
* The "Instruction" API
|
92
|
-
*/
|
93
|
-
|
94
|
-
typedef struct yaml_instruction {
|
95
|
-
yamlbyte_char_t bytecode;
|
96
|
-
yamlbyte_buff_t start;
|
97
|
-
yamlbyte_buff_t finish; /* open range, *finish is _not_ part */
|
98
|
-
} *yamlbyte_inst_t;
|
99
|
-
|
100
|
-
/* producer pushes the instruction with one bytecode event to the
|
101
|
-
* consumer; if the consumer's result is not YAMLBYTE_OK, then
|
102
|
-
* the producer should stop */
|
103
|
-
typedef
|
104
|
-
yamlbyte_result_t
|
105
|
-
(*yamlbyte_push_t)(
|
106
|
-
yamlbyte_consumer_t self,
|
107
|
-
yamlbyte_inst_t inst
|
108
|
-
);
|
109
|
-
|
110
|
-
/* consumer pulls a bytecode instruction from the producer; in this
|
111
|
-
* case the instruction (and is buffer) are owned by the producer and
|
112
|
-
* will remain valid till the pull function is called once again;
|
113
|
-
* if the instruction is NULL, then there are no more results; and
|
114
|
-
* it is important to call the pull function till it returns NULL so
|
115
|
-
* that the producer can clean up its memory allocations */
|
116
|
-
typedef
|
117
|
-
yamlbyte_result_t
|
118
|
-
(*yamlbyte_pull_t)(
|
119
|
-
yamlbyte_producer_t self,
|
120
|
-
yamlbyte_inst_t *inst /* to be filled in by the producer */
|
121
|
-
);
|
122
|
-
|
123
|
-
/*
|
124
|
-
* Buffer based API
|
125
|
-
*/
|
126
|
-
|
127
|
-
/* producer pushes a null terminated buffer filled with one or more
|
128
|
-
* bytecode events to the consumer; if the consumer's result is not
|
129
|
-
* YAMLBYTE_OK, then the producer should stop */
|
130
|
-
typedef
|
131
|
-
yamlbyte_result_t
|
132
|
-
(*yamlbyte_pushbuff_t)(
|
133
|
-
yamlbyte_consumer_t self,
|
134
|
-
yamlbyte_buff_t buff
|
135
|
-
);
|
136
|
-
|
137
|
-
/* consumer pulls bytecode events from the producer; in this case
|
138
|
-
* the buffer is owned by the producer, and will remain valid till
|
139
|
-
* the pull function is called once again; if the buffer pointer
|
140
|
-
* is set to NULL, then there are no more results; it is important
|
141
|
-
* to call the pull function till it returns NULL so that the
|
142
|
-
* producer can clean up its memory allocations */
|
143
|
-
typedef
|
144
|
-
yamlbyte_result_t
|
145
|
-
(*yamlbyte_pullbuff_t)(
|
146
|
-
yamlbyte_producer_t self,
|
147
|
-
yamlbyte_buff_t *buff /* to be filled in by the producer */
|
148
|
-
);
|
149
|
-
|
150
|
-
/* convert a pull interface to a push interface; the reverse process
|
151
|
-
* requires threads and thus is language dependent */
|
152
|
-
#define YAMLBYTE_PULL2PUSH(pull,producer,push,consumer,result) \
|
153
|
-
do { \
|
154
|
-
yamlbyte_pullbuff_t _pull = (pull); \
|
155
|
-
yamlbyte_pushbuff_t _push = (push); \
|
156
|
-
yamlbyte_result_t _result = YAMLBYTE_OK; \
|
157
|
-
yamlbyte_producer_t _producer = (producer); \
|
158
|
-
yamlbyte_consumer_t _consumer = (consumer); \
|
159
|
-
while(1) { \
|
160
|
-
yamlbyte_buff_t buff = NULL; \
|
161
|
-
_result = _pull(_producer,&buff); \
|
162
|
-
if(YAMLBYTE_OK != result || NULL == buff) \
|
163
|
-
break; \
|
164
|
-
_result = _push(_consumer,buff); \
|
165
|
-
if(YAMLBYTE_OK != result) \
|
166
|
-
break; \
|
167
|
-
} \
|
168
|
-
(result) = _result; \
|
169
|
-
} while(0)
|
170
|
-
|
171
|
-
#endif
|