i18nema 0.0.4 → 0.0.5

data/README.md

@@ -3,9 +3,9 @@
 Fast I18n backend to keep things running smoothly.
 
 I18nema is a drop-in replacement for I18n::Backend::Simple, for faster
-lookups and quicker gc runs. Translations are stored outside of the ruby
-heap, and lookups happen in C (rather than the usual inject on nested
-ruby hashes).
+lookups (2x!) and quicker GC runs (ymmv). Translations are stored
+outside of the ruby heap, and lookups happen in C (rather than the usual
+inject on nested ruby hashes).
 
 ## How do I use it?
 
@@ -15,11 +15,113 @@ and then do something like this in an initializer:
 
     I18n.backend = I18nema::Backend.new
 
-As with I18n::Backend::Simple, you can pull in additional features, e.g.
+You can pull in additional features, e.g.
 
     I18nema::Backend.send(:include, I18n::Backend::Fallbacks)
 
-## Notes
+As with regular I18n, you should probably load translations before you
+fork, so that all processes can use the same translations in memory. In
+an initializer, just do `I18n.backend.init_translations`.
+
+## What sort of improvements will I see?
+
+### Faster Startup
+
+Loading all the translations into memory is dramtically faster with
+I18nema (about 4x). While this is just a one time hit, it's pretty
+noticeable when you're waiting on it (e.g. console, specs). In
+[canvas-lms](https://github/com/instructure/canvas-lms), I18nema brings
+it down to just over half a second (from almost 2.5).
+
+### Faster GC Runs
+
+Because there are fewer ruby objects, the periodic GC runs will be
+proportionally faster. How much faster is a question of how many
+translations you have versus how many other ruby objects. Applications
+that are localized in more languages should see a bigger boost (since
+the translations make up a bigger share of the original ObjectSpace).
+
+For example, [canvas-lms](https://github/com/instructure/canvas-lms) is
+translated into seven other languages, and I18nema reduces (startup)
+ObjectSpace by about 18% and GC runtime by about 11%.
+
+I18nema also moves I18n's normalized_key_cache into C structs. This key
+cache grows over time (it eventually holds a key/value for every
+translation key used in the app), so that's another area where I18nema
+is nicer on ObjectSpace than vanilla I18n.
+
+### Faster Translation Lookups
+
+Simple lookups (i.e. no options or interpolation) take about 55% less
+time, and that holds true no matter how deeply scoped your keys are
+(e.g. `foo.bar.baz.lol`).
+
+Lookups with options are also faster (about 35% less time). The gains
+aren't quite as big there yet, since proportionally more of the time is
+spent in unoptimized places like `I18n#interpolate`.
+
+## Show me the benchmarks
+
+Here are some basic ones done with `Benchmark.bmbm` (edited for brevity)
+We run `I18n.translate` 100000 times on 4 different translation keys.
+The `n` in `translate(n)` denotes how many parts there are in the key,
+e.g. `I18n.t('foo') -> 1`, `I18n.t('foo.bar') -> 2`
+
+### simple `translate` (no options)
+
+#### I18nema
+
+                        user     system      total        real
+    translate(1):   0.960000   0.010000   0.970000 (  0.963384)
+    translate(2):   1.070000   0.010000   1.080000 (  1.068789)
+    translate(3):   1.080000   0.000000   1.080000 (  1.083826)
+    translate(4):   1.260000   0.010000   1.270000 (  1.263967)
+
+#### I18n
+
+                        user     system      total        real
+    translate(1):   2.140000   0.000000   2.140000 (  2.137579)
+    translate(2):   2.380000   0.000000   2.380000 (  2.382720)
+    translate(3):   2.420000   0.000000   2.420000 (  2.428245)
+    translate(4):   2.510000   0.000000   2.510000 (  2.514133)
+
+### `translate` with options (locale, interpolation)
+
+#### I18nema
+
+                        user     system      total        real
+    translate(1):   1.770000   0.000000   1.770000 (  1.771934)
+    translate(2):   1.870000   0.010000   1.880000 (  1.880586)
+    translate(3):   1.890000   0.000000   1.890000 (  1.893065)
+    translate(4):   2.080000   0.000000   2.080000 (  2.083093)
+
+#### I18n
+
+                        user     system      total        real
+    translate(1):   2.720000   0.000000   2.720000 (  2.720541)
+    translate(2):   2.950000   0.010000   2.960000 (  2.953175)
+    translate(3):   3.010000   0.010000   3.020000 (  3.028288)
+    translate(4):   3.120000   0.010000   3.130000 (  3.117135)
+
+
+## OK, so what's the catch?
+
+I18nema is still a work in progress, so there are some compatibility
+notes you should be aware of:
+
+I18nema requires ruby 1.9.3 or later.
+
+I18nema only supports `.yml` translation files (no `.rb`).
+
+I18nema requires UTF-8 `.yml` files. That means that your translations
+should actually be in their UTF-8 form (e.g. "Contraseña"), not some
+escaped representation. I18nema uses a simplified syck implementation
+and does not support many optional yml types (e.g. `binary`).
+
+I18nema doesn't yet support symbols as translation *values* (note that
+symbol [keys](http://guides.rubyonrails.org/i18n.html#basic-lookup-scopes-and-nested-keys)
+and [defaults](http://guides.rubyonrails.org/i18n.html#defaults) work
+just fine). Symbol values in your `.yml` file can be used in the same
+way that symbol defaults can, i.e. they tell I18n to find the
+translation under some other key.
 
-You should probably make sure translations are loaded before you fork. In
-an initializer, just do `I18n.backend.init_translations`

data/ext/i18nema/i18nema.c

@@ -1,7 +1,7 @@
 #include <ruby.h>
 #include <ruby/encoding.h>
-#include <syck.h>
-#include "uthash.h"
+#include "vendor/syck.h"
+#include "vendor/uthash.h"
 
 VALUE I18nema = Qnil,
       I18nemaBackend = Qnil,
@@ -12,8 +12,10 @@ struct i_key_value;
 static VALUE array_to_rarray(struct i_object *array);
 static VALUE hash_to_rhash(struct i_object *hash);
 static void merge_hash(struct i_object *hash, struct i_object *other_hash);
-static void delete_hash(struct i_key_value **hash, int delete_child_objects);
-static void delete_object(struct i_object *object, int delete_child_objects);
+static void delete_hash(struct i_key_value **hash, int recurse);
+static void delete_object(struct i_object *object, int recurse);
+static void delete_object_r(struct i_object *object);
+static VALUE normalize_key(VALUE self, VALUE key, VALUE separator);
 
 enum i_object_type {
   i_type_none,
@@ -105,13 +107,37 @@ hash_to_rhash(i_object_t *hash)
 }
 
 static i_object_t*
-root_object_get(VALUE self)
+i_object_get(VALUE self, const char *iv)
 {
-  i_object_t *root_object;
-  VALUE translations;
-  translations = rb_iv_get(self, "@translations");
-  Data_Get_Struct(translations, i_object_t, root_object);
-  return root_object;
+  i_object_t *object;
+  VALUE wrapped;
+  wrapped = rb_iv_get(self, iv);
+  Data_Get_Struct(wrapped, i_object_t, object);
+  return object;
+}
+
+static i_object_t*
+translations_get(VALUE self)
+{
+  return i_object_get(self, "@translations");
+}
+
+static i_object_t*
+normalized_key_cache_get(VALUE self)
+{
+  return i_object_get(self, "@normalized_key_cache");
+}
+
+static i_object_t*
+hash_get(i_object_t *current, VALUE *keys, int num_keys)
+{
+  i_key_value_t *kv = NULL;
+  for (int i = 0; i < num_keys && current != NULL && current->type == i_type_hash; i++) {
+    Check_Type(keys[i], T_STRING);
+    HASH_FIND_STR(current->data.hash, StringValueCStr(keys[i]), kv);
+    current = kv == NULL ? NULL : kv->value;
+  }
+  return current;
 }
 
 /*
@@ -127,37 +153,25 @@ root_object_get(VALUE self)
 static VALUE
 direct_lookup(int argc, VALUE *argv, VALUE self)
 {
-  i_object_t *result = root_object_get(self);;
-  i_key_value_t *kv = NULL;
-  VALUE rs;
-  char *s;
-
-  for (int i = 0; i < argc && result != NULL && result->type == i_type_hash; i++) {
-    rs = rb_funcall(argv[i], s_to_s, 0);
-    s = StringValueCStr(rs);
-    HASH_FIND_STR(result->data.hash, s, kv);
-    result = kv == NULL ? NULL : kv->value;
-  }
-
-  return i_object_to_robject(result);
+  i_object_t *translations = translations_get(self);
+  return i_object_to_robject(hash_get(translations, argv, argc));
 }
 
 static void
-empty_object(i_object_t *object, int delete_child_objects)
+empty_object(i_object_t *object, int recurse)
 {
   if (object == NULL)
     return;
 
   switch (object->type) {
   case i_type_array:
-    if (delete_child_objects) {
+    if (recurse)
       for (unsigned long i = 0; i < object->size; i++)
-        delete_object(object->data.array[i], 1);
-    }
+        delete_object_r(object->data.array[i]);
     xfree(object->data.array);
     break;
   case i_type_hash:
-    delete_hash(&object->data.hash, delete_child_objects);
+    delete_hash(&object->data.hash, recurse);
     break;
   case i_type_none:
     break;
@@ -168,35 +182,35 @@ empty_object(i_object_t *object, int delete_child_objects)
 }
 
 static void
-delete_object_r(i_object_t *object)
+delete_object(i_object_t *object, int recurse)
 {
-  delete_object(object, 1);
+  empty_object(object, recurse);
+  if (object->type != i_type_null && object->type != i_type_true && object->type != i_type_false)
+    xfree(object);
 }
 
 static void
-delete_object(i_object_t *object, int delete_child_objects)
+delete_object_r(i_object_t *object)
 {
-  empty_object(object, delete_child_objects);
-  if (object->type != i_type_null && object->type != i_type_true && object->type != i_type_false)
-    xfree(object);
+  delete_object(object, 1);
 }
 
 static void
 delete_key_value(i_key_value_t *kv, int delete_value)
 {
   if (delete_value)
-    delete_object(kv->value, 1);
+    delete_object_r(kv->value);
   xfree(kv->key);
   xfree(kv);
 }
 
 static void
-delete_hash(i_key_value_t **hash, int delete_child_objects)
+delete_hash(i_key_value_t **hash, int recurse)
 {
   i_key_value_t *kv, *tmp;
   HASH_ITER(hh, *hash, kv, tmp) {
     HASH_DEL(*hash, kv);
-    delete_key_value(kv, delete_child_objects);
+    delete_key_value(kv, recurse);
   }
 }
 
@@ -227,7 +241,7 @@ merge_hash(i_object_t *hash, i_object_t *other_hash)
     HASH_DEL(other_hash->data.hash, kv);
     add_key_value(&hash->data.hash, kv);
   }
-  delete_object(other_hash, 1);
+  delete_object_r(other_hash);
 }
 
 static int
@@ -236,7 +250,7 @@ delete_syck_st_entry(char *key, char *value, char *arg)
   i_object_t *object = (i_object_t *)value;
   // key object whose string we have yoinked into a kv
   if (object->type == i_type_none)
-    delete_object(object, 1);
+    delete_object_r(object);
   return ST_DELETE;
 }
 
@@ -270,18 +284,53 @@ handle_syck_badanchor(SyckParser *parser, char *anchor)
   return NULL;
 }
 
+static char*
+new_string(char *orig, long len)
+{
+  char *str = xmalloc(len + 1);
+  strncpy(str, orig, len);
+  str[len] = '\0';
+  return str;
+}
+
 static i_object_t*
 new_string_object(char *str, long len)
 {
   i_object_t *object = ALLOC(i_object_t);
   object->type = i_type_string;
   object->size = len;
-  object->data.string = xmalloc(len + 1);
-  strncpy(object->data.string, str, len);
-  object->data.string[len] = '\0';
+  object->data.string = new_string(str, len);
+  return object;
+}
+
+static i_object_t*
+new_array_object(long size)
+{
+  i_object_t *object = ALLOC(i_object_t);
+  object->type = i_type_array;
+  object->size = size;
+  object->data.array = ALLOC_N(i_object_t*, size);
   return object;
 }
 
+static i_object_t*
+new_hash_object()
+{
+  i_object_t *object = ALLOC(i_object_t);
+  object->type = i_type_hash;
+  object->data.hash = NULL;
+  return object;
+}
+
+static i_key_value_t*
+new_key_value(char *key, i_object_t *value)
+{
+  i_key_value_t *kv = ALLOC(i_key_value_t);
+  kv->key = key;
+  kv->value = value;
+  return kv;
+}
+
 static SYMID
 handle_syck_node(SyckParser *parser, SyckNode *node)
 {
@@ -315,10 +364,7 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
     }
     break;
   case syck_seq_kind:
-    result = ALLOC(i_object_t);
-    result->type = i_type_array;
-    result->size = node->data.list->idx;
-    result->data.array = ALLOC_N(i_object_t*, node->data.list->idx);
+    result = new_array_object(node->data.list->idx);
     for (long i = 0; i < node->data.list->idx; i++) {
       i_object_t *item = NULL;
 
@@ -330,9 +376,7 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
     }
     break;
   case syck_map_kind:
-    result = ALLOC(i_object_t);
-    result->type = i_type_hash;
-    result->data.hash = NULL;
+    result = new_hash_object();
     for (long i = 0; i < node->data.pairs->idx; i++) {
       i_object_t *key = NULL, *value = NULL;
 
@@ -341,11 +385,8 @@ handle_syck_node(SyckParser *parser, SyckNode *node)
       oid = syck_map_read(node, map_value, i);
       syck_lookup_sym(parser, oid, (void **)&value);
 
-      i_key_value_t *kv;
-      kv = ALLOC(i_key_value_t);
-      kv->key = key->data.string;
+      i_key_value_t *kv = new_key_value(key->data.string, value);
       key->type = i_type_none; // so we know to free this node in delete_syck_st_entry
-      kv->value = value;
       if (value->type == i_type_string)
         current_translation_count++;
       add_key_value(&result->data.hash, kv);
@@ -371,7 +412,7 @@ static VALUE
 load_yml_string(VALUE self, VALUE yml)
 {
   SYMID oid;
-  i_object_t *root_object = root_object_get(self);
+  i_object_t *root_object = translations_get(self);
   i_object_t *new_root_object = NULL;
   current_translation_count = 0;
   SyckParser* parser = syck_new_parser();
@@ -387,7 +428,7 @@ load_yml_string(VALUE self, VALUE yml)
     st_foreach(parser->syms, delete_syck_st_entry, 0);
   syck_free_parser(parser);
   if (new_root_object == NULL || new_root_object->type != i_type_hash) {
-    delete_object(new_root_object, 1);
+    delete_object_r(new_root_object);
     rb_raise(I18nemaBackendLoadError, "root yml node is not a hash");
   }
   merge_hash(root_object, new_root_object);
@@ -409,7 +450,7 @@ available_locales(VALUE self)
 {
   if (!RTEST(rb_iv_get(self, "@initialized")))
     rb_funcall(self, s_init_translations, 0);
-  i_object_t *root_object = root_object_get(self);
+  i_object_t *root_object = translations_get(self);
   i_key_value_t *current = root_object->data.hash;
   VALUE ary = rb_ary_new2(0);
 
@@ -431,22 +472,96 @@ available_locales(VALUE self)
 static VALUE
 reload(VALUE self)
 {
-  i_object_t *root_object = root_object_get(self);
+  i_object_t *root_object = translations_get(self);
   empty_object(root_object, 1);
   rb_iv_set(self, "@initialized", Qfalse);
-  root_object = NULL;
   return Qtrue;
 }
 
+static VALUE
+join_array_key(VALUE self, VALUE key, VALUE separator)
+{
+  long len = RARRAY_LEN(key);
+  if (len == 0)
+    return rb_str_new("", 0);
+
+  VALUE ret = rb_ary_join(normalize_key(self, RARRAY_PTR(key)[0], separator), separator);
+  for (long i = 1; i < len; i++) {
+    rb_str_concat(ret, separator);
+    rb_str_concat(ret, rb_ary_join(normalize_key(self, RARRAY_PTR(key)[i], separator), separator));
+  }
+  return ret;
+}
+
+/*
+ *  call-seq:
+ *     backend.normalize_key(key, separator) -> key
+ *
+ *  Normalizes and splits a key based on the separator.
+ *
+ *     backend.normalize_key "asdf", "."    #=> ["asdf"]
+ *     backend.normalize_key "a.b.c", "."   #=> ["a", "b", "c"]
+ *     backend.normalize_key "a.b.c", ":"   #=> ["a.b.c"]
+ *     backend.normalize_key %{a b.c}, "."  #=> ["a", "b", "c"]
+ */
+
+static VALUE
+normalize_key(VALUE self, VALUE key, VALUE separator)
+{
+  Check_Type(separator, T_STRING);
+
+  i_object_t *key_map = normalized_key_cache_get(self),
+             *sub_map = hash_get(key_map, &separator, 1);
+  if (sub_map == NULL) {
+    sub_map = new_hash_object();
+    char *key = new_string(RSTRING_PTR(separator), RSTRING_LEN(separator));
+    i_key_value_t *kv = new_key_value(key, sub_map);
+    add_key_value(&key_map->data.hash, kv);
+  }
+
+  if (TYPE(key) == T_ARRAY)
+    key = join_array_key(self, key, separator);
+  else if (TYPE(key) != T_STRING)
+    key = rb_funcall(key, s_to_s, 0);
+
+  i_object_t *key_frd = hash_get(sub_map, &key, 1);
+
+  if (key_frd == NULL) {
+    char *sep = StringValueCStr(separator);
+    VALUE parts = rb_str_split(key, sep);
+    long parts_len = RARRAY_LEN(parts),
+         skipped = 0;
+    key_frd = new_array_object(parts_len);
+    for (long i = 0; i < parts_len; i++) {
+      VALUE part = RARRAY_PTR(parts)[i];
+      // TODO: don't alloc for empty strings, since we discard them
+      if (RSTRING_LEN(part) == 0)
+        skipped++;
+      else
+        key_frd->data.array[i - skipped] = new_string_object(RSTRING_PTR(part), RSTRING_LEN(part));
+    }
+    key_frd->size -= skipped;
+
+    char *key_orig = new_string(RSTRING_PTR(key), RSTRING_LEN(key));
+    i_key_value_t *kv = new_key_value(key_orig, key_frd);
+    add_key_value(&sub_map->data.hash, kv);
+  }
+  return i_object_to_robject(key_frd);
+}
+
 static VALUE
 initialize(VALUE self)
 {
-  VALUE translations;
-  i_object_t *root_object = ALLOC(i_object_t);
-  root_object->type = i_type_hash;
-  root_object->data.hash = NULL;
+  VALUE translations, key_cache;
+
+  i_object_t *root_object = new_hash_object();
   translations = Data_Wrap_Struct(I18nemaBackend, 0, delete_object_r, root_object);
   rb_iv_set(self, "@translations", translations);
+
+  i_object_t *key_map = new_hash_object();
+  key_cache = Data_Wrap_Struct(I18nemaBackend, 0, delete_object_r, key_map);
+  rb_iv_set(self, "@normalized_key_cache", key_cache);
+
   return self;
 }
 
@@ -471,4 +586,5 @@ Init_i18nema()
   rb_define_method(I18nemaBackend, "available_locales", available_locales, 0);
   rb_define_method(I18nemaBackend, "reload!", reload, 0);
   rb_define_method(I18nemaBackend, "direct_lookup", direct_lookup, -1);
+  rb_define_method(I18nemaBackend, "normalize_key", normalize_key, 2);
 }

data/lib/i18nema.rb

@@ -18,6 +18,36 @@ module I18nema
       @initialized = true
     end
 
+    RESERVED_KEY_MAP = Hash[I18n::RESERVED_KEYS.map{|k|[k,true]}]
+
+    def translate(locale, key, options = {})
+      raise I18n::InvalidLocale.new(locale) unless locale
+      entry = key && lookup(locale, key, options[:scope], options)
+
+      if options.empty?
+        entry = resolve(locale, key, entry, options)
+      else
+        count, default = options.values_at(:count, :default)
+        # significant speedup over Hash#except
+        values = options.reject { |key, value| RESERVED_KEY_MAP.key?(key) }
+        entry = entry.nil? && default ?
+          default(locale, key, default, options) : resolve(locale, key, entry, options)
+      end
+
+      throw(:exception, I18n::MissingTranslation.new(locale, key, options)) if entry.nil?
+      # no need to dup, since I18nema gives us a new string
+
+      entry = pluralize(locale, entry, count) if count
+      entry = interpolate(locale, entry, values) if values
+      entry
+    end
+
+    alias_method :reload_old, :reload!
+    def reload!
+      debugger
+      reload_old
+    end
+
   protected
     def load_file(filename)
       type = File.extname(filename).tr('.', '').downcase
@@ -35,8 +65,17 @@ module I18nema
 
     def lookup(locale, key, scope = [], options = {})
       init_translations unless initialized?
-      keys = I18n.normalize_keys(locale, key, scope, options[:separator])
+      keys = normalize_keys(locale, key, scope, options[:separator])
       direct_lookup(*keys)
     end
+
+    def normalize_keys(locale, key, scope, separator = nil)
+      separator ||= I18n.default_separator
+
+      keys = [locale.to_s]
+      keys.concat normalize_key(scope, separator) if scope && scope.size > 0
+      keys.concat normalize_key(key, separator)
+      keys
+    end
   end
 end

data/test/i18nema_test.rb

@@ -68,10 +68,6 @@ class I18nemaTest < Test::Unit::TestCase
     assert_equal("root yml node is not a hash", exception.message)
     assert_equal({}, backend.direct_lookup)
 
-    # FIXME ... ruby syck does this differently, and we get a non
-    # i_object_t as the root node, causing delete_object to asplode when
-    # it tries to free a garbage pointer
-    #
     exception = assert_raise(I18nema::Backend::LoadError) {
       backend.load_yml_string("en:\n  foo: \"lol\"\n\tbar: notabs!")
     }
@@ -84,4 +80,18 @@ class I18nemaTest < Test::Unit::TestCase
     assert_match(/bad anchor `a'/, exception.message)
     assert_equal({}, backend.direct_lookup)
   end
+
+  def test_normalize_key
+    backend = I18nema::Backend.new
+    assert_equal %w{asdf},
+                 backend.normalize_key("asdf", ".")
+    assert_equal %w{asdf asdf},
+                 backend.normalize_key("asdf.asdf", ".")
+    assert_equal %w{asdf.asdf},
+                 backend.normalize_key("asdf.asdf", ",")
+    assert_equal %w{asdf asdf},
+                 backend.normalize_key("asdf...asdf", ".")
+    assert_equal %w{asdf asdf asdf},
+                 backend.normalize_key(%w{asdf asdf.asdf}, ".")
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: i18nema
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-30 00:00:00.000000000 Z
+date: 2013-06-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: i18n
@@ -55,11 +55,8 @@ files:
 - Rakefile
 - README.md
 - ext/i18nema/i18nema.c
-- ext/i18nema/extconf.h
-- ext/i18nema/gram.h
-- ext/i18nema/syck.h
-- ext/i18nema/uthash.h
-- ext/i18nema/yamlbyte.h
+- ext/i18nema/vendor/syck.h
+- ext/i18nema/vendor/uthash.h
 - ext/i18nema/extconf.rb
 - ext/i18nema/mkrf_conf.rb
 - lib/i18nema/core_ext/hash.rb

data/ext/i18nema/extconf.h

@@ -1,3 +0,0 @@
-#ifndef EXTCONF_H
-#define EXTCONF_H
-#endif

data/ext/i18nema/gram.h

@@ -1,79 +0,0 @@
-/* A Bison parser, made by GNU Bison 1.875d.  */
-
-/* Skeleton parser for Yacc-like parsing with Bison,
-   Copyright (C) 1984, 1989, 1990, 2000, 2001, 2002, 2003, 2004 Free Software Foundation, Inc.
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU General Public License as published by
-   the Free Software Foundation; either version 2, or (at your option)
-   any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU General Public License
-   along with this program; if not, write to the Free Software
-   Foundation, Inc., 59 Temple Place - Suite 330,
-   Boston, MA 02111-1307, USA.  */
-
-/* As a special exception, when this file is copied by Bison into a
-   Bison output file, you may use that output file without restriction.
-   This special exception was added by the Free Software Foundation
-   in version 1.24 of Bison.  */
-
-/* Tokens.  */
-#ifndef YYTOKENTYPE
-# define YYTOKENTYPE
-   /* Put the tokens into the symbol table, so that GDB and other debuggers
-      know about them.  */
-   enum yytokentype {
-     YAML_ANCHOR = 258,
-     YAML_ALIAS = 259,
-     YAML_TRANSFER = 260,
-     YAML_TAGURI = 261,
-     YAML_ITRANSFER = 262,
-     YAML_WORD = 263,
-     YAML_PLAIN = 264,
-     YAML_BLOCK = 265,
-     YAML_DOCSEP = 266,
-     YAML_IOPEN = 267,
-     YAML_INDENT = 268,
-     YAML_IEND = 269
-   };
-#endif
-#define YAML_ANCHOR 258
-#define YAML_ALIAS 259
-#define YAML_TRANSFER 260
-#define YAML_TAGURI 261
-#define YAML_ITRANSFER 262
-#define YAML_WORD 263
-#define YAML_PLAIN 264
-#define YAML_BLOCK 265
-#define YAML_DOCSEP 266
-#define YAML_IOPEN 267
-#define YAML_INDENT 268
-#define YAML_IEND 269
-
-
-
-
-#if ! defined (YYSTYPE) && ! defined (YYSTYPE_IS_DECLARED)
-#line 35 "gram.y"
-typedef union YYSTYPE {
-    SYMID nodeId;
-    SyckNode *nodeData;
-    char *name;
-} YYSTYPE;
-/* Line 1285 of yacc.c.  */
-#line 71 "gram.h"
-# define yystype YYSTYPE /* obsolescent; will be withdrawn */
-# define YYSTYPE_IS_DECLARED 1
-# define YYSTYPE_IS_TRIVIAL 1
-#endif
-
-
-
-
-

data/ext/i18nema/yamlbyte.h

@@ -1,171 +0,0 @@
-/*  yamlbyte.h
- *
- *  The YAML bytecode "C" interface header file.   See the YAML bytecode
- *  reference for bytecode sequence rules and for the meaning of each
- *  bytecode.
- */
-
-#ifndef YAMLBYTE_H
-#define YAMLBYTE_H
-#include <stddef.h>
-
-/* define what a character is */
-typedef unsigned char yamlbyte_utf8_t;
-typedef unsigned short yamlbyte_utf16_t;
-#ifdef YAMLBYTE_UTF8
-  #ifdef YAMLBYTE_UTF16
-    #error Must only define YAMLBYTE_UTF8 or YAMLBYTE_UTF16
-  #endif
-  typedef yamlbyte_utf8_t yamlbyte_char_t;
-#else
-  #ifdef YAMLBYTE_UTF16
-    typedef yamlbyte_utf16_t yamlbyte_char_t;
-  #else
-    #error Must define YAMLBYTE_UTF8 or YAMLBYTE_UTF16
-  #endif
-#endif
-
-/* specify list of bytecodes */
-#define YAMLBYTE_FINISH          ((yamlbyte_char_t) 0)
-#define YAMLBYTE_DOCUMENT        ((yamlbyte_char_t)'D')
-#define YAMLBYTE_DIRECTIVE       ((yamlbyte_char_t)'V')
-#define YAMLBYTE_PAUSE           ((yamlbyte_char_t)'P')
-#define YAMLBYTE_MAPPING         ((yamlbyte_char_t)'M')
-#define YAMLBYTE_SEQUENCE        ((yamlbyte_char_t)'Q')
-#define YAMLBYTE_END_BRANCH      ((yamlbyte_char_t)'E')
-#define YAMLBYTE_SCALAR          ((yamlbyte_char_t)'S')
-#define YAMLBYTE_CONTINUE        ((yamlbyte_char_t)'C')
-#define YAMLBYTE_NEWLINE         ((yamlbyte_char_t)'N')
-#define YAMLBYTE_NULLCHAR        ((yamlbyte_char_t)'Z')
-#define YAMLBYTE_ANCHOR          ((yamlbyte_char_t)'A')
-#define YAMLBYTE_ALIAS           ((yamlbyte_char_t)'R')
-#define YAMLBYTE_TRANSFER        ((yamlbyte_char_t)'T')
-/* formatting bytecodes */
-#define YAMLBYTE_COMMENT         ((yamlbyte_char_t)'c')
-#define YAMLBYTE_INDENT          ((yamlbyte_char_t)'i')
-#define YAMLBYTE_STYLE           ((yamlbyte_char_t)'s')
-/* other bytecodes */
-#define YAMLBYTE_LINE_NUMBER     ((yamlbyte_char_t)'#')
-#define YAMLBYTE_WHOLE_SCALAR    ((yamlbyte_char_t)'<')
-#define YAMLBYTE_NOTICE          ((yamlbyte_char_t)'!')
-#define YAMLBYTE_SPAN            ((yamlbyte_char_t)')')
-#define YAMLBYTE_ALLOC           ((yamlbyte_char_t)'@')
-
-/* second level style bytecodes, ie "s>" */
-#define YAMLBYTE_FLOW            ((yamlbyte_char_t)'>')
-#define YAMLBYTE_LITERAL         ((yamlbyte_char_t)'|')
-#define YAMLBYTE_BLOCK           ((yamlbyte_char_t)'b')
-#define YAMLBYTE_PLAIN           ((yamlbyte_char_t)'p')
-#define YAMLBYTE_INLINE_MAPPING  ((yamlbyte_char_t)'{')
-#define YAMLBYTE_INLINE_SEQUENCE ((yamlbyte_char_t)'[')
-#define YAMLBYTE_SINGLE_QUOTED   ((yamlbyte_char_t)39)
-#define YAMLBYTE_DOUBLE_QUOTED   ((yamlbyte_char_t)'"')
-
-/*
- * The "C" API has two variants, one based on instructions,
- * with events delivered via pointers; and the other one
- * is character based where one or more instructions are
- * serialized into a buffer.
- *
- * Note: In the instruction based API, WHOLE_SCALAR does
- *       not have the '<here' marshalling stuff.
- */
-
-typedef void * yamlbyte_consumer_t;
-typedef void * yamlbyte_producer_t;
-
-/* push and pull APIs need a way to communicate results */
-typedef enum {
-    YAMLBYTE_OK          = 0,     /* proceed                        */
-    YAMLBYTE_E_MEMORY    = 'M',   /* could not allocate memory      */
-    YAMLBYTE_E_READ      = 'R',   /* input stream read error        */
-    YAMLBYTE_E_WRITE     = 'W',   /* output stream write error      */
-    YAMLBYTE_E_OTHER     = '?',   /* some other error condition     */
-    YAMLBYTE_E_PARSE     = 'P',   /* parse error, check bytecodes   */
-    YAMLBYTE_MAX
-} yamlbyte_result_t;
-
-typedef const yamlbyte_char_t *yamlbyte_buff_t;
-
-/*
- *  The "Instruction" API
- */
-
-typedef struct yaml_instruction {
-    yamlbyte_char_t bytecode;
-    yamlbyte_buff_t start;
-    yamlbyte_buff_t finish;  /* open range, *finish is _not_ part */
-} *yamlbyte_inst_t;
-
-/* producer pushes the instruction with one bytecode event to the
- * consumer; if the consumer's result is not YAMLBYTE_OK, then
- * the producer should stop */
-typedef
-  yamlbyte_result_t
-   (*yamlbyte_push_t)(
-     yamlbyte_consumer_t self,
-     yamlbyte_inst_t  inst
-   );
-
-/* consumer pulls a bytecode instruction from the producer; in this
- * case the instruction (and is buffer) are owned by the producer and
- * will remain valid till the pull function is called once again;
- * if the instruction is NULL, then there are no more results; and
- * it is important to call the pull function till it returns NULL so
- * that the producer can clean up its memory allocations */
-typedef
-   yamlbyte_result_t
-    (*yamlbyte_pull_t)(
-      yamlbyte_producer_t self,
-      yamlbyte_inst_t *inst   /* to be filled in by the producer */
-    );
-
-/*
- *  Buffer based API
- */
-
-/* producer pushes a null terminated buffer filled with one or more
- * bytecode events to the consumer; if the consumer's result is not
- * YAMLBYTE_OK, then the producer should stop */
-typedef
-  yamlbyte_result_t
-   (*yamlbyte_pushbuff_t)(
-     yamlbyte_consumer_t self,
-     yamlbyte_buff_t  buff
-   );
-
-/* consumer pulls bytecode events from the producer; in this case
- * the buffer is owned by the producer, and will remain valid till
- * the pull function is called once again; if the buffer pointer
- * is set to NULL, then there are no more results; it is important
- * to call the pull function till it returns NULL so that the
- * producer can clean up its memory allocations */
-typedef
-   yamlbyte_result_t
-    (*yamlbyte_pullbuff_t)(
-      yamlbyte_producer_t self,
-      yamlbyte_buff_t *buff   /* to be filled in by the producer */
-    );
-
-/* convert a pull interface to a push interface; the reverse process
- * requires threads and thus is language dependent */
-#define YAMLBYTE_PULL2PUSH(pull,producer,push,consumer,result)       \
-    do {                                                         \
-        yamlbyte_pullbuff_t _pull = (pull);                              \
-        yamlbyte_pushbuff_t _push = (push);                              \
-        yamlbyte_result_t _result = YAMLBYTE_OK;                         \
-        yamlbyte_producer_t _producer = (producer);                  \
-        yamlbyte_consumer_t _consumer = (consumer);                  \
-        while(1) {                                               \
-            yamlbyte_buff_t buff = NULL;                           \
-            _result = _pull(_producer,&buff);                    \
-            if(YAMLBYTE_OK != result || NULL == buff)                \
-                break;                                           \
-            _result = _push(_consumer,buff);                     \
-            if(YAMLBYTE_OK != result)                                \
-                break;                                           \
-        }                                                        \
-        (result) = _result;                                      \
-    } while(0)
-
-#endif