tg_geometry 0.1.0 → 0.2.0

data/docs/ERROR_HANDLING.md

@@ -53,3 +53,33 @@ Rtree allocator callbacks return `NULL` on OOM. Rtree search callbacks record fa
 ## Writer safety
 
 Text writers allocate Ruby strings with one extra byte for the null terminator, call the TG writer with that capacity, then set the Ruby string length back to the required content length. WKB allocates exactly the required binary length and associates `ASCII-8BIT` encoding.
+
+## FeatureSource errors and reports
+
+FeatureSource validates the whole JSON source before traversal. Malformed JSON raises `TG::Geometry::ParseError` with the byte offset when available. A root that is not a GeoJSON `FeatureCollection`, or a `features` member that is missing/not an array, also raises `TG::Geometry::ParseError`.
+
+Invalid feature data is fail-fast by default:
+
+- missing/non-object geometry;
+- missing/non-string `geometry.type`;
+- unsupported geometry type;
+- invalid properties type for `read_features_*`;
+- TG parse error for a geometry range.
+
+`on_invalid: :skip` is accepted only with `report: true`. In report mode invalid features increment `:skipped`; filtered geometry types increment `:filtered`.
+
+Missing or `null` ids follow `on_missing_id:`:
+
+- `:raise` raises `TG::Geometry::ArgumentError`;
+- `:skip` requires `report: true` and records a skipped feature;
+- `:ordinal` creates an id like `"feature/12"`.
+
+Invalid id types raise `TG::Geometry::ArgumentError`: boolean, object, array, fractional number, and non-integer exponent number are not accepted FeatureSource ids.
+
+Stored report errors have this shape:
+
+```ruby
+{ feature_index: Integer, byte_offset: Integer, reason: String }
+```
+
+`max_errors:` caps the number of stored error Hashes, not the exact skipped count. TG geometry parse errors copy `tg_geom_error` before freeing the temporary TG error geometry and include feature index / byte offset in the raised message.

data/docs/FEATURE_SOURCE.md

@@ -0,0 +1,166 @@
+# FeatureSource
+
+`TG::Geometry::FeatureSource` reads GeoJSON `FeatureCollection` sources at C level and extracts geometry ranges without building a Ruby `Hash` / `Array` tree for the whole document.
+
+The feature exists for Rails-style import and geofencing workflows where the source data is a real GeoJSON file, not a hand-written polygon string.
+
+## Public API
+
+```ruby
+TG::Geometry::FeatureSource.read_entries_file(path, **opts)
+TG::Geometry::FeatureSource.read_entries_json(json_string, **opts)
+TG::Geometry::FeatureSource.read_entries_io(io, **opts)
+
+TG::Geometry::FeatureSource.read_features_file(path, **opts)
+TG::Geometry::FeatureSource.read_features_json(json_string, **opts)
+TG::Geometry::FeatureSource.read_features_io(io, **opts)
+
+TG::Geometry::FeatureSource.build_index_file(path, **opts)
+TG::Geometry::FeatureSource.build_index_json(json_string, **opts)
+TG::Geometry::FeatureSource.build_index_io(io, **opts)
+```
+
+Source methods are explicit:
+
+- `*_file(path)` treats the argument as a file path only.
+- `*_json(json_string)` treats the argument as raw GeoJSON content only.
+- `*_io(io)` calls `read` and treats the returned content as raw GeoJSON.
+
+There is no path/content auto-detection.
+
+## Return shapes
+
+`read_entries_*` returns pairs suitable for `TG::Geometry::Index.build(..., via: :geojson)`:
+
+```ruby
+[[id, geometry_json_string], ...]
+```
+
+`read_features_*` also returns raw `properties` JSON for database imports:
+
+```ruby
+[[id, geometry_json_string, properties_json_string], ...]
+```
+
+`properties_json_string` is a JSON string, not a Ruby `Hash`.
+
+## Report mode
+
+With `report: true`, read methods return a Hash:
+
+```ruby
+{
+  entries: [[id, geometry_json_string], ...],
+  skipped: 0,
+  filtered: 0,
+  errors: [
+    { feature_index: 1, byte_offset: 1234, reason: "missing geometry" }
+  ]
+}
+```
+
+For `read_features_*`, the collection key is `:features` instead of `:entries`.
+
+`max_errors:` caps stored error Hashes only. `skipped` remains exact.
+
+## Options
+
+```ruby
+id: ["properties", "@id"]
+only: [:polygon, :multipolygon]
+on_invalid: :raise
+on_missing_id: :raise
+report: false
+max_errors: 100
+geometry_index: :ystripes
+```
+
+`id:` may be an `Array<String>` or a simple dot-separated `String`. Array form is the primary API and is required for keys that cannot be represented safely with dot syntax.
+
+Accepted ids:
+
+- JSON string -> Ruby `String`
+- JSON integer number -> Ruby `Integer`
+
+Rejected ids:
+
+- fractional/exponent non-integer number
+- boolean
+- object
+- array
+
+Missing or `null` ids follow `on_missing_id:`:
+
+- `:raise` raises `TG::Geometry::ArgumentError`
+- `:skip` skips the feature and requires `report: true`
+- `:ordinal` creates a string id such as `"feature/12"`
+
+`only:` defaults to polygons and multipolygons. `only: nil` disables geometry type filtering. A mismatch with `only:` is counted as `filtered`, not invalid.
+
+`on_invalid: :skip` requires `report: true`. Silent loss of invalid features is not allowed.
+
+## Direct index build
+
+```ruby
+index = TG::Geometry::FeatureSource.build_index_file(
+  "zones.geojson",
+  id: ["properties", "@id"],
+  strategy: :rtree,
+  predicate: :covers,
+  geometry_index: :ystripes
+)
+```
+
+`build_index_*` returns a frozen `TG::Geometry::Index`.
+
+Unlike `read_entries_* + Index.build`, direct build does not allocate Ruby geometry JSON strings. It parses accepted geometry JSON ranges directly into owned native TG geometries and then builds the immutable index.
+
+`build_index_*` does not accept `report: true` in this release.
+
+## Validation
+
+FeatureSource validates in two layers:
+
+1. JSON syntax and FeatureCollection shape.
+2. Each returned or indexed geometry is parsed with TG via `tg_parse_geojsonn_ix`.
+
+A JSON object being syntactically valid is not enough for index correctness.
+
+## Execution and GVL model
+
+FeatureSource bulk work is split into two phases. The heavy phase runs without the Ruby GVL and uses only C-owned memory:
+
+- `_file` methods copy the Ruby path, then open/read the file in C outside the GVL;
+- JSON validation and `tidwall/json.c` traversal run outside the GVL;
+- geometry validation/parsing with `tg_parse_geojsonn_ix` runs outside the GVL;
+- no Ruby objects are created, no Ruby exceptions are raised, and no `rb_gc_adjust_memory_usage` calls happen in that phase.
+
+After the heavy phase finishes, the GVL is reacquired to materialize Ruby ids/strings/reports or to transfer owned native geometries into a frozen `TG::Geometry::Index`.
+
+FeatureSource deliberately does not implement a manual Fiber scheduler worker in C. The heavy C-only phase is executed with Ruby VM no-GVL APIs: `RB_NOGVL_OFFLOAD_SAFE` when available, otherwise `rb_thread_call_without_gvl`. This means FeatureSource releases the GVL for other Ruby threads. On Rubies without the offload-safe VM API, no explicit Fiber scheduler friendliness is claimed.
+
+## Memory model
+
+FeatureSource reads the whole source into one owned C buffer. `tidwall/json.c` traversal is backed by that buffer.
+
+For `read_entries_*` / `read_features_*`:
+
+- returned geometry strings are copied into Ruby strings;
+- returned properties strings are copied into Ruby strings or literal `"null"`;
+- no returned value points into the freed C buffer.
+
+For `build_index_*`:
+
+- the source buffer remains alive until all geometry ranges are parsed;
+- the final index owns native TG geometries and does not depend on the source buffer;
+- Ruby ids are stored in native entries and marked/compacted by the index wrapper.
+
+## Non-goals in this release
+
+- no streaming parser;
+- no NDGeoJSON / GeoJSONSeq;
+- no gzip decompression;
+- no FlatGeobuf;
+- no Ruby callback/yield API;
+- no property parsing into Ruby Hashes;
+- no general JSON query API.

data/docs/LIMITATIONS.md

@@ -1,61 +1,22 @@
 # Limitations
 
-`tg_geometry` is a small in-process geometry predicate and geofencing-oriented Index gem. It is not a full GIS system.
+`tg_geometry` is not a full GIS system.
 
-## Not included
-
-The first release does not provide:
+Not included:
 
 - geocoding;
 - routing;
 - projections;
-- geodesic distance or area;
-- buffer / union / difference / intersection-result geometry operations;
-- nearest POI index;
-- full PostGIS / GEOS replacement behavior;
-- parsed GeoJSON Feature properties as Ruby Hashes;
-- Line/Ring/Polygon public constructors from Ruby coordinate arrays;
-- user callback APIs;
+- geodesic distance/area;
+- buffer / union / difference / overlay result geometry operations;
+- nearest POI indexing;
+- public callback/search APIs;
 - Ractor support claim;
-- no-allocation point query shortcuts;
-- geodesic/projection helpers;
-- mutable coordinate APIs;
-- global TG environment configuration or allocator override APIs.
-
-## Planar XY only
-
-TG works in planar XY coordinates. If users pass lon/lat, area, length, and perimeter concepts are in input coordinate units, not meters. Real-world distance and area need explicit projection/geodesic tooling outside this first-release core.
-
-## Boundary semantics
-
-Geofencing defaults to `predicate: :covers` because boundary points should count as inside. `predicate: :contains` is stricter and may exclude boundary points.
-
-## Point query performance
-
-The current implementation allocates a temporary TG point geometry per point query. This is intentional for exact `covers` and `contains` semantics. A no-allocation point path can be added later only after tests prove equivalent boundary behavior and benchmarks prove value.
-
-## Build peak memory
-
-`TG::Geometry::Index.build` is atomic and immutable. During reload, memory peak can include both the old Index and the new Index, plus original Ruby entry arrays and temporary build state. This is deliberate: exception safety and read-only concurrency are more important than streaming mutation in the first release.
-
-## IDs are returned by reference
-
-Index query methods return the same Ruby id objects stored in entries. They are not duplicated, frozen, stringified, or copied. If an id object is mutable, user code owns that mutability risk.
-
-## Windows
-
-Windows is not supported in the first release. The intended first-release platforms are Linux and macOS on x86_64 and arm64.
-
-## Expansion limitations
-
-No automatic strategy resolver is enabled in the first public release. For unusual datasets, especially heavily overlapping zones or workloads dominated by first-entry hits, choose `:flat` or `:rtree` explicitly after benchmarking.
-
-`TG::Geometry::Registry` is application sugar, not a distributed registry. It has no Redis dependency, no background reload thread, and no hidden global singleton.
-
-`TG::Geometry::ActiveRecordSource` is optional Ruby helper code. It does not install Rails reload hooks, generators, or background jobs.
+- no-GVL execution claim;
+- automatic strategy selection.
 
-`TG::Geometry::Geom#extra_json` returns raw copied JSON text. It does not parse properties and does not expose Feature child objects. Z/M metadata is readable and point Z/M constructors exist, but broad Line/Ring/Polygon/Multi* construction remains out of scope until a separate ownership model is specified.
+TG works in planar XY coordinates. If lon/lat coordinates are passed in, length, area, and perimeter-style values are in input coordinate units, not meters.
 
-## Expansion Blocks F-H
+FeatureSource reads the full source into memory. It avoids a Ruby `JSON.parse` object tree, but it is not a streaming backend. There is no gzip, NDGeoJSON, GeoJSONSeq, or FlatGeobuf support.
 
-Callback/search APIs, no-allocation point query optimization, and geodesic/projection helpers remain OPEN QUESTION scope. See `docs/EXPANSION_E_TO_H_STATUS.md`.
+FeatureSource returns raw properties JSON strings. It does not parse properties into Ruby Hash objects.

data/docs/MEMORY_OWNERSHIP.md

@@ -85,10 +85,28 @@ For `via: :geojson` and `via: :wkb`, the Index owns each parsed TG geometry. The
 
 ## Low-level child wrappers
 
-Expansion Block E exposes read-only borrowed wrappers for selected TG child types. `TG::Geometry::Line`, `TG::Geometry::Ring`, and `TG::Geometry::Polygon` keep the original parent `TG::Geometry::Geom` Ruby object in `geom_owner`. Their GC callbacks use `rb_gc_mark_movable` and `rb_gc_location`, matching the Index borrowed-geometry model.
+Planned API areas exposes read-only borrowed wrappers for selected TG child types. `TG::Geometry::Line`, `TG::Geometry::Ring`, and `TG::Geometry::Polygon` keep the original parent `TG::Geometry::Geom` Ruby object in `geom_owner`. Their GC callbacks use `rb_gc_mark_movable` and `rb_gc_location`, matching the Index borrowed-geometry model.
 
-Expansion Block J extends this model to borrowed `TG::Geometry::Geom` wrappers returned from GeometryCollection accessors. A borrowed `Geom` stores `owned = false`, `geom_bytes = 0`, and a `geom_owner` reference to the parent wrapper. Its free path only releases the Ruby wrapper struct; it never calls `tg_geom_free` on the borrowed child pointer. This keeps `ObjectSpace.memsize_of` from double-counting parent-owned native geometry.
+Planned API areas extends this model to borrowed `TG::Geometry::Geom` wrappers returned from GeometryCollection accessors. A borrowed `Geom` stores `owned = false`, `geom_bytes = 0`, and a `geom_owner` reference to the parent wrapper. Its free path only releases the Ruby wrapper struct; it never calls `tg_geom_free` on the borrowed child pointer. This keeps `ObjectSpace.memsize_of` from double-counting parent-owned native geometry.
 
 `TG::Geometry::Segment` is different: it stores a `struct tg_segment` by value, not a borrowed pointer. It has no parent owner to mark and no TG deallocator to call.
 
 Line/Ring/Polygon wrappers own only their small Ruby-allocated wrapper structs. They do not own the underlying `const struct tg_line *`, `const struct tg_ring *`, or `const struct tg_poly *`. Cleanup therefore only calls `ruby_xfree` for the wrapper; the parent `TG::Geometry::Geom` remains responsible for the single `tg_geom_free`.
+
+## FeatureSource ownership
+
+FeatureSource adds only method-scope JSON resources. It does not introduce a persistent FeatureSource object type.
+
+Resource | Allocator | Deallocator | Owner | Notes
+--- | --- | --- | --- | ---
+FeatureSource source buffer | C heap (`malloc`/`realloc`) | `free` | FeatureSource method scope | full file/json/io content copy; freed through ensure-style cleanup
+FeatureSource geometry JSON output | Ruby `String` | Ruby GC | returned Array | copied from source buffer before buffer free; UTF-8 encoded
+FeatureSource properties JSON output | Ruby `String` | Ruby GC | returned Array | copied from source buffer or literal `"null"`; UTF-8 encoded
+`tidwall/json.c` `struct json` / raw range | backed by source buffer | none | local traversal scope | invalid after source buffer free; never stored in Ruby object/native index
+FeatureSource validation geometry | `tg_parse_geojsonn_ix` | `tg_geom_free` | local validation scope | used by `read_entries_*` / `read_features_*`; freed immediately
+FeatureSource build index geometry | `tg_parse_geojsonn_ix` | `tg_geom_free` via `index_dispose` | `TG::Geometry::Index` | direct parse from JSON raw range; no Ruby geometry string allocation
+FeatureSource build id | Ruby VM | Ruby GC | `TG::Geometry::Index` entry | wrapper is created before entries are filled; `initialized` controls mark/compact boundary
+
+The source buffer must outlive every JSON range and every `tg_parse_geojsonn_ix` call that receives a pointer into it. After `build_index_*` succeeds, the returned Index owns native TG geometries and no longer depends on the source buffer.
+
+`read_entries_*` and `read_features_*` copy output strings before freeing the buffer. Returned arrays never contain pointers into the native source buffer.

data/ext/tg_geometry/extconf.rb

@@ -12,6 +12,7 @@ end
 VENDOR_DIR = File.expand_path("vendor", __dir__)
 VENDORED_TG_DIR = File.join(VENDOR_DIR, "tg")
 VENDORED_RTREE_DIR = File.join(VENDOR_DIR, "rtree")
+VENDORED_JSON_DIR = File.join(VENDOR_DIR, "json")
 
 required_vendor_files = [
   File.join(VENDORED_TG_DIR, "tg.c"),
@@ -19,7 +20,11 @@ required_vendor_files = [
   File.join(VENDORED_TG_DIR, "VERSION"),
   File.join(VENDORED_RTREE_DIR, "rtree.c"),
   File.join(VENDORED_RTREE_DIR, "rtree.h"),
-  File.join(VENDORED_RTREE_DIR, "VERSION")
+  File.join(VENDORED_RTREE_DIR, "VERSION"),
+  File.join(VENDORED_JSON_DIR, "json.c"),
+  File.join(VENDORED_JSON_DIR, "json.h"),
+  File.join(VENDORED_JSON_DIR, "LICENSE"),
+  File.join(VENDORED_JSON_DIR, "VERSION")
 ]
 
 missing_vendor_files = required_vendor_files.reject { |path| File.file?(path) }
@@ -27,10 +32,44 @@ unless missing_vendor_files.empty?
   abort "tg_geometry requires vendored tidwall/tg and rtree.c sources. Missing: #{missing_vendor_files.join(", ")}. Run `ruby script/vendor_libs.rb --sync`."
 end
 
+def tg_geometry_clang_compiler?
+  cc = RbConfig::CONFIG["CC"].to_s
+  return true if cc.include?("clang")
+
+  # Some build environments use cc as an alias. Detect __clang__ by compiling
+  # a tiny program rather than trusting the command name.
+  try_compile(<<~C)
+    #ifndef __clang__
+    #error not clang
+    #endif
+    int main(void) { return 0; }
+  C
+end
+
+def tg_geometry_sanitize_warnflags!
+  return if tg_geometry_clang_compiler?
+
+  # Ruby builds produced with clang may leak clang-only warning flags into
+  # RbConfig::CONFIG["warnflags"]. GCC prints noisy "unrecognized command-line
+  # option" notes once any real diagnostic is emitted. Keep our Linux CI logs
+  # focused on tg_geometry diagnostics.
+  clang_only_warning_flags = %w[
+    -Wno-constant-logical-operand
+    -Wno-parentheses-equality
+    -Wno-self-assign
+  ]
+
+  $warnflags = $warnflags.to_s.split.reject do |flag|
+    clang_only_warning_flags.include?(flag)
+  end.join(" ")
+end
+
+tg_geometry_sanitize_warnflags!
+
 forbidden_defines = %w[TG_NOATOMICS RTREE_NOATOMICS]
 forbidden_defines.each do |macro|
   if ($CFLAGS.to_s.split + $CPPFLAGS.to_s.split + $defs).any? { |flag| flag.include?(macro) }
-    abort "tg_geometry must not define #{macro}; atomics are required by the release-core contract"
+    abort "tg_geometry must not define #{macro}; atomics are required by the build requirements"
   end
 end
 
@@ -52,7 +91,8 @@ $INCFLAGS = [
   "-isystem $(hdrdir)/ruby/backward",
   "-isystem $(hdrdir)",
   "-I$(srcdir)/vendor/tg",
-  "-I$(srcdir)/vendor/rtree"
+  "-I$(srcdir)/vendor/rtree",
+  "-I$(srcdir)/vendor/json"
 ].join(" ")
 
 unless try_compile(<<~C)
@@ -78,6 +118,7 @@ if have_macro("RB_NOGVL_OFFLOAD_SAFE", "ruby/thread.h")
   $defs << "-DHAVE_RB_NOGVL_OFFLOAD_SAFE"
 end
 
+
 if ENV["TG_DEBUG_TEST"] == "1"
   $defs << "-DTG_DEBUG_TEST"
 end
@@ -85,7 +126,8 @@ end
 $srcs = [
   "tg_geometry_ext.c",
   "tg_geometry_vendor_tg.c",
-  "tg_geometry_vendor_rtree.c"
+  "tg_geometry_vendor_rtree.c",
+  "tg_geometry_vendor_json.c"
 ]
 
 create_makefile("tg_geometry_ext_geometry_ext")