commenter 0.6.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e569c1e1d181c27078d97e4bcaa8a63c54b055aa7aff5dff88646be1237c6a65
-  data.tar.gz: 5d156b1e49c82c9b8002e47f7ab870d98e133622b4830768e02b9bf32c2db993
+  metadata.gz: 116bd5694762b20a730aa2d67d9950e3acedecf5e0e5d2f95a6d183d376c0e97
+  data.tar.gz: d1c34482394b44f5195a91268eb3b46f4db58c986afc214a841d74b908f98edb
 SHA512:
-  metadata.gz: 66daebbb66df3c881828bccd0a36c8cb0e1c524d74c19efef579770f707a366c2f48a51b2871041dd3519db0b26ad4431f0065395b02d85685f08eed9862e39e
-  data.tar.gz: 6ea9b41315c94d87c8356350be9b55235d1dfd07e341bbbc133a6feea27b807f36ac9df05bc32430ed314c73c5b5c8a25d9e1ff5066beb0f36f90032c3b9330a
+  metadata.gz: 1d755c6bfefe119528e4ddb9e65033263f5c03550d972ae29bd62aaaea8df3fa571439305f80429eefb017856d387e5dde23c18a744ecdfb03d2f77d6a6e486e
+  data.tar.gz: 6d297683a681dc1918be0db3a5dd63aded21fca0fce0a63799b36e1342d60f927b9690ca9757c8935c74c530edc3205acb7b510e4930360a7f4a99d595a865fc

data/.rubocop_todo.yml

@@ -30,7 +30,7 @@ Metrics/BlockLength:
 # Offense count: 5
 # Configuration parameters: CountComments, CountAsOne.
 Metrics/ClassLength:
-  Max: 231
+  Max: 270
 
 # Offense count: 8
 # Configuration parameters: AllowedMethods, AllowedPatterns.

data/CLAUDE.md

@@ -49,8 +49,8 @@ Core flow: `Parser` → `CommentSheet` (metadata + `Comment` objects) → YAML /
 - `Parser::OsdXlsxParser` — auto-detects two OSD export variants by header row ("resolved" 17-col starting `Comment ID`; "unresolved" 15-col starting `User name`), extracts metadata (date/reference/stage/titles) from header rows 1–2, maps columns by header name into `Comment` attributes, and synthesizes `observations` from `resolution_status` + `motivation`.
 - `Parser::TrackChangeDocxParser` — redline DOCX import: streams `word/document.xml` with Nokogiri XML::Reader (redlines can exceed 100 MB), captures `w:ins`/`w:del`/`w:moveFrom`/`w:moveTo` as comment entries (`proposed_change` renders the change itself, e.g. `Insert: "text"`; clause resolved from the nearest preceding heading including sub-clauses; element resolved from Table/Figure/Formula/NOTE references in the same clause; self-closing paragraph-mark markers skipped), plus reviewer `w:comment` threads as `-CNNN` remark entries (remark verbatim in `comments`, instruction reworded in `proposed_change`, empty `observations` for the owner). Stamps `observations` from the `observations`/`accept_all` options.
 - `Commenter::Filler` — writes comments into the DOCX template table; maps observation status text (`accept(ed)?`, `noted`, `reject(ed)?`, …) to shading colors.
-- GitHub integration (`lib/commenter/github_integration.rb`) — two classes with parallel structure: `GitHubIssueCreator` (`github-create`) and `GitHubIssueRetriever` (`github-retrieve`). Each loads its own config YAML and Octokit client, and rewrites the comments YAML in place after the run. Duplicate detection searches issue titles for a unique ID rendered from a configurable Liquid `unique_id` template (stage-aware by default: `[DIS] GB-001`), so the same comment ID at different ballot stages creates separate issues.
-- `Commenter::Cli` (`lib/commenter/cli.rb`) — Thor CLI with subcommands `import`, `fill`, `github-create`, `github-retrieve`.
+- GitHub integration — `GitHubIssueCreator` (`github-create`, `lib/commenter/github_integration.rb`), `GitHubIssueRetriever` (`github-retrieve`, `lib/commenter/github_issue_retriever.rb`), and `GitHubSync` (`github-sync`, subclass of the creator). Each loads its own config YAML and Octokit client, and rewrites the comments YAML in place after the run. Duplicate detection verifies issue numbers already recorded in the YAML, then searches issue bodies for a deterministic `urn:commenter:` marker appended to every created issue, then falls back to a title search over a configurable Liquid `unique_id` template (stage-aware by default: `[DIS] GB-001`). The same comment ID at different ballot stages creates separate issues. github-retrieve fills observations from open and closed issues alike.
+- `Commenter::Cli` (`lib/commenter/cli.rb`) — Thor CLI with subcommands `import`, `merge`, `stats`, `diff`, `fill`, `export`, `github-create`, `github-sync`, `github-retrieve`.
 
 ### Schemas
 

data/README.adoc

@@ -294,7 +294,7 @@ Options:
 The commenter gem provides comprehensive GitHub integration for collaborative comment review and tracking. The workflow consists of two main commands:
 
 * `github-create` - Creates GitHub issues from comments and tracks them in YAML
-* `github-retrieve` - Retrieves final observations from closed GitHub issues
+* `github-retrieve` - Retrieves observations recorded in GitHub issues
 
 ==== Creating GitHub issues from comments
 
@@ -431,6 +431,18 @@ detection uses a **configurable unique identifier** that is:
 This ensures that the search pattern always matches what appears in the title,
 preventing synchronization issues.
 
+Every created issue body also carries a deterministic marker line, e.g.
++
+[source]
+----
+`urn:commenter:owner/repo:dis:GB-001`
+----
++
+Duplicate detection searches the issue bodies for this marker first — exact
+and template-independent — and falls back to the title search for issues
+created before the marker existed. Comment numbers already recorded in the
+comments YAML are verified with a direct read before any search runs.
+
 ===== Unique ID template
 
 The unique identifier is defined by the `unique_id` template in your configuration:
@@ -534,7 +546,7 @@ $ commenter github-retrieve --config github_config.yaml comments.yaml
 This command:
 
 * Reads GitHub issue information from the YAML file (no searching required)
-* Fetches observations from closed GitHub issues only
+* Fills in an observation whenever the issue carries one, open or closed
 * Updates the same YAML file with extracted observations
 * Preserves all existing comment data
 
@@ -561,7 +573,7 @@ Alternative shorter syntax:
 
 * `-c, --config FILE` - GitHub configuration YAML file (required)
 * `-o, --output FILE` - Output YAML file (default: update original)
-* `--include-open` - Include observations from open issues (not recommended)
+* `--include-open` - (Deprecated) observations are now filled from open issues too
 * `--dry-run` - Preview observations without updating YAML
 
 ===== Examples

data/lib/commenter/cli.rb

@@ -157,46 +157,8 @@ module Commenter
 
       results = creator.create_issues_from_yaml(input_yaml, github_options)
 
-      if options[:dry_run]
-        puts "DRY RUN - Preview of issues to be created:"
-        puts "=" * 50
-        results.each do |result|
-          puts "\nComment ID: #{result[:comment_id]}"
-          puts "Title: #{result[:title]}"
-          puts "Labels: #{result[:labels].join(", ")}" if result[:labels]&.any?
-          puts "Assignees: #{result[:assignees].join(", ")}" if result[:assignees]&.any?
-          puts "Milestone: #{result[:milestone]}" if result[:milestone]
-          puts "\nBody preview (first 200 chars):"
-          puts result[:body][0...200] + (result[:body].length > 200 ? "..." : "")
-          puts "-" * 30
-        end
-      else
-        puts "GitHub issue creation results:"
-        puts "=" * 40
-
-        created_count = 0
-        skipped_count = 0
-        error_count = 0
-
-        results.each do |result|
-          case result[:status]
-          when :created
-            created_count += 1
-            puts "✓ #{result[:comment_id]}: Created issue ##{result[:issue_number]}"
-            puts "  URL: #{result[:issue_url]}"
-          when :skipped
-            skipped_count += 1
-            puts "- #{result[:comment_id]}: Skipped (#{result[:message]})"
-            puts "  URL: #{result[:issue_url]}" if result[:issue_url]
-          when :error
-            error_count += 1
-            puts "✗ #{result[:comment_id]}: Error - #{result[:message]}"
-          end
-        end
-
-        puts "\nSummary:"
-        puts "Created: #{created_count}, Skipped: #{skipped_count}, Errors: #{error_count}"
-      end
+      error_count = report_issue_results(results, options[:dry_run])
+      exit 1 if error_count.to_i.positive?
     rescue StandardError => e
       puts "Error: #{e.message}"
       exit 1
@@ -341,5 +303,47 @@ module Commenter
       FileUtils.cp(schema_source, schema_target) unless File.expand_path(schema_source) == File.expand_path(schema_target)
       schema_target
     end
+
+    def report_issue_results(results, dry_run)
+      if dry_run
+        puts "DRY RUN - Preview of issues to be created:"
+        puts "=" * 50
+        results.each do |result|
+          puts "\nComment ID: #{result[:comment_id]}"
+          puts "Title: #{result[:title]}"
+          puts "Labels: #{result[:labels].join(", ")}" if result[:labels]&.any?
+          puts "Assignees: #{result[:assignees].join(", ")}" if result[:assignees]&.any?
+          puts "Milestone: #{result[:milestone]}" if result[:milestone]
+          puts "\nBody preview (first 200 chars):"
+          puts result[:body][0...200] + (result[:body].length > 200 ? "..." : "")
+          puts "-" * 30
+        end
+        return nil
+      end
+
+      puts "GitHub issue creation results:"
+      puts "=" * 40
+
+      counts = Hash.new(0)
+      results.each do |result|
+        case result[:status]
+        when :created
+          counts[:created] += 1
+          puts "✓ #{result[:comment_id]}: Created issue ##{result[:issue_number]}"
+          puts "  URL: #{result[:issue_url]}"
+        when :skipped
+          counts[:skipped] += 1
+          puts "- #{result[:comment_id]}: Skipped (#{result[:message]})"
+          puts "  URL: #{result[:issue_url]}" if result[:issue_url]
+        when :error
+          counts[:error] += 1
+          puts "✗ #{result[:comment_id]}: Error - #{result[:message]}"
+        end
+      end
+
+      puts "\nSummary:"
+      puts "Created: #{counts[:created]}, Skipped: #{counts[:skipped]}, Errors: #{counts[:error]}"
+      counts[:error]
+    end
   end
 end

data/lib/commenter/comment_type.rb

@@ -35,5 +35,12 @@ module Commenter
     def display_name(type)
       DISPLAY_NAMES.fetch(type.to_s.strip.downcase) { type || "Unknown" }
     end
+
+    # True when the value is exactly one recognized comment type, in code or
+    # full-name spelling. Combined or free-form values ("ge/te") are not.
+    def known?(type)
+      downcased = type.to_s.strip.downcase
+      FULL_NAMES.key?(downcased) || FULL_NAMES.value?(downcased)
+    end
   end
 end

data/lib/commenter/github_integration.rb

@@ -7,10 +7,15 @@ require "dotenv/load"
 
 module Commenter
   class GitHubIssueCreator
-    def initialize(config_path, title_template_path = nil, body_template_path = nil)
+    THROTTLE_SECONDS = 1
+    RATE_LIMIT_RETRY_DELAY = 60
+    RATE_LIMIT_MAX_RETRIES = 3
+    MARKER_PREFIX = "urn:commenter"
+
+    def initialize(config_path, title_template_path = nil, body_template_path = nil, client: nil)
       @session = GitHubSession.new(config_path)
       @config = @session.config
-      @github_client = @session.client
+      @github_client = client || @session.client
       @repo = @session.repo
 
       @title_template = load_liquid_template(title_template_path || default_title_template_path)
@@ -129,30 +134,36 @@ module Commenter
 
     def create_issue(comment, comment_sheet, options = {})
       puts "[GitHubIssueCreator] Creating issue for comment ID: #{comment.id}"
-      # Check if issue already exists (stage-aware)
-      existing_issue = find_existing_issue(comment, comment_sheet)
-      if existing_issue
-        puts "[GitHubIssueCreator] Issue already exists for comment ID: #{comment.id} at stage #{comment_sheet.stage}, skipping creation."
-        return {
-          comment_id: comment.id,
-          status: :skipped,
-          message: "Issue already exists",
-          issue_url: existing_issue.html_url
-        }
-      end
+      begin
+        # The recorded issue number is verified with a consistent GET; the
+        # title search alone cannot see issues GitHub has not indexed yet.
+        existing_issue = recorded_issue(comment, comment_sheet) ||
+                         find_existing_issue(comment, comment_sheet)
+        if existing_issue
+          puts "[GitHubIssueCreator] Issue already exists for comment ID: #{comment.id} " \
+               "at stage #{comment_sheet.stage}, skipping creation."
+          return {
+            comment_id: comment.id,
+            status: :skipped,
+            message: "Issue already exists",
+            issue_number: existing_issue.number,
+            issue_url: existing_issue.html_url,
+            issue_status: existing_issue.state,
+            issue_created_at: existing_issue.created_at
+          }
+        end
 
-      title = @title_template.render(template_variables(comment, comment_sheet))
-      body = @body_template.render(template_variables(comment, comment_sheet))
+        title = @title_template.render(template_variables(comment, comment_sheet))
+        body = body_with_marker(comment, comment_sheet)
 
-      issue_options = {
-        labels: determine_labels(comment, comment_sheet),
-        assignees: determine_assignees(comment, comment_sheet, options),
-        milestone: determine_milestone(comment, comment_sheet, options)
-      }.compact
+        issue_options = {
+          labels: determine_labels(comment, comment_sheet),
+          assignees: determine_assignees(comment, comment_sheet, options),
+          milestone: determine_milestone(comment, comment_sheet, options)
+        }.compact
 
-      puts "[GitHubIssueCreator] Creating issue with title: #{title}"
-      begin
-        issue = @github_client.create_issue(@repo, title, body, issue_options)
+        puts "[GitHubIssueCreator] Creating issue with title: #{title}"
+        issue = create_issue_with_retry(title, body, issue_options)
         puts "[GitHubIssueCreator] Issue created successfully: #{issue.html_url}"
         {
           comment_id: comment.id,
@@ -170,6 +181,66 @@ module Commenter
       end
     end
 
+    # Deterministic marker embedded in every created issue body, independent
+    # of the user's templates, so duplicate detection never has to rely on
+    # title matching alone (#1).
+    def issue_marker(comment, comment_sheet)
+      "#{MARKER_PREFIX}:#{@repo.downcase}:#{comment_sheet.stage.to_s.downcase}:#{comment.id}"
+    end
+
+    def body_with_marker(comment, comment_sheet)
+      rendered = @body_template.render(template_variables(comment, comment_sheet))
+      "#{rendered}\n\n---\n`#{issue_marker(comment, comment_sheet)}`"
+    end
+
+    def recorded_issue(comment, comment_sheet)
+      return nil unless comment.has_github_issue?
+
+      number = comment.github_issue_number
+      puts "[GitHubIssueCreator] Verifying recorded issue ##{number} for comment ID: #{comment.id}"
+      issue = @github_client.issue(@repo, number)
+      if issue.title.include?(render_unique_id(comment, comment_sheet)) ||
+         issue.body.to_s.include?(issue_marker(comment, comment_sheet))
+        issue
+      else
+        puts "[GitHubIssueCreator] Recorded issue ##{number} does not match, falling back to title search."
+        nil
+      end
+    rescue Octokit::NotFound
+      puts "[GitHubIssueCreator] Recorded issue ##{number} no longer exists, falling back to title search."
+      nil
+    end
+
+    def create_issue_with_retry(title, body, issue_options)
+      attempts = 0
+      begin
+        attempts += 1
+        issue = @github_client.create_issue(@repo, title, body, issue_options)
+        sleep(THROTTLE_SECONDS) if THROTTLE_SECONDS.positive?
+        issue
+      rescue Octokit::TooManyRequests, Octokit::Forbidden => e
+        delay = rate_limit_retry_delay(e)
+        if delay && attempts < RATE_LIMIT_MAX_RETRIES
+          puts "[GitHubIssueCreator] Rate limited (attempt #{attempts}/#{RATE_LIMIT_MAX_RETRIES}), retrying in #{delay}s: #{e.message}"
+          sleep(delay)
+          retry
+        end
+        raise
+      end
+    end
+
+    def rate_limit_retry_delay(error)
+      headers = error.response_headers
+      retry_after = headers && (headers[:retry_after] || headers["retry-after"])
+      return retry_after.to_i if retry_after.to_i.positive?
+
+      return RATE_LIMIT_RETRY_DELAY if error.is_a?(Octokit::TooManyRequests)
+
+      # Secondary ("abuse") rate limits are a 403 whose body names them; a
+      # permissions 403 must not be retried.
+      RATE_LIMIT_RETRY_DELAY if error.message.to_s.include?("secondary rate limit")
+    end
+
     def preview_issue(comment, comment_sheet)
       puts "[GitHubIssueCreator] Previewing issue for comment ID: #{comment.id}"
 
@@ -190,13 +261,15 @@ module Commenter
       unique_id = render_unique_id(comment, comment_sheet)
       puts "[GitHubIssueCreator] Searching for existing issue with unique_id: #{unique_id}"
 
-      # Search for existing issues with the unique_id in the title
-      # The unique_id is configurable and defaults to "[STAGE] COMMENT_ID"
+      # The marker search is exact and template-independent (#1); the title
+      # search remains as a fallback for issues created before the marker.
+      marker = issue_marker(comment, comment_sheet)
+      marker_results = @github_client.search_issues("repo:#{@repo} in:body \"#{marker}\"")
+      return marker_results.items.first if marker_results.items.any?
+
       query = "repo:#{@repo} in:title \"#{unique_id}\""
       results = @github_client.search_issues(query)
       results.items.first
-    rescue Octokit::Error
-      nil
     end
 
     def determine_labels(comment, comment_sheet)
@@ -211,8 +284,9 @@ module Commenter
         labels.concat(stage_labels) if stage_labels
       end
 
-      # Add comment type label
-      labels << comment.type if comment.type
+      # Comment type label: only for a recognized single type — combined or
+      # free-form values ("ge/te") must not mint labels (#3).
+      labels << comment.type if CommentType.known?(comment.type)
 
       labels.uniq
     end
@@ -285,9 +359,12 @@ module Commenter
         comment = comment_sheet.comments.find { |c| c.id == result[:comment_id] }
         next unless comment
 
+        created_at = comment.github_created_at || result[:issue_created_at] || Time.now.utc.iso8601
+        created_at = created_at.iso8601 if created_at.is_a?(Time)
+
         comment.record_github_issue(issue_number: result[:issue_number], issue_url: result[:issue_url],
-                                    status: result[:status] == :created ? "open" : comment.github_status,
-                                    created_at: comment.github_created_at || Time.now.utc.iso8601)
+                                    status: result[:issue_status] || (result[:status] == :created ? "open" : comment.github_status),
+                                    created_at: created_at)
       end
 
       # Write updated YAML
@@ -295,155 +372,4 @@ module Commenter
       File.write(output_file, comment_sheet.to_yaml_document)
     end
   end
-
-  class GitHubIssueRetriever
-    def initialize(config_path)
-      @session = GitHubSession.new(config_path)
-      @config = @session.config
-      @github_client = @session.client
-      @repo = @session.repo
-    end
-
-    def retrieve_observations_from_yaml(yaml_file, options = {})
-      comment_sheet = CommentSheet.from_yaml(File.read(yaml_file))
-
-      results = []
-      comment_sheet.comments.each do |comment|
-        next unless comment.has_github_issue?
-
-        result = if options[:dry_run]
-                   preview_observation_retrieval(comment, options)
-                 else
-                   retrieve_observation(comment, options)
-                 end
-        results << result
-      end
-
-      # Update YAML with observations (unless dry run)
-      update_yaml_with_observations(yaml_file, comment_sheet, options) unless options[:dry_run]
-
-      results
-    end
-
-    private
-
-    def retrieve_observation(comment, options)
-      issue_number = comment.github_issue_number
-
-      begin
-        issue = @github_client.issue(@repo, issue_number)
-
-        # Skip open issues unless explicitly included
-        if issue.state == "open" && !options[:include_open]
-          return {
-            comment_id: comment.id,
-            issue_number: issue_number,
-            status: :skipped,
-            message: "Issue is still open"
-          }
-        end
-
-        # Extract observation from issue comments
-        observation = extract_observation_from_issue(issue_number)
-
-        if observation
-          # Update comment with observation and current status
-          comment.observations = observation
-          comment.github.status = issue.state
-          comment.github.updated_at = Time.now.utc.iso8601
-
-          {
-            comment_id: comment.id,
-            issue_number: issue_number,
-            status: :retrieved,
-            observation: observation
-          }
-        else
-          {
-            comment_id: comment.id,
-            issue_number: issue_number,
-            status: :skipped,
-            message: "No observation found in issue"
-          }
-        end
-      rescue Octokit::Error => e
-        {
-          comment_id: comment.id,
-          issue_number: issue_number,
-          status: :error,
-          message: e.message
-        }
-      end
-    end
-
-    def preview_observation_retrieval(comment, _options)
-      issue_number = comment.github_issue_number
-
-      begin
-        issue = @github_client.issue(@repo, issue_number)
-        observation = extract_observation_from_issue(issue_number)
-
-        {
-          comment_id: comment.id,
-          issue_number: issue_number,
-          status: issue.state,
-          observation: observation
-        }
-      rescue Octokit::Error => e
-        {
-          comment_id: comment.id,
-          issue_number: issue_number,
-          status: :error,
-          message: e.message
-        }
-      end
-    end
-
-    def extract_observation_from_issue(issue_number)
-      comments = @github_client.issue_comments(@repo, issue_number)
-
-      # Look for magic comments with observation markers
-      observation_markers = @config.dig("github", "retrieval", "observation_markers") ||
-                            ["**OBSERVATION:**", "**COMMENTER OBSERVATION:**"]
-
-      # Search comments in reverse order (newest first)
-      comments.reverse_each do |comment|
-        observation = parse_observation_from_comment(comment.body, observation_markers)
-        return observation if observation
-      end
-
-      # Fallback to last comment if configured and no magic comment found
-      return comments.last.body.strip if @config.dig("github", "retrieval",
-                                                     "fallback_to_last_comment") && !comments.empty?
-
-      nil
-    rescue Octokit::Error
-      nil
-    end
-
-    def parse_observation_from_comment(comment_body, markers)
-      markers.each do |marker|
-        # Look for markdown blockquote with the marker
-        pattern = /^>\s*#{Regexp.escape(marker)}\s*\n((?:^>.*\n?)*)/m
-        match = comment_body.match(pattern)
-
-        next unless match
-
-        # Extract the blockquote content and clean it up
-        observation = match[1]
-                      .split("\n")
-                      .map { |line| line.sub(/^>\s?/, "") }
-                      .join("\n")
-                      .strip
-        return observation unless observation.empty?
-      end
-
-      nil
-    end
-
-    def update_yaml_with_observations(yaml_file, comment_sheet, options)
-      output_file = options[:output] || yaml_file
-      File.write(output_file, comment_sheet.to_yaml_document)
-    end
-  end
 end

data/lib/commenter/github_issue_retriever.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+require "octokit"
+require "yaml"
+require "dotenv/load"
+
+module Commenter
+  # Pulls official observations (OBSERVATION: blockquotes) out of the GitHub
+  # issues recorded in the comments YAML and writes them back into the file.
+  class GitHubIssueRetriever
+    def initialize(config_path, client: nil)
+      @session = GitHubSession.new(config_path)
+      @config = @session.config
+      @github_client = client || @session.client
+      @repo = @session.repo
+    end
+
+    def retrieve_observations_from_yaml(yaml_file, options = {})
+      comment_sheet = CommentSheet.from_yaml(File.read(yaml_file))
+
+      results = []
+      comment_sheet.comments.each do |comment|
+        next unless comment.has_github_issue?
+
+        result = if options[:dry_run]
+                   preview_observation_retrieval(comment, options)
+                 else
+                   retrieve_observation(comment, options)
+                 end
+        results << result
+      end
+
+      # Update YAML with observations (unless dry run)
+      update_yaml_with_observations(yaml_file, comment_sheet, options) unless options[:dry_run]
+
+      results
+    end
+
+    private
+
+    def retrieve_observation(comment, _options)
+      issue_number = comment.github_issue_number
+
+      begin
+        issue = @github_client.issue(@repo, issue_number)
+
+        # An observation is filled in whenever it is present, open issue or
+        # not (#4); only the "still waiting" case is skipped.
+        observation = extract_observation_from_issue(issue_number)
+
+        if observation
+          comment.observations = observation
+          comment.github.status = issue.state
+          comment.github.updated_at = Time.now.utc.iso8601
+
+          {
+            comment_id: comment.id,
+            issue_number: issue_number,
+            status: :retrieved,
+            observation: observation
+          }
+        else
+          {
+            comment_id: comment.id,
+            issue_number: issue_number,
+            status: :skipped,
+            message: issue.state == "open" ? "Issue is still open (no observation yet)" : "No observation found in issue"
+          }
+        end
+      rescue Octokit::Error => e
+        {
+          comment_id: comment.id,
+          issue_number: issue_number,
+          status: :error,
+          message: e.message
+        }
+      end
+    end
+
+    def preview_observation_retrieval(comment, _options)
+      issue_number = comment.github_issue_number
+
+      begin
+        issue = @github_client.issue(@repo, issue_number)
+        observation = extract_observation_from_issue(issue_number)
+
+        {
+          comment_id: comment.id,
+          issue_number: issue_number,
+          status: issue.state,
+          observation: observation
+        }
+      rescue Octokit::Error => e
+        {
+          comment_id: comment.id,
+          issue_number: issue_number,
+          status: :error,
+          message: e.message
+        }
+      end
+    end
+
+    def extract_observation_from_issue(issue_number)
+      comments = @github_client.issue_comments(@repo, issue_number)
+
+      # Look for magic comments with observation markers
+      observation_markers = @config.dig("github", "retrieval", "observation_markers") ||
+                            ["**OBSERVATION:**", "**COMMENTER OBSERVATION:**"]
+
+      # Search comments in reverse order (newest first)
+      comments.reverse_each do |comment|
+        observation = parse_observation_from_comment(comment.body, observation_markers)
+        return observation if observation
+      end
+
+      # Fallback to last comment if configured and no magic comment found
+      return comments.last.body.strip if @config.dig("github", "retrieval",
+                                                     "fallback_to_last_comment") && !comments.empty?
+
+      nil
+    rescue Octokit::Error
+      nil
+    end
+
+    def parse_observation_from_comment(comment_body, markers)
+      markers.each do |marker|
+        # Look for markdown blockquote with the marker
+        pattern = /^>\s*#{Regexp.escape(marker)}\s*\n((?:^>.*\n?)*)/m
+        match = comment_body.match(pattern)
+
+        next unless match
+
+        # Extract the blockquote content and clean it up
+        observation = match[1]
+                      .split("\n")
+                      .map { |line| line.sub(/^>\s?/, "") }
+                      .join("\n")
+                      .strip
+        return observation unless observation.empty?
+      end
+
+      nil
+    end
+
+    def update_yaml_with_observations(yaml_file, comment_sheet, options)
+      output_file = options[:output] || yaml_file
+      File.write(output_file, comment_sheet.to_yaml_document)
+    end
+  end
+end

data/lib/commenter/github_sync.rb

@@ -61,7 +61,7 @@ module Commenter
     end
 
     def reconcile(comment, comment_sheet, options)
-      issue = find_existing_issue(comment, comment_sheet)
+      issue = recorded_issue(comment, comment_sheet) || find_existing_issue(comment, comment_sheet)
       return create_issue(comment, comment_sheet, options) unless issue
 
       title = @title_template.render(template_variables(comment, comment_sheet))

data/lib/commenter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Commenter
-  VERSION = "0.6.0"
+  VERSION = "0.6.1"
 end

data/lib/commenter.rb

@@ -20,6 +20,6 @@ module Commenter
   autoload :Parser, "commenter/parser"
   autoload :Filler, "commenter/filler"
   autoload :GitHubIssueCreator, "commenter/github_integration"
-  autoload :GitHubIssueRetriever, "commenter/github_integration"
+  autoload :GitHubIssueRetriever, "commenter/github_issue_retriever"
   autoload :GitHubSync, "commenter/github_sync"
 end

data/spec/commenter/comment_type_spec.rb

@@ -43,3 +43,19 @@ RSpec.describe Commenter::CommentType do
     end
   end
 end
+
+RSpec.describe Commenter::CommentType do
+  describe ".known?" do
+    it "accepts recognized codes and full names" do
+      expect(described_class.known?("te")).to be(true)
+      expect(described_class.known?("Technical")).to be(true)
+      expect(described_class.known?("editorial")).to be(true)
+    end
+
+    it "rejects combined or free-form values" do
+      expect(described_class.known?("ge/te")).to be(false)
+      expect(described_class.known?("")).to be(false)
+      expect(described_class.known?(nil)).to be(false)
+    end
+  end
+end

data/spec/commenter/github_integration_spec.rb

@@ -4,6 +4,76 @@ require "spec_helper"
 require "tempfile"
 require "yaml"
 
+FakeIssue = Struct.new(:number, :title, :body, :html_url, :state, :created_at, keyword_init: true)
+FakeSearchResults = Struct.new(:items, keyword_init: true)
+FakeMilestone = Struct.new(:number, :title, keyword_init: true)
+MarkerIssue = Struct.new(:number, :html_url, :title, :body, :state, :created_at, keyword_init: true)
+MarkerResults = Struct.new(:items, keyword_init: true)
+RetrieverIssue = Struct.new(:number, :state, keyword_init: true)
+RetrieverComment = Struct.new(:body, keyword_init: true)
+
+# Real stand-in for the Octokit surface github-create uses: a seeded issue
+# store plus counters, so duplicate-prevention and retry behavior is
+# exercised against an object with the same protocol as Octokit::Client.
+class FakeGitHubClient
+  attr_accessor :rate_limit_failures, :search_error
+  attr_reader :created
+
+  def initialize(existing_issues = [])
+    @issues = {}
+    existing_issues.each { |issue| @issues[issue.number] = issue }
+    @next_number = @issues.keys.max.to_i
+    @created = []
+    @rate_limit_failures = 0
+  end
+
+  def issue(_repo, number)
+    @issues.fetch(number) { raise Octokit::NotFound }
+  end
+
+  def search_issues(query)
+    raise search_error if search_error
+
+    if (marker = query[/in:body "(.+)"\z/, 1])
+      FakeSearchResults.new(items: @issues.values.select { |i| i.body.to_s.include?(marker) })
+    else
+      unique_id = query[/in:title "(.+)"\z/, 1]
+      FakeSearchResults.new(items: @issues.values.select { |i| i.title.include?(unique_id) })
+    end
+  end
+
+  def milestones(_repo, _options = {})
+    [FakeMilestone.new(number: 1, title: "Test Milestone")]
+  end
+
+  def create_issue(_repo, title, body, _options = {})
+    if @rate_limit_failures.positive?
+      @rate_limit_failures -= 1
+      rate_limited = Octokit::TooManyRequests.new(rate_limited_response)
+      raise rate_limited
+    end
+
+    @next_number += 1
+    issue = FakeIssue.new(number: @next_number, title: title, body: body,
+                          html_url: "https://example.test/issues/#{@next_number}",
+                          state: "open", created_at: "2026-09-07T00:00:00Z")
+    @issues[issue.number] = issue
+    @created << issue
+    issue
+  end
+
+  private
+
+  # Mirrors the response hash Octokit builds from a real 403.
+  def rate_limited_response
+    {
+      status: 403,
+      body: "You have exceeded a secondary rate limit and have been " \
+            "temporarily blocked from content creation."
+    }
+  end
+end
+
 RSpec.describe Commenter::GitHubIssueCreator do
   let(:config_data) do
     {
@@ -131,6 +201,148 @@ RSpec.describe Commenter::GitHubIssueCreator do
       result = results.first
       expect(result[:labels]).not_to include("draft-international-standard")
     end
+
+    context "with an injected client, without dry run" do
+      before do
+        stub_const("Commenter::GitHubIssueCreator::THROTTLE_SECONDS", 0)
+        stub_const("Commenter::GitHubIssueCreator::RATE_LIMIT_RETRY_DELAY", 0)
+      end
+
+      def write_sheet(comments)
+        data = yaml_data
+        data["comments"] = comments
+        file = Tempfile.new(["comments", ".yaml"])
+        file.write(data.to_yaml)
+        file.close
+        file
+      end
+
+      def seeded_issue(number, title)
+        FakeIssue.new(number: number, title: title,
+                      html_url: "https://example.test/issues/#{number}",
+                      state: "open", created_at: "2026-09-06T00:00:00Z")
+      end
+
+      it "skips creation when the YAML records a matching issue" do
+        client = FakeGitHubClient.new([seeded_issue(7, "[DIS] US-001: already posted")])
+        sheet = write_sheet([yaml_data["comments"].first.merge("github" => { "issue_number" => 7 })])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:skipped)
+        expect(results.first[:issue_number]).to eq(7)
+        expect(results.first[:issue_url]).to eq("https://example.test/issues/7")
+        expect(client.created).to be_empty
+        expect(Commenter::CommentSheet.from_yaml(File.read(sheet.path)).comments.first.github_issue_number).to eq(7)
+        sheet.unlink
+      end
+
+      it "falls back to search when the recorded issue no longer exists" do
+        client = FakeGitHubClient.new([seeded_issue(5, "[DIS] US-001: posted earlier")])
+        sheet = write_sheet([yaml_data["comments"].first.merge("github" => { "issue_number" => 99 })])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:skipped)
+        expect(results.first[:issue_number]).to eq(5)
+        expect(client.created).to be_empty
+        expect(Commenter::CommentSheet.from_yaml(File.read(sheet.path)).comments.first.github_issue_number).to eq(5)
+        sheet.unlink
+      end
+
+      it "falls back to search when the recorded issue title does not match the stage" do
+        client = FakeGitHubClient.new([seeded_issue(7, "[WD] US-001: different stage"),
+                                       seeded_issue(5, "[DIS] US-001: posted earlier")])
+        sheet = write_sheet([yaml_data["comments"].first.merge("github" => { "issue_number" => 7 })])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:skipped)
+        expect(results.first[:issue_number]).to eq(5)
+        expect(client.created).to be_empty
+        sheet.unlink
+      end
+
+      it "records the issue in the YAML when skipping via title search" do
+        client = FakeGitHubClient.new([seeded_issue(5, "[DIS] US-001: posted earlier")])
+        sheet = write_sheet(yaml_data["comments"])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:skipped)
+        expect(Commenter::CommentSheet.from_yaml(File.read(sheet.path)).comments.first.github_issue_number).to eq(5)
+        sheet.unlink
+      end
+
+      it "creates and records the issue when nothing exists" do
+        client = FakeGitHubClient.new
+        sheet = write_sheet(yaml_data["comments"])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:created)
+        expect(results.first[:issue_number]).to eq(1)
+        expect(client.created.length).to eq(1)
+        expect(Commenter::CommentSheet.from_yaml(File.read(sheet.path)).comments.first.github_issue_number).to eq(1)
+        sheet.unlink
+      end
+
+      it "retries creation after a rate limit" do
+        client = FakeGitHubClient.new
+        client.rate_limit_failures = 1
+        sheet = write_sheet(yaml_data["comments"])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:created)
+        expect(client.created.length).to eq(1)
+        sheet.unlink
+      end
+
+      it "reports an error when rate limiting persists" do
+        client = FakeGitHubClient.new
+        client.rate_limit_failures = 99
+        sheet = write_sheet(yaml_data["comments"])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:error)
+        expect(client.created).to be_empty
+        sheet.unlink
+      end
+
+      it "reports an error instead of creating when the title search fails" do
+        client = FakeGitHubClient.new
+        client.search_error = Octokit::Forbidden.new(
+          status: 403,
+          body: "You have exceeded a secondary rate limit and have been " \
+                "temporarily blocked from content creation."
+        )
+        sheet = write_sheet(yaml_data["comments"])
+        creator = described_class.new(config_file.path, title_template_file.path,
+                                      body_template_file.path, client: client)
+
+        results = creator.create_issues_from_yaml(sheet.path)
+
+        expect(results.first[:status]).to eq(:error)
+        expect(client.created).to be_empty
+        sheet.unlink
+      end
+    end
   end
 
   describe "template variable generation" do
@@ -214,3 +426,145 @@ RSpec.describe Commenter::GitHubIssueRetriever do
     end
   end
 end
+
+RSpec.describe Commenter::GitHubIssueCreator do
+  let(:config_file) do
+    file = Tempfile.new(["config", ".yaml"])
+    file.write({ "github" => { "repository" => "test-org/test-repo", "token" => "t",
+                               "default_labels" => ["comment-review"] } }.to_yaml)
+    file.close
+    file
+  end
+  let(:client) { Octokit::Client.new(access_token: "dummy") }
+  let(:creator) { described_class.new(config_file.path, client: client) }
+
+  def sheet_with(type)
+    Commenter::CommentSheet.new(stage: "DIS", comments: [
+                                  Commenter::Comment.new(id: "US-001", body: "US", type: type, comments: "Text")
+                                ])
+  end
+
+  after { config_file.unlink }
+
+  describe "comment type labels (#3)" do
+    it "labels recognized types and never mints labels from combined values" do
+      recognized = creator.create_issues_from_yaml(write_yaml(sheet_with("technical")), dry_run: true)
+      combined = creator.create_issues_from_yaml(write_yaml(sheet_with("ge/te")), dry_run: true)
+
+      expect(recognized.first[:labels]).to include("technical")
+      expect(combined.first[:labels]).to eq(["comment-review"])
+    end
+
+    def write_yaml(sheet)
+      file = Tempfile.new(["comments", ".yaml"])
+      file.write(sheet.to_yaml_document)
+      file.close
+      file
+    end
+  end
+
+  describe "issue marker (#1)" do
+    it "embeds the marker in created issue bodies" do
+      empty = MarkerResults.new(items: [])
+      allow(client).to receive(:search_issues).and_return(empty)
+      created = MarkerIssue.new(number: 9, html_url: "https://github.com/test-org/test-repo/issues/9",
+                                title: "T", body: "B", state: "open")
+      allow(client).to receive(:create_issue) { |_repo, _title, _body, _opts| created }
+
+      file = write_yaml_with_github(nil)
+      creator.create_issues_from_yaml(file)
+
+      expect(client).to have_received(:create_issue)
+        .with(anything, anything, %r{urn:commenter:test-org/test-repo:dis:US-001}, anything)
+      file.unlink
+    end
+
+    it "finds existing issues through the marker before the title search" do
+      marker_hit = MarkerIssue.new(number: 9, html_url: "u", title: "t", body: "b", state: "open", created_at: nil)
+      allow(client).to receive(:search_issues).with(/in:body/).and_return(MarkerResults.new(items: [marker_hit]))
+      allow(client).to receive(:search_issues).with(/in:title/).and_raise("title search must not run")
+
+      file = write_yaml_with_github(nil)
+      results = creator.create_issues_from_yaml(file)
+
+      expect(results.first[:status]).to eq(:skipped)
+      expect(results.first[:issue_number]).to eq(9)
+      file.unlink
+    end
+
+    it "verifies recorded issues by marker when the title does not carry the unique id" do
+      recorded = MarkerIssue.new(number: 9, html_url: "u", title: "Custom title", created_at: nil,
+                                 body: "discussed in `urn:commenter:test-org/test-repo:dis:US-001`", state: "open")
+      allow(client).to receive(:issue).with("test-org/test-repo", 9).and_return(recorded)
+      allow(client).to receive(:search_issues).and_raise("search must not run")
+
+      file = write_yaml_with_github(9)
+      results = creator.create_issues_from_yaml(file)
+
+      expect(results.first[:status]).to eq(:skipped)
+      file.unlink
+    end
+
+    def write_yaml_with_github(issue_number)
+      github = issue_number ? { "issue_number" => issue_number, "status" => "open" } : nil
+      file = Tempfile.new(["comments", ".yaml"])
+      file.write({ "version" => "2012-03", "stage" => "DIS",
+                   "comments" => [{ "id" => "US-001", "body" => "US", "comments" => "Text",
+                                    "github" => github }.compact] }.to_yaml)
+      file.close
+      file
+    end
+  end
+end
+
+RSpec.describe Commenter::GitHubIssueRetriever do
+  let(:config_file) do
+    file = Tempfile.new(["config", ".yaml"])
+    file.write({ "github" => { "repository" => "test-org/test-repo", "token" => "t" } }.to_yaml)
+    file.close
+    file
+  end
+  let(:client) { Octokit::Client.new(access_token: "dummy") }
+  let(:retriever) { described_class.new(config_file.path, client: client) }
+
+  after { config_file.unlink }
+
+  def retrieve_with(issue_state, comments)
+    Dir.mktmpdir do |dir|
+      input = File.join(dir, "comments.yaml")
+      File.write(input, { "version" => "2012-03",
+                          "comments" => [{ "id" => "US-001", "comments" => "Text",
+                                           "github" => { "issue_number" => 5, "status" => "open" } }] }.to_yaml)
+      allow(client).to receive(:issue).with("test-org/test-repo", 5)
+                                      .and_return(RetrieverIssue.new(number: 5, state: issue_state))
+      allow(client).to receive(:issue_comments).and_return(comments)
+
+      result = retriever.retrieve_observations_from_yaml(input)[0]
+      sheet = Commenter::CommentSheet.from_yaml(File.read(input))
+      [result, sheet.comments.first]
+    end
+  end
+
+  it "fills an observation from an open issue (#4)" do
+    result, comment = retrieve_with("open", [RetrieverComment.new(body: "> **OBSERVATION:**\n> Accepted.")])
+
+    expect(result[:status]).to eq(:retrieved)
+    expect(comment.observations).to eq("Accepted.")
+  end
+
+  it "skips an open issue only when no observation exists yet" do
+    result, comment = retrieve_with("open", [RetrieverComment.new(body: "Still discussing")])
+
+    expect(result[:status]).to eq(:skipped)
+    expect(result[:message]).to include("still open")
+    expect(comment.observations).to be_nil
+  end
+
+  it "fills observations from closed issues and records the status" do
+    result, comment = retrieve_with("closed", [RetrieverComment.new(body: "> **OBSERVATION:**\n> Noted.")])
+
+    expect(result[:status]).to eq(:retrieved)
+    expect(comment.observations).to eq("Noted.")
+    expect(comment.github_status).to eq("closed")
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: commenter
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.1
 platform: ruby
 authors:
 - Ribose
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2026-08-29 00:00:00.000000000 Z
+date: 2026-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: base64
@@ -163,6 +163,7 @@ files:
 - lib/commenter/filler.rb
 - lib/commenter/github_info.rb
 - lib/commenter/github_integration.rb
+- lib/commenter/github_issue_retriever.rb
 - lib/commenter/github_session.rb
 - lib/commenter/github_sync.rb
 - lib/commenter/parser.rb