wikipedia_scraper 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 39d22e261aa238cf6caf1b1dc29853a712bde81da50d7bb10ec1ae9a9b602110
+  data.tar.gz: 989966ba56260f1106e08944ad2702146018d1ce0d931848a51a0099b26510f9
+SHA512:
+  metadata.gz: da8e01135625d9251c689b5a8d6438084cc379285d298228cd9d8dbaf107b47c14b5077e8ce4ce4349d277b701e3885e888d53977971d68360a0c4cb3c038c93
+  data.tar.gz: daa20c1c153a67f8bb9848e50c3737b6c82b01ad4b16aabdeac241d78b13eb6550e37dfbfd193be602f05204af6ded409b37775ad7377bb2767fd86560bd7dcb

data/.gitignore

@@ -0,0 +1,4 @@
+/spec/examples.txt
+*.gem
+*.json
+Gemfile.lock

data/Gemfile

@@ -0,0 +1,8 @@
+source "https://rubygems.org"
+
+gem 'httparty'
+gem 'nokogiri'
+
+group :test do
+  gem 'rspec'
+end

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2011-2026 The Bootstrap Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,387 @@
+# Wikipedia Scraper
+
+This library can be used for scraping Wikipedia articles into a structured JSON format.
+
+Unlike traditional scrapers that simply extract plain text, this project preserves the structure of a Wikipedia page by converting it into nested sections, Markdown-formatted text, lists and tables.
+
+## Features
+
+- Scrape any public Wikipedia article
+- Convert article content into structured JSON
+- Preserve nested sections
+- Convert formatting to Markdown
+  - **Bold**
+  - *Italic*
+  - Links from anchors
+- Extract
+  - Paragraphs
+  - Ordered & unordered lists
+  - References
+  - Tables (including rowspan/colspan support)
+- Skip Wikipedia navigation, metadata and styling elements
+- Simple CLI interface
+- Can also be used as a Ruby library
+
+---
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone https://github.com/rocketdey/wikipedia_scraper.git
+cd wikipedia_scraper
+```
+
+Install dependencies:
+
+```bash
+bundle install
+```
+
+### Gem installation
+
+```bash
+gem install wikipedia_scraper
+```
+
+## Usage
+
+### Command Line
+
+```bash
+wikipedia_scraper # This will scrape https://en.wikipedia.org/wiki/special:random to current dir
+```
+
+```bash
+wikipedia_scraper https://en.wikipedia.org/wiki/Apple_Inc. ./output/
+```
+
+This will create
+
+```
+output/
+└── Apple_Inc..json
+```
+
+---
+
+### Ruby Library
+
+```ruby
+require "wikipedia_scraper"
+
+page = WikipediaScraper::Page.fetch(
+  "https://en.wikipedia.org/wiki/Alan_Turing"
+)
+
+puts page.title
+
+page.save_json("./output")
+```
+
+---
+
+## Example JSON
+
+```json
+{
+  "title": "David Chase",
+  "short_description": "American writer, director and producer (born 1945)",
+  "url": "https://en.wikipedia.org/wiki/David_Chase",
+  "sections": [
+    {
+      "id": 0,
+      "heading": null,
+      "content": [
+        "**David Henry Chase**[1][2] (born August 22, 1945) is an American writer, producer, and director. ..."
+      ]
+    },
+    {
+      "id": 1,
+      "heading": "Early life",
+      "content": [
+        "Chase was born as an only child to Norma ([née](https://en.wikipedia.org/wiki/Birth_name#Maiden_and_married_names) Bucco) and Enrico \"Henry\" Chase, both born in 1908 and hailing from Italian-American working-class families. ..."
+        {
+          "id": 2,
+          "heading": "Mental health and education",
+          "content": [
+            "Chase struggled with [panic attacks](https://en.wikipedia.org/wiki/Panic_attacks) and [clinical depression](https://en.wikipedia.org/wiki/Clinical_depression) as a teenager, something that he dealt with into adulthood. ..."
+          ]
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "heading": "Career",
+      "content": [
+        "Chase started in Hollywood as a [story editor](https://en.wikipedia.org/wiki/Story_editor) for *[Kolchak: The Night Stalker](https://en.wikipedia.org/wiki/Kolchak:_The_Night_Stalker)* and then produced episodes of *[The Rockford Files](https://en.wikipedia.org/wiki/The_Rockford_Files)* and *[Northern Exposure](https://en.wikipedia.org/wiki/Northern_Exposure)*, among other series. ...",
+        {
+          "id": 4,
+          "heading": "*The Sopranos*",
+          "content": [
+            "Chase worked in relative anonymity before *[The Sopranos](https://en.wikipedia.org/wiki/The_Sopranos)* debuted.[11] The story of *The Sopranos* was initially conceived as a feature film about \"a mobster in therapy having problems with his mother\".[23] Chase got some input from his manager [Lloyd Braun](https://en.wikipedia.org/wiki/Lloyd_Braun_(media_executive)) and decided to adapt it into a television series.[23] ..."
+            [
+              "*The Sopranos* credits",
+              [
+                "Writer",
+                [
+                  "\"[The Sopranos](https://en.wikipedia.org/wiki/The_Sopranos_(pilot_episode))\" *(episode 1.01)*",
+                  "\"[46 Long](https://en.wikipedia.org/wiki/46_Long)\" *(episode 1.02)*",
+                  "..."
+                ],
+                "Director",
+                [
+                  "\"[The Sopranos](https://en.wikipedia.org/wiki/The_Sopranos_(pilot_episode))\" *(episode 1.01)*",
+                  "\"[Made in America](https://en.wikipedia.org/wiki/Made_in_America_(The_Sopranos))\" *(episode 6.21)*"
+                ],
+                "Actor",
+                "Chase appeared as a man sitting at an outdoor cafe in [Naples](https://en.wikipedia.org/wiki/Naples), Italy smoking a cigarette in the season two episode \"[Commendatori](https://en.wikipedia.org/wiki/Commendatori)\". He also appeared as an airline passenger en route to Italy in season six's \"[Luxury Lounge](https://en.wikipedia.org/wiki/Luxury_Lounge)\". His voice was also used over the phone in the episode \"The Test Dream\"."
+              ]
+            ]
+          ]
+        },
+        {
+          "id": 5,
+          "heading": "*Not Fade Away*",
+          "content": [
+            "*[Not Fade Away](https://en.wikipedia.org/wiki/Not_Fade_Away_(film))* (2012), Chase's feature film debut, was released on December 21, 2012. It centers on the lead singer of a teenage [rock 'n' roll](https://en.wikipedia.org/wiki/Rock_and_roll) band (played by [John Magaro](https://en.wikipedia.org/wiki/John_Magaro)) in 1960s New Jersey.[38][39] ..."
+          ]
+        },
+        {
+          "id": 6,
+          "heading": "*The Many Saints of Newark*",
+          "content": [
+            "Although Chase was \"against [the movie] for a long time\",[41] *[Deadline Hollywood](https://en.wikipedia.org/wiki/Deadline_Hollywood)* reported in March 2018 that [New Line Cinema](https://en.wikipedia.org/wiki/New_Line_Cinema) had purchased the script for *[The Many Saints of Newark](https://en.wikipedia.org/wiki/The_Many_Saints_of_Newark)* ..."
+          ]
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "heading": "Unrealized projects",
+      "content": [
+        {
+          "id": 8,
+          "heading": "*A Ribbon of Dreams*",
+          "content": [
+            "Chase has previously developed *A Ribbon of Dreams*, a [miniseries](https://en.wikipedia.org/wiki/Miniseries) for HBO. According to an HBO [press release](https://en.wikipedia.org/wiki/Press_release), the series' pilot would \"begin in 1913 and follow two men, one a college-educated mechanical engineer, the other a cowboy with a violent past, ..."
+          ]
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "heading": "Personal life",
+      "content": [
+        "After graduating from NYU in 1968, Chase moved to California and married his high school sweetheart Denise Kelly.[11] He is the father of actress Michele DeCesare, who appeared in six of *The Sopranos* episodes as [Hunter Scangarelo](https://en.wikipedia.org/wiki/Hunter_Scangarelo).[49]",
+        "..."
+      ]
+    },
+    {
+      "id": 10,
+      "heading": "Select filmography",
+      "content": [
+        {
+          "id": 11,
+          "heading": "Television",
+          "content": [
+            [
+              [
+                "Year",
+                "Title",
+                "Director",
+                "Writer",
+                "Producer",
+                "Creator",
+                "Notes"
+              ],
+              [
+                "1971",
+                "*[The Bold Ones: The Lawyers](https://en.wikipedia.org/wiki/The_Bold_Ones:_The_Lawyers)*",
+                "No",
+                "Yes",
+                "No",
+                "No",
+                "Episode: \"In Defense of Ellen McKay\""
+              ],
+              "..."
+            ]
+          ]
+        },
+        {
+          "id": 12,
+          "heading": "Film",
+          "content": [
+            "..."
+          ]
+        },
+        {
+          "id": 13,
+          "heading": "Other credits",
+          "content": [
+            "..."
+          ]
+        }
+      ]
+    },
+    {
+      "id": 14,
+      "heading": "Awards and recognition",
+      "content": [
+        "..."
+      ]
+    },
+    {
+      "id": 15,
+      "heading": "See also",
+      "content": [
+        "[List of Primetime Emmy Award winners](https://en.wikipedia.org/wiki/List_of_Primetime_Emmy_Award_winners)"
+      ]
+    },
+    {
+      "id": 16,
+      "heading": "References",
+      "content": [
+        [
+          "1. Chase says his name was not David DeCesare at birth in this interview: [https://interviews.televisionacademy.com/interviews/david-chase#](https://interviews.televisionacademy.com/interviews/david-chase#) [Archived](https://web.archive.org/web/20190331030349/https://interviews.televisionacademy.com/interviews/david-chase)March 31, 2019, at the[Wayback Machine](https://en.wikipedia.org/wiki/Wayback_Machine)",
+          "2. Fleming, Mike Jr. (September 7, 2021). [\"David Chase On Reviving 'Sopranos' Spirit With 'The Many Saints Of Newark' And High Interest In Another Prequel Film\"](https://deadline.com/2021/09/david-chase-sopranos-revival-the-many-saints-of-newark-disdain-day-date-bow-interested-in-another-prequel-film-1234828184/). *Deadline*. Retrieved September 8, 2021.",
+          "3. *[Wise Guy: David Chase and the Sopranos](https://en.wikipedia.org/wiki/Wise_Guy:_David_Chase_and_the_Sopranos)*",
+          "..."
+        ]
+      ]
+    },
+    {
+      "id": 17,
+      "heading": "External links",
+      "content": [
+        [
+          "[David Chase](https://www.imdb.com/name/nm0153740/)at[IMDb](https://en.wikipedia.org/wiki/IMDb_(identifier))",
+          "[David Chase](https://interviews.televisionacademy.com/interviews/david-chase)at[The Interviews: An Oral History of Television](https://en.wikipedia.org/wiki/The_Interviews:_An_Oral_History_of_Television)"
+        ]
+      ]
+    }
+  ]
+}
+```
+
+---
+
+## Supported Elements
+
+| Element | Output |
+|---------|--------|
+| Paragraphs | Markdown text |
+| Links | Markdown links |
+| Bold | `**text**` |
+| Italic | `*text*` |
+| Lists | Ruby Arrays |
+| References | Numbered Arrays |
+| Tables | Nested Arrays |
+| Nested Sections | Recursive Hashes |
+
+---
+
+## Project Structure
+
+```
+bin/
+    wikipedia_scraper
+
+lib/
+    wikipedia_scraper/
+        page.rb
+        tag_scraper.rb
+        version.rb
+    wikipedia_scraper.rb
+
+spec/
+    fixtures/
+      Chechen_language.html
+      Chechen_language.json
+      David_Chase.html
+      David_Chase.json
+      The_Off-Season.html
+      The_Off-Season.json
+    page_spec.rb
+    spec_helper.rb
+    tag_scraper_spec.rb
+```
+
+### `Page`
+
+Responsible for
+
+- downloading a Wikipedia page
+- parsing the HTML
+- walking through article sections
+- building the final JSON structure
+- saving the result
+
+### `TagScraper`
+
+Handles HTML parsing and conversion.
+
+Responsibilities include:
+
+- Markdown conversion
+- List parsing
+- Table parsing
+- Reference extraction
+- Link formatting
+
+The module contains no network or file I/O, making it easy to test independently.
+
+---
+
+## Dependencies
+
+- HTTParty
+- Nokogiri
+- JSON
+- RSpec (development)
+
+---
+
+## Running Tests
+
+```bash
+rake spec
+```
+
+---
+
+## Current Limitations
+
+- Infobox parsing is currently disabled.
+- Templates and navigation boxes are intentionally ignored.
+- There may be unknown errors since this library is a WIP (Work in Progress).
+
+---
+
+## Future Improvements
+
+- Infobox parsing
+- Multi-thread operation
+- Parallel page scraping
+
+---
+
+## Why this project?
+
+The goal of this project is to provide a clean, structured representation of Wikipedia articles suitable for:
+
+- LLM datasets
+- Search indexing
+- Knowledge extraction
+- Data analysis
+- Offline archives
+- Markdown generation
+
+Rather than scraping raw HTML, the library attempts to preserve the semantic structure of the article.
+
+---
+
+## License
+
+MIT License

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/wikipedia_scraper

@@ -0,0 +1,11 @@
+#!/usr/bin/env ruby
+
+require 'wikipedia_scraper'
+
+url = (!ARGV[0] || ARGV[0] == 'random') ? 'https://en.wikipedia.org/wiki/special:random' : ARGV[0]
+page = WikipediaScraper::Page.fetch(url)
+
+output_directory = ARGV.length > 1 ? ARGV[-1] : "." # Change the default output location
+page.save_json(output_directory)
+
+puts "Saved #{page.title} to #{output_directory} as #{page.url.split('/').last}.json"

data/lib/wikipedia_scraper/page.rb

@@ -0,0 +1,87 @@
+require 'httparty'
+require 'nokogiri'
+require 'json'
+
+module WikipediaScraper
+  class Page
+    HEADERS = {
+      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:150.0) Gecko/20100101 Firefox/150.0',
+      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+      'Accept-Language': 'en-US,en;q=0.9',
+      'Connection': 'keep-alive',
+      'Upgrade-Insecure-Requests': '1',
+      'Sec-Fetch-Dest': 'document',
+      'Sec-Fetch-Mode': 'navigate',
+      'Sec-Fetch-Site': 'none',
+      'Sec-Fetch-User': '?1',
+      'Priority': 'u=0, i',
+      'TE': 'trailers',
+    }.freeze
+
+    attr_reader :url, :title, :short_description, :sections
+
+    def self.fetch(url)
+      response = HTTParty.get(url, headers: HEADERS)
+      doc = Nokogiri::HTML(response.body)
+      raise "Failed to parse #{url} as HTML" unless doc.html?
+
+      new(doc)
+    end
+
+    def initialize(doc)
+      @url = doc.at_css("link[rel='canonical']")&.attr('href')
+      puts "Scraping #{url}"
+      @title = TagScraper.to_markdown(doc.at_css("#firstHeading"))
+      @short_description = doc.at_css(".shortdescription")&.text
+      @sections = []
+
+      doc.css(".mw-content-ltr > section").each do |section|
+        section_scraper(section, @sections)
+      end
+    end
+
+    def to_h
+      { title: title, short_description: short_description, url: url, sections: sections }
+    end
+
+    def save_json(path)
+      file_path = path[-1] == '/' ? "#{path}#{url.split('/').last}.json" : "#{path}/#{url.split('/').last}.json"
+      File.open(file_path, 'w') do |f|
+        f.write(JSON.pretty_generate(to_h) + "\n")
+      end
+    end
+
+    private
+
+    def section_scraper(section, parent_content)
+      heading_elem = section.at_css("div.mw-heading")
+      new_section = {
+        id: section['data-mw-section-id'].to_i,
+        heading: heading_elem && TagScraper.to_markdown(heading_elem.children[0]),
+        content: []
+      }
+      parent_content << new_section
+
+      section.children.each do |node|
+        if node.name == 'section'
+          section_scraper(node, new_section[:content])
+          next
+        end
+
+        next unless !node.content.strip.empty? &&
+        ['p', 'div', 'ol', 'ul', 'table'].include?(node.name) &&
+        !node.classes.any? { |c| ['shortdescription', 'metadata', 'hatnote', 'sistersitebox', 'navbox', 'navbox-styles', 'mw-heading'].include?(c) }
+
+        scraped_node = TagScraper.scrape(node)
+        scraped_node = simplify_array(scraped_node)
+        next if [nil, "", []].include?(scraped_node)
+        new_section[:content] << scraped_node
+      end
+    end
+
+    def simplify_array(item)
+      return item unless item.is_a?(Array)
+      item.size == 1 ? simplify_array(item.first) : item.map { |i| simplify_array(i) }
+    end
+  end
+end

data/lib/wikipedia_scraper/tag_scraper.rb

@@ -0,0 +1,269 @@
+module TagScraper
+
+  def self.scrape(element)
+    return nil if element.nil? || element.classes.any? { |c| ['mw-cite-backlink', 'noprint'].include?(c) } || (!element.children.any? && element.text.empty?)
+    case element.name
+    when "p", "a", "i", "b", "br", "span"
+      to_markdown(element)
+    when "ol", "ul"
+      parse_list(element)
+    when "table"
+      if !element.classes.include?('wikitable')
+        if element.classes.include?('infobox')
+          #return parse_infobox(element)
+          return nil
+        end
+
+        element_list = element.css(".wikitable, .mw-heading")
+        if element_list.empty?
+          parse_table([element])
+        elsif element_list.any? { |e| e.classes.include?('mw-heading')}
+          content_list = []
+          if element_list[0].classes.include?('wikitable')
+            last_heading = nil
+          else
+            last_heading = element_list[0].at_css('h2, h3, h4').content
+          end
+          tables = []
+          section = {heading: last_heading, content: tables}
+          element_list.drop(1).each do |e|
+            if e.classes.include?('mw-heading')
+              section = {heading: last_heading, content: tables}
+              content_list << section
+              last_heading = e.at_css('h2, h3, h4').content
+              tables = []
+            else
+              tables << parse_table([e])
+            end
+          end
+          content_list
+        else
+          parse_table(element_list)
+        end
+      else
+        parse_table([element])
+      end
+    when "div"
+      if element.classes.include?('hatnote')
+        to_markdown(element)
+      else
+        result = []
+        text_inside_div = ''
+        element.children.each do |child|
+          if ["p", "a", "i", "b", "br", "span", "text"].include?(child.name)
+            text_inside_div << (to_markdown(child) || '')
+          else
+            result << text_inside_div.strip && text_inside_div = '' unless text_inside_div.strip.empty?
+            result << self.scrape(child)
+          end
+        end
+        result << text_inside_div.strip unless text_inside_div.strip.empty?
+        result.compact
+      end
+    when "style", "figure", "sup"
+      nil
+    else
+      if !element.children.empty?
+        element.children.map do |child|
+          self.scrape(child)
+        end.compact
+      else
+        to_markdown(element)
+      end
+    end
+  end
+
+  def self.to_markdown(element)
+
+    unless element.nil? || element.classes.any? { |c| ['mw-cite-backlink', 'noprint'].include?(c) }
+      case element.name
+      when 'text'
+        content = element.content.gsub(/\s+/, ' ')
+        return content.strip.empty? ? nil : content
+      when "ol", "ul"
+        return parse_list(element).join(', ')
+      end
+
+      inner = element.children.map { |node| markdown_node(node) }.join
+      inner = inner.gsub(/\s+/, ' ')
+      inner = inner.strip unless inner == ' '
+
+      case element.name
+      when "b"
+        "**#{inner}**"
+      when "a"
+        parse_anchor(element, inner)
+      when "i"
+        "*#{inner}*"
+      else
+        inner
+      end
+    end
+  end
+
+  def self.markdown_node(node)
+
+    return '' if node.classes.any? { |c| ['mw-cite-backlink', 'noprint'].include?(c) }
+
+    case node.name
+    when "text"
+      node.content
+    when "br"
+      ' '
+    when "span"
+      to_markdown(node)
+    when "sup"
+      return nil if node['style'] == "display:none;"
+      node.at_css('a').nil? ? to_markdown(node) : parse_anchor(node.at_css('a'))
+    when "style"
+      nil
+    else
+      to_markdown(node)
+    end
+  end
+
+  def self.parse_anchor(element, inner = nil)
+    return nil if ['Wikipedia:Citation needed', 'Edit this at Wikidata'].include?(element['title'])
+
+    url = element['href']
+    if !inner.nil? && !inner.strip.empty?
+      url_text = inner
+    elsif element.content.empty?
+      url_text = element['title']
+    else
+      url_text = element.content
+    end
+
+    if img = element.at_css('img')
+      if img['alt'] || img['title']
+        url_text = img['alt'] || img['title']
+      else
+        return '' if img['src']&.include?('/thumb/')
+      end
+    end
+
+    if url.include?('&action=edit')
+      "https://en.wikipedia.org#{url[18..]}"
+    elsif url.include?('#cite_note')
+      element.content
+    elsif !url_text.nil? && !url_text.empty?
+      url.include?('://') ? "[#{url_text}](#{url})".strip : "[#{url_text}](https://en.wikipedia.org#{url[18..]})".strip
+    else
+      url.include?('://') ? url : "https://en.wikipedia.org#{url[18..]}"
+    end
+  end
+
+  def self.parse_list(element)
+    items = []
+    element.css('> li').each do |node|
+      if reference_number = node['data-mw-footnote-number']
+        items << "#{reference_number}. #{to_markdown(node)}"
+      elsif node.matches?('.gallerybox')
+        img_node = node.at_css('img')
+        url = img_node.parent['href']
+        if node.content.empty?
+          items << [url.include?('://') ? url : "https://en.wikipedia.org#{url[18..]}", img_node['alt']]
+        else
+          items << [url.include?('://') ? url : "https://en.wikipedia.org#{url[18..]}", to_markdown(node.at_css(".gallerytext"))]
+        end
+      else
+        items << to_markdown(node)
+      end
+    end
+
+    items
+  end
+
+  def self.parse_table(element_list)
+    main_tables = []
+    element_list.each do |element|
+      rows = element.css('tr')
+      table_caption = to_markdown(element.at_css('caption'))
+      table_caption = nil if table_caption&.empty?
+      element.css('style').each(&:remove)
+      col_length = rows[0].css('th:not([style*="display:none"])', 'td:not([style*="display:none"])').sum { |c| c['colspan'].nil? ? 1 : c['colspan'].to_i}
+      table_data = Array.new(rows.length) { Array.new(col_length, nil) }
+      rows.each_with_index do |row_element, row_index|
+        unless row_element.css('table').empty?
+          table_data = parse_table(row_element.css('table'))
+          break
+        end
+        row = row_element.css('th:not([style*="display:none"])', 'td:not([style*="display:none"])')
+        cursor_index = col_index = 0
+        while col_index < row.length
+          data = row[col_index]
+          data_content = to_markdown(data)
+          if data_content.is_a?(Array)
+            table_data[row_index][cursor_index] << data_content
+          elsif !table_data[row_index][cursor_index].nil?
+            col_index += 1 if data_content.nil? || data_content.empty?
+            cursor_index += 1
+            break if cursor_index > col_length
+            next
+          else
+            colspan = row[col_index]['colspan'].nil? ? 1 : row[col_index]['colspan'].to_i
+            rowspan = row[col_index]['rowspan'].nil? ? 1 : row[col_index]['rowspan'].to_i
+            (0..colspan - 1).each do |c|
+              (0..rowspan - 1).each do |r|
+                table_data[r + row_index][c + cursor_index] = data_content
+              end
+            end
+          end
+          cursor_index += colspan
+          col_index += 1
+        end
+      end
+      table_data.each do |row|
+        row.map! { |a| a == '' ? nil : a} if row.is_a?(Array)
+      end
+      table_data.reject! { |a| a.all?(nil) || a.empty? }
+      table_caption.nil? ? main_tables << table_data : main_tables << [table_caption] + table_data
+    end
+    main_tables.length > 1 ? main_tables : main_tables[0]
+  end
+
+  def self.parse_infobox(element)
+    main_tables = []
+    rows = element.css('tr')
+    
+    rows.each do |row|
+      table_data = []
+      label = row.at_css("th.infobox-label")
+      if label
+        row_dict = {label: to_markdown(label), content: []}
+        combined_text = ''
+        row.at_css('td').children.each do |row_elem|
+          case row_elem.name
+          when "p", "a", "i", "b", "br", "span", "text"
+            combined_text << scrape(row_elem)
+          else
+            unless combined_text.empty?
+              row_dict[:content][-1].nil? ? row_dict[:content] << combined_text : row_dict[:content][-1] << combined_text
+            end
+            combined_text = ''
+            row_dict[:content].push(*scrape(row_elem))
+          end
+        end
+        main_tables << row_dict
+      else
+        combined_text = ''
+        row.at_css('td').children.each do |row_elem|
+          puts combined_text
+          case row_elem.name
+          when "p", "a", "i", "b", "br", "span", "text"
+            combined_text << scrape(row_elem)
+          else
+            unless combined_text.empty?
+              table_data[-1].nil? ? table_data << combined_text : table_data[-1] << combined_text
+            end
+            combined_text = ''
+            data = scrape(row_elem)
+            data.is_a?(Array) ? table_data.push(*data) : table_data << data
+          end
+        end
+      end
+      main_tables << table_data
+    end
+    {id: nil, heading: 'Infobox', title: to_markdown(element.at_css('caption')), content: main_tables}
+  end
+end

data/lib/wikipedia_scraper/version.rb

@@ -0,0 +1,3 @@
+module WikipediaScraper
+  VERSION = "0.1.1"
+end

data/lib/wikipedia_scraper.rb

@@ -0,0 +1,3 @@
+require 'wikipedia_scraper/version'
+require 'wikipedia_scraper/tag_scraper'
+require 'wikipedia_scraper/page'

metadata

@@ -0,0 +1,112 @@
+--- !ruby/object:Gem::Specification
+name: wikipedia_scraper
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- rocketdey
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: httparty
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.21'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.21'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+description: Scrape Wikipedia articles into structured JSON, preserving nested sections,
+  Markdown-formatted text, lists, tables, and references. Ideal for LLM datasets,
+  search indexing, and knowledge extraction.
+email:
+- kursataydin165@gmail.com
+executables:
+- wikipedia_scraper
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/wikipedia_scraper
+- lib/wikipedia_scraper.rb
+- lib/wikipedia_scraper/page.rb
+- lib/wikipedia_scraper/tag_scraper.rb
+- lib/wikipedia_scraper/version.rb
+homepage: https://github.com/rocketdey/wikipedia_scraper
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/rocketdey/wikipedia_scraper
+  source_code_uri: https://github.com/rocketdey/wikipedia_scraper
+  changelog_uri: https://github.com/rocketdey/wikipedia_scraper/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.20
+specification_version: 4
+summary: Scrapes Wikipedia articles into structured markdown/JSON.
+test_files: []