mddir 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 28e9c0901404556fd2aeeef6ccd427029da4d3f50f7fe0e5a55fa66b118cce1e
+  data.tar.gz: f74a0a254c16cf555c14497a36cc2925e791dcdf3bb5ebcb189cb6b0d2a1fdf1
+SHA512:
+  metadata.gz: a593cff246076103d9348d5281063f63abd56fde7540d5596fd97adae0384f9e7a128546fc22ada3c1349ab1557ac5b4190ceb9ff64ce0c6f68befb1cb629997
+  data.tar.gz: ce4d88969d6e2054670b32780cd6492d3fb90d41e456322869754d993f8bd74d0934e88b6d6d68c780e5f8e3ac9fc17ef901499d354613e88a118dbcbc2bc112

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-02-28
+
+- Fetch web pages and convert to clean markdown
+- Organize saved pages into named collections
+- Full-text search across all collections or within a single collection
+- Built-in web UI for browsing, reading, and searching
+- YAML frontmatter with metadata and token counts on every entry
+- Cookie support for fetching pages behind authentication
+- Configurable via `~/.mddir.yml` (base directory, port, editor, user agent)
+- SQLite-backed search index with automatic reindexing

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Ali Hamdi Ali Fadel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,160 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/AliOsm/mddir/main/mddir-logo.png" alt="mddir logo" width="200">
+</p>
+
+# mddir
+
+Your web, saved locally — a markdown knowledge base for humans and agents.
+
+[![Gem Version](https://img.shields.io/gem/v/mddir)](https://rubygems.org/gems/mddir)
+[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.2.0-red)](https://www.ruby-lang.org)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://opensource.org/licenses/MIT)
+
+mddir fetches web pages, converts them to clean markdown, and organizes them into local collections. It includes full-text search, a built-in web UI, and works with AI agents out of the box. Everything is stored as plain markdown files you own.
+
+![Home](https://raw.githubusercontent.com/AliOsm/mddir/main/screenshots/home.png)
+![Reader](https://raw.githubusercontent.com/AliOsm/mddir/main/screenshots/reader.png)
+
+## Installation
+
+Requires Ruby >= 3.2.0. If you don't have Ruby installed, [mise](https://mise.jdx.dev) is the easiest way to get it:
+
+```bash
+mise use --global ruby@3
+```
+
+Then install the gem:
+
+```bash
+gem install mddir
+```
+
+## Quick Start
+
+```bash
+# Save a page to a collection
+mddir add ruby https://docs.ruby-lang.org/en/3.3/String.html
+
+# List your collections
+mddir ls
+
+# Search across everything
+mddir search "freeze"
+
+# Open the web UI
+mddir open
+```
+
+## CLI Reference
+
+### Core Commands
+
+| Command | Description |
+|---|---|
+| `mddir add COLLECTION URL [URL...]` | Fetch web pages and save to a collection |
+| `mddir search [COLLECTION] QUERY` | Search entries for a query string |
+| `mddir ls [COLLECTION]` | List collections or entries in a collection |
+| `mddir open` | Start the web UI and open in browser |
+| `mddir serve` | Start the web UI server |
+
+### Management Commands
+
+| Command | Description |
+|---|---|
+| `mddir rm COLLECTION [ENTRY]` | Remove a collection or entry |
+| `mddir collection create NAME` | Create a new empty collection |
+| `mddir reindex` | Rebuild the global index |
+| `mddir config` | Open configuration file in editor |
+
+### Flags
+
+| Flag | Applies to | Description |
+|---|---|---|
+| `--cookies PATH` | `add` | Path to a cookies file for authenticated fetching |
+
+## Web UI
+
+Run `mddir open` to launch the built-in web UI at `http://localhost:7768`. Browse collections, read saved pages, and search your knowledge base from the browser.
+
+## Agent / LLM Integration
+
+mddir is designed to work as a knowledge retrieval tool for AI agents. All content lives under `~/.mddir/` as plain markdown files with YAML frontmatter:
+
+```yaml
+---
+url: https://example.com/article
+title: Article Title
+token_count: 1523
+token_estimated: true
+---
+```
+
+Agents can search the knowledge base directly:
+
+```bash
+mddir search "concurrency patterns"
+mddir search ruby "freeze"
+```
+
+## Configuration
+
+Settings live in `~/.mddir.yml`. Run `mddir config` to open it in your editor.
+
+| Option | Default | Description |
+|---|---|---|
+| `base_dir` | `~/.mddir` | Where collections are stored |
+| `port` | `7768` | Web UI port |
+| `editor` | `$EDITOR` or `vi` | Editor for `mddir config` |
+| `user_agent` | Chrome UA string | User agent for fetching pages |
+
+## Cookie Support
+
+Pass a cookies file to fetch pages behind authentication:
+
+```bash
+mddir add docs https://private.example.com/page --cookies ~/cookies.txt
+```
+
+## Data Storage
+
+```
+~/.mddir/
+├── search.db
+├── index.yml
+├── ruby/
+│   ├── index.yml
+│   ├── string-a1b2c3.md
+│   └── array-d4e5f6.md
+└── golang/
+    ├── index.yml
+    └── concurrency-patterns-f7a8b9.md
+```
+
+## Todos
+
+- [ ] Add local HTML and markdown files to collections
+- [ ] Headless browser rendering for JavaScript-heavy pages
+- [ ] Download and embed images locally instead of linking to remote URLs
+- [ ] Re-fetch command to update stale entries
+- [ ] Export a collection to a single combined markdown file
+- [ ] Merge collections into one
+- [ ] Fuzzy search and ranking improvements
+- [ ] Archive mode to save raw HTML alongside markdown
+- [ ] Migration command to relocate the base directory
+- [ ] Collection pinning and sorting in the web UI
+
+## Development
+
+```bash
+bundle install
+rake test
+rubocop
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/AliOsm/mddir. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/AliOsm/mddir/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/exe/mddir

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "mddir"
+
+Mddir::CLI.start(ARGV)

data/lib/mddir/cli.rb

@@ -0,0 +1,230 @@
+# frozen_string_literal: true
+
+require "thor"
+
+module Mddir
+  module CLIConfig
+    private
+
+    def config
+      @config ||= Config.new
+    end
+  end
+
+  class CollectionCLI < Thor
+    include CLIConfig
+
+    namespace "collection"
+
+    desc "create NAME", "Create a new empty collection"
+    def create(name)
+      collection = Collection.new(name, config)
+
+      if collection.exist?
+        say "Collection '#{collection.name}' already exists"
+        return
+      end
+
+      collection.create!
+      say "Created collection '#{collection.name}'"
+    end
+  end
+
+  class CLI < Thor # rubocop:disable Metrics/ClassLength
+    include CLIConfig
+
+    desc "collection SUBCOMMAND", "Manage collections"
+    subcommand "collection", CollectionCLI
+
+    desc "add COLLECTION URL [URL...]", "Fetch web pages and save to a collection"
+    method_option :cookies, type: :string, desc: "Path to a cookies file"
+    def add(collection_name, *urls)
+      if urls.empty?
+        say_error "Error: provide at least one URL"
+        return
+      end
+
+      collection = Collection.new(collection_name, config)
+      collection.create! unless collection.exist?
+
+      fetcher = Fetcher.new(config, cookies_path: options[:cookies])
+      urls.each { |url| fetch_and_save(url, collection, fetcher) }
+    end
+
+    desc "ls [COLLECTION]", "List collections or entries in a collection"
+    def ls(collection_name = nil)
+      if collection_name
+        list_collection_entries(collection_name)
+      else
+        list_collections
+      end
+    end
+
+    desc "rm COLLECTION [ENTRY]", "Remove a collection or entry"
+    def rm(collection_name, entry_identifier = nil)
+      collection = Collection.new(collection_name, config)
+
+      unless collection.exist?
+        say_error "Error: collection '#{collection.name}' not found"
+        return
+      end
+
+      if entry_identifier
+        remove_entry(collection, entry_identifier)
+      else
+        remove_collection(collection)
+      end
+    end
+
+    desc "search [COLLECTION] QUERY", "Search entries for a query string"
+    def search(*args)
+      if args.empty?
+        say "Usage: mddir search [collection] <query>"
+        return
+      end
+
+      collection_name = args.length >= 2 ? args[0] : nil
+      query = args.length >= 2 ? args[1..].join(" ") : args[0]
+
+      results = Search.new(config).search(query, collection_name:)
+      results.empty? ? say("No matches found") : print_search_results(results)
+    end
+
+    desc "config", "Open configuration file in editor"
+    map "config" => :edit_config
+    def edit_config
+      config.create_default_config!
+      system(config.editor, config.path)
+    end
+
+    desc "reindex", "Rebuild the global index from per-collection indexes"
+    def reindex
+      data = GlobalIndex.update!(config)
+      collection_count = data["collections"]&.length || 0
+      total = data["total_entries"] || 0
+      say "Reindexed #{collection_count} collections, #{total} entries"
+    end
+
+    desc "serve", "Start the web UI server"
+    def serve
+      require_relative "server"
+      Server.start(config)
+    end
+
+    desc "open", "Start the web UI server and open in browser"
+    def open
+      require_relative "server"
+
+      url = "http://localhost:#{config.port}"
+      Thread.new do
+        sleep 1
+        open_browser(url)
+      end
+
+      Server.start(config)
+    end
+
+    private
+
+    def fetch_and_save(url, collection, fetcher)
+      if collection.entries.any? { |entry| entry["url"] == url }
+        say "Skipped (duplicate): #{url}"
+        return
+      end
+
+      entry = fetcher.fetch(url)
+      entry.save_to(collection.path)
+      collection.add_entry(entry.to_index_entry)
+      say "Saved: #{entry.filename} → #{collection.name} (#{entry.conversion})"
+    rescue FetchError, StandardError => e
+      say_error "Error fetching #{url}: #{e.message}"
+    end
+
+    def list_collections
+      collections = GlobalIndex.load(config)["collections"] || {}
+
+      if collections.empty?
+        say "No collections"
+        return
+      end
+
+      collections.each do |name, info|
+        count = info["entry_count"] || 0
+        label = count == 1 ? "entry" : "entries"
+        say "#{name.ljust(16)} | #{count} #{label}"
+      end
+    end
+
+    def list_collection_entries(collection_name)
+      collection = Collection.new(collection_name, config)
+
+      unless collection.exist?
+        say_error "Error: collection '#{collection.name}' not found"
+        return
+      end
+
+      entries = collection.entries
+      count = entries.length
+      label = count == 1 ? "entry" : "entries"
+      say "#{collection.name} (#{count} #{label})\n\n"
+
+      entries.each_with_index { |entry, idx| print_entry(entry, idx + 1) }
+    end
+
+    def print_entry(entry, number)
+      say "  #{number}. #{entry["title"]}"
+      say "     #{entry["description"]}" unless entry["description"].to_s.empty?
+      say "     #{entry["filename"]}"
+      say "     #{entry["url"]}"
+      say
+    end
+
+    def remove_entry(collection, identifier)
+      entry = collection.remove_entry(identifier)
+
+      if entry
+        say "Removed: #{entry["title"]}"
+      else
+        say_error "Error: entry not found"
+      end
+    end
+
+    def remove_collection(collection)
+      count = collection.entry_count
+      label = count == 1 ? "entry" : "entries"
+
+      if yes?("Remove collection '#{collection.name}' with #{count} #{label}? [y/N]")
+        collection.remove!
+        say "Removed collection '#{collection.name}'"
+      else
+        say "Cancelled"
+      end
+    end
+
+    def print_search_results(results)
+      total_matches = results.sum { |result| result.matches.length }
+      say "\nFound #{total_matches} matches in #{results.length} files\n\n"
+
+      results.each { |result| print_search_result(result) }
+    end
+
+    def print_search_result(result)
+      say "[#{result.collection_name}] #{result.entry["title"]}"
+      say "  #{result.entry["filename"]}"
+      say "  #{result.entry["url"]}"
+      result.matches.each { |match| say "  Line #{match.line_number}: #{match.snippet}" }
+      say
+    end
+
+    def open_browser(url)
+      case RUBY_PLATFORM
+      when /darwin/
+        system("open", url)
+      when /linux/
+        system("xdg-open", url)
+      when /mswin|mingw/
+        system("start", url)
+      end
+    end
+  end
+end

data/lib/mddir/collection.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+require "yaml"
+require "fileutils"
+
+module Mddir
+  class Collection
+    attr_reader :name, :config, :path
+
+    def initialize(name, config)
+      @name = Utils.slugify(name)
+      @config = config
+      @path = File.join(config.base_dir, @name)
+    end
+
+    def self.all(config)
+      base = config.base_dir
+      return [] unless Dir.exist?(base)
+
+      Dir.children(base)
+         .select { |child| File.directory?(File.join(base, child)) }
+         .sort
+         .map { |folder| new(folder, config) }
+    end
+
+    def exist?
+      Dir.exist?(@path)
+    end
+
+    def entries
+      load_index
+    end
+
+    def entry_count
+      entries.length
+    end
+
+    def last_added
+      entries_list = entries
+      return nil if entries_list.empty?
+
+      entries_list.map { |entry| entry["saved_at"] }.compact.max
+    end
+
+    def find_entry(identifier)
+      entries_list = entries
+
+      if identifier.match?(/\A\d+\z/)
+        index = identifier.to_i - 1
+        entries_list[index] if index >= 0 && index < entries_list.length
+      else
+        slug = identifier.delete_suffix(".md")
+        entries_list.find { |entry| entry["slug"] == slug }
+      end
+    end
+
+    def index_path
+      File.join(@path, "index.yml")
+    end
+
+    def create!
+      FileUtils.mkdir_p(@path)
+      write_index([]) unless File.exist?(index_path)
+      GlobalIndex.update!(config)
+      self
+    end
+
+    def add_entry(entry_data) # rubocop:disable Naming/PredicateMethod
+      entries_list = entries
+      return false if entries_list.any? { |entry| entry["url"] == entry_data["url"] }
+
+      entries_list << entry_data
+      write_index(entries_list)
+      GlobalIndex.update!(config)
+      true
+    end
+
+    def remove_entry(identifier)
+      entry = find_entry(identifier)
+      return nil unless entry
+
+      file_path = File.join(@path, entry["filename"])
+      FileUtils.rm_f(file_path)
+
+      entries_list = entries.reject { |list_entry| list_entry["filename"] == entry["filename"] }
+      write_index(entries_list)
+      GlobalIndex.update!(config)
+      entry
+    end
+
+    def remove!
+      FileUtils.rm_rf(@path)
+      GlobalIndex.update!(config)
+      SearchIndex.open(config) { |index| index.remove_collection!(name) }
+    end
+
+    private
+
+    def load_index
+      return [] unless File.exist?(index_path)
+
+      data = YAML.safe_load_file(index_path, permitted_classes: [Time])
+      data.is_a?(Array) ? data : []
+    rescue Psych::SyntaxError
+      warn "Warning: corrupted index.yml in collection '#{@name}', treating as empty"
+      []
+    end
+
+    def write_index(entries_list)
+      File.write(index_path, YAML.dump(entries_list))
+    end
+  end
+end

data/lib/mddir/config.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require "yaml"
+
+module Mddir
+  class Config
+    DEFAULT_PATH = File.expand_path("~/.mddir.yml")
+
+    DEFAULTS = {
+      "base_dir" => "~/.mddir",
+      "port" => 7768,
+      "editor" => ENV.fetch("EDITOR", "vi"),
+      "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" # rubocop:disable Layout/LineLength
+    }.freeze
+
+    attr_reader :path
+
+    def initialize(path: DEFAULT_PATH)
+      @path = path
+      @data = DEFAULTS.dup
+      load_config if File.exist?(path)
+    end
+
+    def base_dir
+      File.expand_path(@data["base_dir"])
+    end
+
+    def port
+      @data["port"].to_i
+    end
+
+    def editor
+      @data["editor"]
+    end
+
+    def user_agent
+      @data["user_agent"]
+    end
+
+    def create_default_config!
+      return if File.exist?(path)
+
+      File.write(path, YAML.dump(DEFAULTS))
+    end
+
+    private
+
+    def load_config
+      loaded = YAML.safe_load_file(path)
+      @data.merge!(loaded) if loaded.is_a?(Hash)
+    rescue Psych::SyntaxError
+      # Use defaults if config is corrupted
+    end
+  end
+end

data/lib/mddir/entry.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+require "digest"
+require "time"
+require "yaml"
+
+module Mddir
+  class Entry
+    attr_reader :url,
+                :title,
+                :description,
+                :slug,
+                :filename,
+                :markdown,
+                :conversion,
+                :token_count,
+                :token_estimated,
+                :saved_at
+
+    def initialize(url:, title:, description:, markdown:, conversion:, token_count:, token_estimated:) # rubocop:disable Metrics/ParameterLists
+      @url = url
+      @title = title.to_s.encode("UTF-8", invalid: :replace, undef: :replace)
+      @description = description.to_s.encode("UTF-8", invalid: :replace, undef: :replace)
+      @markdown = markdown.encode("UTF-8", invalid: :replace, undef: :replace)
+      @conversion = conversion
+      @token_count = token_count
+      @token_estimated = token_estimated
+      @saved_at = Time.now.utc.iso8601
+      @slug = generate_slug
+      @filename = "#{@slug}.md"
+    end
+
+    def to_index_entry
+      {
+        "url" => @url,
+        "title" => @title,
+        "description" => @description,
+        "filename" => @filename,
+        "slug" => @slug,
+        "saved_at" => @saved_at,
+        "conversion" => @conversion,
+        "token_count" => @token_count,
+        "token_estimated" => @token_estimated
+      }
+    end
+
+    def to_markdown_with_frontmatter
+      frontmatter = {
+        "url" => @url,
+        "title" => @title,
+        "description" => @description,
+        "slug" => @slug,
+        "saved_at" => @saved_at,
+        "conversion" => @conversion,
+        "token_count" => @token_count,
+        "token_estimated" => @token_estimated
+      }
+
+      content = Utils.strip_frontmatter(@markdown)
+      yaml_str = YAML.dump(frontmatter).delete_prefix("---\n").chomp
+      "---\n#{yaml_str}\n---\n\n#{content}"
+    end
+
+    def save_to(collection_path)
+      file_path = File.join(collection_path, @filename)
+      File.write(file_path, to_markdown_with_frontmatter)
+    end
+
+    private
+
+    def generate_slug
+      title_slug = Utils.slugify(@title.empty? ? "untitled" : @title)
+      hash = Digest::SHA256.hexdigest(@url)[0, 6]
+      "#{title_slug}-#{hash}"
+    end
+  end
+end