automatic 14.12.2 → 26.08

data/doc/QUICKSTART.md

@@ -0,0 +1,256 @@
+# Quick Start
+
+This guide takes four public pages through one short Automatic Ruby pipeline and
+leaves what they publish as one Markdown document. It needs no account, no
+credential, no paid service and no database server.
+
+It does need two of the optional gems — because of the plugins the Recipe
+names, not because of the framework — and installing exactly those is a step of
+this guide rather than a footnote to it. Automatic Ruby installs what the
+framework needs and leaves a plugin's gems to the operator who uses that
+plugin, so "which plugins does this Recipe name, and what do they need" is a
+question every Recipe asks. Skipping it is the usual way a first run stops half
+way through.
+
+## 1. Install
+
+Use Ruby 3.3 through 4.0 on a Unix-like system:
+
+```sh
+ruby -v
+gem install automatic
+automatic --version
+```
+
+Working from a Git checkout instead is this same guide with one step done
+differently; "From a source checkout" at the end is that step.
+
+## 2. Create the user directory
+
+```sh
+automatic scaffold
+```
+
+This creates `~/.automatic` and copies the shipped examples to
+`~/.automatic/config/example`. Existing files and directories are not
+overwritten.
+
+## 3. Write the Recipe
+
+A Recipe is one job: the plugins it runs, in order, with their settings. This
+one reads four public index pages as HTML and makes a feed of the articles each
+lists — which is what to do for a page whose feed you do not have — keeps a
+record of what it has already seen, and appends the rest to a Markdown
+document.
+
+Write it to `~/.automatic/config/web2markdown.yml`:
+
+```yaml
+plugins:
+  - module: CustomFeedWeb
+    config:
+      retry: 2
+      interval: 2
+      sites:
+        # Articles: https://blog.python.org/2026/08/python-3147-31315/
+        - name: Python Insider
+          url: https://blog.python.org/
+          link_selector: 'a[href]'
+          include:
+            - '^https://blog\.python\.org/20[0-9]{2}/[0-9]{2}/[^/]+/?$'
+          same_host: true
+          fetch_items: 20
+
+        # Articles: https://blog.rust-lang.org/2026/08/04/enabling-polonius-alpha-on-nightly/
+        - name: Rust Blog
+          url: https://blog.rust-lang.org/
+          link_selector: 'a[href]'
+          include:
+            - '^https://blog\.rust-lang\.org/20[0-9]{2}/[0-9]{2}/[0-9]{2}/[^/]+/?$'
+          same_host: true
+          fetch_items: 20
+
+        # Articles: https://go.dev/blog/pkgsite-api
+        - name: The Go Blog
+          url: https://go.dev/blog/
+          link_selector: 'a[href]'
+          include:
+            - '^https://go\.dev/blog/[^/]+$'
+          same_host: true
+          fetch_items: 20
+
+        # Alerts: https://www.jpcert.or.jp/at/2026/at260021.html
+        - name: JPCERT/CC Alerts
+          url: https://www.jpcert.or.jp/at/2026.html
+          link_selector: 'a[href]'
+          include:
+            - '^https://www\.jpcert\.or\.jp/at/20[0-9]{2}/at[0-9]+\.html$'
+          same_host: true
+          fetch_items: 20
+
+  - module: StoreDigest
+    config:
+      db: web2markdown.db
+      fields:
+        - title
+        - link
+
+  - module: PublishMarkdown
+    config:
+      file: ~/.automatic/markdown/web.md
+      mode: append
+```
+
+Three plugins, and each hands its result to the next:
+
+- **`CustomFeedWeb`** fetches each page and makes a feed of the article links it
+  lists. `include` is what tells an article from a navigation link, `interval`
+  is the pause between requests, and one run makes one request per site —
+  nothing here follows a link or reads an article body.
+- **`StoreDigest`** records a digest of each item and passes on only the items
+  whose digest it had not recorded already. It is what makes the Recipe safe to
+  run repeatedly. `db` is a **file name**, kept under `~/.automatic/db`.
+- **`PublishMarkdown`** appends what is left to a plain-text document, creating
+  the directory if it is missing. Its `file`, unlike `db`, is a path.
+
+## 4. Install what the Recipe needs
+
+Read the Recipe you have just written, plugin by plugin, and look each one up in
+the table of optional plugin dependencies in [`DEPLOYMENT.md`](DEPLOYMENT.md).
+That table gives, for these three:
+
+- **`CustomFeedWeb`** — `nokogiri`, which it reads the pages with. In a
+  checkout, the group `html`.
+- **`StoreDigest`** — `activerecord` and `sqlite3`, which it keeps its record
+  in. In a checkout, the group `store`.
+- **`PublishMarkdown`** — nothing of its own.
+
+```sh
+gem install nokogiri
+gem install activerecord sqlite3
+```
+
+**Install what the whole Recipe needs, not what its first plugin needs.** A
+plugin is loaded when the pipeline reaches it, so installing only `nokogiri`
+would let `CustomFeedWeb` fetch its pages, hand its feeds on, and stop the run
+where the next plugin is loaded:
+
+```text
+automatic: The `activerecord` gem is not installed. It is needed by the store
+plugins. Install it with `gem install activerecord`, or in a source checkout add
+its group to the bundle; see the optional plugin dependencies in
+doc/DEPLOYMENT.md. (cannot load such file -- active_record)
+```
+
+`PublishMarkdown` is the entry worth reading twice: it needs no gem of its own,
+using an HTML parser where one is installed and its own substitution where none
+is. A plugin's row in that table is the answer, not a guess from what the
+plugin does.
+
+`nokogiri` and `sqlite3` build a native extension where no binary package
+matches your platform. Install a build environment only if one of them says it
+needs one.
+
+## 5. Run it
+
+```sh
+automatic -c ~/.automatic/config/web2markdown.yml
+```
+
+A bare name is resolved inside `~/.automatic/config`, so `automatic -c
+web2markdown.yml` is the same command. The log names each page as it is fetched,
+each digest as it is saved, and the document as it is written.
+
+## 6. Read the result
+
+```sh
+sed -n '1,80p' ~/.automatic/markdown/web.md
+```
+
+Each item is a level-2 heading followed by the metadata the feed carries:
+
+```markdown
+## Extending the pkgsite API
+
+- Link: <https://go.dev/blog/pkgsite-api>
+- Date: 2026-08-17 09:00:00 +0900
+```
+
+## 7. Run it again
+
+```sh
+automatic -c ~/.automatic/config/web2markdown.yml
+```
+
+The second run appends nothing: the pages still list the same articles, and
+`StoreDigest` has the digest of every one of them. An article published between
+the two runs is the one thing that would be added — which is what makes this
+Recipe safe to put in `cron`, and what to check before any Recipe with an
+effect.
+
+If instead everything is appended a second time, the store plugin is writing
+somewhere other than where you think. The `Using Database:` line of the log
+names the file it opened.
+
+## 8. Run it from cron
+
+Create the log directory once, then use the absolute path reported by
+`command -v automatic`:
+
+```sh
+mkdir -p ~/.automatic/log
+command -v automatic
+```
+
+```crontab
+0 * * * * /usr/local/bin/automatic -c $HOME/.automatic/config/web2markdown.yml >> $HOME/.automatic/log/web2markdown.log 2>&1
+```
+
+Automatic Ruby runs once and exits; `cron` supplies the schedule.
+
+## From a source checkout
+
+The normal installation above remains the quickest way to use Automatic Ruby. To
+try the development version or change the source, first follow
+[README's checkout setup](../README.md#from-a-checkout). Every `automatic` above
+then becomes `bundle exec bin/automatic`, run from the checkout directory:
+
+```sh
+cd ~/automaticruby
+bundle exec bin/automatic scaffold
+bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml
+```
+
+Step 4 is the step that differs, because a checkout resolves its gems through
+Bundler rather than through RubyGems. Each optional gem is in a Bundler group,
+and the Recipe's groups — `html` and `store`, from step 4 — are selected
+together and installed once:
+
+```sh
+bundle config set --local with "html store"
+bundle install
+```
+
+`gem install nokogiri` does **not** work here: the gem installs, and the
+checkout still reports it as missing, because it is not in the bundle. That
+difference, the commands that show what the bundle holds, and the same three
+plugins taken step by step through choosing their groups are in
+[`DEPLOYMENT.md`](DEPLOYMENT.md) under "Working out what a Recipe needs, in a
+checkout".
+
+## Next
+
+The Recipe is ordinary YAML. Add a site, change the Markdown path, or put
+`PublishConsoleLink` at the end to see what the pipeline holds without writing
+anything. The shipped `feed2markdown.yml` and `feed2console.yml` in
+`~/.automatic/config/example` are the same shape over an ordinary feed, for a
+site that publishes one.
+
+Taking this same pipeline further — the article bodies, one joined text, and an
+AI service asked one question about it — is
+[`AI_TUTORIAL.md`](AI_TUTORIAL.md).
+
+What each plugin does is [`PLUGINS.md`](PLUGINS.md) section 6; installing,
+scheduling and operating a Recipe is [`DEPLOYMENT.md`](DEPLOYMENT.md); writing a
+small plugin of your own is
+[`PLUGIN_DEVELOPMENT.md`](PLUGIN_DEVELOPMENT.md).

data/doc/RELEASING.md

@@ -0,0 +1,381 @@
+# Releasing Automatic Ruby
+
+This is the maintainer runbook for building and publishing the `automatic` gem.
+It describes a manual release. Building and inspecting a gem does not publish
+it; `gem push` does, and must be run only for an explicitly approved release.
+
+The governing principles are:
+
+- Build locally.
+- Verify before publishing.
+- Keep credentials out of the repository.
+- Publish only an explicitly approved release.
+
+Do not restore the historical Jeweler tasks. `automatic.gemspec` is maintained
+by hand, `Rakefile` contains test tasks only, and RubyGems builds the package
+directly from the gemspec.
+
+## 1. Prerequisites
+
+The releaser needs:
+
+- a supported Ruby, currently Ruby 3.3 through 4.0;
+- current Bundler and RubyGems versions compatible with that Ruby;
+- write access to the source repository and permission to create and push tags;
+- ownership of `automatic` on RubyGems.org, or an API key with push permission;
+- access to the RubyGems.org account and its required MFA method.
+
+Check the local tools before changing release metadata:
+
+```sh
+ruby -v
+gem --version
+bundle --version
+```
+
+The release Ruby need not reproduce the complete CI matrix locally. Required
+GitHub Actions checks must be green on Ruby 3.3, 3.4 and 4.0 before publication.
+
+## 2. Authentication and credentials
+
+RubyGems authenticates publishing operations with an API key. Confirm the
+account is an owner of the package before release; gem ownership grants the
+ability to push versions and yank releases. Use the least privilege needed for
+the release and follow the current RubyGems.org account and API-key guidance.
+
+The standard RubyGems sign-in command stores a key in the RubyGems credentials
+file:
+
+```sh
+gem signin
+```
+
+RubyGems commonly uses `~/.gem/credentials`; with the XDG directory layout it
+may report another user-specific path. Use the path reported by the installed
+RubyGems. The file contains secrets and must be readable and writable only by
+its owner:
+
+```sh
+chmod 0600 ~/.gem/credentials
+```
+
+Never put an API key, password, OTP or credentials file in this repository, the
+Gemfile, the gemspec, a shell script, README, a commit or a GitHub artifact. Do
+not show a real key in an example. Do not pass a key as a literal command-line
+argument, because process listings and shell history may retain it.
+
+The gemspec sets `rubygems_mfa_required` to `true`. If RubyGems.org requires
+MFA, complete the interactive prompt from `gem signin` or `gem push`. RubyGems
+also supports `--otp CODE` and the short-lived `GEM_HOST_OTP_CODE` environment
+variable. Do not store an OTP in the repository or a persistent script.
+
+Before the release window, verify access through the RubyGems.org web account.
+Do not change owners or create, rotate or revoke production keys as an
+incidental part of this runbook.
+
+## 3. Prepare the source version
+
+Start from a clean checkout of the release branch and fetch the authoritative
+remote. Review the complete release diff and the top entry in `doc/VERSIONS`.
+
+Automatic Ruby uses a calendar version taken from the release date:
+
+```text
+X.YY        release in year X, month YY
+X.YY.PATCH  correction to an earlier release in the same month
+```
+
+Section 10 of `POLICY.md` is authoritative. A release updates these three
+places together in one release-metadata commit:
+
+- `VERSION` contains the package version;
+- `lib/automatic/version.rb` defines `Automatic::VERSION` for the CLI;
+- the current heading in `doc/VERSIONS` contains the same version.
+
+`automatic.gemspec` does not contain another version literal. It reads and
+strips the repository-root `VERSION` file. Confirm all three values agree:
+
+```sh
+version=$(cat VERSION)
+ruby -Ilib -e "require 'automatic/version'; abort unless Automatic::VERSION == '$version'"
+ruby -e 'spec = Gem::Specification.load("automatic.gemspec")
+  abort unless spec.version.to_s == File.read("VERSION").strip'
+```
+
+Finalize the current unreleased `doc/VERSIONS` entry as a release summary, not
+a development diary. Change `(Release Date: TBD)` to the actual release date.
+Do not add a bullet merely for finalizing the date. Remove or consolidate an
+unreleased bullet only when it does not describe the source being published.
+
+## 4. Run release checks
+
+Install the default development bundle and run the same default suite used by
+the repository:
+
+```sh
+bundle install
+bundle exec rake
+bundle exec ruby -Ilib -e "require 'automatic'"
+bundle exec bin/automatic --version
+bundle exec bin/automatic --help
+```
+
+Also confirm that all required GitHub Actions checks are green for the release
+commit. Do not publish from an uncommitted or dirty tree:
+
+```sh
+git status --short
+```
+
+## 5. Build the gem
+
+Build from the hand-maintained gemspec at the repository root:
+
+```sh
+gem build automatic.gemspec
+```
+
+This creates `automatic-X.Y.Z.gem` in the current directory. Here and below,
+`X.Y.Z` means the exact value in `VERSION`; the project version may have two or
+three components. The generated archive is ignored by Git and is not source.
+Never commit it.
+
+For copy-and-paste-safe inspection commands, capture its exact name:
+
+```sh
+version=$(cat VERSION)
+package="automatic-${version}.gem"
+test -f "$package"
+```
+
+## 6. Inspect package contents
+
+First inspect the archive's recorded file list without installing it:
+
+```sh
+gem specification "$package" files
+```
+
+For a filesystem view, unpack it into a fresh temporary directory:
+
+```sh
+inspect_dir=$(mktemp -d)
+gem unpack "$package" --target "$inspect_dir"
+find "$inspect_dir/automatic-$version" -type f | sort
+```
+
+Confirm that the package contains, at minimum:
+
+- `lib/` and `lib/automatic/version.rb`;
+- `bin/automatic`, recorded as the `automatic` executable;
+- the maintained files under `plugins/`;
+- the shipped configuration and assets;
+- maintained documentation, including `README.md` and `doc/RELEASING.md`;
+- `doc/LICENSE.md`, `doc/COPYING` and `doc/COPYING.LESSER`;
+- `VERSION` and `automatic.gemspec`.
+
+Confirm that it does not contain:
+
+- a generated `.gem` archive or `pkg/` output;
+- `.git`, `.github`, `.bundle`, `Gemfile.lock` or local Bundler directories;
+- `Gemfile`, `Rakefile`, `script/`, `spec/`, `test/` or `vendor/`;
+- credentials, databases, editor files, temporary files or coverage output;
+- documents deleted or superseded in the maintained source tree.
+
+The gemspec deliberately derives its list from files visible to Git, then
+filters development and generated paths. Inspect every release archive anyway:
+the build result, rather than the intended filter, is what would be published.
+
+## 7. Inspect package metadata
+
+Read metadata from the built archive, not only from the source gemspec:
+
+```sh
+gem specification "$package" name
+gem specification "$package" version
+gem specification "$package" summary
+gem specification "$package" homepage
+gem specification "$package" metadata
+gem specification "$package" required_ruby_version
+gem specification "$package" licenses
+gem specification "$package" authors
+gem specification "$package" dependencies
+```
+
+Confirm that:
+
+- the name is `automatic` and the version equals `VERSION`;
+- the summary and homepage describe this project;
+- `source_code_uri` is the Automatic Ruby repository;
+- the required Ruby version is `>= 3.3.0`;
+- licenses contain both `GPL-3.0-only` and `LGPL-3.0-only`;
+- the authors are correct;
+- runtime and development dependencies match `automatic.gemspec`, and the
+  runtime ones are the framework's own — no gem that belongs to a plugin has
+  found its way in ([`POLICY.md`](POLICY.md) section 9.1);
+- `rubygems_mfa_required` is `true`.
+
+## 8. Test an isolated local installation
+
+Install the archive into a temporary gem home so the smoke test does not load
+the checkout's `lib/` directory. Dependency downloads may require network
+access if they are not already cached.
+
+```sh
+gem_home=$(mktemp -d)
+env -u RUBYLIB GEM_HOME="$gem_home" GEM_PATH="$gem_home" \
+  gem install "./$package" --no-document
+env -u RUBYLIB GEM_HOME="$gem_home" GEM_PATH="$gem_home" \
+  "$gem_home/bin/automatic" --version
+env -u RUBYLIB GEM_HOME="$gem_home" GEM_PATH="$gem_home" \
+  "$gem_home/bin/automatic" --help
+```
+
+Confirm the installed version equals `VERSION`. Then run a Recipe that needs no
+network and no optional gem, with a temporary home:
+
+```sh
+smoke_home=$(mktemp -d)
+mkdir -p "$smoke_home/.automatic/config"
+cat >"$smoke_home/.automatic/config/smoke.yml" <<'YAML'
+plugins:
+  - module: SubscriptionText
+    config:
+      feeds:
+        - title: release smoke test
+          url: https://example.com/
+  - module: PublishConsoleLink
+YAML
+env -u RUBYLIB HOME="$smoke_home" GEM_HOME="$gem_home" GEM_PATH="$gem_home" \
+  "$gem_home/bin/automatic" -c smoke.yml
+```
+
+The command must print the example link without loading a file from the source
+checkout. Remove the temporary directories when inspection is complete:
+
+```sh
+rm -rf "$inspect_dir" "$gem_home" "$smoke_home"
+```
+
+## 9. Prepare the release commit and tag
+
+After tests and package checks pass, commit the finalized release metadata.
+Wait for required CI on that exact commit. Create an annotated tag named with
+the version prefixed by `v`, as required by the existing version policy:
+
+```sh
+version=$(cat VERSION)
+git tag -a "v$version" -m "Release $version"
+git push origin HEAD
+git push origin "v$version"
+```
+
+The tag identifies the source used to build the gem. Do not move or force-update
+a release tag after publication. If the tag is wrong before publication, stop
+and correct it openly. If publication has happened, preserve the source history
+and make the correction in a new commit and version.
+
+The repository has no established requirement for a GitHub Release separate
+from its Git tag. If maintainers create one, create it from the same immutable
+tag and use the finalized `doc/VERSIONS` entry as its notes. The tag records the
+source; the RubyGems version records the published package. Neither replaces
+the other.
+
+## 10. Publish to RubyGems.org
+
+Reconfirm approval, the package name and the version immediately before this
+step. The following command performs the real publication to RubyGems.org:
+
+```sh
+gem push "automatic-X.Y.Z.gem"
+```
+
+Replace the placeholder with the archive already inspected. **Running this
+command actually publishes the gem.** Do not run it for a dry run, a test, or
+merely because the build succeeded. Complete any MFA prompt using the approved
+RubyGems.org account.
+
+RubyGems.org does not provide an overwrite operation for an existing name,
+version and platform. A published version is immutable: never rebuild the same
+version and expect a second push to replace it.
+
+## 11. Verify publication
+
+Open <https://rubygems.org/gems/automatic> and confirm the new version appears.
+Verify its version, both licenses, required Ruby version, dependency metadata,
+homepage and source link against the archive inspected above.
+
+Allow for normal index propagation, then install from RubyGems.org into another
+fresh gem home rather than reusing the local-package test:
+
+```sh
+verify_home=$(mktemp -d)
+version=X.Y.Z
+GEM_HOME="$verify_home" GEM_PATH="$verify_home" \
+  gem install automatic -v "$version" --no-document
+GEM_HOME="$verify_home" GEM_PATH="$verify_home" \
+  "$verify_home/bin/automatic" --version
+GEM_HOME="$verify_home" GEM_PATH="$verify_home" \
+  "$verify_home/bin/automatic" --help
+rm -rf "$verify_home"
+```
+
+The installed version must equal the tag, `VERSION`, `Automatic::VERSION`, the
+`doc/VERSIONS` heading and the version shown by RubyGems.org.
+
+## 12. Publication failures and corrections
+
+Stop on an error and diagnose it; do not bypass RubyGems security controls.
+
+- **Authentication failure:** run `gem signin` for the correct account, check
+  the credentials path and permissions, and verify that the key is active and
+  has push scope. Never paste the key into source or a committed script.
+- **MFA failure:** use the current OTP for the account and retry only the failed
+  push. Check the account's MFA configuration rather than disabling MFA.
+- **Ownership failure:** have an existing owner grant the intended account
+  ownership or an appropriately scoped key. Do not publish under another name.
+- **Version already exists:** do not attempt to overwrite it. Determine whether
+  it was already published correctly; otherwise fix the source, increment the
+  version according to `POLICY.md`, rebuild and repeat every verification step.
+- **Malformed metadata or rejected gem:** fix the gemspec and release metadata,
+  increment the version if that version reached RubyGems.org, then rebuild,
+  inspect and test the new archive. Do not use `--force` to hide validation.
+
+`gem yank automatic -v X.Y.Z` removes a published version from normal index use.
+Yank only for an exceptional serious mispublication, such as exposed secrets or
+a release that must not be installed. It is not the normal correction process,
+does not make the version reusable, and does not erase every downloaded copy.
+The normal response is to correct the problem, bump the version and publish a
+new release. Rotate any exposed secret immediately as well as yanking.
+
+Publication state and Git history are separate. Never rewrite a published
+commit, force-push the release branch, or move a release tag to simulate a
+rollback. Preserve a traceable record and publish a corrected version.
+
+Future automation may use a publishing mechanism currently supported by
+RubyGems.org, but introducing automatic publication or trusted publishing is a
+separate, explicitly reviewed change.
+
+## Official RubyGems references
+
+Recheck these official references when preparing a release, because account and
+security behavior can change independently of this repository:
+
+- [Publishing your gem](https://guides.rubygems.org/publishing/)
+- [RubyGems command reference](https://guides.rubygems.org/command-reference/)
+- [API key scopes](https://guides.rubygems.org/api-key-scopes/)
+- [MFA requirement opt-in](https://guides.rubygems.org/mfa-requirement-opt-in/)
+- [Removing a published gem](https://guides.rubygems.org/removing-a-published-gem/)
+
+## Quick checklist
+
+- [ ] `VERSION`, `Automatic::VERSION` and the `doc/VERSIONS` heading agree.
+- [ ] The release date and release summary are finalized.
+- [ ] The working tree is clean and required CI is green.
+- [ ] `bundle exec rake` passes and the gem builds.
+- [ ] Package contents and metadata have been inspected.
+- [ ] GPLv3/LGPLv3, Ruby version and dependencies are correct.
+- [ ] The isolated local install and offline smoke test pass.
+- [ ] The release commit and immutable `vX.Y.Z` tag identify the built source.
+- [ ] Publication has explicit approval and `gem push` succeeds.
+- [ ] RubyGems.org metadata and a clean remote install are verified.