webctx 0.1.0

package/AGENTS.md

@@ -0,0 +1,84 @@
+# AGENTS.md
+
+Guidance for coding agents working in `webctx`.
+
+## Purpose
+
+This repo contains the pure Go port of the `webctx` CLI.
+
+It is no longer a generic starter template. Treat the current CLI behavior as the source of truth unless the user explicitly asks to change it.
+
+## Architecture
+
+- `cmd/webctx/main.go`: process entrypoint, exit-code based dispatch.
+- `internal/app/app.go`: CLI parsing and top-level command routing.
+- `internal/app/tools.go`: search provider clients, ranking, formatting, and HTTP helpers.
+- `internal/app/scrape.go`: GitHub raw-content optimization, `.md` fetch path, Firecrawl queue, and env loading.
+- `internal/app/app_test.go`: unit tests for CLI behavior and core helpers.
+- `bin/webctx.js`: npm shim that invokes the packaged native binary.
+- `scripts/postinstall.js`: downloads release binary on install, falls back to `go build`.
+- `.github/workflows/release.yml`: tag-driven release pipeline.
+- `docs/porting-status.md`: progress log and remaining work for future agents.
+
+## Local commands
+
+Use `make` targets:
+
+- `make fmt`
+- `make test`
+- `make vet`
+- `make lint`
+- `make check`
+- `make build`
+- `make build-all`
+- `make install-local`
+
+Direct commands:
+
+- `go test ./...`
+- `go vet ./...`
+- `npm run lint`
+
+## Current CLI contract
+
+Preserve these commands unless the user explicitly asks to change them:
+
+- `webctx search <query> [--exclude domain1,domain2] [--keyword phrase]`
+- `webctx read-link <url>`
+- `webctx map-site <url>`
+- `webctx --version`
+
+Behavioral expectations:
+
+- `search` combines Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring.
+- `read-link` keeps the current GitHub raw-content fast path, `.md` fast path, and Firecrawl fallback settings.
+- `map-site` keeps the current Firecrawl map request settings.
+- The CLI should remain agent-friendly and emit plain markdown/text output.
+
+## How to change things safely
+
+1. Keep binary naming convention unchanged unless you also update postinstall/workflow:
+- release assets: `<cli>_<goos>_<goarch>[.exe]`
+- npm-installed binary path: `bin/<cli>-bin` (or `.exe` on Windows)
+
+2. If changing search behavior, compare against the TypeScript porting notes in `docs/porting-status.md` first.
+
+3. If adding dependencies, commit `go.sum` and make sure the workflow still passes on a clean checkout.
+
+4. If you change release artifacts or version plumbing, update `Makefile`, `.github/workflows/release.yml`, and `scripts/postinstall.js` together.
+
+## Release contract
+
+Release pipeline triggers on `v*` tags and expects:
+
+- `NPM_TOKEN` GitHub secret present.
+- npm package name in `package.json` is publishable under your account/org.
+- repository URL matches the release origin used by `scripts/postinstall.js`.
+
+Release binaries should embed the tagged version into `internal/buildinfo.Version` so `webctx --version` matches the release tag.
+
+## Guardrails
+
+- Prefer additive changes and keep the CLI output stable.
+- Do not silently change Firecrawl request settings unless the user explicitly wants behavioral changes.
+- Do not reintroduce MCP/server code unless requested; this repo is intentionally CLI-only.

package/CONTRIBUTORS.md

@@ -0,0 +1,93 @@
+# CONTRIBUTORS.md
+
+Maintainer notes for `webctx`.
+
+## Prerequisites
+
+- Go `1.26+`
+- Node `18+`
+- npm account with publish rights for the package name in `package.json`
+- GitHub repo admin access
+
+## Local development
+
+```bash
+make check
+make build
+./dist/webctx --help
+```
+
+Example command checks:
+
+```bash
+./dist/webctx --version
+./dist/webctx search "golang http client"
+./dist/webctx read-link https://github.com/amxv/webctx-ts/blob/main/cli.ts
+./dist/webctx map-site https://example.com
+```
+
+Install command locally:
+
+```bash
+make install-local
+webctx --help
+```
+
+## Release process
+
+1. Ensure `main` is green:
+
+```bash
+make check
+```
+
+2. Confirm the release workflow is targeting `webctx` and that `package.json` still points to the correct GitHub repository.
+
+3. Prepare release tag:
+
+```bash
+make release-tag VERSION=0.1.0
+```
+
+4. GitHub Actions `release` workflow runs automatically:
+- quality checks
+- cross-platform binary build
+- GitHub release publish
+- npm publish
+
+## Required GitHub secret
+
+- `NPM_TOKEN`: npm automation token with publish rights for your package.
+
+Set via GitHub CLI:
+
+```bash
+gh secret set NPM_TOKEN --repo amxv/webctx
+```
+
+## npm token setup
+
+Create token at npm:
+
+- Profile -> Access Tokens -> Create New Token
+- Use an automation/granular token scoped to required package/org
+
+Validate auth locally:
+
+```bash
+npm whoami
+```
+
+## Notes on package naming
+
+`webctx` is already configured. If you ever rename or move the package, update all of the following together:
+
+- `package.json`
+- `bin/webctx.js`
+- `scripts/postinstall.js`
+- `.github/workflows/release.yml`
+- `Makefile`
+
+## Porting reference
+
+The repo includes `docs/porting-status.md` as the running reference for what was ported from `webctx-ts`, what was intentionally excluded, and what future agents should verify before making behavior changes.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 amxv
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/Makefile

@@ -0,0 +1,69 @@
+SHELL := /bin/bash
+
+GO ?= go
+GOFMT ?= gofmt
+BIN_NAME ?= webctx
+CMD_PATH ?= ./cmd/$(BIN_NAME)
+DIST_DIR ?= dist
+BIN_PATH ?= $(DIST_DIR)/$(BIN_NAME)
+VERSION ?= $(shell node -p "require('./package.json').version" 2>/dev/null)
+LDFLAGS ?= -s -w -X github.com/amxv/webctx/internal/buildinfo.Version=$(if $(VERSION),$(VERSION),dev)
+
+.PHONY: help fmt test vet lint check build build-all install-local clean release-tag
+
+help:
+	@echo "webctx command runner"
+	@echo ""
+	@echo "Targets:"
+	@echo "  make fmt          - format Go files"
+	@echo "  make test         - run go test ./..."
+	@echo "  make vet          - run go vet ./..."
+	@echo "  make lint         - run Node script checks"
+	@echo "  make check        - fmt + test + vet + lint"
+	@echo "  make build        - build local binary to dist/webctx"
+	@echo "  make build-all    - build release binaries for 5 target platforms"
+	@echo "  make install-local - install CLI to ~/.local/bin/webctx"
+	@echo "  make clean        - remove dist artifacts"
+	@echo "  make release-tag  - create and push git tag (requires VERSION=x.y.z)"
+
+fmt:
+	@$(GOFMT) -w $$(find . -type f -name '*.go' -not -path './dist/*')
+
+test:
+	@$(GO) test ./...
+
+vet:
+	@$(GO) vet ./...
+
+lint:
+	@npm run lint
+
+check: fmt test vet lint
+
+build:
+	@mkdir -p $(DIST_DIR)
+	@$(GO) build -trimpath -ldflags="$(LDFLAGS)" -o $(BIN_PATH) $(CMD_PATH)
+
+build-all:
+	@mkdir -p $(DIST_DIR)
+	@for target in "darwin amd64" "darwin arm64" "linux amd64" "linux arm64" "windows amd64"; do \
+		set -- $$target; \
+		GOOS=$$1; GOARCH=$$2; \
+		EXT=""; \
+		if [ "$$GOOS" = "windows" ]; then EXT=".exe"; fi; \
+		echo "Building $(BIN_NAME) for $$GOOS/$$GOARCH"; \
+		CGO_ENABLED=0 GOOS=$$GOOS GOARCH=$$GOARCH $(GO) build -trimpath -ldflags="$(LDFLAGS)" -o "$(DIST_DIR)/$(BIN_NAME)_$${GOOS}_$${GOARCH}$${EXT}" $(CMD_PATH); \
+	done
+
+install-local: build
+	@mkdir -p $$HOME/.local/bin
+	@install -m 755 $(BIN_PATH) $$HOME/.local/bin/$(BIN_NAME)
+	@echo "Installed $(BIN_NAME) to $$HOME/.local/bin/$(BIN_NAME)"
+
+clean:
+	@rm -rf $(DIST_DIR)
+
+release-tag:
+	@test -n "$(VERSION)" || (echo "Usage: make release-tag VERSION=x.y.z" && exit 1)
+	@git tag "v$(VERSION)"
+	@git push origin "v$(VERSION)"

package/README.md

@@ -0,0 +1,95 @@
+# webctx
+
+`webctx` is a pure Go CLI for agent-friendly web search and page extraction.
+
+## What it does
+
+- `search`: combines Brave, Tavily, and Exa search results, deduplicates them, and re-ranks them
+- `read-link`: returns clean markdown for a single URL using a GitHub raw-content path, a `.md` fast path, and Firecrawl scraping fallback
+- `map-site`: returns a sitemap-style list of URLs and metadata from Firecrawl
+
+## Install
+
+Global npm install:
+
+```bash
+npm i -g webctx
+webctx --help
+```
+
+Build from source:
+
+```bash
+git clone https://github.com/amxv/webctx.git
+cd webctx
+make build
+./dist/webctx --help
+```
+
+## Commands
+
+```bash
+webctx --help
+webctx --version
+webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
+webctx read-link <url>
+webctx map-site <url>
+```
+
+Examples:
+
+```bash
+webctx search "next.js server components"
+webctx search "react hooks" --exclude youtube.com,vimeo.com
+webctx search "drizzle orm" --keyword "migration guide"
+webctx read-link https://docs.example.com/guide
+webctx map-site https://example.com
+```
+
+## Environment variables
+
+The CLI loads `.env.local` when present and reads provider credentials from the environment.
+
+Quick start:
+
+```bash
+cp .env.local.example .env.local
+```
+
+Required by command:
+
+- `search`
+  - `BRAVE_API_KEY`
+  - `TAVILY_API_KEY`
+  - `EXA_API_KEY`
+- `read-link`
+  - `FIRECRAWL_API_KEY` for non-GitHub / non-`.md` URLs
+- `map-site`
+  - `FIRECRAWL_API_KEY`
+
+## Release and distribution
+
+This repo publishes in two ways:
+
+- GitHub Releases for native binaries
+- npm for `npm i -g webctx`
+
+The release workflow triggers on `v*` tags and does the following:
+
+1. runs Go and Node quality checks
+2. builds cross-platform binaries
+3. creates a GitHub Release with those assets
+4. publishes the npm package using the tag version
+
+## Project layout
+
+- `cmd/webctx/main.go`: CLI entrypoint
+- `internal/app/`: CLI parsing, search, ranking, scrape, and Firecrawl queue logic
+- `internal/buildinfo/`: build-time version plumbing for `--version`
+- `bin/webctx.js`: npm shim that invokes the packaged native binary
+- `scripts/postinstall.js`: downloads the release binary on install and falls back to local `go build`
+- `.github/workflows/release.yml`: tag-driven release pipeline
+- `AGENTS.md`: guidance for coding agents
+- `CONTRIBUTORS.md`: maintainer/release notes
+
+See `AGENTS.md` and `CONTRIBUTORS.md` for repo-specific implementation and maintenance details.

package/bin/webctx.js

@@ -0,0 +1,28 @@
+#!/usr/bin/env node
+
+const fs = require("node:fs");
+const path = require("node:path");
+const { spawnSync } = require("node:child_process");
+
+const pkg = require("../package.json");
+const cliName = pkg.config?.cliBinaryName || "webctx";
+const executableName = process.platform === "win32" ? `${cliName}.exe` : `${cliName}-bin`;
+const executablePath = path.join(__dirname, executableName);
+
+if (!fs.existsSync(executablePath)) {
+  console.error(`${cliName} binary is not installed. Re-run: npm rebuild -g ${pkg.name}`);
+  process.exit(1);
+}
+
+const child = spawnSync(executablePath, process.argv.slice(2), { stdio: "inherit" });
+
+if (child.error) {
+  console.error(child.error.message);
+  process.exit(1);
+}
+
+if (child.signal) {
+  process.kill(process.pid, child.signal);
+}
+
+process.exit(child.status ?? 1);

package/cmd/webctx/main.go

@@ -0,0 +1,11 @@
+package main
+
+import (
+	"os"
+
+	"github.com/amxv/webctx/internal/app"
+)
+
+func main() {
+	os.Exit(app.Run(os.Args[1:], os.Stdout, os.Stderr))
+}

package/docs/porting-status.md

@@ -0,0 +1,173 @@
+# webctx TypeScript -> Go porting status
+
+This document is the handoff reference for future agents working on `amxv/webctx`.
+
+## Goal
+
+Port the CLI behavior from `amxv/webctx-ts` into pure Go while keeping the command-line interface and provider behavior effectively one-to-one for the CLI use case.
+
+Scope intentionally excludes the MCP/server/dashboard pieces from the TypeScript repo.
+
+## Source areas reviewed in `webctx-ts`
+
+- `cli.ts`
+- `tools/search.ts`
+- `tools/read-link.ts`
+- `tools/map-site.ts`
+- `lib/search/brave.ts`
+- `lib/search/tavily.ts`
+- `lib/search/exa.ts`
+- `lib/ranking.ts`
+- `lib/utils.ts`
+- `lib/scraping.ts`
+- `lib/firecrawl-queue.ts`
+- `lib/rate-limiter.ts`
+
+## Completed
+
+### CLI surface
+
+Implemented in Go:
+
+- `webctx search <query> [--exclude domain1,domain2] [--keyword phrase]`
+- `webctx read-link <url>`
+- `webctx map-site <url>`
+- `webctx --help`
+- `webctx --version`
+
+Notes:
+
+- `--version` now prints the bare version string, matching the TypeScript CLI.
+- Error handling is exit-code based in Go rather than promise rejection based.
+
+### Search port
+
+Implemented:
+
+- Brave HTTP client
+- Tavily HTTP client using direct HTTP API instead of the TypeScript SDK
+- Exa HTTP client
+- provider fan-out with per-provider timeout
+- duplicate-aware reranking
+- excluded-domain filtering
+- HTML entity decoding
+- keyword truncation to 5 words for Exa include-text mode
+- top 35 result output limit
+
+Current behavior matches the TypeScript CLI design:
+
+- normal search mode queries Brave + Tavily + Exa
+- keyword mode queries Exa only
+- user/domain exclusions are applied after provider collection, matching the TypeScript tool flow
+
+### Read-link port
+
+Implemented:
+
+- GitHub raw-content fast path
+- `.md` fast path with HEAD probe
+- Firecrawl scrape fallback with the same agent-oriented request settings
+- PDF parser enablement for `.pdf` URLs
+
+Kept settings aligned with the TypeScript CLI:
+
+- `formats: ["markdown"]`
+- `onlyMainContent: true`
+- `skipTlsVerification: true`
+- `blockAds: true`
+- `removeBase64Images: true`
+- `maxAge: 600000`
+- same excluded tags list
+
+### Map-site port
+
+Implemented with the same Firecrawl map settings:
+
+- `sitemap: "include"`
+- `includeSubdomains: true`
+- `ignoreQueryParameters: true`
+- `limit: 5000`
+
+### Firecrawl queue / rate limiting
+
+Implemented in Go:
+
+- singleton-style queue wrapper
+- token bucket limiter at 10 requests/minute
+- serialized queue processing for Firecrawl operations
+
+This is not a literal line-by-line port, but preserves the same operational intent.
+
+### Release/publish setup
+
+Updated to be release-ready for the real `webctx` CLI:
+
+- GitHub Actions workflow now builds `webctx` instead of the old template placeholder
+- release binaries embed the tagged version into `internal/buildinfo.Version`
+- npm metadata now describes the actual CLI instead of the template
+- README / agent / maintainer docs updated for the real repo
+
+## Intentionally not ported
+
+- MCP/server behavior
+- Next.js app/dashboard code
+- database/logging layers unrelated to the CLI
+
+These can be added later only if explicitly requested.
+
+## Current repo files of interest
+
+- `cmd/webctx/main.go`
+- `internal/app/app.go`
+- `internal/app/tools.go`
+- `internal/app/scrape.go`
+- `internal/app/app_test.go`
+- `.github/workflows/release.yml`
+- `scripts/postinstall.js`
+- `README.md`
+
+## Verification already completed
+
+- `go test ./...`
+- `go build ./cmd/webctx`
+
+## Live validation notes
+
+Live CLI validation was run against a real `.env.local` on the Sprite machine.
+
+Confirmed working live:
+
+- combined `search` path returns real web results
+- public GitHub blob `read-link` fast path works
+- Firecrawl-backed `read-link` works
+- Firecrawl-backed `map-site` works
+
+Observed external/provider constraints during live validation:
+
+- `search --keyword` currently depends on Exa-only results and could not be fully validated because the live Exa account returned `NO_MORE_CREDITS`
+- private GitHub blob URLs are not readable via unauthenticated raw-content fetch, so they fall through to the general scrape path
+
+These findings were from live provider behavior, not from compile/test failures in the Go port.
+
+## Good next checks for future agents
+
+1. Run live end-to-end checks against real provider keys for:
+   - normal multi-provider search
+   - Exa keyword-only search mode
+   - GitHub raw-content read-link
+   - `.md` fast path read-link
+   - Firecrawl scrape fallback
+   - Firecrawl map-site
+
+2. Compare a handful of live outputs from `webctx-ts` and Go `webctx` for formatting parity.
+
+3. If performance tuning is needed, focus on:
+   - HTTP client reuse
+   - provider timeout tuning
+   - Firecrawl queue behavior under concurrent use
+
+## Constraints to preserve
+
+- Keep the CLI output simple and agent-friendly.
+- Keep the Firecrawl request settings stable unless explicitly asked to change them.
+- Keep the release asset naming contract stable unless postinstall/workflow are updated together.

package/go.mod

@@ -0,0 +1,3 @@
+module github.com/amxv/webctx
+
+go 1.26