yam-flow 2.4.0 → 2.5.0

package/AGENTS.md

@@ -24,18 +24,20 @@
 
 ## Latest Update Trigger
 
-When the user says exactly `yam 최신 업데이트 사항 반영해줘` in a chat for this project, treat that phrase as explicit authorization to update the globally installed package and refresh the installed yam skills.
+When the user says exactly `yam 최신 업데이트 사항 반영해줘` in a chat for this project, treat that phrase as explicit authorization to check and apply reviewed updates for yam, Scrapling, and Insane Search.
 
 Run these commands sequentially from the yam project directory:
 
 ```bash
-npm install -g yam-flow
-yam install
+yam update check --json
+yam update apply --all --json
 yam version
 yam status
 ```
 
 - Request any environment approval required for network access or writes outside the workspace.
 - Stop at the first failed command, report the blocker, and do not claim that later steps ran.
-- After all four commands pass, report the installed version and skill status, then tell the user that Codex can be restarted to reload the updated skills.
+- Treat that exact phrase as one-time authorization for this run only; do not create an unattended updater, hook, or automation.
+- Never edit `.codex/plugins/cache` directly. If Insane Search cannot be updated in place through the official Codex CLI, report `manual_plugin_update_required` and stop.
+- After all four commands pass, summarize every component receipt, report the installed yam version and skill status, then tell the user that Codex can be restarted to reload updated skills/plugins.
 - Do not restart Codex automatically.

package/CHANGELOG.md

@@ -2,6 +2,34 @@
 
 All notable changes to yam-flow are documented here.
 
+## v2.5.0 - 2026-07-30
+
+### Added
+
+- Added read-only `yam update check [--json]` for yam, Scrapling, and Insane Search with official registry/manifest versions and source-revision metadata.
+- Added explicitly authorized component/all update application with per-component receipts, a concurrent-run lock, exact stable versions, verification evidence, and rollback guidance.
+- Added isolated Scrapling version environments with `pip check`, HTTP extraction, browser extraction, and atomic executable switching; the previous environment remains available for rollback.
+- Added deterministic temp-only updater smoke coverage, including receipt-failure symlink rollback, manual Codex plugin fallback, strict no-remove/cache behavior, and yam-last `--all` ordering.
+
+### Improved
+
+- The build now preserves executable mode on `dist/bin/yam.js`, and dist freshness fails if a global npm install would expose a non-executable `yam` target.
+
+### Security
+
+- Insane Search updates pin the official manifest to the exact observed Git revision and use only official Codex CLI commands. A missing safe in-place path returns `manual_plugin_update_required`; an unreadable post-upgrade state fails closed before `add`; yam never removes the plugin first or writes `.codex/plugins/cache` directly.
+- External commands use argument arrays without a shell, version/component allowlists, redacted captured output, and fail-closed apply locking.
+- Existing Scrapling links require a matching yam ownership marker, and an unwritable receipt stops `--all` before any later component mutation.
+
+### Verification
+
+- `npm run external-updates-smoke`
+- `npm run typecheck`
+- `npm run verify:self`
+- `npm run cli-smoke`
+- `npm run dist:freshness`
+- `npm run package-boundary:check`
+
 ## v2.4.0 - 2026-07-27
 
 ### Added

package/COMMANDS.md

@@ -143,6 +143,19 @@ yam measure quick --files 3 --commands 1 --report-lines 5 --seconds 40
 yam template tuning
 ```
 
+## External Updates
+
+```bash
+yam update check
+yam update check --json
+yam update apply --component scrapling --json
+yam update apply --component insane-search --json
+yam update apply --component yam --json
+yam update apply --all --json
+```
+
+`yam update check` is read-only. Every `apply` needs explicit authorization, uses a concurrent-run lock, records a component receipt, and stops on failed/manual results. `--all` runs yam last. Insane Search is never remove-first and `.codex/plugins/cache` is never edited directly.
+
 ## Tool Doctor / Proof / Safety
 
 Read-only readiness and evidence helpers. These do not install, deploy, query databases, or run verification by themselves.

package/README.md

@@ -130,6 +130,54 @@ Manual copying can still make a skill loadable, but it intentionally remains unv
 
 Restart Codex after installing so skills reload.
 
+## External Updates
+
+Update checks are read-only and never authorize a mutation by themselves:
+
+```bash
+yam update check
+yam update check --json
+```
+
+Apply requires an explicit component or `--all`:
+
+```bash
+yam update apply --component scrapling
+yam update apply --component insane-search
+yam update apply --component yam
+yam update apply --all --json
+```
+
+`--all` processes Scrapling, then Insane Search, then yam so the running updater replaces itself last. It stops at the first failed or manual-only component. Each attempt writes a component receipt under `~/.local/state/yam/update-receipts/` with the old/new version, source revision, checks, side effects, outcome, and rollback guidance. A lock prevents concurrent apply runs.
+
+Scrapling updates install an exact stable PyPI version into a new versioned virtual environment. `pip check`, ordinary HTTP extraction, and headless browser extraction must pass before the executable symlink changes atomically; the previous environment is retained. An existing environment must have a matching `.yam-scrapling-install.json` ownership marker before yam will replace its link. Insane Search uses only the official `codex plugin` commands. yam never edits `.codex/plugins/cache`, never removes the plugin first, and returns `manual_plugin_update_required` when the installed Codex CLI cannot update it safely in place.
+
+There is no unattended updater. The exact project trigger in `AGENTS.md` grants one-time authorization for a checked `--all` run, and Codex is never restarted automatically.
+
+### Scrapling Usage
+
+The isolated executable can be used directly from the shell:
+
+```bash
+scrapling --version
+scrapling extract get https://example.com example.txt --css-selector h1
+scrapling extract stealthy-fetch https://example.com example-browser.txt --css-selector h1 --headless
+```
+
+From Python inside the installed environment:
+
+```python
+from scrapling.fetchers import Fetcher, StealthyFetcher
+
+page = Fetcher.get("https://example.com")
+print(page.css("h1::text").get())
+
+browser_page = StealthyFetcher.fetch("https://example.com", headless=True)
+print(browser_page.css("h1::text").get())
+```
+
+Use `Fetcher` for ordinary HTTP pages and `StealthyFetcher` only when a real browser or stronger anti-bot behavior is needed; the browser path is heavier.
+
 ## Uninstall
 
 ```bash

package/ROADMAP.md

@@ -97,6 +97,7 @@ Tasks:
 - Add symlink-friendly local install.
 - Add version command. Done.
 - Add staged transactional install, rollback, receipt, and drift-aware status. Done.
+- Add checked component updates for yam, Scrapling, and Insane Search with explicit authorization, receipts, and rollback guidance. Done.
 - Consider npm packaging only after route behavior stabilizes.
 
 ### 8. Scout / Research Workflow

package/dist/bin/yam.js

@@ -11,6 +11,7 @@ import { fileURLToPath } from 'node:url';
 import { TRUTH_STATUSES, buildLoopReport, buildMediaGenerationProof, buildMissionCompletionGate, buildMissionPatchEnvelope, buildMissionSubagentReceipt, buildRollbackHint, buildRuntimeBackendEvidence, buildStudyNote, buildUeyeDesignCompletionGate, buildUeyeVisualProvenance, buildUeyeRunReport, buildUeyeSurfaceContext, buildYamCompletionProof, detectDbSafetyText as detectTrustDbSafetyText, isTruthStatus } from '../lib/trust-kernel.js';
 import { archiveUeyeRevision, inspectImageFile, upsertUeyeAsset, verifyUeyeAssetManifest, verifyUeyeRevisionHistory } from '../lib/ueye-artifacts.js';
 import { inspectSkillInstallation, installSkillSetTransactional, uninstallSkillSetSafely } from '../lib/skill-installation.js';
+import { EXTERNAL_UPDATE_COMPONENTS, applyExternalUpdates, checkExternalUpdates } from '../lib/external-updates.js';
 function errorMessage(error) {
     return error instanceof Error ? error.message : String(error);
 }
@@ -146,6 +147,9 @@ Usage:
   yam mission queue [--agent-id id] [--scope text] [--changed file] [--verification-hint text] [--json]
   yam mission receipt [--thread-id id] [--role reviewer] [--lifecycle stopped] [--outcome passed] [--evidence text] [--json]
   yam mission gate [--expected-thread id] [--receipt file] [--json]
+  yam update check [--json]
+  yam update apply --component yam|scrapling|insane-search [--json]
+  yam update apply --all [--json]
   yam benchmark report [--baseline n] [--current n] [--unit ms] [--target lower|higher] [--json]
   yam release report [--json]
   yam safety [text...]
@@ -3556,6 +3560,86 @@ async function mission(args = []) {
     console.error(`unknown mission command: ${subcommand}`);
     return missionUsage();
 }
+async function updateCommand(args = []) {
+    const subcommand = args[0] || 'help';
+    if (subcommand === 'help' || subcommand === '--help' || subcommand === '-h')
+        return updateUsage();
+    if (subcommand === 'check') {
+        const rest = args.slice(1);
+        const invalid = rest.filter((arg) => arg !== '--json');
+        if (invalid.length || rest.filter((arg) => arg === '--json').length > 1) {
+            throw new Error(`invalid update check option: ${invalid[0] || '--json repeated'}`);
+        }
+        const report = await checkExternalUpdates(VERSION);
+        printJsonOrHuman(report, rest.includes('--json'), 'External update check');
+        if (!report.success)
+            process.exitCode = 1;
+        return;
+    }
+    if (subcommand !== 'apply') {
+        throw new Error(`unknown update command: ${subcommand}`);
+    }
+    const parsed = parseUpdateApplyArgs(args.slice(1));
+    const report = await applyExternalUpdates(VERSION, {
+        component: parsed.component,
+        all: parsed.all
+    });
+    printJsonOrHuman(report, parsed.json, 'External update apply');
+    if (!report.success)
+        process.exitCode = 1;
+}
+function parseUpdateApplyArgs(args) {
+    let component;
+    let all = false;
+    let json = false;
+    for (let index = 0; index < args.length; index += 1) {
+        const arg = String(args[index] || '');
+        if (arg === '--json') {
+            if (json)
+                throw new Error('--json may be supplied only once');
+            json = true;
+            continue;
+        }
+        if (arg === '--all') {
+            if (all)
+                throw new Error('--all may be supplied only once');
+            all = true;
+            continue;
+        }
+        if (arg === '--component' || arg.startsWith('--component=')) {
+            if (component)
+                throw new Error('--component may be supplied only once');
+            const value = arg === '--component' ? String(args[index + 1] || '') : arg.slice('--component='.length);
+            if (arg === '--component')
+                index += 1;
+            if (!EXTERNAL_UPDATE_COMPONENTS.includes(value)) {
+                throw new Error('component must be one of: yam, scrapling, insane-search');
+            }
+            component = value;
+            continue;
+        }
+        throw new Error(`invalid update apply option: ${arg}`);
+    }
+    if ((all && component) || (!all && !component)) {
+        throw new Error('choose exactly one of --component yam|scrapling|insane-search or --all');
+    }
+    return { component, all, json };
+}
+function updateUsage() {
+    console.log(`yam update
+
+Usage:
+  yam update check [--json]
+  yam update apply --component yam|scrapling|insane-search [--json]
+  yam update apply --all [--json]
+
+Safety:
+  check is read-only. apply requires an explicit component or --all.
+  --all updates Scrapling and Insane Search before yam, then stops at the first failed/manual component.
+  Insane Search is updated only through the official Codex CLI; plugin caches are never edited directly.
+  Every apply attempt records component checks, outcome, side effects, and rollback guidance.
+`);
+}
 function missionUsage() {
     console.log(`yam mission
 
@@ -5313,6 +5397,7 @@ function showRequestedHelp(args) {
         media: mediaUsage,
         runtime: runtimeUsage,
         mission: missionUsage,
+        update: updateUsage,
         benchmark: benchmarkUsage,
         memory: memoryUsage
     };
@@ -5360,6 +5445,8 @@ async function main() {
         return runtime(process.argv.slice(3));
     if (command === 'mission')
         return mission(process.argv.slice(3));
+    if (command === 'update')
+        return updateCommand(process.argv.slice(3));
     if (command === 'benchmark')
         return benchmark(process.argv.slice(3));
     if (command === 'release')

package/dist/lib/external-updates.d.ts

@@ -0,0 +1,103 @@
+export declare const EXTERNAL_UPDATE_COMPONENTS: readonly ["yam", "scrapling", "insane-search"];
+export type ExternalUpdateComponent = typeof EXTERNAL_UPDATE_COMPONENTS[number];
+type JsonRecord = Record<string, any>;
+export interface ExternalCommandResult {
+    ok: boolean;
+    status: number | null;
+    stdout: string;
+    stderr: string;
+}
+export interface ExternalCommandOptions {
+    cwd?: string;
+    env?: NodeJS.ProcessEnv;
+    timeout?: number;
+}
+export interface ExternalUpdatePaths {
+    stateDir: string;
+    receiptDir: string;
+    lockFile: string;
+    scraplingRoot: string;
+    scraplingBin: string;
+}
+export interface ExternalUpdateDependencies {
+    run?: (command: string, args: string[], options?: ExternalCommandOptions) => Promise<ExternalCommandResult> | ExternalCommandResult;
+    fetchJson?: (url: string) => Promise<any>;
+    now?: () => Date;
+    homeDir?: string;
+    env?: NodeJS.ProcessEnv;
+    paths?: Partial<ExternalUpdatePaths>;
+    removeCandidate?: (candidate: string, root: string) => Promise<void>;
+}
+export interface ExternalComponentCheck {
+    component: ExternalUpdateComponent;
+    installed_version: string;
+    latest_version: string;
+    update_available: boolean;
+    status: 'up_to_date' | 'update_available' | 'not_installed' | 'check_failed';
+    source: string;
+    source_revision?: {
+        local: string;
+        remote: string;
+        drift: boolean;
+        note: string;
+    };
+    executable?: string;
+    error?: string;
+}
+export interface ExternalComponentReceipt {
+    schema: 'yam.external-update-receipt.v1';
+    generated_at: string;
+    component: ExternalUpdateComponent;
+    outcome: 'up_to_date' | 'updated' | 'failed' | 'manual_plugin_update_required';
+    previous_version: string;
+    requested_version: string;
+    installed_version: string;
+    source: string;
+    source_revision: JsonRecord | null;
+    checks: Array<{
+        id: string;
+        status: 'passed' | 'failed' | 'skipped';
+        note: string;
+    }>;
+    rollback_hint: {
+        automatic: boolean;
+        previous_target: string;
+        guidance: string;
+    };
+    side_effects: string[];
+    error: string;
+    truth_status: 'verified' | 'partial' | 'blocked';
+    persistence?: 'written' | 'failed';
+    receipt_path?: string;
+    receipt_error?: string;
+}
+export declare function checkExternalUpdates(currentYamVersion: string, dependencies?: ExternalUpdateDependencies): Promise<{
+    schema: string;
+    generated_at: string;
+    mutation_authorized: boolean;
+    components: [ExternalComponentCheck, ExternalComponentCheck, ExternalComponentCheck];
+    update_available: boolean;
+    success: boolean;
+    truth_status: string;
+    next_action: string;
+}>;
+export declare function applyExternalUpdates(currentYamVersion: string, selection: {
+    component?: ExternalUpdateComponent;
+    all?: boolean;
+}, dependencies?: ExternalUpdateDependencies): Promise<{
+    schema: string;
+    generated_at: string;
+    mutation_authorized: boolean;
+    requested_components: ("yam" | "scrapling" | "insane-search")[];
+    applied_components: ("yam" | "scrapling" | "insane-search")[];
+    receipts: ExternalComponentReceipt[];
+    stopped_early: boolean;
+    lock: {
+        path: string;
+        released: boolean;
+    };
+    success: boolean;
+    truth_status: string;
+    next_action: string;
+}>;
+export {};