@tokmizer/plugin 0.9.61 → 0.9.63

package/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokmizer",
-  "version": "0.9.61",
+  "version": "0.9.63",
   "description": "Tokmizer reduces token consumption for AI agents.",
   "author": {
     "name": "Tokmizer",

package/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokmizer",
-  "version": "0.9.61",
+  "version": "0.9.63",
   "description": "Tokmizer reduces token consumption for OpenAI Codex.",
   "author": {
     "name": "Tokmizer",

package/LICENSE

@@ -0,0 +1,176 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

package/LICENSE-ENGINE

@@ -0,0 +1,107 @@
+Business Source License 1.1
+
+License text copyright (c) 2020 MariaDB Corporation Ab, All Rights Reserved.
+"Business Source License" is a trademark of MariaDB Corporation Ab.
+
+-----------------------------------------------------------------------------
+
+Parameters
+
+Licensor:             Impulse Creation
+
+Licensed Work:        Tokmizer compression engine and rule catalogue, version
+                      as published: the contents of the src/engine/ and
+                      filters/ directories of the @tokmizer/plugin package
+                      and its source repository, including the default
+                      rules built from filters/ and embedded in the
+                      compiled package.
+                      The Licensed Work is (c) 2026 Impulse Creation.
+
+Additional Use Grant: You may make production use of the Licensed Work,
+                      without charge, including as a component of the
+                      Tokmizer command-line tools, hooks and plugins, and
+                      inside your own internal tools and workflows. This
+                      grant does not permit offering the Licensed Work, or a
+                      derivative of it (including the engine or the rules,
+                      alone or together), to third parties as part of a
+                      product or service, paid or hosted, whose purpose is
+                      to reduce, compress or optimize the tokens or the
+                      context sent to AI models and that competes with
+                      Tokmizer. For such use, contact contact@tokmizer.com.
+
+Change Date:          Four years after the date on which this version of
+                      the Licensed Work is first publicly distributed by the
+                      Licensor (for a version first distributed on
+                      2026-10-15, the Change Date is 2030-10-15)
+
+Change License:       Apache License, Version 2.0
+
+For information about alternative licensing arrangements for the Licensed
+Work, please contact contact@tokmizer.com.
+
+-----------------------------------------------------------------------------
+
+Terms
+
+The Licensor hereby grants you the right to copy, modify, create derivative
+works, redistribute, and make non-production use of the Licensed Work. The
+Licensor may make an Additional Use Grant, above, permitting limited
+production use.
+
+Effective on the Change Date, or the fourth anniversary of the first publicly
+available distribution of a specific version of the Licensed Work under this
+License, whichever comes first, the Licensor hereby grants you rights under
+the terms of the Change License, and the rights granted in the paragraph
+above terminate.
+
+If your use of the Licensed Work does not comply with the requirements
+currently in effect as described in this License, you must purchase a
+commercial license from the Licensor, its affiliated entities, or authorized
+resellers, or you must refrain from using the Licensed Work.
+
+All copies of the original and modified Licensed Work, and derivative works
+of the Licensed Work, are subject to this License. This License applies
+separately for each version of the Licensed Work and the Change Date may vary
+for each version of the Licensed Work released by Licensor.
+
+You must conspicuously display this License on each original or modified copy
+of the Licensed Work. If you receive the Licensed Work in original or
+modified form from a third party, the terms and conditions set forth in this
+License apply to your use of that work.
+
+Any use of the Licensed Work in violation of this License will automatically
+terminate your rights under this License for the current and all other
+versions of the Licensed Work.
+
+This License does not grant you any right in any trademark or logo of
+Licensor or its affiliates (provided that you may use a trademark or logo of
+Licensor as expressly required by this License).
+
+TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
+AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
+EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
+TITLE.
+
+-----------------------------------------------------------------------------
+
+Covenants of Licensor
+
+In consideration of the right to use this License's text and the "Business
+Source License" name and trademark, Licensor covenants to MariaDB, and to all
+other recipients of the licensed work to be provided by Licensor:
+
+1. To specify as the Change License the GPL Version 2.0 or any later version,
+   or a license that is compatible with GPL Version 2.0 or a later version,
+   where "compatible" means that software provided under the Change License can
+   be included in a program with software provided under GPL Version 2.0 or a
+   later version. Licensor may specify additional Change Licenses without
+   limitation.
+
+2. To either: (a) specify an additional grant of rights to use that does not
+   impose any additional restriction on the right granted in this License, as
+   the Additional Use Grant; or (b) insert the text "None".
+
+3. To specify a Change Date.
+
+4. Not to modify this License in any other way.

package/LICENSES.md

@@ -0,0 +1,29 @@
+# Licences
+
+Tokmizer uses two licences, split by directory. No source file carries a licence header: the directory decides.
+
+Copyright 2026 Impulse Creation, the copyright holder and the Licensor named in the BUSL-1.1 files.
+
+| Path | Licence | Text |
+|---|---|---|
+| `src/engine/**` (compression engine: pipeline, routing, matcher, filter schema and loader, transforms, structured compaction, test-failure folding) | BUSL-1.1, converts to Apache-2.0 on the Change Date | [`src/engine/LICENSE`](src/engine/LICENSE) |
+| `filters/**` (TOML rule catalogue), including the default rules built from it and embedded in `dist/` | BUSL-1.1, same Licensed Work as the engine | [`filters/LICENSE`](filters/LICENSE) |
+| Everything else: CLI (`src/bin`, `src/*.ts`), hooks (`hooks/`, `src/hooks-host`), agent adapters (`src/adapters`, `agent-plugins/`), daemon, launcher and shims (`src/daemon`, `src/exec`, `src/shim`), native launcher (`native/`), setup commands (`command-templates/`, `codex-plugin/`, `.claude-plugin/`, install scripts), tests and fixtures (`src/__tests__`), build scripts (`scripts/`) and documentation | Apache-2.0 | [`LICENSE`](LICENSE) |
+| `dist/` and the npm package `@tokmizer/plugin` | `Apache-2.0 AND BUSL-1.1`: the bundled files contain code and rules under both licences | both texts above |
+| Third-party code bundled into `dist/` and into the native launcher | its own licence | [`THIRD-PARTY-NOTICES.md`](THIRD-PARTY-NOTICES.md) |
+
+The BUSL-1.1 text is the same file in `src/engine/LICENSE`, `filters/LICENSE` and, at the root, [`LICENSE-ENGINE`](LICENSE-ENGINE).
+
+## What BUSL-1.1 allows here
+
+You can read, copy, modify and redistribute the engine and the rules, and use them in production, including inside the Tokmizer CLI, hooks and plugins and in your own internal tools. You cannot offer the engine or the rules, alone or together, or a derivative of them, to third parties as part of a product or service that reduces, compresses or optimizes AI-model tokens or context and competes with Tokmizer. The exact terms are in [`LICENSE-ENGINE`](LICENSE-ENGINE).
+
+Each published version becomes Apache-2.0 on its Change Date: four years after that version's first public release, or earlier if the licence file of that version says so.
+
+## Your own rules
+
+Rules you write yourself, in your project or user configuration, are yours. Writing or reading rules in the same TOML format with your own tools needs no licence from us; only the engine source and the rule catalogue published here are BUSL-1.1.
+
+## Contributions
+
+Contributions are accepted under the Tokmizer Contributor License Agreement, which grants Impulse Creation the right to distribute them under the licence of the directory they change and under other licences, including commercial ones. The agreement is `CLA.md` at the root of the public repository; the process is described in its `CONTRIBUTING.md`.

package/NOTICE

@@ -0,0 +1,18 @@
+Tokmizer
+Copyright 2026 Impulse Creation
+
+This product is licensed under the Apache License, Version 2.0 (see LICENSE),
+except the compression engine in src/engine/ and the rule catalogue in
+filters/, which are licensed under the Business Source License 1.1 (see
+LICENSE-ENGINE, src/engine/LICENSE and filters/LICENSE).
+LICENSES.md maps each directory to its licence.
+
+The TOML filter rule format and parts of the rule catalogue in filters/
+(rule names, some descriptions and some line patterns) are derived from rtk
+(https://github.com/rtk-ai/rtk), Copyright Patrick Szymkowiak and the rtk
+contributors, licensed under the Apache License, Version 2.0. Those derived
+portions remain available under the Apache License, Version 2.0; the
+Business Source License 1.1 applies to the changes and additions made here.
+
+The npm package and the native launcher bundle third-party software. Their
+copyright notices and licence texts are reproduced in THIRD-PARTY-NOTICES.md.

package/README.md

@@ -100,7 +100,7 @@ Each command adds one `PreToolUse` hook for the agent's shell tool. It rewrites
 
 Permission rules stay in force. Agents match their allow, ask and deny rules against the command text, so Tokmizer leaves a command untouched whenever one of your deny or ask rules matches it. Tokmizer reads those rules for Claude Code (`permissions.deny` and `permissions.ask` in user, project, local and managed settings; a command matched by one of your `permissions.allow` rules is left untouched too, so that rule keeps approving it), OpenCode (`permission` in `opencode.json`), Cursor CLI (`permissions.deny`), Gemini CLI (policy files and `tools.exclude`) and Antigravity CLI (`permissions.deny` and `permissions.ask`). For the other agents, and on top of the Cursor, Gemini and Antigravity rules, only inspection, build and test commands (`cat`, `rg`, `git log`, `npm test`, `cargo build` and similar; inspection only for Codex) are routed; anything else, such as `git push` or `rm`, runs exactly as typed. If a settings file cannot be read, nothing is routed. Shell functions, aliases, builtins and anything that is not a program on your `PATH` always run exactly as typed.
 
-In Claude Code, a command is rewritten only when that adds no permission prompt: in `bypassPermissions` mode, or when a plain `Bash` allow rule in your user or managed settings applies and the session is in default, `acceptEdits` or `dontAsk` mode. Otherwise it runs exactly as typed, so Claude Code still approves read-only commands such as `ls`, `cat` or `git status` on its own, and Tokmizer shortens the output once the command has succeeded. Claude Code runs that after-command hook only for successful commands, so in this default mode a failing command reaches Claude in full. When commands are rewritten, failing output is shortened too, with its failure details kept.
+In Claude Code, a command is rewritten only when that adds no permission prompt: in `bypassPermissions` mode, or when a plain `Bash` allow rule in your user or managed settings applies and the session is in default, `acceptEdits` or `dontAsk` mode. Otherwise it runs exactly as typed, so Claude Code still approves read-only commands such as `ls`, `cat` or `git status` on its own, and Tokmizer shortens the output once the command has succeeded. Claude Code runs that after-command hook only for successful commands, so in this default mode a failing command reaches Claude in full. A single command typed on its own (no `&&`, `;`, pipe, redirection, `$( )` or wildcard) is also rewritten when one of your own allow rules for it, such as `Bash(pytest:*)` or `Bash(cargo build:*)` in your user or managed settings, applies and no deny or ask rule matches: Tokmizer then gives the approval Claude Code would have given to the command as typed, so no new prompt appears. When commands are rewritten, failing output is shortened too, with its failure details kept.
 
 Aider, Cline and Windsurf have no hooks. Start them through Tokmizer, for example `tkr wrap aider`: only that session goes through Tokmizer's command shims. Your own terminals are not changed; `tkr shim global-path on` routes every terminal through the shims by adding a PATH line to your shell profile, and `tkr shim global-path off` removes it.
 
@@ -320,10 +320,26 @@ Every changed file is backed up under `~/.tokmizer/migrate-rtk/`. Running `--yes
 
 This README is shared by the npm package and the public GitHub repository.
 
-- The npm package includes the CLI, Claude Code hooks and command templates, and the Codex manifest.
-- The public repository contains installation scripts, distribution manifests, the `/tkr` entry command, and selected UI, usage, and redaction helpers.
+- The npm package includes the CLI, Claude Code hooks and command templates, the Codex manifest and the compiled compression engine.
+- The public repository contains the full source of the package: CLI, hooks, agent adapters, native launcher, setup commands, the TOML rule catalogue and the compression engine.
 
-The optimization engine and API client ship compiled with [`@tokmizer/plugin`](https://www.npmjs.com/package/@tokmizer/plugin). Their source code is not in the public repository.
+## Build from source
+
+Requirements: Node.js 20.11 or later and npm. The native launcher is optional and needs a Rust toolchain.
+
+```
+git clone https://github.com/norwelltw/Tokmizer
+cd Tokmizer
+npm install
+npm run build
+npm test
+```
+
+`npm run build` type-checks the sources and bundles them into `packages/plugin/dist/`. Run the result with `node packages/plugin/dist/bin/tkr.js help`.
+
+To build the native launcher for your platform: `npm --workspace packages/plugin run build:native`, then `npm run build` again to copy it into `dist/native/`.
+
+To try your build in Claude Code, install the folder as a local plugin or link the CLI with `npm --workspace packages/plugin link`.
 
 ## Network & privacy
 
@@ -374,7 +390,7 @@ The switch does not cover Claude Code's own marketplace auto-update, which `/tkr
 ### Local data
 
 - Restoration copies are local and kept up to 30 days. A restore hint identifies available cached content; there is no guarantee that every removed detail has a saved copy. YOLO creates no new restoration copies.
-- The public [redaction helpers](https://github.com/norwelltw/Tokmizer/blob/main/src/learn/redact.ts) are available for inspection.
+- The [redaction helpers](https://github.com/norwelltw/Tokmizer/blob/main/packages/plugin/src/learn/redact.ts) are available for inspection, like the rest of the source.
 - Local state lives in `~/.tokmizer/`.
 
 ## Uninstall
@@ -394,4 +410,4 @@ In Claude Code: `/plugin uninstall tokmizer@tokmizer`. Your savings history stay
 
 ## License
 
-Proprietary. See [LICENSE](LICENSE) and the EULA at [tokmizer.com/eula](https://tokmizer.com/eula).
+Apache-2.0, except the compression engine in `src/engine/` and the rule catalogue in `filters/`, which are under the Business Source License 1.1 and become Apache-2.0 four years after each release. Production use is allowed, including inside your own tools; offering the engine or the rules as part of a competing token-optimization product or hosted service is not. Rules you write yourself stay yours. See [LICENSES.md](LICENSES.md), [LICENSE](LICENSE), [LICENSE-ENGINE](LICENSE-ENGINE) and [NOTICE](NOTICE).