arachna 1.0.1__tar.gz

arachna-1.0.1/CHANGELOG.md

@@ -0,0 +1,194 @@
+# Changelog
+
+## v1.0.1 — Windows test fixes
+
+- tests/cache: _make_entry uses real SHA256 hash instead of "dummy"
+- tests/cache: time.sleep(0.01) in test_get_changed_files_modified and test_get_changed_files_mixed
+- tests/formatter: test_permission_denied skipped on Windows (chmod 0o000 unsupported)
+- tests/gatherer: time.sleep(0.01) in test_collect_sections_incremental_detects_modified
+- tests/hook: S_IXUSR check only on Unix (Windows does not support executable bits)
+- __init__.py: bump __version__ to 1.0.1
+- pyproject.toml: bump version to 1.0.1
+
+## v1.0.0 — Public release
+
+- First public release on PyPI
+- __init__.py: bump __version__ to 1.0.0
+- pyproject.toml: bump version to 1.0.0
+
+## v0.9.5 — GitHub prep
+
+- pyproject.toml: URLs updated to github.com/dead-duke/arachna
+- README.md: added repo link, badges, safety section, doctor, hooks, full commands and options
+- __init__.py: bump __version__ to 0.9.5
+
+## v0.9.4 — Final polish
+
+- runner.py: import json вынесен на верхний уровень модуля
+- gatherer.py: _assemble_content — общая функция сборки контента для collect и dry_run
+- collector.py: collect использует _assemble_content, убран дублирующийся код
+- __init__.py: bump __version__ to 0.9.4
+- pyproject.toml: bump version to 0.9.4
+
+## v0.9.3 — Final fixes
+
+- __main__.py: _cmd_validate использует get_profile() для консистентной валидации
+- cache.py, gitignore.py: комментарии к _MAX_HASH_SIZE и _MAX_GITIGNORE_SIZE
+- gitignore.py: обработка ValueError от relative_to на всех вызовах
+- tests/runner: subprocess.CompletedProcess вместо MagicMock
+- __init__.py: bump __version__ to 0.9.3
+
+## v0.9.2 — Pre-release fixes
+
+- hook.py: git_dir.exists() → git_dir.is_dir()
+- doctor.py: проверка project_root.is_dir() перед load_gitignore_patterns
+- gitignore.py: обработка ValueError от relative_to для симлинков
+- __main__.py: _cmd_doctor и _cmd_install_hook без неиспользуемых параметров
+- tests/doctor: тесты на _cmd_doctor и _cmd_install_hook с проверкой sys.exit
+- __init__.py: bump __version__ to 0.9.2
+
+## v0.9.1 — Version sync
+
+- __init__.py: bump __version__ to 0.9.1
+- pyproject.toml: bump version to 0.9.1
+
+## v0.9.0 — Infrastructure
+
+- PyPI-упаковка: authors, keywords, urls в pyproject.toml
+- Кроссплатформенные тесты (Windows CI)
+
+## v0.8.5 — Sandbox
+
+- runner.py: dry-run + интерактивное подтверждение для недоверенных команд
+- runner.py: _is_safe_command для проверки безопасности в dry-run режиме
+
+## v0.8.4 — Merge
+
+- collector.py: --merge для --profile, добавление вывода к существующему манифесту
+- collector.py: _find_next_part_num для нумерации в merge режиме
+
+## v0.8.3 — Git hooks
+
+- hook.py: arachna --install-hook, установка post-commit хука
+- hook.py: настраиваемая команда через .arachna.json hook.post-commit
+- hook.py: --force для перезаписи существующего хука
+
+## v0.8.2 — Doctor
+
+- doctor.py: arachna --doctor, проверка конфига и корректности контекста
+- doctor.py: run_doctor и print_doctor для программного использования
+
+## v0.8.1 — Low fixes
+
+- config.py: DEFAULT_EXCLUDE генерируется из _COMMON_EXCLUDE_DIRS
+- splitter.py: токенизаторное усечение через бинарный поиск вместо CHARS_PER_TOKEN
+- tests/splitter: тесты на проброс кастомного токенизатора (MagicMock)
+
+## v0.8.0 — God function
+
+- gatherer.py: декомпозиция _collect_named_sections
+- gatherer.py: _collect_directory_sections и _collect_file_sections
+
+## v0.7.5 — Truncation API + shlex
+
+- splitter.py: was_truncated через logger.warning вместо print
+- runner.py: проверка пустой строки и непарных кавычек перед shlex.split
+
+## v0.7.4 — Sandbox pipe fix
+
+- runner.py: проверка частей пайпа по отдельности в _validate_command
+- runner.py: _resolve_base вместо _resolve_command
+
+## v0.7.3 — Test stability
+
+- tests: замена os.chdir на tmp_path/monkeypatch (все модули)
+- tests/runner: замокать subprocess.run
+- tests/config: изоляция от родительского .arachna.json
+- tests/gatherer/test_incremental.py: переписан на интеграционный тест
+
+## v0.7.2 — Architecture cleanup
+
+- gatherer.py: удалено глобальное _TOKENIZE, get_tokenizer, set_tokenizer
+- collector.py: убран fallback write_text в save_manifest
+- splitter.py: вынесен CHARS_PER_TOKEN, добавлен флаг truncated в _handle_single
+- config.py + gitignore.py: унифицированы EXCLUDED_DIRS
+- CHANGELOG.md: исправлена дезинформация и дублирование
+
+## v0.7.1 — Critical fixes
+
+- runner.py: удалены интерпретаторы (python, node, ruby, perl, php) из _ALLOWED_COMMANDS
+- splitter.py: исправлен проброс tokenizer в _build_parts (keyword args)
+- __main__.py: _apply_args_to_profile возвращает копию, не мутирует оригинал
+
+## v0.7.0 — Security sandbox, architecture cleanup
+
+- runner.py: sandbox-валидация _validate_command с _BLOCKED_PATTERNS и _ALLOWED_COMMANDS
+- runner.py: аудит-лог команд в .arachna_commands.log
+- cache.py: атомарная запись через tempfile + os.replace
+- gitignore.py: ограничение размера, фильтрация EXCLUDED_DIRS, детект бинарных файлов
+- formatter.py: проверка размера до read_text, verbose skip reasons
+- __main__.py: рефакторинг _cmd_all и _cmd_single через _run_profile
+- gatherer.py: set_tokenizer/get_tokenizer deprecated
+- 179 tests, 90% coverage
+
+## v0.6.0 — Pluggable tokenizer
+
+- tokenizer.py: load_tokenizer(spec)
+- tokenizer field в profile
+- Проброшен через collector → gatherer
+- 179 tests, 90% coverage
+
+## v0.5.0 — Tests, safety, audit fixes
+
+- Тесты: cache, completion, init, formatter xml/json, incremental, manifest
+- Убран compress_indent (небезопасный для Python)
+- Безопасное сжатие: пустые строки + trailing spaces
+- Shell security warning в README
+- LICENSE (MIT)
+- 175 tests, 90% coverage
+
+## v0.4.2 — Audit fixes
+
+- Убран мёртвый код в gatherer.py
+- Исправлены CJK token тесты
+- README: рекомендация по token margin
+
+## v0.4.1 — Table of contents + manifest
+
+- TOC в каждой части: список файлов
+- chat-manifest.md: сводка всех собранных файлов
+
+## v0.4.0 — Shell completion + hooks
+
+- bash и zsh completion (arachna --completion bash|zsh)
+- post_commands в профиле: запуск после коллекта
+- 144 tests, 70% coverage
+
+## v0.3.0 — Compress, incremental, formats, binary
+
+- Whitespace compression (--compress)
+- Инкрементальный режим: mtime кэш (--incremental)
+- section_format: markdown (default), xml, json (--format)
+- include_binary: base64 с фильтрами по размеру и расширению
+- 140 tests
+
+## v0.2.2 — Git split marker, per-profile manifest cleanup
+
+- git split_marker: \n=== COMMIT:
+- --all: очистка всех файлов, пересборка всех профилей
+- --profile: очистка только этого профиля
+
+## v0.2.1 — arachna init
+
+- --init interactive + --defaults auto-detect
+
+## v0.2.0 — Single file output, manifest, test reorg
+
+- chat-code.md, manifest, arachna_context/, 129 tests, 90% coverage
+
+## v0.1.5 — Shebang Detection
+## v0.1.4 — Tests & Bugfixes
+## v0.1.3 — Validate & Gitignore
+## v0.1.2 — Dry-run, renderer, pre-commit
+## v0.1.1 — Tests + fixes
+## v0.1.0 — MVP

arachna-1.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Artem Terenin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arachna-1.0.1/MANIFEST.in

@@ -0,0 +1,7 @@
+include README.md
+include CHANGELOG.md
+include LICENSE
+include TODO.md
+include requirements-dev.txt
+recursive-include src *.py
+recursive-include tests *.py

arachna-1.0.1/PKG-INFO

@@ -0,0 +1,255 @@
+Metadata-Version: 2.4
+Name: arachna
+Version: 1.0.1
+Summary: Context collector for AI — gathers project files into token-limited chunks
+Author: Artem Terenin
+License: MIT
+Project-URL: Homepage, https://github.com/dead-duke/arachna
+Project-URL: Repository, https://github.com/dead-duke/arachna
+Project-URL: Issues, https://github.com/dead-duke/arachna/issues
+Project-URL: Changelog, https://github.com/dead-duke/arachna/blob/main/CHANGELOG.md
+Keywords: ai,context,tokenizer,code-collector,llm
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Utilities
+Classifier: Intended Audience :: Developers
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# arachna
+
+[![PyPI version](https://img.shields.io/pypi/v/arachna)](https://pypi.org/project/arachna/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+
+Context collector for AI — gathers project files into token-limited chunks.
+
+## What is arachna
+
+arachna is a command-line tool that collects your project's source code and documentation into files ready to be sent to an AI. It understands tokens (not lines) and splits output smartly so nothing gets cut in the middle.
+
+## Why arachna
+
+- Token-aware splitting: other tools split by lines, arachna splits by tokens
+- Zero dependencies: just Python stdlib
+- Multiple profiles: code, docs, tests, git history
+- Smart defaults: arachna --init detects your project in seconds
+
+## Install
+```
+pip install arachna
+```
+
+## Quick start
+```
+cd your-project
+arachna --init
+arachna --all
+```
+Creates arachna_context/ folder with .md files ready for AI.
+
+## Commands
+```
+arachna --init              interactive setup
+arachna --init --defaults   auto-detect everything
+arachna --all               collect all profiles
+arachna --profile code      collect one profile
+arachna --all --dry-run     preview without writing
+arachna --clean             remove collected files
+arachna --list              show profiles
+arachna --validate          check config for errors
+arachna --doctor            run full diagnostic
+arachna --install-hook      install git post-commit hook
+```
+
+## Options
+
+| Option | Description |
+|--------|-------------|
+| `--output-dir path` | where to write (default: arachna_context/) |
+| `--verbose` | show skipped files |
+| `--compress` | remove blank lines and trailing spaces |
+| `--incremental` | only files changed since last run |
+| `--format xml` | markdown (default), xml, or json |
+| `--merge` | append to existing output instead of replacing |
+| `--dry-run` | preview without writing files |
+| `--force` | force overwrite with `--install-hook` |
+
+## Safety
+
+Commands in `.arachna.json` (pre_commands, post_commands, command) are validated before execution. Unknown or dangerous commands are blocked by default. Use `--dry-run` to preview what will be executed before running.
+
+## Doctor
+
+`arachna --doctor` runs a full diagnostic of your configuration — validates all profiles, checks that directories and files exist, and verifies `.gitignore` integration. Use it when something doesn't work as expected.
+
+## Git hooks
+
+`arachna --install-hook` installs a post-commit hook that automatically runs arachna after each commit. Configure the command in `.arachna.json`:
+
+```json
+{
+  "hook": {
+    "post-commit": "arachna --all --incremental"
+  }
+}
+```
+
+## Configuration (.arachna.json)
+
+arachna uses profiles to define what and how to collect.
+
+Example for a Python project:
+
+```json
+{
+  "project_name": "MyProject",
+  "profiles": {
+    "code": {
+      "split_mode": "by_file",
+      "directories": ["src", "app"],
+      "patterns": ["*.py"],
+      "files": ["pyproject.toml", "requirements.txt"],
+      "pre_commands": ["tree src app"],
+      "max_tokens": 16000
+    },
+    "tests": {
+      "split_mode": "by_file",
+      "directories": ["tests"],
+      "patterns": ["*.py"],
+      "max_tokens": 16000
+    },
+    "docs": {
+      "split_mode": "by_file",
+      "files": ["README.md", "TODO.md", "CHANGELOG.md"],
+      "max_tokens": 16000
+    },
+    "git": {
+      "split_mode": "by_marker",
+      "split_marker": "\n=== COMMIT:",
+      "command": "git log --reverse --format='=== COMMIT: %h ===%nTITLE: %s%n%nMESSAGE:%n%b%n'",
+      "max_tokens": 16000
+    }
+  }
+}
+```
+
+## Split modes
+
+by_file: code and docs, each file stays intact (default)
+by_paragraph: logs, splits on blank lines
+by_marker: git history, splits on custom marker
+single: everything in one file, truncates if too big
+
+## All config fields
+
+split_mode: by_file, by_paragraph, by_marker, or single
+split_marker: string for by_marker mode
+directories: folders to scan
+patterns: glob patterns like ["*.py"]
+files: specific files to include
+exclude_patterns: glob patterns to skip
+pre_commands: shell commands before collection
+post_commands: shell commands after collection
+command: use command output instead of files
+max_tokens: token limit per output file
+section_format: markdown, xml, or json
+compress: enable safe whitespace compression (blank lines, trailing spaces). Does not modify indentation.
+include_binary: include binaries as base64 (true/false)
+binary_extensions: whitelist like [".png"]
+binary_max_mb: max binary file size in MB
+
+## Output
+
+Files go to arachna_context/ (configurable):
+
+arachna_context/
+  .arachna_manifest.json
+  chat-manifest.md          # summary of all files
+  chat-code.md
+  chat-tests.md
+  chat-docs.md
+  chat-git.md
+
+When content exceeds max_tokens, files are numbered: chat-code_1.md, chat-code_2.md...
+
+## Manifest and cleanup
+
+Every created file is tracked in .arachna_manifest.json. Running --all again removes old files automatically. With --profile, only that profile's files are cleaned.
+
+## Incremental mode
+
+With --incremental, arachna skips files unchanged since last run. Uses .arachna_cache.json.
+
+## Tokenizer
+
+arachna uses a conservative estimate: 4 characters = 1 token.
+This works for any model with a 20-30% safety margin.
+
+### Built-in (default)
+
+No dependencies. Always works. Set max_tokens below your model's context window:
+- 8192 window → max_tokens: 6000
+- 32768 window → max_tokens: 24000
+
+### Custom tokenizer
+
+Add to your .arachna.json:
+
+  "tokenizer": "my_module:count_tokens"
+
+Your module must export count_tokens(text) -> int. Example:
+
+  # my_tok.py
+  def count_tokens(text: str) -> int:
+      return max(1, len(text) // 4)  # your logic here
+
+### Cloud models
+
+For exact token counts with cloud APIs, install tiktoken:
+
+  pip install tiktoken
+
+  "tokenizer": "tiktoken:cl100k_base"    # GPT-4, DeepSeek
+  "tokenizer": "tiktoken:o200k_base"     # GPT-4o
+
+### Local models
+
+For exact token counts with HuggingFace tokenizers, install transformers:
+
+  pip install transformers
+
+  "tokenizer": "transformers:Qwen/Qwen2.5-7B-Instruct"
+  "tokenizer": "transformers:mistralai/Mistral-7B-Instruct-v0.3"
+  "tokenizer": "transformers:google/gemma-7b"
+
+Note: transformers is a heavy dependency (gigabytes). Use only if you need exact counts.
+For most local models, the built-in estimate with safety margin is sufficient.
+
+## Supported project types
+
+arachna --init auto-detects:
+
+Python: src/, app/, tests/, *.py, pyproject.toml, requirements.txt
+JS/TS: src/, tests/, *.js, *.ts, package.json
+Go: cmd/, pkg/, *.go, go.mod
+Rust: src/, tests/, *.rs, Cargo.toml
+
+Also: README.md, TODO.md, CHANGELOG.md, Makefile, config/, docs/, data/prompts/.
+
+## Links
+
+- [GitHub Repository](https://github.com/dead-duke/arachna)
+- [Issue Tracker](https://github.com/dead-duke/arachna/issues)
+- [Changelog](https://github.com/dead-duke/arachna/blob/main/CHANGELOG.md)
+
+## License
+
+MIT