de-arsenal 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- de_arsenal-0.2.0/.gitignore +12 -0
- de_arsenal-0.2.0/PKG-INFO +68 -0
- de_arsenal-0.2.0/README.md +41 -0
- de_arsenal-0.2.0/pyproject.toml +41 -0
- de_arsenal-0.2.0/src/arsenal/__init__.py +12 -0
- de_arsenal-0.2.0/src/arsenal/cli.py +183 -0
- de_arsenal-0.2.0/src/arsenal/project.py +68 -0
- de_arsenal-0.2.0/src/arsenal/py.typed +0 -0
- de_arsenal-0.2.0/src/arsenal/recipes/api-to-postgres/README.md +11 -0
- de_arsenal-0.2.0/src/arsenal/recipes/api-to-postgres/arsenal.yaml +3 -0
- de_arsenal-0.2.0/src/arsenal/recipes/api-to-postgres/collect.yaml +16 -0
- de_arsenal-0.2.0/src/arsenal/recipes/csv-cleanup/README.md +7 -0
- de_arsenal-0.2.0/src/arsenal/recipes/csv-cleanup/arsenal.yaml +5 -0
- de_arsenal-0.2.0/src/arsenal/recipes/csv-cleanup/collect.yaml +8 -0
- de_arsenal-0.2.0/src/arsenal/recipes/csv-cleanup/transform.yaml +6 -0
- de_arsenal-0.2.0/src/arsenal/recipes/github-issues/README.md +7 -0
- de_arsenal-0.2.0/src/arsenal/recipes/github-issues/arsenal.yaml +5 -0
- de_arsenal-0.2.0/src/arsenal/recipes/github-issues/collect.yaml +9 -0
- de_arsenal-0.2.0/src/arsenal/recipes/github-issues/transform.yaml +7 -0
- de_arsenal-0.2.0/tests/test_cli.py +242 -0
- de_arsenal-0.2.0/tests/test_import_arsenal.py +24 -0
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: de-arsenal
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Arsenal — 원클릭 데이터 스택. 수집→검증→변환→쿼리를 명령 하나로 (pugio+gladius 우산 CLI)
|
|
5
|
+
Project-URL: Homepage, https://github.com/KooEric/de-arsenal
|
|
6
|
+
Project-URL: Repository, https://github.com/KooEric/de-arsenal
|
|
7
|
+
License-Expression: Apache-2.0
|
|
8
|
+
Classifier: Intended Audience :: Developers
|
|
9
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
15
|
+
Requires-Python: >=3.11
|
|
16
|
+
Requires-Dist: arsenal-core<0.3,>=0.2
|
|
17
|
+
Requires-Dist: de-gladius<0.3,>=0.2
|
|
18
|
+
Requires-Dist: pugio<0.3,>=0.2
|
|
19
|
+
Requires-Dist: typer>=0.12
|
|
20
|
+
Provides-Extra: dbt
|
|
21
|
+
Requires-Dist: dbt-duckdb<2,>=1.8; extra == 'dbt'
|
|
22
|
+
Provides-Extra: dlt
|
|
23
|
+
Requires-Dist: pugio[dlt]; extra == 'dlt'
|
|
24
|
+
Provides-Extra: postgres
|
|
25
|
+
Requires-Dist: pugio[postgres]; extra == 'postgres'
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
|
|
28
|
+
# Arsenal (`de-arsenal`)
|
|
29
|
+
|
|
30
|
+
> 우산 CLI — Pugio(수집)와 Gladius(변환)를 한 진입점에서 부른다. 새 로직 없음,
|
|
31
|
+
> 위임만 한다(Unix 철학: 개별 도구는 독립적으로 완결됨).
|
|
32
|
+
|
|
33
|
+
`arsenal init <recipe>` → `arsenal run` → `arsenal query`, 세 명령으로
|
|
34
|
+
수집→검증→변환→쿼리를 재현하는 원클릭 데이터 스택. 배포명은 `de-arsenal`,
|
|
35
|
+
설치되는 명령어는 `arsenal`이다.
|
|
36
|
+
|
|
37
|
+
## 설치
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
uv tool install de-arsenal
|
|
41
|
+
# 또는
|
|
42
|
+
pipx install de-arsenal
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## 사용 예
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
arsenal init csv-cleanup # 레시피로 프로젝트 스캐폴드 (arsenal init --list로 목록)
|
|
49
|
+
arsenal run # arsenal.yaml 순서대로 수집→변환 일괄 실행. 재실행 = 재개
|
|
50
|
+
arsenal query "SELECT * FROM './data/clean/*.parquet' LIMIT 10"
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
`arsenal.yaml` 매니페스트:
|
|
54
|
+
|
|
55
|
+
```yaml
|
|
56
|
+
name: my-project
|
|
57
|
+
pipelines: # pugio로 순서대로 실행
|
|
58
|
+
- collect.yaml
|
|
59
|
+
transforms: # gladius로 순서대로 실행
|
|
60
|
+
- transform.yaml
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## 문서
|
|
64
|
+
|
|
65
|
+
- [docs/01-scope.md](../../docs/01-scope.md) — M4 통합·릴리스 범위
|
|
66
|
+
- [docs/08-limits.md](../../docs/08-limits.md) — 정직한 한계선
|
|
67
|
+
- [examples/quickstart](../../examples/quickstart) — 10분 collect → transform → query 데모
|
|
68
|
+
- [저장소 루트 README](../../README.md)
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Arsenal (`de-arsenal`)
|
|
2
|
+
|
|
3
|
+
> 우산 CLI — Pugio(수집)와 Gladius(변환)를 한 진입점에서 부른다. 새 로직 없음,
|
|
4
|
+
> 위임만 한다(Unix 철학: 개별 도구는 독립적으로 완결됨).
|
|
5
|
+
|
|
6
|
+
`arsenal init <recipe>` → `arsenal run` → `arsenal query`, 세 명령으로
|
|
7
|
+
수집→검증→변환→쿼리를 재현하는 원클릭 데이터 스택. 배포명은 `de-arsenal`,
|
|
8
|
+
설치되는 명령어는 `arsenal`이다.
|
|
9
|
+
|
|
10
|
+
## 설치
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
uv tool install de-arsenal
|
|
14
|
+
# 또는
|
|
15
|
+
pipx install de-arsenal
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## 사용 예
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
arsenal init csv-cleanup # 레시피로 프로젝트 스캐폴드 (arsenal init --list로 목록)
|
|
22
|
+
arsenal run # arsenal.yaml 순서대로 수집→변환 일괄 실행. 재실행 = 재개
|
|
23
|
+
arsenal query "SELECT * FROM './data/clean/*.parquet' LIMIT 10"
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`arsenal.yaml` 매니페스트:
|
|
27
|
+
|
|
28
|
+
```yaml
|
|
29
|
+
name: my-project
|
|
30
|
+
pipelines: # pugio로 순서대로 실행
|
|
31
|
+
- collect.yaml
|
|
32
|
+
transforms: # gladius로 순서대로 실행
|
|
33
|
+
- transform.yaml
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## 문서
|
|
37
|
+
|
|
38
|
+
- [docs/01-scope.md](../../docs/01-scope.md) — M4 통합·릴리스 범위
|
|
39
|
+
- [docs/08-limits.md](../../docs/08-limits.md) — 정직한 한계선
|
|
40
|
+
- [examples/quickstart](../../examples/quickstart) — 10분 collect → transform → query 데모
|
|
41
|
+
- [저장소 루트 README](../../README.md)
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
[project]
|
|
2
|
+
name = "de-arsenal"
|
|
3
|
+
version = "0.2.0"
|
|
4
|
+
description = "Arsenal — 원클릭 데이터 스택. 수집→검증→변환→쿼리를 명령 하나로 (pugio+gladius 우산 CLI)"
|
|
5
|
+
readme = "README.md"
|
|
6
|
+
requires-python = ">=3.11"
|
|
7
|
+
license = "Apache-2.0"
|
|
8
|
+
dependencies = [
|
|
9
|
+
"arsenal-core>=0.2,<0.3",
|
|
10
|
+
"pugio>=0.2,<0.3",
|
|
11
|
+
"de-gladius>=0.2,<0.3",
|
|
12
|
+
"typer>=0.12",
|
|
13
|
+
]
|
|
14
|
+
classifiers = [
|
|
15
|
+
"Programming Language :: Python :: 3",
|
|
16
|
+
"Programming Language :: Python :: 3.11",
|
|
17
|
+
"Programming Language :: Python :: 3.12",
|
|
18
|
+
"License :: OSI Approved :: Apache Software License",
|
|
19
|
+
"Operating System :: OS Independent",
|
|
20
|
+
"Intended Audience :: Developers",
|
|
21
|
+
"Topic :: Software Development :: Libraries :: Python Modules",
|
|
22
|
+
]
|
|
23
|
+
|
|
24
|
+
[project.optional-dependencies]
|
|
25
|
+
postgres = ["pugio[postgres]"]
|
|
26
|
+
dlt = ["pugio[dlt]"]
|
|
27
|
+
dbt = ["dbt-duckdb>=1.8,<2"]
|
|
28
|
+
|
|
29
|
+
[project.urls]
|
|
30
|
+
Homepage = "https://github.com/KooEric/de-arsenal"
|
|
31
|
+
Repository = "https://github.com/KooEric/de-arsenal"
|
|
32
|
+
|
|
33
|
+
[project.scripts]
|
|
34
|
+
arsenal = "arsenal.cli:app"
|
|
35
|
+
|
|
36
|
+
[build-system]
|
|
37
|
+
requires = ["hatchling"]
|
|
38
|
+
build-backend = "hatchling.build"
|
|
39
|
+
|
|
40
|
+
[tool.hatch.build.targets.wheel]
|
|
41
|
+
packages = ["src/arsenal"]
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
"""Arsenal — 우산 CLI. 단일 진입점, 새 로직 없음 (pugio·gladius 위임).
|
|
2
|
+
|
|
3
|
+
원클릭 경험의 실체: init(레시피 스캐폴드) → run(수집→변환 일괄) → query(즉석 확인).
|
|
4
|
+
구현: docs/plans/2026-07-08-m4-integration-release.md Task 4.0~4.1
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from importlib.metadata import PackageNotFoundError, version
|
|
8
|
+
|
|
9
|
+
try:
|
|
10
|
+
__version__ = version("de-arsenal")
|
|
11
|
+
except PackageNotFoundError:
|
|
12
|
+
__version__ = "0.2.0"
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
"""Arsenal CLI — 단일 진입점. 위임만 하고 로직을 갖지 않는다.
|
|
2
|
+
|
|
3
|
+
구현: docs/plans/2026-07-08-m4-integration-release.md Task 4.0
|
|
4
|
+
"""
|
|
5
|
+
|
|
6
|
+
import contextlib
|
|
7
|
+
import importlib
|
|
8
|
+
import json
|
|
9
|
+
import os
|
|
10
|
+
import sys
|
|
11
|
+
from collections.abc import Generator
|
|
12
|
+
from importlib import resources
|
|
13
|
+
from pathlib import Path
|
|
14
|
+
from typing import Any
|
|
15
|
+
|
|
16
|
+
import duckdb
|
|
17
|
+
import pyarrow.csv as pa_csv
|
|
18
|
+
import typer
|
|
19
|
+
|
|
20
|
+
from arsenal.project import DbtProjectRef, load_project
|
|
21
|
+
from arsenal_core.errors import ArsenalError, FatalError
|
|
22
|
+
from arsenal_core.spec import load_pipeline
|
|
23
|
+
from gladius import engine as gladius_engine
|
|
24
|
+
from gladius.cli import app as gladius_app
|
|
25
|
+
from gladius.loader import load_transform
|
|
26
|
+
from pugio.cli import app as pugio_app
|
|
27
|
+
from pugio.runner import run_pipeline as _pugio_run_pipeline
|
|
28
|
+
|
|
29
|
+
app = typer.Typer(help="Arsenal — 원클릭 데이터 스택. init → run → query.")
|
|
30
|
+
|
|
31
|
+
_RECIPES_DIRNAME = "recipes"
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def _fatal(e: ArsenalError) -> typer.Exit:
|
|
35
|
+
"""ArsenalError → `error: {e}` 출력 + exit(1). 커맨드에서는 `raise _fatal(e) from e`로 사용."""
|
|
36
|
+
typer.echo(f"error: {e}", err=True)
|
|
37
|
+
return typer.Exit(1)
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def _available_recipes() -> list[str]:
|
|
41
|
+
root = resources.files("arsenal") / _RECIPES_DIRNAME
|
|
42
|
+
return sorted(p.name for p in root.iterdir() if p.is_dir())
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
def _scaffold_recipe(recipe: str, dest: Path) -> None:
|
|
46
|
+
"""레시피 패키지 데이터를 dest로 복사. 기존 파일이 하나라도 있으면 통째로 거부."""
|
|
47
|
+
available = _available_recipes()
|
|
48
|
+
if recipe not in available:
|
|
49
|
+
raise FatalError(f"unknown recipe: {recipe!r} (available: {', '.join(available)})")
|
|
50
|
+
|
|
51
|
+
src_root = resources.files("arsenal") / _RECIPES_DIRNAME / recipe
|
|
52
|
+
entries = [p for p in src_root.iterdir() if p.is_file()]
|
|
53
|
+
|
|
54
|
+
try:
|
|
55
|
+
dest.mkdir(parents=True, exist_ok=True)
|
|
56
|
+
for entry in entries:
|
|
57
|
+
target = dest / entry.name
|
|
58
|
+
if target.exists():
|
|
59
|
+
raise FatalError(f"refusing to overwrite existing file: {target}")
|
|
60
|
+
for entry in entries:
|
|
61
|
+
(dest / entry.name).write_bytes(entry.read_bytes())
|
|
62
|
+
except OSError as e:
|
|
63
|
+
raise FatalError(f"cannot scaffold recipe into {dest}: {e}") from e
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
@app.command()
|
|
67
|
+
def init(
|
|
68
|
+
recipe: str | None = typer.Argument(None, help="레시피 이름 (예: csv-cleanup)"),
|
|
69
|
+
dest: Path = Path("."),
|
|
70
|
+
list_recipes: bool = typer.Option(False, "--list", help="사용 가능한 레시피 목록 출력"),
|
|
71
|
+
) -> None:
|
|
72
|
+
"""레시피로 프로젝트 스캐폴드 (예: arsenal init csv-cleanup). --list 로 목록 확인."""
|
|
73
|
+
if list_recipes:
|
|
74
|
+
for name in _available_recipes():
|
|
75
|
+
typer.echo(name)
|
|
76
|
+
return
|
|
77
|
+
try:
|
|
78
|
+
if recipe is None:
|
|
79
|
+
raise FatalError("RECIPE argument required (use --list to see available recipes)")
|
|
80
|
+
_scaffold_recipe(recipe, dest)
|
|
81
|
+
except ArsenalError as e:
|
|
82
|
+
raise _fatal(e) from e
|
|
83
|
+
typer.echo(f"done: {recipe} scaffolded into {dest}")
|
|
84
|
+
typer.echo(f" 1. cd {dest}")
|
|
85
|
+
typer.echo(" 2. edit collect.yaml / transform.yaml as needed")
|
|
86
|
+
typer.echo(" 3. arsenal run")
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
@contextlib.contextmanager
|
|
90
|
+
def _chdir(path: Path) -> Generator[None, None, None]:
|
|
91
|
+
"""상대 경로(sink `./data/...` 등)가 프로젝트 디렉터리 기준으로 풀리게 한다."""
|
|
92
|
+
prev = Path.cwd()
|
|
93
|
+
os.chdir(path)
|
|
94
|
+
try:
|
|
95
|
+
yield
|
|
96
|
+
finally:
|
|
97
|
+
os.chdir(prev)
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def _run_pipeline(path: Path) -> None:
|
|
101
|
+
"""pugio 파이프라인 하나 실행 — 위임 지점(테스트가 monkeypatch하는 seam)."""
|
|
102
|
+
spec = load_pipeline(path)
|
|
103
|
+
report = _pugio_run_pipeline(spec)
|
|
104
|
+
typer.echo(
|
|
105
|
+
f" collect {path.name}: "
|
|
106
|
+
f"fetched={report.fetched} written={report.written} skipped={report.skipped}"
|
|
107
|
+
)
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def _run_transform(path: Path) -> None:
|
|
111
|
+
"""gladius 변환 하나 실행 — 위임 지점(테스트가 monkeypatch하는 seam)."""
|
|
112
|
+
spec = load_transform(path)
|
|
113
|
+
out = gladius_engine.run_transform(spec)
|
|
114
|
+
typer.echo(f" transform {path.name}: {out}")
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def _run_dbt(path: Path) -> None:
|
|
118
|
+
"""Run a dbt-duckdb project through dbt's in-process CLI API."""
|
|
119
|
+
if not path.is_dir():
|
|
120
|
+
raise FatalError(f"dbt project not found: {path}")
|
|
121
|
+
try:
|
|
122
|
+
dbt_main: Any = importlib.import_module("dbt.cli.main")
|
|
123
|
+
dbt_runner: Any = dbt_main.dbtRunner
|
|
124
|
+
except (ImportError, AttributeError) as e:
|
|
125
|
+
raise FatalError("dbt stage requires the 'dbt' extra: pip install de-arsenal[dbt]") from e
|
|
126
|
+
try:
|
|
127
|
+
result: Any = dbt_runner().invoke(["run", "--project-dir", str(path)])
|
|
128
|
+
except Exception as e:
|
|
129
|
+
raise FatalError(f"dbt run failed for {path}: {e}") from e
|
|
130
|
+
if not result.success:
|
|
131
|
+
detail = str(getattr(result, "exception", "unknown dbt error"))
|
|
132
|
+
raise FatalError(f"dbt run failed for {path}: {detail}")
|
|
133
|
+
typer.echo(f" dbt {path}: run succeeded")
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
@app.command()
|
|
137
|
+
def run(project: Path = Path(".")) -> None:
|
|
138
|
+
"""arsenal.yaml 순서대로 수집→변환 일괄 실행. 재실행 = 재개."""
|
|
139
|
+
try:
|
|
140
|
+
proj_dir = project if project.is_absolute() else project.absolute()
|
|
141
|
+
proj = load_project(proj_dir / "arsenal.yaml")
|
|
142
|
+
with _chdir(proj_dir):
|
|
143
|
+
for pipeline_path in proj.pipelines:
|
|
144
|
+
_run_pipeline(pipeline_path)
|
|
145
|
+
for transform in proj.transforms:
|
|
146
|
+
if isinstance(transform, DbtProjectRef):
|
|
147
|
+
_run_dbt(transform.dbt)
|
|
148
|
+
else:
|
|
149
|
+
_run_transform(transform)
|
|
150
|
+
except ArsenalError as e:
|
|
151
|
+
raise _fatal(e) from e
|
|
152
|
+
typer.echo(f"done: {len(proj.pipelines)} pipeline(s), {len(proj.transforms)} transform(s)")
|
|
153
|
+
|
|
154
|
+
|
|
155
|
+
@app.command()
|
|
156
|
+
def query(sql: str, fmt: str = typer.Option("table", "--format")) -> None:
|
|
157
|
+
"""수집·변환 결과에 즉석 SQL (gladius.engine.query 위임)."""
|
|
158
|
+
try:
|
|
159
|
+
result = gladius_engine.query(sql)
|
|
160
|
+
except ArsenalError as e:
|
|
161
|
+
raise _fatal(e) from e
|
|
162
|
+
|
|
163
|
+
if fmt == "table":
|
|
164
|
+
duckdb.sql("SELECT * FROM result").show() # pyright: ignore[reportUnknownMemberType]
|
|
165
|
+
elif fmt == "csv":
|
|
166
|
+
pa_csv.write_csv( # pyright: ignore[reportPrivateImportUsage, reportUnknownMemberType]
|
|
167
|
+
result, sys.stdout.buffer
|
|
168
|
+
)
|
|
169
|
+
elif fmt == "jsonl":
|
|
170
|
+
for row in result.to_pylist():
|
|
171
|
+
typer.echo(json.dumps(row))
|
|
172
|
+
else:
|
|
173
|
+
typer.echo(f"error: unsupported --format: {fmt!r} (table | csv | jsonl)", err=True)
|
|
174
|
+
raise typer.Exit(1)
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
# 개별 도구는 서브커맨드로 그대로 노출 — 우산은 감싸되 가리지 않는다
|
|
178
|
+
app.add_typer(pugio_app, name="collect", help="수집 (pugio)")
|
|
179
|
+
app.add_typer(gladius_app, name="transform", help="변환·쿼리 (gladius)")
|
|
180
|
+
|
|
181
|
+
|
|
182
|
+
if __name__ == "__main__":
|
|
183
|
+
app()
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
"""arsenal.yaml — 프로젝트 매니페스트. `arsenal run`의 실행 순서 선언.
|
|
2
|
+
|
|
3
|
+
구현: docs/plans/2026-07-08-m4-integration-release.md Task 4.0
|
|
4
|
+
경로는 매니페스트 파일 기준 상대 경로로 해석한다.
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from pathlib import Path
|
|
8
|
+
|
|
9
|
+
import yaml
|
|
10
|
+
from pydantic import BaseModel, ConfigDict, ValidationError
|
|
11
|
+
|
|
12
|
+
from arsenal_core.errors import FatalError
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
class DbtProjectRef(BaseModel):
|
|
16
|
+
model_config = ConfigDict(frozen=True, extra="forbid")
|
|
17
|
+
|
|
18
|
+
dbt: Path
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
class ArsenalProject(BaseModel):
|
|
22
|
+
model_config = ConfigDict(frozen=True, extra="forbid")
|
|
23
|
+
|
|
24
|
+
name: str
|
|
25
|
+
pipelines: list[Path] = [] # pugio 스펙들 — 순서대로 실행
|
|
26
|
+
transforms: list[Path | DbtProjectRef] = [] # gladius/dbt 단계 — 순서대로
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
def load_project(manifest_path: Path) -> ArsenalProject:
|
|
30
|
+
"""arsenal.yaml 로드 + 상대 경로를 매니페스트 기준 절대 경로로 해석.
|
|
31
|
+
|
|
32
|
+
참조된 파이프라인/변환 스펙 파일의 실존 여부는 검사하지 않는다 — 경로
|
|
33
|
+
해석만 한다 (존재 검사는 각 스펙을 실제로 로드하는 시점의 책임).
|
|
34
|
+
"""
|
|
35
|
+
try:
|
|
36
|
+
text = manifest_path.read_text(encoding="utf-8")
|
|
37
|
+
except FileNotFoundError as e:
|
|
38
|
+
raise FatalError(f"manifest not found: {manifest_path}") from e
|
|
39
|
+
except OSError as e:
|
|
40
|
+
raise FatalError(f"cannot read manifest {manifest_path}: {e}") from e
|
|
41
|
+
|
|
42
|
+
try:
|
|
43
|
+
raw = yaml.safe_load(text)
|
|
44
|
+
except yaml.YAMLError as e:
|
|
45
|
+
raise FatalError(f"invalid manifest {manifest_path}: {e}") from e
|
|
46
|
+
|
|
47
|
+
try:
|
|
48
|
+
parsed = ArsenalProject.model_validate(raw)
|
|
49
|
+
except ValidationError as e:
|
|
50
|
+
lines = [f"{'.'.join(str(p) for p in err['loc'])}: {err['msg']}" for err in e.errors()]
|
|
51
|
+
raise FatalError(f"invalid manifest {manifest_path}:\n " + "\n ".join(lines)) from e
|
|
52
|
+
|
|
53
|
+
base = manifest_path.parent
|
|
54
|
+
if not base.is_absolute():
|
|
55
|
+
base = base.absolute()
|
|
56
|
+
|
|
57
|
+
transforms: list[Path | DbtProjectRef] = []
|
|
58
|
+
for transform in parsed.transforms:
|
|
59
|
+
if isinstance(transform, Path):
|
|
60
|
+
transforms.append(base / transform)
|
|
61
|
+
else:
|
|
62
|
+
transforms.append(DbtProjectRef(dbt=base / transform.dbt))
|
|
63
|
+
|
|
64
|
+
return ArsenalProject(
|
|
65
|
+
name=parsed.name,
|
|
66
|
+
pipelines=[base / p for p in parsed.pipelines],
|
|
67
|
+
transforms=transforms,
|
|
68
|
+
)
|
|
File without changes
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# api-to-postgres 레시피
|
|
2
|
+
|
|
3
|
+
REST API를 받아 검증 게이트를 통과시킨 뒤 Postgres에 멱등 upsert한다.
|
|
4
|
+
|
|
5
|
+
0. 설치: 이 레시피는 postgres extra(`psycopg`)가 필요하다 —
|
|
6
|
+
`uv tool install "de-arsenal[postgres]"` (또는 `pip install "pugio[postgres]"`).
|
|
7
|
+
extra 없이 실행하면 postgres sink가 `FatalError`로 즉시 안내한다(모듈을 찾을 수
|
|
8
|
+
없다는 원문 traceback이 아니다).
|
|
9
|
+
1. 바꿀 것: `collect.yaml`의 API URL, 환경변수 `PG_DSN`(Postgres 접속 문자열), `merge_key`
|
|
10
|
+
2. 실행: `arsenal run` — 실행 전 `PG_DSN`이 가리키는 Postgres에 접속 가능해야 한다
|
|
11
|
+
3. 확인: 위반 레코드는 적재되지 않고 `.arsenal/dlq/api-to-postgres/`에 격리된다 (`pugio dlq list`)
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: api-to-postgres
|
|
2
|
+
source:
|
|
3
|
+
type: rest
|
|
4
|
+
url: https://api.example.com/records # <- 바꿀 것: 실제 API 엔드포인트
|
|
5
|
+
pagination: { mode: page, param: page, size_param: per_page, size: 100 }
|
|
6
|
+
validate:
|
|
7
|
+
rules:
|
|
8
|
+
- field: id
|
|
9
|
+
not_null: true
|
|
10
|
+
unique: true
|
|
11
|
+
on_violation: quarantine
|
|
12
|
+
sink:
|
|
13
|
+
type: postgres
|
|
14
|
+
dsn_env: PG_DSN # <- 바꿀 것: Postgres DSN을 담은 환경변수 이름
|
|
15
|
+
table: records
|
|
16
|
+
merge_key: [id]
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
name: github-issues
|
|
2
|
+
source:
|
|
3
|
+
type: rest
|
|
4
|
+
url: https://api.github.com/repos/duckdb/duckdb/issues # <- 저장소를 바꾸세요
|
|
5
|
+
headers: { Authorization: "Bearer ${GITHUB_TOKEN}" }
|
|
6
|
+
pagination: { mode: offset, param: page, size_param: per_page, size: 100 }
|
|
7
|
+
sink:
|
|
8
|
+
type: parquet
|
|
9
|
+
path: ./data/issues
|
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
"""Arsenal 우산 CLI 테스트 (M4 Task 4.0) — init/run/query, 위임만 하고 로직은 없다."""
|
|
2
|
+
|
|
3
|
+
import json
|
|
4
|
+
from pathlib import Path
|
|
5
|
+
|
|
6
|
+
import pyarrow as pa
|
|
7
|
+
import pytest
|
|
8
|
+
from arsenal.cli import app
|
|
9
|
+
from arsenal.project import load_project
|
|
10
|
+
from typer.testing import CliRunner
|
|
11
|
+
|
|
12
|
+
from arsenal_core.errors import FatalError
|
|
13
|
+
|
|
14
|
+
runner = CliRunner()
|
|
15
|
+
|
|
16
|
+
MANIFEST = """\
|
|
17
|
+
name: my-project
|
|
18
|
+
pipelines:
|
|
19
|
+
- collect.yaml
|
|
20
|
+
transforms:
|
|
21
|
+
- transform.yaml
|
|
22
|
+
"""
|
|
23
|
+
|
|
24
|
+
DBT_MANIFEST = """\
|
|
25
|
+
name: my-project
|
|
26
|
+
pipelines: []
|
|
27
|
+
transforms:
|
|
28
|
+
- dbt: ./dbt_project
|
|
29
|
+
"""
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
def test_manifest_parses_and_paths_resolve_relative_to_file(tmp_path: Path) -> None:
|
|
33
|
+
(tmp_path / "arsenal.yaml").write_text(MANIFEST, encoding="utf-8")
|
|
34
|
+
proj = load_project(tmp_path / "arsenal.yaml")
|
|
35
|
+
assert proj.name == "my-project"
|
|
36
|
+
assert proj.pipelines[0] == tmp_path / "collect.yaml"
|
|
37
|
+
assert proj.transforms[0] == tmp_path / "transform.yaml"
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def test_manifest_resolves_dbt_project_reference(tmp_path: Path) -> None:
|
|
41
|
+
(tmp_path / "arsenal.yaml").write_text(DBT_MANIFEST, encoding="utf-8")
|
|
42
|
+
proj = load_project(tmp_path / "arsenal.yaml")
|
|
43
|
+
assert len(proj.transforms) == 1
|
|
44
|
+
assert proj.transforms[0].dbt == tmp_path / "dbt_project" # type: ignore[union-attr]
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def test_load_project_missing_manifest_raises_fatal_error(tmp_path: Path) -> None:
|
|
48
|
+
with pytest.raises(FatalError):
|
|
49
|
+
load_project(tmp_path / "arsenal.yaml")
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def test_load_project_bad_yaml_raises_fatal_error(tmp_path: Path) -> None:
|
|
53
|
+
(tmp_path / "arsenal.yaml").write_text("name: [unterminated", encoding="utf-8")
|
|
54
|
+
with pytest.raises(FatalError):
|
|
55
|
+
load_project(tmp_path / "arsenal.yaml")
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
def test_load_project_missing_required_field_raises_fatal_error(tmp_path: Path) -> None:
|
|
59
|
+
(tmp_path / "arsenal.yaml").write_text("pipelines: []\n", encoding="utf-8")
|
|
60
|
+
with pytest.raises(FatalError, match="name"):
|
|
61
|
+
load_project(tmp_path / "arsenal.yaml")
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
def test_run_executes_pipelines_then_transforms(
|
|
65
|
+
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
66
|
+
) -> None:
|
|
67
|
+
(tmp_path / "arsenal.yaml").write_text(MANIFEST, encoding="utf-8")
|
|
68
|
+
calls: list[str] = []
|
|
69
|
+
|
|
70
|
+
def fake_run_pipeline(p: Path) -> None:
|
|
71
|
+
calls.append(f"collect:{p.name}")
|
|
72
|
+
|
|
73
|
+
def fake_run_transform(p: Path) -> None:
|
|
74
|
+
calls.append(f"transform:{p.name}")
|
|
75
|
+
|
|
76
|
+
monkeypatch.setattr("arsenal.cli._run_pipeline", fake_run_pipeline)
|
|
77
|
+
monkeypatch.setattr("arsenal.cli._run_transform", fake_run_transform)
|
|
78
|
+
result = runner.invoke(app, ["run", "--project", str(tmp_path)])
|
|
79
|
+
assert result.exit_code == 0, result.output
|
|
80
|
+
assert calls == ["collect:collect.yaml", "transform:transform.yaml"]
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
def test_run_executes_dbt_reference_after_pipelines(
|
|
84
|
+
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
85
|
+
) -> None:
|
|
86
|
+
(tmp_path / "arsenal.yaml").write_text(DBT_MANIFEST, encoding="utf-8")
|
|
87
|
+
(tmp_path / "dbt_project").mkdir()
|
|
88
|
+
calls: list[str] = []
|
|
89
|
+
|
|
90
|
+
def fake_run_dbt(path: Path) -> None:
|
|
91
|
+
calls.append(path.name)
|
|
92
|
+
|
|
93
|
+
monkeypatch.setattr("arsenal.cli._run_dbt", fake_run_dbt)
|
|
94
|
+
result = runner.invoke(app, ["run", "--project", str(tmp_path)])
|
|
95
|
+
assert result.exit_code == 0, result.output
|
|
96
|
+
assert calls == ["dbt_project"]
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
def test_run_bad_manifest_exits_with_clean_error(tmp_path: Path) -> None:
|
|
100
|
+
result = runner.invoke(app, ["run", "--project", str(tmp_path)])
|
|
101
|
+
assert result.exit_code == 1
|
|
102
|
+
assert "error:" in result.output
|
|
103
|
+
assert "Traceback" not in result.output
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
def test_run_missing_pipeline_spec_exits_clean_and_names_file(tmp_path: Path) -> None:
|
|
107
|
+
"""valid arsenal.yaml → collect.yaml referenced but absent — no raw traceback."""
|
|
108
|
+
(tmp_path / "arsenal.yaml").write_text(MANIFEST, encoding="utf-8")
|
|
109
|
+
# collect.yaml intentionally not created
|
|
110
|
+
result = runner.invoke(app, ["run", "--project", str(tmp_path)])
|
|
111
|
+
assert result.exit_code == 1
|
|
112
|
+
assert result.output.startswith("error:")
|
|
113
|
+
assert "Traceback" not in result.output
|
|
114
|
+
assert "collect.yaml" in result.output
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def test_run_missing_transform_spec_exits_clean_and_names_file(
|
|
118
|
+
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
119
|
+
) -> None:
|
|
120
|
+
"""valid arsenal.yaml → transform.yaml referenced but absent — no raw traceback."""
|
|
121
|
+
(tmp_path / "arsenal.yaml").write_text(MANIFEST, encoding="utf-8")
|
|
122
|
+
(tmp_path / "collect.yaml").write_text(
|
|
123
|
+
"unused", encoding="utf-8"
|
|
124
|
+
) # pipeline step is stubbed below
|
|
125
|
+
|
|
126
|
+
def fake_run_pipeline(p: Path) -> None:
|
|
127
|
+
pass
|
|
128
|
+
|
|
129
|
+
monkeypatch.setattr("arsenal.cli._run_pipeline", fake_run_pipeline)
|
|
130
|
+
# transform.yaml intentionally not created
|
|
131
|
+
result = runner.invoke(app, ["run", "--project", str(tmp_path)])
|
|
132
|
+
assert result.exit_code == 1
|
|
133
|
+
assert result.output.startswith("error:")
|
|
134
|
+
assert "Traceback" not in result.output
|
|
135
|
+
assert "transform.yaml" in result.output
|
|
136
|
+
|
|
137
|
+
|
|
138
|
+
def test_init_happy_path_copies_recipe_and_prints_guidance(tmp_path: Path) -> None:
|
|
139
|
+
dest = tmp_path / "proj"
|
|
140
|
+
result = runner.invoke(app, ["init", "csv-cleanup", "--dest", str(dest)])
|
|
141
|
+
assert result.exit_code == 0, result.output
|
|
142
|
+
assert (dest / "arsenal.yaml").exists()
|
|
143
|
+
assert (dest / "collect.yaml").exists()
|
|
144
|
+
assert (dest / "transform.yaml").exists()
|
|
145
|
+
assert (dest / "README.md").exists()
|
|
146
|
+
assert "arsenal run" in result.output
|
|
147
|
+
|
|
148
|
+
|
|
149
|
+
def test_init_refuses_to_overwrite_existing_files(tmp_path: Path) -> None:
|
|
150
|
+
dest = tmp_path / "proj"
|
|
151
|
+
dest.mkdir()
|
|
152
|
+
(dest / "arsenal.yaml").write_text("keep me", encoding="utf-8")
|
|
153
|
+
result = runner.invoke(app, ["init", "csv-cleanup", "--dest", str(dest)])
|
|
154
|
+
assert result.exit_code == 1
|
|
155
|
+
assert "error:" in result.output
|
|
156
|
+
assert (dest / "arsenal.yaml").read_text(encoding="utf-8") == "keep me"
|
|
157
|
+
assert not (dest / "collect.yaml").exists() # 부분 복사 없음 — 전부 아니면 전무
|
|
158
|
+
|
|
159
|
+
|
|
160
|
+
def test_init_list_prints_available_recipes() -> None:
|
|
161
|
+
result = runner.invoke(app, ["init", "--list"])
|
|
162
|
+
assert result.exit_code == 0
|
|
163
|
+
assert "github-issues" in result.output
|
|
164
|
+
assert "csv-cleanup" in result.output
|
|
165
|
+
|
|
166
|
+
|
|
167
|
+
def test_init_unknown_recipe_lists_available(tmp_path: Path) -> None:
|
|
168
|
+
result = runner.invoke(app, ["init", "does-not-exist", "--dest", str(tmp_path / "x")])
|
|
169
|
+
assert result.exit_code == 1
|
|
170
|
+
assert "error:" in result.output
|
|
171
|
+
assert "github-issues" in result.output
|
|
172
|
+
|
|
173
|
+
|
|
174
|
+
def test_init_without_recipe_or_list_is_a_clean_error() -> None:
|
|
175
|
+
result = runner.invoke(app, ["init"])
|
|
176
|
+
assert result.exit_code == 1
|
|
177
|
+
assert "error:" in result.output
|
|
178
|
+
|
|
179
|
+
|
|
180
|
+
def test_query_delegates_to_gladius_engine(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
181
|
+
table = pa.table({"a": [1, 2]})
|
|
182
|
+
|
|
183
|
+
def fake_query(sql: str) -> pa.Table:
|
|
184
|
+
return table
|
|
185
|
+
|
|
186
|
+
monkeypatch.setattr("gladius.engine.query", fake_query)
|
|
187
|
+
result = runner.invoke(app, ["query", "SELECT 1", "--format", "jsonl"])
|
|
188
|
+
assert result.exit_code == 0, result.output
|
|
189
|
+
rows = [json.loads(line) for line in result.output.strip().splitlines()]
|
|
190
|
+
assert rows == [{"a": 1}, {"a": 2}]
|
|
191
|
+
|
|
192
|
+
|
|
193
|
+
def test_query_wraps_arsenal_error_as_clean_exit(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
194
|
+
def boom(sql: str) -> pa.Table:
|
|
195
|
+
raise FatalError("bad sql")
|
|
196
|
+
|
|
197
|
+
monkeypatch.setattr("gladius.engine.query", boom)
|
|
198
|
+
result = runner.invoke(app, ["query", "not sql"])
|
|
199
|
+
assert result.exit_code == 1
|
|
200
|
+
assert "error: bad sql" in result.output
|
|
201
|
+
|
|
202
|
+
|
|
203
|
+
def test_query_format_table_renders_via_duckdb(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
204
|
+
table = pa.table({"a": [1, 2]})
|
|
205
|
+
|
|
206
|
+
def fake_query(sql: str) -> pa.Table:
|
|
207
|
+
return table
|
|
208
|
+
|
|
209
|
+
monkeypatch.setattr("gladius.engine.query", fake_query)
|
|
210
|
+
result = runner.invoke(app, ["query", "SELECT 1", "--format", "table"])
|
|
211
|
+
assert result.exit_code == 0, result.output
|
|
212
|
+
assert "a" in result.output
|
|
213
|
+
assert "1" in result.output
|
|
214
|
+
assert "2" in result.output
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
def test_query_format_csv(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
218
|
+
table = pa.table({"a": [1, 2]})
|
|
219
|
+
|
|
220
|
+
def fake_query(sql: str) -> pa.Table:
|
|
221
|
+
return table
|
|
222
|
+
|
|
223
|
+
monkeypatch.setattr("gladius.engine.query", fake_query)
|
|
224
|
+
result = runner.invoke(app, ["query", "SELECT 1", "--format", "csv"])
|
|
225
|
+
assert result.exit_code == 0, result.output
|
|
226
|
+
lines = result.output.strip().splitlines()
|
|
227
|
+
assert lines[0].strip('"') == "a"
|
|
228
|
+
assert lines[1] == "1"
|
|
229
|
+
assert lines[2] == "2"
|
|
230
|
+
|
|
231
|
+
|
|
232
|
+
def test_query_format_unsupported_is_clean_error(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
233
|
+
table = pa.table({"a": [1, 2]})
|
|
234
|
+
|
|
235
|
+
def fake_query(sql: str) -> pa.Table:
|
|
236
|
+
return table
|
|
237
|
+
|
|
238
|
+
monkeypatch.setattr("gladius.engine.query", fake_query)
|
|
239
|
+
result = runner.invoke(app, ["query", "SELECT 1", "--format", "bogus"])
|
|
240
|
+
assert result.exit_code == 1
|
|
241
|
+
assert "error:" in result.output
|
|
242
|
+
assert "bogus" in result.output
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
"""스켈레톤 배선 검증 — 공개 API가 전부 import 가능하고 이름이 살아 있어야 한다."""
|
|
2
|
+
|
|
3
|
+
from importlib import resources
|
|
4
|
+
|
|
5
|
+
import arsenal
|
|
6
|
+
from arsenal.cli import app
|
|
7
|
+
from arsenal.project import ArsenalProject, load_project
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
def test_version() -> None:
|
|
11
|
+
assert arsenal.__version__
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
def test_public_api_is_wired() -> None:
|
|
15
|
+
assert callable(load_project)
|
|
16
|
+
assert app.info is not None
|
|
17
|
+
assert ArsenalProject is not None
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
def test_recipe_shape_example_ships_with_package() -> None:
|
|
21
|
+
"""레시피는 패키지 데이터다 — github-issues가 형태 예시로 포함되어야 한다."""
|
|
22
|
+
recipe = resources.files("arsenal") / "recipes" / "github-issues"
|
|
23
|
+
names = {p.name for p in recipe.iterdir()}
|
|
24
|
+
assert {"arsenal.yaml", "collect.yaml", "transform.yaml", "README.md"} <= names
|