pseek 3.1.2__tar.gz → 3.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pseek-3.1.2 → pseek-3.2.0}/PKG-INFO +66 -11
- {pseek-3.1.2 → pseek-3.2.0}/README.md +65 -10
- {pseek-3.1.2 → pseek-3.2.0}/pseek/archive.py +19 -26
- {pseek-3.1.2 → pseek-3.2.0}/pseek/cli.py +45 -32
- {pseek-3.1.2 → pseek-3.2.0}/pseek/parser.py +0 -2
- {pseek-3.1.2 → pseek-3.2.0}/pseek/searcher.py +132 -54
- {pseek-3.1.2 → pseek-3.2.0}/pseek/structs.py +61 -43
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/PKG-INFO +66 -11
- {pseek-3.1.2 → pseek-3.2.0}/pyproject.toml +1 -1
- {pseek-3.1.2 → pseek-3.2.0}/LICENSE +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek/__init__.py +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek/compound_extensions +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek/utils.py +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/SOURCES.txt +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/dependency_links.txt +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/entry_points.txt +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/requires.txt +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/pseek.egg-info/top_level.txt +0 -0
- {pseek-3.1.2 → pseek-3.2.0}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pseek
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.2.0
|
|
4
4
|
Summary: Pseek is a Python CLI toolkit to search files, folders, and text
|
|
5
5
|
Author: Groot
|
|
6
6
|
License-Expression: MIT
|
|
@@ -112,12 +112,14 @@ psk -- --path
|
|
|
112
112
|
| `--re-include`, `--re-exclude` | Limit search results to specific directories or files with regex |
|
|
113
113
|
| `--word` | Match the whole word only (except when `--expr` is enabled, in which case you can make it match whole word by putting `w` before term: `w"foo"`) |
|
|
114
114
|
| `--expr` | Enable boolean query expressions. Example: `r"foo.*bar" and ("bar" or "baz") and not "qux"`. Prefixes: `r=regex`, `c=case-sensitive`, `w=whole-word`, `f=fuzzy` |
|
|
115
|
+
| `--context` | Show N context lines before and after matches, or use N:M to specify before:after |
|
|
116
|
+
| `--depth` | Limit directory traversal to given depth range. By default, there is no limit on search depth |
|
|
115
117
|
| `--timeout` | Stop the search after the specified number of seconds |
|
|
116
118
|
| `--fuzzy` | Enable fuzzy search (Highlighting and counting matches are disabled in this mode if `--word` is not enabled to prevent the program from slowing down). except when `--expr` is enabled, in which case you can make it fuzzy by putting `f` before term: `f"foo"` |
|
|
117
119
|
| `--fuzzy-level` | Fuzzy matching threshold (0-99). Higher values require closer matches (default: `80`) |
|
|
118
120
|
| `--size` | Limit results based on the size of files |
|
|
119
121
|
| `--archive` | Enable search within archive files (e.g. `zip`, `rar`, `7z`, `gz`, `bz2`, `xz`, `tar`, `tar.gz`, `tar.bz2`, `tar.xz`) |
|
|
120
|
-
| `--depth` |
|
|
122
|
+
| `--arc-depth` | Limit nested archive to given depth range. By default, there is no limit |
|
|
121
123
|
| `--arc-ext`, `--arc-exc-ext` | Filter by file extension inside archive files |
|
|
122
124
|
| `--arc-include`, `--arc-exclude` | Limit search results to specific set of directories or files inside archive files |
|
|
123
125
|
| `--arc-size` | Limit results based on the size of files in the archive |
|
|
@@ -176,7 +178,7 @@ Example:
|
|
|
176
178
|
psk "hello world"
|
|
177
179
|
```
|
|
178
180
|
|
|
179
|
-
**Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
181
|
+
> **Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
180
182
|
|
|
181
183
|
### Case Sensitive Search
|
|
182
184
|
|
|
@@ -364,7 +366,51 @@ simultaneously.
|
|
|
364
366
|
|
|
365
367
|
Allowed modes: `r`, `c`, `w`, `f`, `rc`, `cr`, `cw`, `wc`, `cf`, `fc`, `wf`, `fw`, `cwf`, `cfw`, `wcf`, `wfc`, `fcw`, `fwc`
|
|
366
368
|
|
|
367
|
-
**Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
369
|
+
> **Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
370
|
+
|
|
371
|
+
## Context
|
|
372
|
+
|
|
373
|
+
Show context lines around each match. The value is specified as `BEFORE:AFTER`:
|
|
374
|
+
|
|
375
|
+
```bash
|
|
376
|
+
--context 2 # 2 lines before and after
|
|
377
|
+
--context 2:0 # 2 lines before, none after
|
|
378
|
+
--context 0:2 # none before, 2 lines after
|
|
379
|
+
--context 2:5 # 2 lines before, 5 lines after
|
|
380
|
+
--context 2: # 2 lines before, none after
|
|
381
|
+
--context :2 # none before, 2 lines after
|
|
382
|
+
--context 0 # matching lines only
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
Nearby matches whose context ranges overlap or directly touch are combined into a single group.
|
|
386
|
+
|
|
387
|
+
## Depth
|
|
388
|
+
|
|
389
|
+
Controls how deep the search goes inside directories.
|
|
390
|
+
|
|
391
|
+
A depth of `0` means that only base path's immediate contents are searched; subdirectories are not entered.
|
|
392
|
+
|
|
393
|
+
```text
|
|
394
|
+
project/ ← base directory
|
|
395
|
+
├── file.txt ← depth 0
|
|
396
|
+
├── folder1/ ← depth 0
|
|
397
|
+
│ ├── file.txt ← depth 1
|
|
398
|
+
│ └── folder2/ ← depth 1
|
|
399
|
+
│ └── file.txt ← depth 2
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
`--depth` can be specified multiple times to search non-contiguous depth ranges:
|
|
403
|
+
|
|
404
|
+
```bash
|
|
405
|
+
--depth 0 # Direct contents only
|
|
406
|
+
--depth 1 # One level inside
|
|
407
|
+
--depth 2:4 # Depths 2 through 4
|
|
408
|
+
--depth 2 --depth 5:7 # Depths 2 and 5 through 7
|
|
409
|
+
--depth :2 # Depths 0 through 2
|
|
410
|
+
--depth 3: # Depth 3 and deeper
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
If `--depth` is not specified, the search has no depth limit.
|
|
368
414
|
|
|
369
415
|
## Extension Filters
|
|
370
416
|
|
|
@@ -402,7 +448,7 @@ psk TODO \
|
|
|
402
448
|
|
|
403
449
|
Skip those paths.
|
|
404
450
|
|
|
405
|
-
**Note:** The include and exclude paths will be combined with path argument.
|
|
451
|
+
> **Note:** The include and exclude paths will be combined with path argument.
|
|
406
452
|
|
|
407
453
|
## Regex Path Filters
|
|
408
454
|
|
|
@@ -455,7 +501,7 @@ psk config \
|
|
|
455
501
|
|
|
456
502
|
Files between `1 MiB` and `10 MiB` will not match.
|
|
457
503
|
|
|
458
|
-
**Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
504
|
+
> **Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
459
505
|
|
|
460
506
|
## Archive Search
|
|
461
507
|
|
|
@@ -487,15 +533,24 @@ backup.zip::source.7z::notes.txt
|
|
|
487
533
|
|
|
488
534
|
### Archive Depth
|
|
489
535
|
|
|
490
|
-
|
|
536
|
+
The concept of depth in archive is different from `--depth`. Depth in archive is calculated based on nested archives. Each archive that is inside another archive counts as one level of depth.
|
|
537
|
+
|
|
538
|
+
| Syntax | Meaning |
|
|
539
|
+
| ------ | -------------------------------------------------------- |
|
|
540
|
+
| `2` | exactly `2nd` depth |
|
|
541
|
+
| `:2` | `2nd` depth or lower |
|
|
542
|
+
| `2:` | `2nd` depth or higher |
|
|
543
|
+
| `2:4` | between `2` and `4` (Both range boundaries are included) |
|
|
544
|
+
|
|
545
|
+
`--arc-depth` can be specified multiple times:
|
|
491
546
|
|
|
492
547
|
```bash
|
|
493
|
-
psk TODO --archive --depth 2
|
|
548
|
+
psk TODO --archive --arc-depth :2 --arc-depth 5:
|
|
494
549
|
```
|
|
495
550
|
|
|
496
|
-
Meaning `archive level
|
|
551
|
+
Meaning `archive level 3` and `archive level 4` won't be searched.
|
|
497
552
|
|
|
498
|
-
**Note:** `--depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
553
|
+
> **Note:** `--arc-depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
499
554
|
|
|
500
555
|
## Archive Filters
|
|
501
556
|
|
|
@@ -535,7 +590,7 @@ This works exactly like the [--size](#size-filtering) option.
|
|
|
535
590
|
psk TODO --archive --arc-size 10m:40m
|
|
536
591
|
```
|
|
537
592
|
|
|
538
|
-
**Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
593
|
+
> **Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
539
594
|
|
|
540
595
|
## RAR Backend
|
|
541
596
|
|
|
@@ -83,12 +83,14 @@ psk -- --path
|
|
|
83
83
|
| `--re-include`, `--re-exclude` | Limit search results to specific directories or files with regex |
|
|
84
84
|
| `--word` | Match the whole word only (except when `--expr` is enabled, in which case you can make it match whole word by putting `w` before term: `w"foo"`) |
|
|
85
85
|
| `--expr` | Enable boolean query expressions. Example: `r"foo.*bar" and ("bar" or "baz") and not "qux"`. Prefixes: `r=regex`, `c=case-sensitive`, `w=whole-word`, `f=fuzzy` |
|
|
86
|
+
| `--context` | Show N context lines before and after matches, or use N:M to specify before:after |
|
|
87
|
+
| `--depth` | Limit directory traversal to given depth range. By default, there is no limit on search depth |
|
|
86
88
|
| `--timeout` | Stop the search after the specified number of seconds |
|
|
87
89
|
| `--fuzzy` | Enable fuzzy search (Highlighting and counting matches are disabled in this mode if `--word` is not enabled to prevent the program from slowing down). except when `--expr` is enabled, in which case you can make it fuzzy by putting `f` before term: `f"foo"` |
|
|
88
90
|
| `--fuzzy-level` | Fuzzy matching threshold (0-99). Higher values require closer matches (default: `80`) |
|
|
89
91
|
| `--size` | Limit results based on the size of files |
|
|
90
92
|
| `--archive` | Enable search within archive files (e.g. `zip`, `rar`, `7z`, `gz`, `bz2`, `xz`, `tar`, `tar.gz`, `tar.bz2`, `tar.xz`) |
|
|
91
|
-
| `--depth` |
|
|
93
|
+
| `--arc-depth` | Limit nested archive to given depth range. By default, there is no limit |
|
|
92
94
|
| `--arc-ext`, `--arc-exc-ext` | Filter by file extension inside archive files |
|
|
93
95
|
| `--arc-include`, `--arc-exclude` | Limit search results to specific set of directories or files inside archive files |
|
|
94
96
|
| `--arc-size` | Limit results based on the size of files in the archive |
|
|
@@ -147,7 +149,7 @@ Example:
|
|
|
147
149
|
psk "hello world"
|
|
148
150
|
```
|
|
149
151
|
|
|
150
|
-
**Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
152
|
+
> **Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
151
153
|
|
|
152
154
|
### Case Sensitive Search
|
|
153
155
|
|
|
@@ -335,7 +337,51 @@ simultaneously.
|
|
|
335
337
|
|
|
336
338
|
Allowed modes: `r`, `c`, `w`, `f`, `rc`, `cr`, `cw`, `wc`, `cf`, `fc`, `wf`, `fw`, `cwf`, `cfw`, `wcf`, `wfc`, `fcw`, `fwc`
|
|
337
339
|
|
|
338
|
-
**Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
340
|
+
> **Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
341
|
+
|
|
342
|
+
## Context
|
|
343
|
+
|
|
344
|
+
Show context lines around each match. The value is specified as `BEFORE:AFTER`:
|
|
345
|
+
|
|
346
|
+
```bash
|
|
347
|
+
--context 2 # 2 lines before and after
|
|
348
|
+
--context 2:0 # 2 lines before, none after
|
|
349
|
+
--context 0:2 # none before, 2 lines after
|
|
350
|
+
--context 2:5 # 2 lines before, 5 lines after
|
|
351
|
+
--context 2: # 2 lines before, none after
|
|
352
|
+
--context :2 # none before, 2 lines after
|
|
353
|
+
--context 0 # matching lines only
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Nearby matches whose context ranges overlap or directly touch are combined into a single group.
|
|
357
|
+
|
|
358
|
+
## Depth
|
|
359
|
+
|
|
360
|
+
Controls how deep the search goes inside directories.
|
|
361
|
+
|
|
362
|
+
A depth of `0` means that only base path's immediate contents are searched; subdirectories are not entered.
|
|
363
|
+
|
|
364
|
+
```text
|
|
365
|
+
project/ ← base directory
|
|
366
|
+
├── file.txt ← depth 0
|
|
367
|
+
├── folder1/ ← depth 0
|
|
368
|
+
│ ├── file.txt ← depth 1
|
|
369
|
+
│ └── folder2/ ← depth 1
|
|
370
|
+
│ └── file.txt ← depth 2
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
`--depth` can be specified multiple times to search non-contiguous depth ranges:
|
|
374
|
+
|
|
375
|
+
```bash
|
|
376
|
+
--depth 0 # Direct contents only
|
|
377
|
+
--depth 1 # One level inside
|
|
378
|
+
--depth 2:4 # Depths 2 through 4
|
|
379
|
+
--depth 2 --depth 5:7 # Depths 2 and 5 through 7
|
|
380
|
+
--depth :2 # Depths 0 through 2
|
|
381
|
+
--depth 3: # Depth 3 and deeper
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
If `--depth` is not specified, the search has no depth limit.
|
|
339
385
|
|
|
340
386
|
## Extension Filters
|
|
341
387
|
|
|
@@ -373,7 +419,7 @@ psk TODO \
|
|
|
373
419
|
|
|
374
420
|
Skip those paths.
|
|
375
421
|
|
|
376
|
-
**Note:** The include and exclude paths will be combined with path argument.
|
|
422
|
+
> **Note:** The include and exclude paths will be combined with path argument.
|
|
377
423
|
|
|
378
424
|
## Regex Path Filters
|
|
379
425
|
|
|
@@ -426,7 +472,7 @@ psk config \
|
|
|
426
472
|
|
|
427
473
|
Files between `1 MiB` and `10 MiB` will not match.
|
|
428
474
|
|
|
429
|
-
**Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
475
|
+
> **Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
430
476
|
|
|
431
477
|
## Archive Search
|
|
432
478
|
|
|
@@ -458,15 +504,24 @@ backup.zip::source.7z::notes.txt
|
|
|
458
504
|
|
|
459
505
|
### Archive Depth
|
|
460
506
|
|
|
461
|
-
|
|
507
|
+
The concept of depth in archive is different from `--depth`. Depth in archive is calculated based on nested archives. Each archive that is inside another archive counts as one level of depth.
|
|
508
|
+
|
|
509
|
+
| Syntax | Meaning |
|
|
510
|
+
| ------ | -------------------------------------------------------- |
|
|
511
|
+
| `2` | exactly `2nd` depth |
|
|
512
|
+
| `:2` | `2nd` depth or lower |
|
|
513
|
+
| `2:` | `2nd` depth or higher |
|
|
514
|
+
| `2:4` | between `2` and `4` (Both range boundaries are included) |
|
|
515
|
+
|
|
516
|
+
`--arc-depth` can be specified multiple times:
|
|
462
517
|
|
|
463
518
|
```bash
|
|
464
|
-
psk TODO --archive --depth 2
|
|
519
|
+
psk TODO --archive --arc-depth :2 --arc-depth 5:
|
|
465
520
|
```
|
|
466
521
|
|
|
467
|
-
Meaning `archive level
|
|
522
|
+
Meaning `archive level 3` and `archive level 4` won't be searched.
|
|
468
523
|
|
|
469
|
-
**Note:** `--depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
524
|
+
> **Note:** `--arc-depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
470
525
|
|
|
471
526
|
## Archive Filters
|
|
472
527
|
|
|
@@ -506,7 +561,7 @@ This works exactly like the [--size](#size-filtering) option.
|
|
|
506
561
|
psk TODO --archive --arc-size 10m:40m
|
|
507
562
|
```
|
|
508
563
|
|
|
509
|
-
**Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
564
|
+
> **Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
510
565
|
|
|
511
566
|
## RAR Backend
|
|
512
567
|
|
|
@@ -39,7 +39,7 @@ def archive_should_skip(path_info: Path, config, p_size: float, file_ext, is_dir
|
|
|
39
39
|
return False
|
|
40
40
|
|
|
41
41
|
|
|
42
|
-
def extract_names_from_archive(file_path: Path, config, depth: int
|
|
42
|
+
def extract_names_from_archive(file_path: Path, config, depth: int = -1,
|
|
43
43
|
file_bytes: bytes | None = None, parent_label: list[str] | None = None):
|
|
44
44
|
"""
|
|
45
45
|
Recursively extract files and directories name from archive files.
|
|
@@ -57,8 +57,7 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
57
57
|
|
|
58
58
|
file_ext = get_path_suffix(file_path)
|
|
59
59
|
label_prefix = [*parent_label, str(file_path)] if parent_label is not None else []
|
|
60
|
-
|
|
61
|
-
depth = config.depth
|
|
60
|
+
new_depth = depth + 1
|
|
62
61
|
|
|
63
62
|
try:
|
|
64
63
|
# Decide the stream source: from disk or memory
|
|
@@ -72,8 +71,6 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
72
71
|
name = Path(info.filename)
|
|
73
72
|
new_path_ext = get_path_suffix(name)
|
|
74
73
|
is_dir = info.is_dir()
|
|
75
|
-
# At each recursion, subtract 1 from depth if it's set
|
|
76
|
-
new_depth = None if depth is None else depth - 1
|
|
77
74
|
|
|
78
75
|
if not archive_should_skip(
|
|
79
76
|
name,
|
|
@@ -82,10 +79,11 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
82
79
|
new_path_ext,
|
|
83
80
|
is_dir
|
|
84
81
|
):
|
|
85
|
-
|
|
82
|
+
if any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
83
|
+
yield label_prefix, name, is_dir
|
|
86
84
|
|
|
87
85
|
# Check if this is a nested archive
|
|
88
|
-
if new_path_ext in ARCHIVE_EXTS[:-3] and (new_depth
|
|
86
|
+
if new_path_ext in ARCHIVE_EXTS[:-3] and any(new_depth < d[1] for d in config.arc_depth):
|
|
89
87
|
yield from extract_names_from_archive(
|
|
90
88
|
name,
|
|
91
89
|
config,
|
|
@@ -100,7 +98,6 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
100
98
|
name = Path(info.filename)
|
|
101
99
|
new_path_ext = get_path_suffix(name)
|
|
102
100
|
is_dir = info.is_directory
|
|
103
|
-
new_depth = None if depth is None else depth - 1
|
|
104
101
|
|
|
105
102
|
if not archive_should_skip(
|
|
106
103
|
name,
|
|
@@ -109,9 +106,10 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
109
106
|
new_path_ext,
|
|
110
107
|
is_dir
|
|
111
108
|
):
|
|
112
|
-
|
|
109
|
+
if any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
110
|
+
yield label_prefix, name, is_dir
|
|
113
111
|
|
|
114
|
-
if new_path_ext in ARCHIVE_EXTS[:-3] and (new_depth
|
|
112
|
+
if new_path_ext in ARCHIVE_EXTS[:-3] and any(new_depth < d[1] for d in config.arc_depth):
|
|
115
113
|
file_data = z.read([info.filename]).get(info.filename)
|
|
116
114
|
if file_data is None:
|
|
117
115
|
continue
|
|
@@ -138,7 +136,6 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
138
136
|
name = Path(member.name)
|
|
139
137
|
new_path_ext = get_path_suffix(name)
|
|
140
138
|
is_dir = member.isdir()
|
|
141
|
-
new_depth = None if depth is None else depth - 1
|
|
142
139
|
|
|
143
140
|
if not archive_should_skip(
|
|
144
141
|
name,
|
|
@@ -147,9 +144,10 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
147
144
|
new_path_ext,
|
|
148
145
|
is_dir
|
|
149
146
|
):
|
|
150
|
-
|
|
147
|
+
if any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
148
|
+
yield label_prefix, name, is_dir
|
|
151
149
|
|
|
152
|
-
if new_path_ext in ARCHIVE_EXTS[:-3] and (new_depth
|
|
150
|
+
if new_path_ext in ARCHIVE_EXTS[:-3] and any(new_depth < d[1] for d in config.arc_depth):
|
|
153
151
|
f = tf.extractfile(member)
|
|
154
152
|
if f is None:
|
|
155
153
|
continue
|
|
@@ -165,7 +163,7 @@ def extract_names_from_archive(file_path: Path, config, depth: int | None = None
|
|
|
165
163
|
return # silently skip invalid or unreadable archives
|
|
166
164
|
|
|
167
165
|
|
|
168
|
-
def extract_text_from_archive(file_path: Path, config, depth: int
|
|
166
|
+
def extract_text_from_archive(file_path: Path, config, depth: int = -1,
|
|
169
167
|
file_bytes: bytes | None = None, parent_label: list[str] | None = None):
|
|
170
168
|
"""
|
|
171
169
|
Recursively extract (path_label, text_content) from any archive file.
|
|
@@ -183,8 +181,7 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
183
181
|
|
|
184
182
|
file_ext = get_path_suffix(file_path)
|
|
185
183
|
label_prefix = [*parent_label, str(file_path)] if parent_label is not None else []
|
|
186
|
-
|
|
187
|
-
depth = config.depth
|
|
184
|
+
new_depth = depth + 1
|
|
188
185
|
|
|
189
186
|
try:
|
|
190
187
|
# Decide the stream source: from disk or memory
|
|
@@ -199,8 +196,6 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
199
196
|
continue
|
|
200
197
|
|
|
201
198
|
file_name = Path(info.filename)
|
|
202
|
-
# At each recursion, subtract 1 from depth if it's set
|
|
203
|
-
new_depth = None if depth is None else depth - 1
|
|
204
199
|
new_path_ext = get_path_suffix(file_name)
|
|
205
200
|
|
|
206
201
|
if archive_should_skip(
|
|
@@ -222,9 +217,9 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
222
217
|
|
|
223
218
|
# Check if this is a nested archive
|
|
224
219
|
if new_path_ext in ARCHIVE_EXTS:
|
|
225
|
-
if (new_depth
|
|
220
|
+
if any(new_depth < d[1] for d in config.arc_depth):
|
|
226
221
|
yield from extract_text_from_archive(file_name, config, new_depth, data, label_prefix)
|
|
227
|
-
|
|
222
|
+
elif any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
228
223
|
yield [*label_prefix, str(file_name)], data
|
|
229
224
|
# Handle 7Z archives
|
|
230
225
|
elif file_ext == '7z':
|
|
@@ -238,7 +233,6 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
238
233
|
continue
|
|
239
234
|
|
|
240
235
|
file_name = Path(info.filename)
|
|
241
|
-
new_depth = None if depth is None else depth - 1
|
|
242
236
|
new_path_ext = get_path_suffix(file_name)
|
|
243
237
|
|
|
244
238
|
if archive_should_skip(
|
|
@@ -255,9 +249,9 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
255
249
|
|
|
256
250
|
data = head + file_data.read()
|
|
257
251
|
if new_path_ext in ARCHIVE_EXTS:
|
|
258
|
-
if (new_depth
|
|
252
|
+
if any(new_depth < d[1] for d in config.arc_depth):
|
|
259
253
|
yield from extract_text_from_archive(file_name, config, new_depth, data, label_prefix)
|
|
260
|
-
|
|
254
|
+
elif any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
261
255
|
yield [*label_prefix, str(file_name)], data
|
|
262
256
|
# Handle TAR and compressed TAR formats
|
|
263
257
|
elif file_ext in ('tar', 'tar.gz', 'tar.bz2', 'tar.xz'):
|
|
@@ -278,7 +272,6 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
278
272
|
continue
|
|
279
273
|
|
|
280
274
|
file_name = Path(member.name)
|
|
281
|
-
new_depth = None if depth is None else depth - 1
|
|
282
275
|
new_path_ext = get_path_suffix(file_name)
|
|
283
276
|
|
|
284
277
|
if archive_should_skip(
|
|
@@ -295,9 +288,9 @@ def extract_text_from_archive(file_path: Path, config, depth: int | None = None,
|
|
|
295
288
|
|
|
296
289
|
data = head + f.read()
|
|
297
290
|
if new_path_ext in ARCHIVE_EXTS:
|
|
298
|
-
if (new_depth
|
|
291
|
+
if any(new_depth < d[1] for d in config.arc_depth):
|
|
299
292
|
yield from extract_text_from_archive(file_name, config, new_depth, data, label_prefix)
|
|
300
|
-
|
|
293
|
+
elif any(mi <= new_depth <= ma for mi, ma in config.arc_depth):
|
|
301
294
|
yield [*label_prefix, str(file_name)], data
|
|
302
295
|
# Handle single compressed files like .gz, .bz2, .xz
|
|
303
296
|
elif file_ext in ARCHIVE_EXTS[-3:]:
|
|
@@ -11,6 +11,10 @@ BACKEND_PATH = Path(__file__).parent / "RARBackend"
|
|
|
11
11
|
|
|
12
12
|
def merge_matches(matches: list[tuple[int, int]]):
|
|
13
13
|
"""Merge overlapping matches (for example, if one match was inside another match)"""
|
|
14
|
+
|
|
15
|
+
# sort by start position
|
|
16
|
+
matches.sort()
|
|
17
|
+
|
|
14
18
|
merged = []
|
|
15
19
|
for start, end in matches:
|
|
16
20
|
if not merged or start > merged[-1][1]: # No overlap
|
|
@@ -32,14 +36,14 @@ def build_highlight(text, matches):
|
|
|
32
36
|
|
|
33
37
|
for start, end in matches:
|
|
34
38
|
parts.append(text[last:start])
|
|
35
|
-
parts.append(click.style(text[start:end], fg='
|
|
39
|
+
parts.append(click.style(text[start:end], fg='red', bold=True))
|
|
36
40
|
last = end
|
|
37
41
|
parts.append(text[last:])
|
|
38
42
|
|
|
39
43
|
return ''.join(parts)
|
|
40
44
|
|
|
41
45
|
|
|
42
|
-
def echo(results: dict):
|
|
46
|
+
def echo(results: dict, context: bool):
|
|
43
47
|
"""Display results with a specific format and color scheme"""
|
|
44
48
|
for match_type, datas in results.items():
|
|
45
49
|
if datas:
|
|
@@ -62,26 +66,31 @@ def echo(results: dict):
|
|
|
62
66
|
)]
|
|
63
67
|
)
|
|
64
68
|
)
|
|
65
|
-
|
|
66
|
-
# Print file lines
|
|
67
|
-
for line in data.lines:
|
|
68
|
-
matches = merge_matches(line.matches)
|
|
69
|
-
# Keep the original count so overlapping matches are still counted separately
|
|
70
|
-
count = len(line.matches)
|
|
71
|
-
|
|
72
|
-
# Show a note if the pattern repeats 3 or more times
|
|
73
|
-
count_query = f' ({count} matches)' if count >= 3 else ''
|
|
74
|
-
|
|
75
|
-
prefix = click.style(
|
|
76
|
-
f"Line {line.number}{count_query}: ",
|
|
77
|
-
fg="magenta",
|
|
78
|
-
)
|
|
79
|
-
output_line = prefix + build_highlight(line.text, matches)
|
|
80
69
|
|
|
81
|
-
|
|
82
|
-
|
|
70
|
+
counter = 0
|
|
71
|
+
for group in data.groups:
|
|
72
|
+
if any(context):
|
|
73
|
+
if counter != 0:
|
|
74
|
+
print('---')
|
|
75
|
+
else:
|
|
76
|
+
counter = 1
|
|
77
|
+
|
|
78
|
+
for line in group.lines:
|
|
79
|
+
if line.number in group.matching_lines:
|
|
80
|
+
matches = merge_matches(line.matches)
|
|
81
|
+
|
|
82
|
+
prefix = click.style(
|
|
83
|
+
f'{line.number}' + (':' if line.number in group.matching_lines else '-'),
|
|
84
|
+
fg='magenta'
|
|
85
|
+
)
|
|
86
|
+
output_line = prefix + (build_highlight(line.text, matches) \
|
|
87
|
+
if line.number in group.matching_lines \
|
|
88
|
+
else line.text)
|
|
89
|
+
|
|
90
|
+
print(output_line)
|
|
91
|
+
|
|
83
92
|
# Print a blank line to separate results
|
|
84
|
-
if data.
|
|
93
|
+
if data.groups:
|
|
85
94
|
print()
|
|
86
95
|
else: # Print a file or directory match, including archive paths
|
|
87
96
|
matches = merge_matches(data.matches)
|
|
@@ -93,13 +102,13 @@ def echo(results: dict):
|
|
|
93
102
|
matches
|
|
94
103
|
)
|
|
95
104
|
|
|
96
|
-
|
|
105
|
+
print(
|
|
97
106
|
separator.join(
|
|
98
107
|
[data.path, *data.virtual_path[0:-1], virtual_file_name]
|
|
99
108
|
)
|
|
100
109
|
)
|
|
101
110
|
else:
|
|
102
|
-
|
|
111
|
+
print(
|
|
103
112
|
build_highlight(data.path, matches)
|
|
104
113
|
)
|
|
105
114
|
|
|
@@ -120,11 +129,11 @@ def echo_stats(config, results, metrics, elapsed_time=None):
|
|
|
120
129
|
if config.content:
|
|
121
130
|
click.echo(INDENT + f'Files with matched content: {len(results["content"]):,}')
|
|
122
131
|
|
|
123
|
-
lines_matched = sum(len(
|
|
132
|
+
lines_matched = sum(len(group.matching_lines) for file in results["content"] for group in file.groups)
|
|
124
133
|
matches = sum(
|
|
125
134
|
len(line.matches)
|
|
126
135
|
for file in results["content"]
|
|
127
|
-
for
|
|
136
|
+
for group in file.groups for line in group.lines
|
|
128
137
|
)
|
|
129
138
|
|
|
130
139
|
if lines_matched:
|
|
@@ -305,7 +314,7 @@ def check_rar_backend(archive_enabled: bool, tool_path: str, backend: str):
|
|
|
305
314
|
@click.option('-d', '--directory', is_flag=True, help='Search only in directory names.')
|
|
306
315
|
@click.option('-c', '--content', is_flag=True, help='Search within file contents.')
|
|
307
316
|
# Additional options
|
|
308
|
-
@click.option('-
|
|
317
|
+
@click.option('-s', '--case-sensitive', is_flag=True,
|
|
309
318
|
help='Make the search case-sensitive '
|
|
310
319
|
'(except when --expr is enabled, '
|
|
311
320
|
'in which case you can make it case sensitive by putting c before term: c"foo")')
|
|
@@ -320,13 +329,17 @@ def check_rar_backend(archive_enabled: bool, tool_path: str, backend: str):
|
|
|
320
329
|
@click.option('--expr', is_flag=True,
|
|
321
330
|
help='Enable boolean query expressions. Example: r"foo.*bar" and ("bar" or "baz") and not "qux". '
|
|
322
331
|
'Prefixes: r=regex, c=case-sensitive, w=whole-word, f=fuzzy.')
|
|
323
|
-
@click.option('--timeout', type=click.FloatRange(min=0, min_open=True),
|
|
324
|
-
help='Stop the search after the specified number of seconds.')
|
|
325
332
|
@click.option('--fuzzy', is_flag=True, help='Enable fuzzy search (approximate matching). '
|
|
326
333
|
'except when --expr is enabled, '
|
|
327
334
|
'in which case you can make it fuzzy by putting f before term: f"foo"')
|
|
328
335
|
@click.option('--fuzzy-level', type=click.IntRange(1, 99), default=80, show_default=True,
|
|
329
336
|
help='Fuzzy matching threshold (1-99). Higher values require closer matches.')
|
|
337
|
+
@click.option('-C', '--context', help='Show N context lines before and after matches, '
|
|
338
|
+
'or use N:M to specify before:after.')
|
|
339
|
+
@click.option('--depth', multiple=True, help='Limit directory traversal to given depth range. '
|
|
340
|
+
'By default, there is no limit on search depth.')
|
|
341
|
+
@click.option('--timeout', type=click.FloatRange(min=0, min_open=True),
|
|
342
|
+
help='Stop the search after the specified number of seconds.')
|
|
330
343
|
# Extension filters
|
|
331
344
|
@click.option('--ext', multiple=True, type=click.STRING,
|
|
332
345
|
help='Include files with these extensions. Example: --ext py --ext js')
|
|
@@ -346,8 +359,8 @@ def check_rar_backend(archive_enabled: bool, tool_path: str, backend: str):
|
|
|
346
359
|
# Archive options
|
|
347
360
|
@click.option('--archive', is_flag=True,
|
|
348
361
|
help='Enable search within archive files (e.g. zip, rar, 7z, gz, bz2, xz, tar, tar.gz, tar.bz2, tar.xz)')
|
|
349
|
-
@click.option('--depth',
|
|
350
|
-
help='
|
|
362
|
+
@click.option('--arc-depth', multiple=True,
|
|
363
|
+
help='Limit nested archive to given depth range. By default, there is no limit.')
|
|
351
364
|
@click.option('--arc-ext', multiple=True, type=click.STRING,
|
|
352
365
|
help='Include files with these extensions inside archive files. Example: --arc-ext py --arc-ext js')
|
|
353
366
|
@click.option('--arc-exc-ext', multiple=True, type=click.STRING,
|
|
@@ -363,7 +376,7 @@ def check_rar_backend(archive_enabled: bool, tool_path: str, backend: str):
|
|
|
363
376
|
# Output option
|
|
364
377
|
@click.option('-a', '--absolute-path', is_flag=True, help='Display full paths for results.')
|
|
365
378
|
@click.option('--paths-only', is_flag=True, help='Only show matching file paths for content search.')
|
|
366
|
-
@click.option('-
|
|
379
|
+
@click.option('-S', '--stats', is_flag=True,
|
|
367
380
|
help='Show search statistics including result counts and search time.')
|
|
368
381
|
def search(**kwargs):
|
|
369
382
|
"""Search for files, directories, and file content based on the query."""
|
|
@@ -389,7 +402,7 @@ def search(**kwargs):
|
|
|
389
402
|
results, metrics, elapsed, timed_out = search_with_timeout(config)
|
|
390
403
|
|
|
391
404
|
# Even if the search timed out, display all results found before termination.
|
|
392
|
-
echo(results)
|
|
405
|
+
echo(results, config.context)
|
|
393
406
|
|
|
394
407
|
if timed_out:
|
|
395
408
|
click.secho(
|
|
@@ -404,7 +417,7 @@ def search(**kwargs):
|
|
|
404
417
|
results, metrics = seek(config)
|
|
405
418
|
elapsed = time.perf_counter() - start
|
|
406
419
|
|
|
407
|
-
echo(results)
|
|
420
|
+
echo(results, config.context)
|
|
408
421
|
if config.stats:
|
|
409
422
|
echo_stats(config, results, metrics, elapsed)
|
|
410
423
|
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
import mmap, os
|
|
2
2
|
from pathlib import Path
|
|
3
|
-
from collections import defaultdict
|
|
3
|
+
from collections import defaultdict, deque
|
|
4
4
|
from .utils import get_path_suffix, is_binary
|
|
5
5
|
from .parser import parse_query_expression, TermNode, find_matches
|
|
6
6
|
from .archive import ARCHIVE_EXTS, extract_names_from_archive, extract_text_from_archive
|
|
7
|
-
from .structs import FileDirResult, ContentResult,
|
|
7
|
+
from .structs import FileDirResult, ContentResult, MatchGroup, Line
|
|
8
8
|
|
|
9
9
|
|
|
10
10
|
def should_skip(config, p: Path, file_ext: str, p_size: int) -> bool:
|
|
@@ -88,7 +88,7 @@ def search_file_and_dir(config, matches: dict, pattern, p: Path,
|
|
|
88
88
|
elif is_dir:
|
|
89
89
|
add_metric(metrics, result_queue, 'directories_scanned', full_virtual_path)
|
|
90
90
|
|
|
91
|
-
# We cannot use "full_virtual_path[-1]" because it may be limited by --depth
|
|
91
|
+
# We cannot use "full_virtual_path[-1]" because it may be limited by --arc-depth
|
|
92
92
|
# and may not enter the archive at all to scan inside it
|
|
93
93
|
if len(full_virtual_path) >= 3 and get_path_suffix(full_virtual_path[-2]) in ARCHIVE_EXTS[:-3]:
|
|
94
94
|
add_metric(metrics, result_queue, 'archives_scanned', full_virtual_path)
|
|
@@ -129,13 +129,78 @@ def content_contains(content, binary_pattern) -> bool:
|
|
|
129
129
|
return binary_pattern.search(content) is not None
|
|
130
130
|
|
|
131
131
|
|
|
132
|
+
class ContextCollector:
|
|
133
|
+
"""
|
|
134
|
+
Collects matching lines and their surrounding context into
|
|
135
|
+
non-overlapping MatchGroups in a single pass.
|
|
136
|
+
"""
|
|
137
|
+
|
|
138
|
+
def __init__(self, context: tuple[int, int]):
|
|
139
|
+
# Lines immediately preceding the current search position
|
|
140
|
+
self.before_buffer = deque(maxlen=context[0])
|
|
141
|
+
# Lines after the latest match, kept until the group is finalized
|
|
142
|
+
# or another match extends the same group
|
|
143
|
+
self.after_buffer = []
|
|
144
|
+
self.before = context[0]
|
|
145
|
+
self.after = context[1]
|
|
146
|
+
self.lines = []
|
|
147
|
+
self.matching_lines = set()
|
|
148
|
+
self.groups = []
|
|
149
|
+
|
|
150
|
+
def _finish_group(self):
|
|
151
|
+
self.lines.extend(self.after_buffer[:self.after])
|
|
152
|
+
self.groups.append(
|
|
153
|
+
MatchGroup(
|
|
154
|
+
self.lines,
|
|
155
|
+
self.matching_lines
|
|
156
|
+
)
|
|
157
|
+
)
|
|
158
|
+
|
|
159
|
+
self.before_buffer.clear()
|
|
160
|
+
# Preserve the latest lines as before-context for the next group.
|
|
161
|
+
self.before_buffer.extend(self.after_buffer)
|
|
162
|
+
self.after_buffer.clear()
|
|
163
|
+
# Because of ownership, we can't use .clear() here.
|
|
164
|
+
# Using it could also clear values inside MatchGroup.
|
|
165
|
+
self.lines = []
|
|
166
|
+
self.matching_lines = set()
|
|
167
|
+
|
|
168
|
+
def process(self, line: Line, matched: bool):
|
|
169
|
+
if matched:
|
|
170
|
+
if self.matching_lines:
|
|
171
|
+
self.lines.extend(self.after_buffer)
|
|
172
|
+
self.lines.append(line)
|
|
173
|
+
|
|
174
|
+
self.after_buffer.clear()
|
|
175
|
+
else:
|
|
176
|
+
self.lines.extend(self.before_buffer)
|
|
177
|
+
self.lines.append(line)
|
|
178
|
+
|
|
179
|
+
self.matching_lines.add(line.number)
|
|
180
|
+
else:
|
|
181
|
+
if self.matching_lines:
|
|
182
|
+
self.after_buffer.append(line)
|
|
183
|
+
|
|
184
|
+
# Keep groups together while their context ranges overlap or touch
|
|
185
|
+
if len(self.after_buffer) > self.after + self.before:
|
|
186
|
+
self._finish_group()
|
|
187
|
+
else:
|
|
188
|
+
self.before_buffer.append(line)
|
|
189
|
+
|
|
190
|
+
def finish(self) -> list[MatchGroup]:
|
|
191
|
+
if self.matching_lines:
|
|
192
|
+
self._finish_group()
|
|
193
|
+
|
|
194
|
+
return self.groups
|
|
195
|
+
|
|
196
|
+
|
|
132
197
|
def search_content(config, matches: dict, pattern, binary_pattern,
|
|
133
198
|
p: Path, p_ext: str, metrics, result_queue):
|
|
134
199
|
"""Search within the contents of system files and files inside archive files"""
|
|
135
200
|
|
|
136
201
|
path_str = str(p)
|
|
137
202
|
|
|
138
|
-
#
|
|
203
|
+
# Search files content inside archives separately.
|
|
139
204
|
if config.archive and p_ext in ARCHIVE_EXTS:
|
|
140
205
|
for virtual_path, content in extract_text_from_archive(p, config):
|
|
141
206
|
if config.stats:
|
|
@@ -161,12 +226,11 @@ def search_content(config, matches: dict, pattern, binary_pattern,
|
|
|
161
226
|
except UnicodeDecodeError:
|
|
162
227
|
continue
|
|
163
228
|
|
|
164
|
-
|
|
229
|
+
collector = ContextCollector(config.context)
|
|
165
230
|
for num, line in enumerate(decoded_content.splitlines(), 1):
|
|
166
|
-
|
|
167
|
-
continue
|
|
231
|
+
matched = pattern.evaluate(line)
|
|
168
232
|
|
|
169
|
-
if config.paths_only:
|
|
233
|
+
if config.paths_only and matched:
|
|
170
234
|
add_result(
|
|
171
235
|
matches,
|
|
172
236
|
result_queue,
|
|
@@ -176,15 +240,17 @@ def search_content(config, matches: dict, pattern, binary_pattern,
|
|
|
176
240
|
virtual_path=virtual_path
|
|
177
241
|
)
|
|
178
242
|
)
|
|
179
|
-
|
|
180
243
|
break
|
|
181
244
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
245
|
+
line_instance = Line(num, line)
|
|
246
|
+
|
|
247
|
+
if matched:
|
|
248
|
+
line_instance.matches = find_matches(pattern, line)
|
|
186
249
|
|
|
187
|
-
|
|
250
|
+
collector.process(line_instance, matched)
|
|
251
|
+
|
|
252
|
+
groups = collector.finish()
|
|
253
|
+
if groups:
|
|
188
254
|
add_result(
|
|
189
255
|
matches,
|
|
190
256
|
result_queue,
|
|
@@ -192,14 +258,14 @@ def search_content(config, matches: dict, pattern, binary_pattern,
|
|
|
192
258
|
ContentResult(
|
|
193
259
|
path=path_str,
|
|
194
260
|
virtual_path=virtual_path,
|
|
195
|
-
|
|
261
|
+
groups=groups
|
|
196
262
|
)
|
|
197
263
|
)
|
|
198
264
|
|
|
199
265
|
# Skip next block to avoid searching the contents of archive files
|
|
200
266
|
return
|
|
201
267
|
|
|
202
|
-
|
|
268
|
+
collector = ContextCollector(config.context)
|
|
203
269
|
# Memory-map the file for efficient access
|
|
204
270
|
with open(p, 'rb') as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
|
|
205
271
|
head = mm[:8192]
|
|
@@ -211,72 +277,84 @@ def search_content(config, matches: dict, pattern, binary_pattern,
|
|
|
211
277
|
# Iterate over each line in the file
|
|
212
278
|
for num, line in enumerate(iter(mm.readline, b''), 1):
|
|
213
279
|
try:
|
|
214
|
-
|
|
215
|
-
line_decoded = line.decode('utf-8').strip()
|
|
280
|
+
line_decoded = line.decode('utf-8').rstrip('\r\n')
|
|
216
281
|
except UnicodeDecodeError:
|
|
217
282
|
# Skip lines that can't be decoded
|
|
218
283
|
continue
|
|
219
284
|
|
|
220
|
-
|
|
221
|
-
if pattern.evaluate(line_decoded):
|
|
222
|
-
# Avoid searching through the entire file content if the fast-content flag is True
|
|
223
|
-
if config.paths_only:
|
|
224
|
-
add_result(
|
|
225
|
-
matches,
|
|
226
|
-
result_queue,
|
|
227
|
-
'content',
|
|
228
|
-
ContentResult(path=path_str)
|
|
229
|
-
)
|
|
285
|
+
matched = pattern.evaluate(line_decoded)
|
|
230
286
|
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
287
|
+
# Avoid searching through the entire file content if the paths-only flag is True
|
|
288
|
+
if config.paths_only and matched:
|
|
289
|
+
add_result(
|
|
290
|
+
matches,
|
|
291
|
+
result_queue,
|
|
292
|
+
'content',
|
|
293
|
+
ContentResult(path=path_str)
|
|
235
294
|
)
|
|
295
|
+
return
|
|
296
|
+
|
|
297
|
+
line_instance = Line(num, line_decoded)
|
|
298
|
+
|
|
299
|
+
if matched:
|
|
300
|
+
line_instance.matches = find_matches(pattern, line_decoded)
|
|
236
301
|
|
|
237
|
-
|
|
302
|
+
collector.process(line_instance, matched)
|
|
303
|
+
|
|
304
|
+
groups = collector.finish()
|
|
305
|
+
if groups:
|
|
238
306
|
add_result(
|
|
239
307
|
matches,
|
|
240
308
|
result_queue,
|
|
241
309
|
'content',
|
|
242
310
|
ContentResult(
|
|
243
311
|
path=path_str,
|
|
244
|
-
|
|
312
|
+
groups=groups
|
|
245
313
|
)
|
|
246
314
|
)
|
|
247
315
|
|
|
248
316
|
|
|
249
|
-
def walk_logic(path: Path,
|
|
317
|
+
def walk_logic(path: Path, config, depth: int):
|
|
250
318
|
"""Recursively walk a directory while pruning excluded subtrees"""
|
|
319
|
+
|
|
320
|
+
new_depth = depth + 1
|
|
321
|
+
|
|
322
|
+
try:
|
|
323
|
+
with os.scandir(path) as entries:
|
|
324
|
+
for entry in entries:
|
|
325
|
+
entry_path = Path(entry.path)
|
|
326
|
+
|
|
327
|
+
# Excluded directories are pruned before recursion
|
|
328
|
+
if entry_path in config.exclude:
|
|
329
|
+
continue
|
|
330
|
+
|
|
331
|
+
if any(mi <= new_depth <= ma for mi, ma in config.depth):
|
|
332
|
+
yield entry_path
|
|
251
333
|
|
|
252
|
-
with os.scandir(path) as entries:
|
|
253
|
-
for entry in entries:
|
|
254
|
-
entry_path = Path(entry.path)
|
|
255
|
-
if not absolute_path:
|
|
256
334
|
try:
|
|
257
|
-
|
|
335
|
+
is_dir = entry.is_dir(follow_symlinks=False)
|
|
258
336
|
except OSError:
|
|
259
337
|
continue
|
|
260
|
-
else:
|
|
261
|
-
entry_path_resolved = entry_path
|
|
262
|
-
|
|
263
|
-
if entry_path_resolved in exclude:
|
|
264
|
-
continue
|
|
265
338
|
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
339
|
+
if is_dir and any(new_depth < d[1] for d in config.depth):
|
|
340
|
+
yield from walk_logic(entry_path, config, new_depth)
|
|
341
|
+
except OSError:
|
|
342
|
+
pass
|
|
269
343
|
|
|
270
344
|
|
|
271
|
-
def walk(
|
|
345
|
+
def walk(config):
|
|
272
346
|
"""Include paths define traversal roots, avoiding unnecessary traversal
|
|
273
347
|
of unrelated parts of the tree"""
|
|
274
|
-
roots = include or {path}
|
|
348
|
+
roots = config.include or {config.path}
|
|
275
349
|
|
|
276
350
|
for root in roots:
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
351
|
+
depth = len(root.relative_to(config.path).parts) - 1
|
|
352
|
+
|
|
353
|
+
if root.is_dir() and not root.is_symlink() and any(depth < d[1] for d in config.depth):
|
|
354
|
+
yield from walk_logic(root, config, depth)
|
|
355
|
+
|
|
356
|
+
if any(mi <= depth <= ma for mi, ma in config.depth):
|
|
357
|
+
yield root
|
|
280
358
|
|
|
281
359
|
|
|
282
360
|
def seek(config, result_queue=None) -> dict:
|
|
@@ -292,7 +370,7 @@ def seek(config, result_queue=None) -> dict:
|
|
|
292
370
|
matches = {'file': [], 'directory': [], 'content': []}
|
|
293
371
|
metrics = defaultdict(set)
|
|
294
372
|
|
|
295
|
-
for p in walk(config
|
|
373
|
+
for p in walk(config):
|
|
296
374
|
try:
|
|
297
375
|
p_ext = get_path_suffix(p)
|
|
298
376
|
p_size = p.stat().st_size
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
from click import BadParameter
|
|
2
|
-
import re
|
|
2
|
+
import re
|
|
3
3
|
from pathlib import Path
|
|
4
4
|
from dataclasses import dataclass, field
|
|
5
5
|
from math import isfinite
|
|
@@ -27,8 +27,6 @@ def normalize_paths(base_path: Path, paths: tuple[str], param_hint: str, absolut
|
|
|
27
27
|
|
|
28
28
|
if absolute_path:
|
|
29
29
|
p = p.resolve()
|
|
30
|
-
else:
|
|
31
|
-
p = p.resolve() if param_hint == 'exclude' else p
|
|
32
30
|
|
|
33
31
|
# Already covered by an existing parent.
|
|
34
32
|
if any(p.is_relative_to(r) for r in result):
|
|
@@ -76,42 +74,47 @@ def extract_size(size: str, param_hint: str) -> int:
|
|
|
76
74
|
return int(number * SUFFIXES[size_suffix])
|
|
77
75
|
|
|
78
76
|
|
|
79
|
-
def
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
if size.startswith(':'):
|
|
86
|
-
maximum = extract_size(size[1:], param_hint)
|
|
87
|
-
ranges.append((0, maximum))
|
|
88
|
-
elif size.endswith(':'):
|
|
89
|
-
minimum = extract_size(size[:-1], param_hint)
|
|
90
|
-
ranges.append((minimum, float('inf')))
|
|
91
|
-
elif ':' in size:
|
|
92
|
-
min_str, max_str = size.split(':', 1)
|
|
93
|
-
|
|
94
|
-
if not min_str or not max_str:
|
|
95
|
-
raise BadParameter(
|
|
96
|
-
f'Invalid size range: {size}',
|
|
97
|
-
param_hint=f'--{param_hint}',
|
|
98
|
-
)
|
|
99
|
-
|
|
100
|
-
minimum = extract_size(min_str, param_hint)
|
|
101
|
-
maximum = extract_size(max_str, param_hint)
|
|
77
|
+
def extract_int(num: str, param_hint: str) -> int:
|
|
78
|
+
if not num.isdigit():
|
|
79
|
+
raise BadParameter(
|
|
80
|
+
f'Invalid value: {num}',
|
|
81
|
+
param_hint=f'--{param_hint}',
|
|
82
|
+
)
|
|
102
83
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
84
|
+
return int(num)
|
|
85
|
+
|
|
86
|
+
|
|
87
|
+
def parse_range(data: str, extract_func, param_hint: str) -> tuple[int, int | float]:
|
|
88
|
+
"""Parse str into a range of int or float"""
|
|
89
|
+
|
|
90
|
+
if data.startswith(':'):
|
|
91
|
+
maximum = extract_func(data[1:], param_hint)
|
|
92
|
+
return (0, maximum)
|
|
93
|
+
elif data.endswith(':'):
|
|
94
|
+
minimum = extract_func(data[:-1], param_hint)
|
|
95
|
+
return (minimum, 0 if param_hint == 'context' else float('inf'))
|
|
96
|
+
elif ':' in data:
|
|
97
|
+
min_str, max_str = data.split(':', 1)
|
|
98
|
+
|
|
99
|
+
if not min_str or not max_str:
|
|
100
|
+
raise BadParameter(
|
|
101
|
+
f'Invalid range: {data}',
|
|
102
|
+
param_hint=f'--{param_hint}',
|
|
103
|
+
)
|
|
104
|
+
|
|
105
|
+
minimum = extract_func(min_str, param_hint)
|
|
106
|
+
maximum = extract_func(max_str, param_hint)
|
|
107
|
+
|
|
108
|
+
if minimum > maximum and param_hint != 'context':
|
|
109
|
+
raise BadParameter(
|
|
110
|
+
'Minimum cannot be greater than maximum',
|
|
111
|
+
param_hint=f'--{param_hint}',
|
|
112
|
+
)
|
|
113
|
+
|
|
114
|
+
return (minimum, maximum)
|
|
115
|
+
else:
|
|
116
|
+
exact = extract_func(data, param_hint)
|
|
117
|
+
return (exact, exact)
|
|
115
118
|
|
|
116
119
|
|
|
117
120
|
@dataclass
|
|
@@ -125,6 +128,7 @@ class SearchConfig:
|
|
|
125
128
|
regex: bool
|
|
126
129
|
word: bool
|
|
127
130
|
expr: bool
|
|
131
|
+
depth: list[tuple]
|
|
128
132
|
timeout: int | None
|
|
129
133
|
fuzzy: bool
|
|
130
134
|
fuzzy_level: int
|
|
@@ -136,7 +140,7 @@ class SearchConfig:
|
|
|
136
140
|
re_exclude: re.Pattern | None
|
|
137
141
|
size: list[tuple]
|
|
138
142
|
archive: bool
|
|
139
|
-
|
|
143
|
+
arc_depth: list[tuple]
|
|
140
144
|
arc_ext: set[str]
|
|
141
145
|
arc_exc_ext: set[str | None]
|
|
142
146
|
arc_include: set[Path]
|
|
@@ -146,6 +150,7 @@ class SearchConfig:
|
|
|
146
150
|
absolute_path: bool
|
|
147
151
|
paths_only: bool
|
|
148
152
|
stats: bool
|
|
153
|
+
context: tuple[int]
|
|
149
154
|
|
|
150
155
|
def __post_init__(self):
|
|
151
156
|
"""Post-initialization processing to normalize and validate inputs"""
|
|
@@ -188,8 +193,15 @@ class SearchConfig:
|
|
|
188
193
|
self.re_exclude = compile_regex(self.re_exclude)
|
|
189
194
|
|
|
190
195
|
# Normalize sizes
|
|
191
|
-
self.size =
|
|
192
|
-
self.arc_size =
|
|
196
|
+
self.size = [parse_range(s, extract_size, 'size') for s in self.size]
|
|
197
|
+
self.arc_size = [parse_range(s, extract_size, 'arc-size') for s in self.arc_size]
|
|
198
|
+
|
|
199
|
+
# Normalize depth
|
|
200
|
+
# If depth isn't define, don't impose limit
|
|
201
|
+
self.depth = [(0, float('inf'))] if not self.depth else [parse_range(d, extract_int, 'depth') for d in self.depth]
|
|
202
|
+
self.arc_depth = [(0, float('inf'))] if not self.arc_depth else [parse_range(d, extract_int, 'arc-depth') for d in self.arc_depth]
|
|
203
|
+
|
|
204
|
+
self.context = parse_range(self.context, extract_int, 'context') if self.context else (0, 0)
|
|
193
205
|
|
|
194
206
|
|
|
195
207
|
@dataclass
|
|
@@ -204,12 +216,18 @@ class FileDirResult(SearchResult):
|
|
|
204
216
|
|
|
205
217
|
|
|
206
218
|
@dataclass
|
|
207
|
-
class
|
|
219
|
+
class Line:
|
|
208
220
|
number: int
|
|
209
221
|
text: str
|
|
210
222
|
matches: list[tuple[int, int]] = field(default_factory=list)
|
|
211
223
|
|
|
212
224
|
|
|
225
|
+
@dataclass
|
|
226
|
+
class MatchGroup:
|
|
227
|
+
lines: list[Line] = field(default_factory=list)
|
|
228
|
+
matching_lines: set[int] = field(default_factory=set)
|
|
229
|
+
|
|
230
|
+
|
|
213
231
|
@dataclass
|
|
214
232
|
class ContentResult(SearchResult):
|
|
215
|
-
|
|
233
|
+
groups: list[MatchGroup] = field(default_factory=list, kw_only=True)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pseek
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.2.0
|
|
4
4
|
Summary: Pseek is a Python CLI toolkit to search files, folders, and text
|
|
5
5
|
Author: Groot
|
|
6
6
|
License-Expression: MIT
|
|
@@ -112,12 +112,14 @@ psk -- --path
|
|
|
112
112
|
| `--re-include`, `--re-exclude` | Limit search results to specific directories or files with regex |
|
|
113
113
|
| `--word` | Match the whole word only (except when `--expr` is enabled, in which case you can make it match whole word by putting `w` before term: `w"foo"`) |
|
|
114
114
|
| `--expr` | Enable boolean query expressions. Example: `r"foo.*bar" and ("bar" or "baz") and not "qux"`. Prefixes: `r=regex`, `c=case-sensitive`, `w=whole-word`, `f=fuzzy` |
|
|
115
|
+
| `--context` | Show N context lines before and after matches, or use N:M to specify before:after |
|
|
116
|
+
| `--depth` | Limit directory traversal to given depth range. By default, there is no limit on search depth |
|
|
115
117
|
| `--timeout` | Stop the search after the specified number of seconds |
|
|
116
118
|
| `--fuzzy` | Enable fuzzy search (Highlighting and counting matches are disabled in this mode if `--word` is not enabled to prevent the program from slowing down). except when `--expr` is enabled, in which case you can make it fuzzy by putting `f` before term: `f"foo"` |
|
|
117
119
|
| `--fuzzy-level` | Fuzzy matching threshold (0-99). Higher values require closer matches (default: `80`) |
|
|
118
120
|
| `--size` | Limit results based on the size of files |
|
|
119
121
|
| `--archive` | Enable search within archive files (e.g. `zip`, `rar`, `7z`, `gz`, `bz2`, `xz`, `tar`, `tar.gz`, `tar.bz2`, `tar.xz`) |
|
|
120
|
-
| `--depth` |
|
|
122
|
+
| `--arc-depth` | Limit nested archive to given depth range. By default, there is no limit |
|
|
121
123
|
| `--arc-ext`, `--arc-exc-ext` | Filter by file extension inside archive files |
|
|
122
124
|
| `--arc-include`, `--arc-exclude` | Limit search results to specific set of directories or files inside archive files |
|
|
123
125
|
| `--arc-size` | Limit results based on the size of files in the archive |
|
|
@@ -176,7 +178,7 @@ Example:
|
|
|
176
178
|
psk "hello world"
|
|
177
179
|
```
|
|
178
180
|
|
|
179
|
-
**Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
181
|
+
> **Note:** To use case-sensitive, whole-word matching, regular expression search, and fuzzy search when `--expr` is enabled, we can use [expression prefixes](#expression-prefixes).
|
|
180
182
|
|
|
181
183
|
### Case Sensitive Search
|
|
182
184
|
|
|
@@ -364,7 +366,51 @@ simultaneously.
|
|
|
364
366
|
|
|
365
367
|
Allowed modes: `r`, `c`, `w`, `f`, `rc`, `cr`, `cw`, `wc`, `cf`, `fc`, `wf`, `fw`, `cwf`, `cfw`, `wcf`, `wfc`, `fcw`, `fwc`
|
|
366
368
|
|
|
367
|
-
**Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
369
|
+
> **Note:** Whole word matching and regex matching cannot be used at the same time, because we can use `\b` in regex to enable whole word matching: `r"\btext\b"`
|
|
370
|
+
|
|
371
|
+
## Context
|
|
372
|
+
|
|
373
|
+
Show context lines around each match. The value is specified as `BEFORE:AFTER`:
|
|
374
|
+
|
|
375
|
+
```bash
|
|
376
|
+
--context 2 # 2 lines before and after
|
|
377
|
+
--context 2:0 # 2 lines before, none after
|
|
378
|
+
--context 0:2 # none before, 2 lines after
|
|
379
|
+
--context 2:5 # 2 lines before, 5 lines after
|
|
380
|
+
--context 2: # 2 lines before, none after
|
|
381
|
+
--context :2 # none before, 2 lines after
|
|
382
|
+
--context 0 # matching lines only
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
Nearby matches whose context ranges overlap or directly touch are combined into a single group.
|
|
386
|
+
|
|
387
|
+
## Depth
|
|
388
|
+
|
|
389
|
+
Controls how deep the search goes inside directories.
|
|
390
|
+
|
|
391
|
+
A depth of `0` means that only base path's immediate contents are searched; subdirectories are not entered.
|
|
392
|
+
|
|
393
|
+
```text
|
|
394
|
+
project/ ← base directory
|
|
395
|
+
├── file.txt ← depth 0
|
|
396
|
+
├── folder1/ ← depth 0
|
|
397
|
+
│ ├── file.txt ← depth 1
|
|
398
|
+
│ └── folder2/ ← depth 1
|
|
399
|
+
│ └── file.txt ← depth 2
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
`--depth` can be specified multiple times to search non-contiguous depth ranges:
|
|
403
|
+
|
|
404
|
+
```bash
|
|
405
|
+
--depth 0 # Direct contents only
|
|
406
|
+
--depth 1 # One level inside
|
|
407
|
+
--depth 2:4 # Depths 2 through 4
|
|
408
|
+
--depth 2 --depth 5:7 # Depths 2 and 5 through 7
|
|
409
|
+
--depth :2 # Depths 0 through 2
|
|
410
|
+
--depth 3: # Depth 3 and deeper
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
If `--depth` is not specified, the search has no depth limit.
|
|
368
414
|
|
|
369
415
|
## Extension Filters
|
|
370
416
|
|
|
@@ -402,7 +448,7 @@ psk TODO \
|
|
|
402
448
|
|
|
403
449
|
Skip those paths.
|
|
404
450
|
|
|
405
|
-
**Note:** The include and exclude paths will be combined with path argument.
|
|
451
|
+
> **Note:** The include and exclude paths will be combined with path argument.
|
|
406
452
|
|
|
407
453
|
## Regex Path Filters
|
|
408
454
|
|
|
@@ -455,7 +501,7 @@ psk config \
|
|
|
455
501
|
|
|
456
502
|
Files between `1 MiB` and `10 MiB` will not match.
|
|
457
503
|
|
|
458
|
-
**Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
504
|
+
> **Note:** Directory sizes are not calculated recursively. When `--size` is used, directories are automatically excluded from the search results.
|
|
459
505
|
|
|
460
506
|
## Archive Search
|
|
461
507
|
|
|
@@ -487,15 +533,24 @@ backup.zip::source.7z::notes.txt
|
|
|
487
533
|
|
|
488
534
|
### Archive Depth
|
|
489
535
|
|
|
490
|
-
|
|
536
|
+
The concept of depth in archive is different from `--depth`. Depth in archive is calculated based on nested archives. Each archive that is inside another archive counts as one level of depth.
|
|
537
|
+
|
|
538
|
+
| Syntax | Meaning |
|
|
539
|
+
| ------ | -------------------------------------------------------- |
|
|
540
|
+
| `2` | exactly `2nd` depth |
|
|
541
|
+
| `:2` | `2nd` depth or lower |
|
|
542
|
+
| `2:` | `2nd` depth or higher |
|
|
543
|
+
| `2:4` | between `2` and `4` (Both range boundaries are included) |
|
|
544
|
+
|
|
545
|
+
`--arc-depth` can be specified multiple times:
|
|
491
546
|
|
|
492
547
|
```bash
|
|
493
|
-
psk TODO --archive --depth 2
|
|
548
|
+
psk TODO --archive --arc-depth :2 --arc-depth 5:
|
|
494
549
|
```
|
|
495
550
|
|
|
496
|
-
Meaning `archive level
|
|
551
|
+
Meaning `archive level 3` and `archive level 4` won't be searched.
|
|
497
552
|
|
|
498
|
-
**Note:** `--depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
553
|
+
> **Note:** `--arc-depth 0` means perform the search only within this current archive and don't enter nested archives.
|
|
499
554
|
|
|
500
555
|
## Archive Filters
|
|
501
556
|
|
|
@@ -535,7 +590,7 @@ This works exactly like the [--size](#size-filtering) option.
|
|
|
535
590
|
psk TODO --archive --arc-size 10m:40m
|
|
536
591
|
```
|
|
537
592
|
|
|
538
|
-
**Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
593
|
+
> **Note:** Archive directory sizes are usually reported as zero by archive formats, so directory search is disabled if this filter is enabled to avoid incorrect results.
|
|
539
594
|
|
|
540
595
|
## RAR Backend
|
|
541
596
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|