rouge-xql 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +38 -0
- data/LICENSE +21 -0
- data/README.md +90 -0
- data/lib/rouge/demos/xql +10 -0
- data/lib/rouge/lexers/xql/builtins.rb +51 -0
- data/lib/rouge/lexers/xql.rb +281 -0
- data/lib/rouge-xql.rb +8 -0
- metadata +74 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: c8498c1b6b5b8343b0c4697c926cb1d7802e477bcec93b83d3b9466c303153d3
|
|
4
|
+
data.tar.gz: 978ccccd64668854812bc8917fce25231c1f5f2d8fa814a4aa716e34eead26e2
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 8b50cc65c64c569318a18ea20ffd9bd937810bcd37569ea211ed322866bdd921512df1eeec5cfa2d0d33fb02be5edb65d666c46d8a44a2fc25c9e2138949739e
|
|
7
|
+
data.tar.gz: 902b9f14103808ab065b537085275851fc5e2e199ee0c80d64b21bbb5677b9781a4066f600839b322618fdd84cbf11b8d7387ce286bf0ee3f7e715e6b16e6b79
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.0]
|
|
10
|
+
|
|
11
|
+
Initial release.
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- `Rouge::Lexers::XQL`, registered under the tag `xql` with aliases `xif` and `xsiam`,
|
|
16
|
+
and associated with `*.xql` and `*.xif`.
|
|
17
|
+
- Support for both XQL dialects in one lexer: pipeline queries
|
|
18
|
+
(`dataset = x | filter ... | alter ...`) and parsing/modeling rule files
|
|
19
|
+
(`[INGEST:...]`, `[RULE: NAME]`, `[MODEL:...]`, `[CONST]`).
|
|
20
|
+
- Highlighting for the constructs that generic highlighters tend to miss: `xdm.*` model
|
|
21
|
+
fields as single tokens, `ENUM.*` and `XDM_CONST.*` constants, `$` parameter and
|
|
22
|
+
constant references, backtick-quoted identifiers, time literals (`30d`, `24H`, `3mo`),
|
|
23
|
+
the `->` JSON-path operator with its `[]` and `{}` forms, and section markers.
|
|
24
|
+
- A string directly after `~=` / `!~=` is lexed as a regex, with its escapes picked out.
|
|
25
|
+
- Only standard Rouge token types are emitted, so any stock theme styles the output with
|
|
26
|
+
no extra CSS.
|
|
27
|
+
|
|
28
|
+
### Notes
|
|
29
|
+
|
|
30
|
+
The string rules follow what the language actually accepts rather than the more widely
|
|
31
|
+
copied approximations: inside `"..."` the only escape is `\"`, so `\d` and `\s` are
|
|
32
|
+
ordinary characters and regexes are written raw; `"""..."""` takes the JSON escape set and
|
|
33
|
+
is the only string form that may span a newline; and because `\\` is not an escape, a
|
|
34
|
+
string ending `...\\"` resolves by longest valid token. There are no hex literals, no
|
|
35
|
+
exponent floats, no `=>` or `<>`, and no single-quoted strings.
|
|
36
|
+
|
|
37
|
+
[Unreleased]: https://github.com/vanhecke/rouge-xql/compare/v0.1.0...HEAD
|
|
38
|
+
[0.1.0]: https://github.com/vanhecke/rouge-xql/releases/tag/v0.1.0
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Joris
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# rouge-xql
|
|
2
|
+
|
|
3
|
+
A [Rouge](https://github.com/rouge-ruby/rouge) lexer for **XQL**, the Cortex XSIAM/XDR query
|
|
4
|
+
language — covering both of its dialects:
|
|
5
|
+
|
|
6
|
+
* **XQL Queries** — `config timeframe = 30d | dataset = xdr_data | filter a ~= "b"`
|
|
7
|
+
* **Parsing & Modeling rules** — `[INGEST:...]`, `[RULE: NAME]`, `[MODEL:...]`, `[CONST]`
|
|
8
|
+
|
|
9
|
+
Both register under one tag, `xql`, because they share the entire expression language and
|
|
10
|
+
real-world tooling feeds both through the same highlighter.
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
```ruby
|
|
15
|
+
# Gemfile
|
|
16
|
+
gem "rouge-xql"
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
```ruby
|
|
20
|
+
require "rouge-xql"
|
|
21
|
+
|
|
22
|
+
Rouge::Lexer.find("xql") # => Rouge::Lexers::XQL
|
|
23
|
+
Rouge.highlight(source, "xql", "html")
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Requiring the gem is all that is needed: Rouge's `tag` DSL registers the class as a side
|
|
27
|
+
effect of defining it. `xif` and `xsiam` work as aliases, and `*.xql` / `*.xif` are
|
|
28
|
+
recognised by `Rouge::Lexer.guess`.
|
|
29
|
+
|
|
30
|
+
Pair it with any Rouge theme — the lexer emits only standard token types, so stock
|
|
31
|
+
stylesheets (including a plain `rougify style github` dump) style it correctly with no
|
|
32
|
+
extra CSS.
|
|
33
|
+
|
|
34
|
+
## Notes on the language
|
|
35
|
+
|
|
36
|
+
XQL has a few properties that highlighters routinely get wrong. This lexer follows what
|
|
37
|
+
the language actually accepts, and each case is covered by a test:
|
|
38
|
+
|
|
39
|
+
* Inside `"..."` **the only escape is `\"`**. `\d`, `\s` and `\\` are ordinary characters,
|
|
40
|
+
which is why regexes are written raw. `"""..."""` is different — it takes the JSON escape
|
|
41
|
+
set and is the only string form that may span a newline.
|
|
42
|
+
* Because `\\` is not an escape, `"...\\"` is ambiguous. It resolves by longest valid
|
|
43
|
+
token: `\"` counts as an escape only when a closing quote is still reachable on the line.
|
|
44
|
+
Get this wrong and a regex ending in a backslash swallows the rest of the line.
|
|
45
|
+
* `-` binds greedily into a number: `a-1` is `Name(a) Num(-1)`, but `fields -foo` is
|
|
46
|
+
`Keyword(fields) Operator(-) Name(foo)`.
|
|
47
|
+
* `xdm.a.b.c` is a single token. `ENUM.X` and `XDM_CONST.X` are three, but are highlighted
|
|
48
|
+
as one unit since that is what they mean.
|
|
49
|
+
* Backticks are an identifier quote, not a string — that is how a reserved word becomes a
|
|
50
|
+
field name (`` `target` ``).
|
|
51
|
+
* `[]`, `{}` and `->` are single tokens; a lone `{` or `}` is invalid.
|
|
52
|
+
* There are **no** hex literals, no exponent floats, no `=>` or `<>`, and no single-quoted
|
|
53
|
+
strings. All are common in third-party XQL highlighting definitions and none is real.
|
|
54
|
+
* Every keyword is case-insensitive; `OR`, `AND` and `IN` are commonly uppercase in the
|
|
55
|
+
wild.
|
|
56
|
+
|
|
57
|
+
Two spellings are deliberately **not** treated as keywords: `colddataset` and `callmacro`.
|
|
58
|
+
Some tooling lists them, but the language accepts only `cold_dataset` and `call_macro`, so
|
|
59
|
+
highlighting them as commands would hide a real syntax error.
|
|
60
|
+
|
|
61
|
+
Field names are not baked in. They are per-deployment schema rather than language vocabulary,
|
|
62
|
+
and the language itself draws no distinction between a field and any other identifier.
|
|
63
|
+
Fields are highlighted structurally instead: `xdm.*` paths, `XDM_CONST.*` / `ENUM.*`
|
|
64
|
+
constants, `$refs` and backtick-quoted names each get their own token type.
|
|
65
|
+
|
|
66
|
+
### Known limitation
|
|
67
|
+
|
|
68
|
+
A string directly after `~=` / `!~=` is lexed as a regex (`Str::Regex`), with its escapes
|
|
69
|
+
picked out. The regex argument of `regextract(field, "...")` and `replex` is **not** —
|
|
70
|
+
finding it needs paren-depth tracking through nested calls like
|
|
71
|
+
`regextract(to_string(_time), "...")`. Those render as ordinary strings.
|
|
72
|
+
|
|
73
|
+
## Development
|
|
74
|
+
|
|
75
|
+
```sh
|
|
76
|
+
bundle install
|
|
77
|
+
bundle exec rake # specs
|
|
78
|
+
TEST=spec/lexers/xql_spec.rb bundle exec rake spec # one file
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
The suite covers the token types, the string and comment edge cases above, filename and
|
|
82
|
+
mimetype guessing, and asserts that both the demo and the visual sample lex with zero
|
|
83
|
+
`Error` tokens and round-trip byte for byte.
|
|
84
|
+
|
|
85
|
+
`spec/visual/samples/xql` is a single file exercising both dialects and most of the
|
|
86
|
+
language; it is the quickest way to eyeball a change. Every identifier in it is invented.
|
|
87
|
+
|
|
88
|
+
## Licence
|
|
89
|
+
|
|
90
|
+
MIT. Not affiliated with, or endorsed by, Palo Alto Networks.
|
data/lib/rouge/demos/xql
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
config timeframe = 30d
|
|
2
|
+
| dataset = xdr_data // agent telemetry
|
|
3
|
+
| filter event_type = ENUM.NETWORK and lowercase(actor_process_image_name) in ("winword.exe", "excel.exe")
|
|
4
|
+
| filter action_remote_ip !~= "^10\.|^192\.168\."
|
|
5
|
+
| alter host = arrayindex(regextract(_raw_log, "from (\S+) port"), 0),
|
|
6
|
+
user = coalesce(`target` -> [0].alternateId, actor_effective_username)
|
|
7
|
+
| comp count(event_id) as connections by host, user, xdm.source.host.hostname
|
|
8
|
+
| filter connections >= 5 and user != null
|
|
9
|
+
| sort desc connections
|
|
10
|
+
| limit 10
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# -*- coding: utf-8 -*- #
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# The 139 built-in functions of XQL, as listed in the Cortex XQL
|
|
5
|
+
# reference documentation. Generated -- do not edit by hand.
|
|
6
|
+
|
|
7
|
+
module Rouge
|
|
8
|
+
module Lexers
|
|
9
|
+
class XQL < RegexLexer
|
|
10
|
+
def self.functions
|
|
11
|
+
@functions ||= Set.new %w(
|
|
12
|
+
abs acos add approx_count
|
|
13
|
+
approx_quantiles approx_top array_all array_any
|
|
14
|
+
array_length arrayconcat arraycreate arraydistinct
|
|
15
|
+
arrayfilter arrayindex arrayindexof arraymap
|
|
16
|
+
arraymerge arrayrange arraystring asin
|
|
17
|
+
avg bitwise_and bitwise_or bitwise_sleft
|
|
18
|
+
bitwise_sright bitwise_xor cbrt ceil
|
|
19
|
+
coalesce concat convert_from_base_64 convert_to_base_64
|
|
20
|
+
cos cosine_distance cot count
|
|
21
|
+
count_distinct csc current_time date_floor
|
|
22
|
+
divide earliest euclidean_distance exp
|
|
23
|
+
extract_time extract_url_host extract_url_pub_suffix extract_url_registered_domain
|
|
24
|
+
first first_value floor format_string
|
|
25
|
+
format_timestamp greatest hierarchy_match if
|
|
26
|
+
incidr incidr6 incidrlist int_to_ip
|
|
27
|
+
ip_to_int is_ipv4 is_ipv6 is_known_private_ipv4
|
|
28
|
+
is_known_private_ipv6 json_extract json_extract_array json_extract_scalar
|
|
29
|
+
json_extract_scalar_array json_path_extract lag last
|
|
30
|
+
last_value latest least len
|
|
31
|
+
list ln log log10
|
|
32
|
+
lowercase ltrim max md5
|
|
33
|
+
median min mod multiply
|
|
34
|
+
object_create object_merge parse_epoch parse_timestamp
|
|
35
|
+
pow rand range_bucket rank
|
|
36
|
+
regextract replace replex round
|
|
37
|
+
row_number rtrim safe_add safe_divide
|
|
38
|
+
safe_ip_to_int safe_multiply safe_negate safe_subtract
|
|
39
|
+
sec sha1 sha256 sha512
|
|
40
|
+
sign sin split sqrt
|
|
41
|
+
stddev_population stddev_sample string_count subtract
|
|
42
|
+
sum tan time_frame_end timestamp_diff
|
|
43
|
+
timestamp_seconds to_boolean to_epoch to_float
|
|
44
|
+
to_integer to_json_string to_number to_string
|
|
45
|
+
to_timestamp trim trunc uppercase
|
|
46
|
+
values var wildcard_match
|
|
47
|
+
)
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
# -*- coding: utf-8 -*- #
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require "rouge"
|
|
5
|
+
|
|
6
|
+
module Rouge
|
|
7
|
+
module Lexers
|
|
8
|
+
# Lexer for XQL, the Cortex XSIAM/XDR query language.
|
|
9
|
+
#
|
|
10
|
+
# Covers both dialects, which share one expression language:
|
|
11
|
+
#
|
|
12
|
+
# * pipeline queries `config timeframe = 30d | dataset = x | filter a ~= "b"`
|
|
13
|
+
# * XIF rule files `[INGEST:vendor="x", ...]` / `[RULE: NAME]` / `[MODEL: dataset="x"]`
|
|
14
|
+
#
|
|
15
|
+
# Three properties of the language drive most of the rule ordering:
|
|
16
|
+
#
|
|
17
|
+
# * inside `"..."` the ONLY escape is `\"` -- `\d`, `\s` and `\\` are ordinary
|
|
18
|
+
# characters, which is why regexes are written raw. `"""..."""` is different: it
|
|
19
|
+
# takes the JSON escape set, and is the only string form that may span newlines.
|
|
20
|
+
# * `-` binds greedily into a number, so `a-1` is two tokens: `a` and `-1`.
|
|
21
|
+
# * the language resolves ambiguity by longest match where Rouge takes the first
|
|
22
|
+
# match, so `4625abc` is a single identifier and the number rules need a trailing
|
|
23
|
+
# `(?!#{WORD_CHAR})` guard or they would split it.
|
|
24
|
+
class XQL < RegexLexer
|
|
25
|
+
title "XQL"
|
|
26
|
+
desc "Cortex XSIAM/XDR Query Language and XIF parsing/modeling rules"
|
|
27
|
+
tag "xql"
|
|
28
|
+
aliases "xif", "xsiam"
|
|
29
|
+
filenames "*.xql", "*.xif"
|
|
30
|
+
mimetypes "text/x-xql"
|
|
31
|
+
|
|
32
|
+
# Required for an out-of-tree lexer: Lexer.demo_file defaults to a path inside
|
|
33
|
+
# rouge's own lib/rouge, so `.demo` would raise Errno::ENOENT without this.
|
|
34
|
+
demo_file File.expand_path("../demos/xql", __dir__)
|
|
35
|
+
|
|
36
|
+
lazy { require_relative "xql/builtins" }
|
|
37
|
+
|
|
38
|
+
# Stage commands -- the word that opens a query or follows a pipe.
|
|
39
|
+
# The last six are valid only in XIF rule files.
|
|
40
|
+
def self.commands
|
|
41
|
+
@commands ||= Set.new %w(
|
|
42
|
+
aihelper search preset call call_macro dataset cold_dataset datamodel
|
|
43
|
+
filter sort limit fields alter comp windowcomp target dedup bin union
|
|
44
|
+
join pivot view arrayexpand replacenull config iploc transaction tag
|
|
45
|
+
top getrole transpose hl highlight table graph
|
|
46
|
+
drop storeset storeget switch case default
|
|
47
|
+
)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# Deliberately NOT commands: `colddataset` and `callmacro`. Some tooling lists these
|
|
51
|
+
# spellings, but the language accepts only `cold_dataset` and `call_macro`.
|
|
52
|
+
# Highlighting them as commands would hide a real syntax error.
|
|
53
|
+
|
|
54
|
+
# Words that modify a stage rather than opening one.
|
|
55
|
+
def self.clause_keywords
|
|
56
|
+
@clause_keywords ||= Set.new %w(
|
|
57
|
+
as by asc desc for span bins suffix timeshift timezone
|
|
58
|
+
)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def self.operator_words
|
|
62
|
+
@operator_words ||= Set.new %w(
|
|
63
|
+
and or not in contains like between incidr incidr6
|
|
64
|
+
)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def self.constants
|
|
68
|
+
@constants ||= Set.new %w(true false null)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# Only consulted directly before a `(`, where these are join types
|
|
72
|
+
# (`join type = inner (...)`) rather than calls. None is a builtin function.
|
|
73
|
+
def self.join_types
|
|
74
|
+
@join_types ||= Set.new %w(inner left right outer full cross)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
# An identifier is one or more of: digit, ASCII letter, underscore, or a character in
|
|
78
|
+
# [\u0080-\u201B] / [\u201E-\uFFFE] -- note the hole, which excludes the smart quotes
|
|
79
|
+
# U+201C and U+201D. Backtick-quoted names are handled by their own rule.
|
|
80
|
+
WORD_CHAR = /[0-9A-Za-z_\u0080-\u201B\u201E-\uFFFE]/.freeze
|
|
81
|
+
|
|
82
|
+
# A double-quoted string, which cannot span a newline.
|
|
83
|
+
QUOTED = /"(?:\\"|[^"\r\n])*"/.freeze
|
|
84
|
+
|
|
85
|
+
# `\"` is the only escape, and `\\` is NOT one -- so `"\\"` is ambiguous: the last
|
|
86
|
+
# two characters are either an escaped quote (leaving the string open) or a literal
|
|
87
|
+
# backslash and the closing quote. The language resolves this by longest valid token,
|
|
88
|
+
# i.e. `\"` is an escape only when a closing quote is still reachable on the same
|
|
89
|
+
# line. This lookahead, applied at the opening quote and at each `\"`, reproduces
|
|
90
|
+
# that. Regexes ending in a backslash depend on it.
|
|
91
|
+
CLOSING_QUOTE_AHEAD = /(?=(?:\\"|[^"\r\n])*")/.freeze
|
|
92
|
+
|
|
93
|
+
def self.detect?(text)
|
|
94
|
+
# An XIF section header is unambiguous; nothing else opens a file this way.
|
|
95
|
+
return true if text =~ /\A\s*(?:\/\*.*?\*\/|\/\/[^\n]*\n|\s)*\[\s*(?:INGEST|MODEL|RULE|CONST)\b/im
|
|
96
|
+
|
|
97
|
+
false
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
state :root do
|
|
101
|
+
mixin :comments
|
|
102
|
+
|
|
103
|
+
# [RULE: NAME] -- matched whole so the macro name can be picked out. Every other
|
|
104
|
+
# header form falls through to the generic rule below.
|
|
105
|
+
rule %r/(\[)(\s*)(RULE)(\s*)(:)(\s*)(#{WORD_CHAR}+)(\s*)(\])/i do
|
|
106
|
+
groups Punctuation, Text::Whitespace, Keyword::Namespace, Text::Whitespace,
|
|
107
|
+
Punctuation, Text::Whitespace, Name::Function, Text::Whitespace, Punctuation
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
# [INGEST:...] [MODEL:...] [CONST] [CALL ...]. Not distinct tokens in the language
|
|
111
|
+
# -- they lex as `[` name `:` ... `]` -- but they are the file's structure.
|
|
112
|
+
rule %r/(\[)(\s*)(INGEST|MODEL|CONST|CALL|RULE)\b/i do
|
|
113
|
+
groups Punctuation, Text::Whitespace, Keyword::Namespace
|
|
114
|
+
push :section_header
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
mixin :expression
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
state :section_header do
|
|
121
|
+
rule %r/\]/, Punctuation, :pop!
|
|
122
|
+
mixin :comments
|
|
123
|
+
rule %r/\b(?:keep|drop)\b/i, Keyword::Constant
|
|
124
|
+
rule QUOTED, Str::Double
|
|
125
|
+
rule %r/#{WORD_CHAR}+(?=\s*=)/, Name::Attribute
|
|
126
|
+
rule %r/#{WORD_CHAR}+/, Name
|
|
127
|
+
rule %r/=/, Operator
|
|
128
|
+
rule %r/[:,]/, Punctuation
|
|
129
|
+
rule %r/./m, Error
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
state :comments do
|
|
133
|
+
rule %r/\s+/m, Text::Whitespace
|
|
134
|
+
# A section-marker comment. Highest priority of all rules, and conventionally
|
|
135
|
+
# rendered in the background colour, i.e. invisible.
|
|
136
|
+
rule %r/§.*?§/m, Comment::Special
|
|
137
|
+
rule %r(//[^\n]*), Comment::Single
|
|
138
|
+
# Non-greedy and does NOT nest: `/* a /* b */` ends at the first `*/`.
|
|
139
|
+
rule %r(/\*.*?\*/)m, Comment::Multiline
|
|
140
|
+
# A dangling delimiter is always an error.
|
|
141
|
+
rule %r(/\*|\*/), Error
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
state :expression do
|
|
145
|
+
mixin :comments
|
|
146
|
+
|
|
147
|
+
# -- strings ---------------------------------------------------------------
|
|
148
|
+
# Triple quotes first: they take the JSON escape set and may span newlines.
|
|
149
|
+
rule %r/"""/, Str::Double, :triple_string
|
|
150
|
+
rule %r/"#{CLOSING_QUOTE_AHEAD}/, Str::Double, :string
|
|
151
|
+
# No closing quote on this line, so this cannot be a string at all.
|
|
152
|
+
rule %r/"[^\r\n]*/, Error
|
|
153
|
+
|
|
154
|
+
# The identifier quote, used to name a field that collides with a reserved word or
|
|
155
|
+
# contains a space -- `filter `target` = 1`.
|
|
156
|
+
rule %r/`(?:\\`|[^`\r\n])*`/, Str::Backtick
|
|
157
|
+
rule %r/`/, Error
|
|
158
|
+
|
|
159
|
+
# -- names -----------------------------------------------------------------
|
|
160
|
+
# A query parameter ($user) or an XIF [CONST] reference ($MY_REGEX).
|
|
161
|
+
rule %r/\$#{WORD_CHAR}*(?:\.#{WORD_CHAR}+)*/, Name::Variable
|
|
162
|
+
|
|
163
|
+
# Three tokens in the language (name, dot, name), but one visual unit.
|
|
164
|
+
rule %r/\b(?:ENUM|XDM_CONST)\.#{WORD_CHAR}+/i, Name::Constant
|
|
165
|
+
|
|
166
|
+
# `xdm.source.user.username` really is a single token.
|
|
167
|
+
rule %r/\bxdm\.#{WORD_CHAR}+(?:\.#{WORD_CHAR}+)*/i, Name::Attribute
|
|
168
|
+
|
|
169
|
+
# -- numbers ---------------------------------------------------------------
|
|
170
|
+
# Time literals before plain numbers, and `ms`/`mo` before the single letters.
|
|
171
|
+
rule %r/-?\d+(?:\.\d+)?(?:ms|mo|[smhdwy])(?!#{WORD_CHAR})/i, Literal::Date
|
|
172
|
+
rule %r/-?\d+\.\d+(?!#{WORD_CHAR})/, Num::Float
|
|
173
|
+
rule %r/-?\d+(?!#{WORD_CHAR})/, Num::Integer
|
|
174
|
+
|
|
175
|
+
# -- words -----------------------------------------------------------------
|
|
176
|
+
# Set lookups rather than Rouge's `keywords` DSL, which is not present in 5.1.0 and
|
|
177
|
+
# this gem supports rouge >= 4.0. It also avoids building a large alternation.
|
|
178
|
+
#
|
|
179
|
+
# Directly before a `(`. Commands and word operators keep their own colour here
|
|
180
|
+
# (`filter (a or b)`, `not (...)`, `in (...)`).
|
|
181
|
+
rule %r/#{WORD_CHAR}+(?=\s*\()/ do |m|
|
|
182
|
+
word = m[0].downcase
|
|
183
|
+
token(if self.class.commands.include?(word) then Keyword::Declaration
|
|
184
|
+
elsif self.class.operator_words.include?(word) then Operator::Word
|
|
185
|
+
elsif self.class.join_types.include?(word) then Keyword
|
|
186
|
+
elsif self.class.functions.include?(word) then Name::Builtin
|
|
187
|
+
else Name::Function
|
|
188
|
+
end)
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
# Everywhere else. Builtin function names are deliberately absent: bare `count`
|
|
192
|
+
# or `min` is a field name, not a call.
|
|
193
|
+
rule %r/#{WORD_CHAR}+/ do |m|
|
|
194
|
+
word = m[0].downcase
|
|
195
|
+
token(if self.class.commands.include?(word) then Keyword::Declaration
|
|
196
|
+
elsif self.class.clause_keywords.include?(word) then Keyword
|
|
197
|
+
elsif self.class.operator_words.include?(word) then Operator::Word
|
|
198
|
+
elsif self.class.constants.include?(word) then Keyword::Constant
|
|
199
|
+
else Name
|
|
200
|
+
end)
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
# -- operators and punctuation --------------------------------------------
|
|
204
|
+
rule %r/->/, Operator
|
|
205
|
+
# A regex match takes a regex on the right, so colour that string as one.
|
|
206
|
+
rule %r/!~=|~=/, Operator, :regex_expected
|
|
207
|
+
rule %r/!=|<=|>=/, Operator
|
|
208
|
+
rule %r/[=<>|*+-]/, Operator
|
|
209
|
+
# `[]` (whole array) and `{}` (whole object) are single tokens; a lone brace is not.
|
|
210
|
+
rule %r/\[\]|\{\}/, Punctuation
|
|
211
|
+
rule %r/[()\[\],.:;]/, Punctuation
|
|
212
|
+
|
|
213
|
+
# `#`, `%`, `@`, `'`, `&`, `^`, `?`, `{`, `}`, `\` and the smart quotes are not
|
|
214
|
+
# valid anywhere outside a string.
|
|
215
|
+
rule %r/./m, Error
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
# After `~=` / `!~=`: colour a string literal here as a regex, then resume.
|
|
219
|
+
# `goto` rather than `push` so the string state pops straight back to :expression.
|
|
220
|
+
state :regex_expected do
|
|
221
|
+
mixin :comments
|
|
222
|
+
|
|
223
|
+
rule %r/"""/ do
|
|
224
|
+
token Str::Regex
|
|
225
|
+
goto :triple_regex
|
|
226
|
+
end
|
|
227
|
+
|
|
228
|
+
rule %r/"#{CLOSING_QUOTE_AHEAD}/ do
|
|
229
|
+
token Str::Regex
|
|
230
|
+
goto :regex
|
|
231
|
+
end
|
|
232
|
+
|
|
233
|
+
rule %r/"[^\r\n]*/, Error, :pop!
|
|
234
|
+
|
|
235
|
+
# Not a literal (a field, a $param, a call) -- nothing to do.
|
|
236
|
+
rule %r/(?=\S)/ do
|
|
237
|
+
pop!
|
|
238
|
+
end
|
|
239
|
+
end
|
|
240
|
+
|
|
241
|
+
state :string do
|
|
242
|
+
rule %r/\\"#{CLOSING_QUOTE_AHEAD}/, Str::Escape
|
|
243
|
+
rule %r/"/, Str::Double, :pop!
|
|
244
|
+
rule %r/[^"\\\r\n]+/, Str::Double
|
|
245
|
+
# A backslash that is not part of `\"` is an ordinary character here.
|
|
246
|
+
rule %r/\\/, Str::Double
|
|
247
|
+
# Defensive only: the entry rule already diverts an unterminated string to Error.
|
|
248
|
+
rule %r/\r?\n/, Error, :pop!
|
|
249
|
+
end
|
|
250
|
+
|
|
251
|
+
# Same content, but every backslash escape gets picked out -- these are regexes.
|
|
252
|
+
state :regex do
|
|
253
|
+
rule %r/\\"#{CLOSING_QUOTE_AHEAD}/, Str::Escape
|
|
254
|
+
rule %r/"/, Str::Regex, :pop!
|
|
255
|
+
# Deliberately excludes `\\`: consuming a backslash pair here would step over the
|
|
256
|
+
# `\"` above and close the string early on a regex ending in `...\\"`.
|
|
257
|
+
rule %r/\\[^"\\\r\n]/, Str::Escape
|
|
258
|
+
rule %r/[^"\\\r\n]+/, Str::Regex
|
|
259
|
+
rule %r/\\/, Str::Regex
|
|
260
|
+
# Defensive only: the entry rule already diverts an unterminated string to Error.
|
|
261
|
+
rule %r/\r?\n/, Error, :pop!
|
|
262
|
+
end
|
|
263
|
+
|
|
264
|
+
# Takes the JSON escape set (\" \/ \\ \b \f \n \r \t) and, unlike a plain
|
|
265
|
+
# double-quoted string, may span newlines.
|
|
266
|
+
state :triple_string do
|
|
267
|
+
rule %r/"""/, Str::Double, :pop!
|
|
268
|
+
rule %r(\\["/\\bfnrt]), Str::Escape
|
|
269
|
+
rule %r/[^"\\]+/m, Str::Double
|
|
270
|
+
rule %r/[\\"]/, Str::Double
|
|
271
|
+
end
|
|
272
|
+
|
|
273
|
+
state :triple_regex do
|
|
274
|
+
rule %r/"""/, Str::Regex, :pop!
|
|
275
|
+
rule %r(\\["/\\bfnrt]), Str::Escape
|
|
276
|
+
rule %r/[^"\\]+/m, Str::Regex
|
|
277
|
+
rule %r/[\\"]/, Str::Regex
|
|
278
|
+
end
|
|
279
|
+
end
|
|
280
|
+
end
|
|
281
|
+
end
|
data/lib/rouge-xql.rb
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# -*- coding: utf-8 -*- #
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require "rouge"
|
|
5
|
+
|
|
6
|
+
# Defining the class registers it: Rouge::Lexer.tag calls Lexer.register as a side
|
|
7
|
+
# effect, so after this require Rouge::Lexer.find("xql") resolves. Nothing else to wire up.
|
|
8
|
+
require_relative "rouge/lexers/xql"
|
metadata
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: rouge-xql
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Joris Vanhecke
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: rouge
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '4.0'
|
|
19
|
+
- - "<"
|
|
20
|
+
- !ruby/object:Gem::Version
|
|
21
|
+
version: '6.0'
|
|
22
|
+
type: :runtime
|
|
23
|
+
prerelease: false
|
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
25
|
+
requirements:
|
|
26
|
+
- - ">="
|
|
27
|
+
- !ruby/object:Gem::Version
|
|
28
|
+
version: '4.0'
|
|
29
|
+
- - "<"
|
|
30
|
+
- !ruby/object:Gem::Version
|
|
31
|
+
version: '6.0'
|
|
32
|
+
description: 'A Rouge syntax-highlighting lexer for XQL. Registers under the tag "xql".
|
|
33
|
+
|
|
34
|
+
'
|
|
35
|
+
email:
|
|
36
|
+
- joris@joris.be
|
|
37
|
+
executables: []
|
|
38
|
+
extensions: []
|
|
39
|
+
extra_rdoc_files: []
|
|
40
|
+
files:
|
|
41
|
+
- CHANGELOG.md
|
|
42
|
+
- LICENSE
|
|
43
|
+
- README.md
|
|
44
|
+
- lib/rouge-xql.rb
|
|
45
|
+
- lib/rouge/demos/xql
|
|
46
|
+
- lib/rouge/lexers/xql.rb
|
|
47
|
+
- lib/rouge/lexers/xql/builtins.rb
|
|
48
|
+
homepage: https://github.com/vanhecke/rouge-xql
|
|
49
|
+
licenses:
|
|
50
|
+
- MIT
|
|
51
|
+
metadata:
|
|
52
|
+
homepage_uri: https://github.com/vanhecke/rouge-xql
|
|
53
|
+
source_code_uri: https://github.com/vanhecke/rouge-xql
|
|
54
|
+
bug_tracker_uri: https://github.com/vanhecke/rouge-xql/issues
|
|
55
|
+
changelog_uri: https://github.com/vanhecke/rouge-xql/blob/main/CHANGELOG.md
|
|
56
|
+
rubygems_mfa_required: 'true'
|
|
57
|
+
rdoc_options: []
|
|
58
|
+
require_paths:
|
|
59
|
+
- lib
|
|
60
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
61
|
+
requirements:
|
|
62
|
+
- - ">="
|
|
63
|
+
- !ruby/object:Gem::Version
|
|
64
|
+
version: '3.0'
|
|
65
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
66
|
+
requirements:
|
|
67
|
+
- - ">="
|
|
68
|
+
- !ruby/object:Gem::Version
|
|
69
|
+
version: '0'
|
|
70
|
+
requirements: []
|
|
71
|
+
rubygems_version: 4.0.16
|
|
72
|
+
specification_version: 4
|
|
73
|
+
summary: Rouge lexer for XQL, the Cortex XSIAM/XDR query and rule language
|
|
74
|
+
test_files: []
|