kramdown-syntax_tree_sitter 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +295 -0
- data/ext/Rakefile +5 -0
- data/ext/tasks.rake +34 -0
- data/ext/tree_sitter_adapter/Cargo.lock +924 -0
- data/ext/tree_sitter_adapter/Cargo.toml +16 -0
- data/ext/tree_sitter_adapter/src/lib.rs +62 -0
- data/ext/tree_sitter_adapter/src/tree_sitter_adapter.rs +171 -0
- data/lib/kramdown/converter/syntax_highlighter/tree_sitter.rb +35 -0
- data/lib/kramdown/syntax_tree_sitter/version.rb +12 -0
- data/lib/kramdown/syntax_tree_sitter.rb +4 -0
- data/lib/tree_sitter_adapter.rb +8 -0
- metadata +168 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 02e58047b1cac1778ad13d25f88a64a42f0714faeba928322851a78afeb4ce66
|
4
|
+
data.tar.gz: 87cc78c75fb83a9a56cca8bb663f72b1d3abbd03f3920e37bad6ac629c699d69
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 3b609ade16479c2e09504248970b1f95eb496b982cc641e766881dbfc74bedd580a45bc5f0c24f9b859535a3967a74e51126e6dfdf8df7448a5284da259e3105
|
7
|
+
data.tar.gz: ae2cdab47a180b7435fc9c01bff81aefb377a353af341dc98f3f5bd8685105f6b976d35c25a900fcc7caa8744e3c34ef3fdbb63fb09dc27d52ee63f67644466e
|
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
MIT License
|
2
|
+
|
3
|
+
Copyright (c) 2022 Andrew T. Biehl
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,295 @@
|
|
1
|
+
# Kramdown Tree-sitter Highlighter
|
2
|
+
|
3
|
+
***Syntax highlight code with [Tree-sitter](https://tree-sitter.github.io/tree-sitter)
|
4
|
+
via [Kramdown](https://kramdown.gettalong.org).***
|
5
|
+
|
6
|
+
This is a syntax highlighter plugin for [Kramdown](https://kramdown.gettalong.org) that
|
7
|
+
leverages
|
8
|
+
[Tree-sitter's native syntax highlighter](https://tree-sitter.github.io/tree-sitter/syntax-highlighting)
|
9
|
+
to highlight code blocks (and spans) when rendering HTML.
|
10
|
+
|
11
|
+
## Getting started
|
12
|
+
|
13
|
+
### Requirements and compatibility
|
14
|
+
|
15
|
+
This plugin is built for [Kramdown](https://kramdown.gettalong.org) and hence requires a
|
16
|
+
compatible [Ruby](https://www.ruby-lang.org) installation to function. It is also
|
17
|
+
essentially an adapter for the
|
18
|
+
[Tree-sitter highlight library](https://crates.io/crates/tree-sitter-highlight) and
|
19
|
+
hence also requires a compatible [Rust](https://www.rust-lang.org) installation to
|
20
|
+
function. It is officially compatible with the following environments:
|
21
|
+
|
22
|
+
- **Ruby**: 2.7, 3.0, 3.1
|
23
|
+
- **Rust**: 1.61, 1.62, 1.63, 1.64, 1.65
|
24
|
+
- **Platforms**: MacOS, Linux
|
25
|
+
|
26
|
+
### Installation
|
27
|
+
|
28
|
+
For projects using [Bundler](https://bundler.io) for dependency management, run the
|
29
|
+
following command to both install the gem and add it to the Gemfile:
|
30
|
+
|
31
|
+
```shell
|
32
|
+
bundle add kramdown-syntax_tree_sitter --github andrewtbiehl/kramdown-syntax_tree_sitter
|
33
|
+
```
|
34
|
+
|
35
|
+
Otherwise, download this project's repository and then run the following command from
|
36
|
+
within it to build and install the gem:
|
37
|
+
|
38
|
+
```shell
|
39
|
+
gem build && gem install kramdown-syntax_tree_sitter
|
40
|
+
```
|
41
|
+
|
42
|
+
## Usage
|
43
|
+
|
44
|
+
### Quickstart
|
45
|
+
|
46
|
+
For the following example to function, the
|
47
|
+
[Tree-sitter Python parser library](https://github.com/tree-sitter/tree-sitter-python)
|
48
|
+
must be present inside a directory called `tree_sitter_parsers`, which in turn must be
|
49
|
+
located in the home directory. See the subsequent
|
50
|
+
['Tree-sitter parsers'](#tree-sitter-parsers) section for more information.
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
require 'kramdown'
|
54
|
+
require 'kramdown/syntax_tree_sitter'
|
55
|
+
|
56
|
+
text = <<~MARKDOWN
|
57
|
+
~~~source.python
|
58
|
+
print('Hello, World!')
|
59
|
+
~~~
|
60
|
+
MARKDOWN
|
61
|
+
|
62
|
+
Kramdown::Document.new(text, syntax_highlighter: :'tree-sitter').to_html
|
63
|
+
```
|
64
|
+
|
65
|
+
### Usage with Jekyll
|
66
|
+
|
67
|
+
This plugin can be used with the popular static site generator
|
68
|
+
[Jekyll](https://jekyllrb.com). Jekyll projects using Kramdown for Markdown rendering
|
69
|
+
(the default setting for Jekyll), can enable this plugin by installing the gem (most
|
70
|
+
likely via Bundler, as described in the ['Installation'](#installation) section) and
|
71
|
+
then adding the following lines to the Jekyll project's configuration file
|
72
|
+
(`_config.yml`):
|
73
|
+
|
74
|
+
```yaml
|
75
|
+
plugins:
|
76
|
+
- kramdown/syntax_tree_sitter
|
77
|
+
# Other Jekyll plugins...
|
78
|
+
kramdown:
|
79
|
+
syntax_highlighter: tree-sitter
|
80
|
+
# Other Kramdown options...
|
81
|
+
```
|
82
|
+
|
83
|
+
To highlight every code block in a Jekyll project via the plugin, make sure that every
|
84
|
+
language identifier is expressed with the correct Tree-sitter scope and that every
|
85
|
+
language referenced has a corresponding Tree-sitter parser library installed. See the
|
86
|
+
subsequent ['Language identifiers'](#language-identifiers) and
|
87
|
+
['Tree-sitter parsers'](#tree-sitter-parsers) sections for more information.
|
88
|
+
|
89
|
+
Also, there are multiple ways to render highlighted code blocks with Jekyll, as
|
90
|
+
illustrated in the following table:
|
91
|
+
|
92
|
+
<table>
|
93
|
+
<tr>
|
94
|
+
<th>Method name</th>
|
95
|
+
<th>Example</th>
|
96
|
+
<th>Supported by this plugin</th>
|
97
|
+
</tr>
|
98
|
+
<tr>
|
99
|
+
<td>Fenced code block</td>
|
100
|
+
<td>
|
101
|
+
|
102
|
+
---
|
103
|
+
````
|
104
|
+
```source.python
|
105
|
+
print('Hello, World!')
|
106
|
+
```
|
107
|
+
|
108
|
+
or
|
109
|
+
|
110
|
+
~~~source.python
|
111
|
+
print('Hello, World!')
|
112
|
+
~~~
|
113
|
+
````
|
114
|
+
---
|
115
|
+
|
116
|
+
</td>
|
117
|
+
<td>:white_check_mark:</td>
|
118
|
+
</tr>
|
119
|
+
<tr>
|
120
|
+
<td>Indented code block</td>
|
121
|
+
<td>
|
122
|
+
|
123
|
+
---
|
124
|
+
```
|
125
|
+
print('Hello, World!')
|
126
|
+
{: class="language-source.python" }
|
127
|
+
```
|
128
|
+
---
|
129
|
+
|
130
|
+
</td>
|
131
|
+
<td>:white_check_mark:</td>
|
132
|
+
</tr>
|
133
|
+
<tr>
|
134
|
+
<td>Liquid highlight tag</td>
|
135
|
+
<td>
|
136
|
+
|
137
|
+
---
|
138
|
+
```
|
139
|
+
{% highlight source.python %}
|
140
|
+
print('Hello, World!')
|
141
|
+
{% endhighlight %}
|
142
|
+
```
|
143
|
+
---
|
144
|
+
|
145
|
+
</td>
|
146
|
+
<td>:x:</td>
|
147
|
+
</tr>
|
148
|
+
</table>
|
149
|
+
|
150
|
+
Since Jekyll does not defer to Kramdown to render Liquid highlight tags, this plugin
|
151
|
+
does not support highlighting code using that method. Therefore, ***code blocks must be
|
152
|
+
represented in either fenced or indented notation*** in order to be rendered via this
|
153
|
+
plugin.
|
154
|
+
|
155
|
+
### Tree-sitter parsers
|
156
|
+
|
157
|
+
Tree-sitter relies on external parser libraries to understand each language grammar.
|
158
|
+
Thus, in order to syntax highlight a given language using this plugin, that language's
|
159
|
+
Tree-sitter parser library must be installed to the correct directory on your machine.
|
160
|
+
This directory is set as `~/tree_sitter_parsers` by default but is also configurable
|
161
|
+
(see the ['Configuration'](#configuration) section for details).
|
162
|
+
|
163
|
+
For most such parser libraries, installation simply amounts to downloading the
|
164
|
+
repository into the configured Tree-sitter parsers directory.
|
165
|
+
|
166
|
+
A partial list of languages for which Tree-sitter parser libraries have been developed
|
167
|
+
can be found on
|
168
|
+
[the official Tree-sitter website](https://tree-sitter.github.io/tree-sitter/#available-parsers).
|
169
|
+
|
170
|
+
### Language identifiers
|
171
|
+
|
172
|
+
Tree-sitter uses a string-based identifier called a 'scope' to identify each language.
|
173
|
+
For example, the scope string for Python is 'source.python', whereas for HTML it is
|
174
|
+
'text.html.basic'. Currently, this plugin follows this same convention, so a given code
|
175
|
+
block will only be correctly highlighted if the language identifier provided for that
|
176
|
+
code block is its language's corresponding Tree-sitter scope string. This is illustrated
|
177
|
+
by the code block used in the [Quickstart](#quickstart) example.
|
178
|
+
|
179
|
+
### Configuration
|
180
|
+
|
181
|
+
This Kramdown plugin currently supports the following options when provided as sub-keys
|
182
|
+
of the Kramdown option `syntax_highlighter_opts`:
|
183
|
+
|
184
|
+
| Key | Description | Default value |
|
185
|
+
| :-- | :-- | :-- |
|
186
|
+
| `tree_sitter_parsers_dir` | The path to the Tree-sitter language parsers directory. | `~/tree_sitter_parsers` |
|
187
|
+
|
188
|
+
## Contributing
|
189
|
+
|
190
|
+
Contributions are welcome!
|
191
|
+
|
192
|
+
### Development
|
193
|
+
|
194
|
+
To set up a compatible local development environment, please first refer to the
|
195
|
+
['Requirements and Compatibility'](#requirements-and-compatibility) section of this
|
196
|
+
document.
|
197
|
+
|
198
|
+
This project also depends on Git submodules to run some of its tests. Accordingly, make
|
199
|
+
sure to initialize recursive submodules when cloning the project for development
|
200
|
+
purposes, for example with the following command:
|
201
|
+
|
202
|
+
```shell
|
203
|
+
git clone --recurse-submodules https://github.com/andrewtbiehl/kramdown-syntax_tree_sitter.git
|
204
|
+
```
|
205
|
+
|
206
|
+
After checking out the project, run `bundle install` from within it to install
|
207
|
+
dependencies. Then run `bundle exec rake --tasks` to list all available Rake tasks. Each
|
208
|
+
task can be invoked via `bundle exec rake <task name>`. For example,
|
209
|
+
`bundle exec rake test` runs unit tests and `bundle exec rake smoke_test` installs the
|
210
|
+
gem and runs a smoke test against it.
|
211
|
+
|
212
|
+
This project uses [GitHub Actions](https://github.com/features/actions) workflows to
|
213
|
+
facilitate continuous integration. The 'Quality Control' workflow runs any time new
|
214
|
+
commits are pushed to GitHub on any branch. This workflow runs the `rubocop`, `test`,
|
215
|
+
and `smoke_test` Rake tasks to verify that new changes meet the project's code quality
|
216
|
+
standards, so it is strongly recommended that these tasks are first run locally against
|
217
|
+
new changes before such changes are pushed.
|
218
|
+
|
219
|
+
The 'Unreleased' section of the changelog should also be updated accordingly whenever
|
220
|
+
significant changes are introduced.
|
221
|
+
|
222
|
+
#### Release process
|
223
|
+
|
224
|
+
1. Determine the new version number based on the changes being introduced. This project
|
225
|
+
follows the [Semantic Versioning](https://semver.org/spec/v2.0.0.html) versioning
|
226
|
+
standard.
|
227
|
+
|
228
|
+
1. Make a commit to the trunk branch with the following changes:
|
229
|
+
|
230
|
+
- Update the project's version number (located in
|
231
|
+
[`lib/kramdown/syntax_tree_sitter/version.rb`](https://github.com/andrewtbiehl/kramdown-syntax_tree_sitter/blob/main/lib/kramdown/syntax_tree_sitter/version.rb)).
|
232
|
+
|
233
|
+
- Update the project's changelog
|
234
|
+
([`CHANGELOG.md`](https://github.com/andrewtbiehl/kramdown-syntax_tree_sitter/blob/main/CHANGELOG.md))
|
235
|
+
with documentation for the new
|
236
|
+
version by modifying the current 'Unreleased' section accordingly.
|
237
|
+
|
238
|
+
- Add a copy of the following release template to the top of the changelog and update
|
239
|
+
the 'Unreleased' section link to restart the process for the next release.
|
240
|
+
|
241
|
+
```markdown
|
242
|
+
## [Unreleased]
|
243
|
+
|
244
|
+
### Added
|
245
|
+
<!-- For new features -->
|
246
|
+
|
247
|
+
### Changed
|
248
|
+
<!-- For changes in existing functionality -->
|
249
|
+
|
250
|
+
### Deprecated
|
251
|
+
<!-- For soon-to-be removed features -->
|
252
|
+
|
253
|
+
### Removed
|
254
|
+
<!-- For now removed features -->
|
255
|
+
|
256
|
+
### Fixed
|
257
|
+
<!-- For any bug fixes -->
|
258
|
+
|
259
|
+
### Security
|
260
|
+
<!-- In case of vulnerabilities -->
|
261
|
+
```
|
262
|
+
|
263
|
+
1. [Draft and publish a new GitHub release](https://github.com/andrewtbiehl/kramdown-syntax_tree_sitter/releases/new)
|
264
|
+
with a new trunk branch tag and title corresponding to the new version number and a
|
265
|
+
description copied over from the changelog. This will trigger the 'Gem Publication'
|
266
|
+
GitHub Actions workflow, which will push the new version to
|
267
|
+
[RubyGems.org](https://rubygems.org).
|
268
|
+
|
269
|
+
## About
|
270
|
+
|
271
|
+
[Tree-sitter](https://tree-sitter.github.io/tree-sitter) is a modern, general-purpose
|
272
|
+
parsing library that outclasses many existing tools at the task of syntax highlighting.
|
273
|
+
This plugin adapts Tree-sitter's native highlighter for
|
274
|
+
[Kramdown](https://kramdown.gettalong.org), so that Tree-sitter's superior highlighting
|
275
|
+
capabilities can be easily leveraged in the context of rendering Markdown.
|
276
|
+
|
277
|
+
The basic functionality of this plugin was originally presented as a blog post:
|
278
|
+
*["Syntax highlight your Jekyll site with Tree-sitter!"](https://andrewtbiehl.com/blog/jekyll-tree-sitter)*.
|
279
|
+
This article explains the original use case and inspiration for the project, walks
|
280
|
+
through its implementation, and even provides some fun examples of syntax highlighting
|
281
|
+
with Tree-sitter.
|
282
|
+
|
283
|
+
### Disclaimer
|
284
|
+
|
285
|
+
Neither this plugin nor its author are affiliated with Tree-sitter or Kramdown in any
|
286
|
+
way. For any information particular to
|
287
|
+
[Tree-sitter](https://tree-sitter.github.io/tree-sitter) or
|
288
|
+
[Kramdown](https://kramdown.gettalong.org), please refer to their respective
|
289
|
+
documentation directly.
|
290
|
+
|
291
|
+
## License
|
292
|
+
|
293
|
+
This project is released under the MIT License.
|
294
|
+
|
295
|
+
The text for this license can be found in [the project's LICENSE.txt file](LICENSE.txt).
|
data/ext/Rakefile
ADDED
data/ext/tasks.rake
ADDED
@@ -0,0 +1,34 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'English'
|
4
|
+
|
5
|
+
MANIFEST_PATH = File.join __dir__, 'tree_sitter_adapter', 'Cargo.toml'
|
6
|
+
|
7
|
+
MISSING_RUST_ERROR_MSG = <<~TEXT
|
8
|
+
This gem requires a standard Rust installation to install and function.
|
9
|
+
See https://www.rust-lang.org/tools/install for details on installing Rust.
|
10
|
+
TEXT
|
11
|
+
|
12
|
+
namespace :extensions do
|
13
|
+
desc 'Build gem extensions in place'
|
14
|
+
task build: :'rust:build'
|
15
|
+
end
|
16
|
+
|
17
|
+
namespace :rust do
|
18
|
+
task :exists do # rubocop:disable Rake/Desc
|
19
|
+
command_exists?('cargo') || abort(MISSING_RUST_ERROR_MSG)
|
20
|
+
end
|
21
|
+
|
22
|
+
task build: :exists do # rubocop:disable Rake/Desc
|
23
|
+
build_rust_project MANIFEST_PATH
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def command_exists?(name)
|
28
|
+
`which #{name}`
|
29
|
+
$CHILD_STATUS.success?
|
30
|
+
end
|
31
|
+
|
32
|
+
def build_rust_project(manifest_path)
|
33
|
+
sh "cargo build --release --manifest-path #{manifest_path}"
|
34
|
+
end
|