@korap/cmc-tagger 1.1.0

package/CHANGELOG.md

@@ -0,0 +1,31 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## 1.1.0 - 2026-04-11
+
+### Added
+
+- Hashtag tagging with the `HST` tag.
+- The npm CLI is also exposed under the alias `cmc-tagger`.
+- CLI version output via `-V` / `--version`.
+- Regression tests for hashtag tagging and `ADR` tagging behavior.
+- Expanded documentation covering the tagset, CoNLL-U output examples, limitations, performance, and application contexts.
+
+### Changed
+
+- `ADR` is now emitted whenever a token matches the `@`-address pattern, regardless of existing POS values in the input.
+- Purely numeric tokens such as `#10` are no longer tagged as `HST`; hashtags must contain at least one Unicode letter.
+- Standalone `pkg` builds and release artifacts are now emitted as `cmc-tagger` / `cmc-tagger.exe` instead of `conllu2cmc`.
+- Test tooling was updated by upgrading Jest from `^29.7.0` to `^30.3.0`; `.gitlab-ci-local` is now ignored during Jest module discovery to avoid local naming collisions.
+- Documentation examples were revised and anonymized for public release.
+
+### Fixed
+
+- Sparse mode now respects stdout backpressure, avoiding Node.js heap exhaustion on very large corpora with many matches.
+- Hashtags containing Unicode letters such as umlauts are now tagged as `HST`.
+- Emoji-name values in the `n` FEATS field no longer insert spurious underscores after separators such as `:` and `,`; examples now use forms like `thumbs_up:light_skin_tone` and `family:man,man,boy`.
+
+## 1.0.0
+
+- Initial release.

package/Readme.md

@@ -0,0 +1,187 @@
+# conllu-cmc
+
+[![Docker](https://img.shields.io/docker/v/korap/conllu-cmc?label=Docker&sort=semver)](https://hub.docker.com/r/korap/conllu-cmc)
+
+Reads CoNLL-U format from stdin and annotates Unicode emojis, ASCII emoticons, Wikipedia emoji templates, action words, hashtags, URLs, email addresses, and @names with CMC-oriented STTS-IBK tags in the XPOS column (Beißwenger/Bartsch/Evert/Würzner 2016). Writes CoNLL-U format to stdout.
+
+For Unicode emojis (`EMOIMG`), the base emoji without skin tone modifiers
+is written to the LEMMA column and Unicode emoji metadata is added to the FEATS column:
+
+<!-- markdownlint-disable MD010 -->
+```tsv
+# text = 😂
+1	😂	😂	_	EMOIMG	g=smileys_&_emotion|s=face_smiling|q=fully_qualified|v=E0.6|n=face_with_tears_of_joy	_	_	_	_
+```
+<!-- markdownlint-enable MD010 -->
+
+The FEATS field encodes: `g` (group), `s` (subgroup), `q` (qualification status), `v` (Unicode version first introduced), `n` (emoji name – including skin tone). In `n`, separators such as `:` and `,` are preserved without following spaces, e.g. `thumbs_up:light_skin_tone` or `family:man,man,boy`. See <https://www.unicode.org/Public/UCD/latest/emoji/emoji-test.txt> for details.
+
+## Tagset
+
+All CMC annotations are written to the XPOS column (column 5 in CoNLL-U). For `EMOIMG`, the tagger also normalizes the LEMMA column to the base emoji and enriches FEATS with emoji metadata when available.
+
+| Tag | Phenomenon | Example token | Output behavior |
+| --- | --- | --- | --- |
+| `EMOWIKI` | Wikipedia emoji templates | `[_EMOJI:{{S\|;)}}_]` | Writes `EMOWIKI` to XPOS |
+| `EMOIMG` | Unicode emoji tokens | `😂`, `😇` | Writes `EMOIMG` to XPOS, normalizes LEMMA to the base emoji, and adds FEATS metadata |
+| `AKW` | Action words / inflectives | `:grins:` | Writes `AKW` to XPOS |
+| `EMOASC` | ASCII emoticons | `:)`, `<3` | Writes `EMOASC` to XPOS |
+| `HST` | Hashtags | `#KorAP`, `#3D` | Writes `HST` to XPOS when the hashtag contains at least one letter |
+| `URL` | URLs | `https://korap.ids-mannheim.de` | Writes `URL` to XPOS |
+| `EML` | Email addresses | `mail@example.org` | Writes `EML` to XPOS |
+| `ADR` | `@`-names / addresses | `@markup` | Writes `ADR` to XPOS |
+
+Numeric-only forms such as `#10` are not tagged as `HST`.
+
+## CoNLL-U Output Examples
+
+The following example shows how the different tags appear in CoNLL-U output. In all cases, the annotation is written to XPOS; only `EMOIMG` additionally changes LEMMA and FEATS.
+
+<!-- markdownlint-disable MD010 -->
+```tsv
+# foundry = cmc
+# text_id = readme-demo
+# text = [_EMOJI:{{cool}}_] 😂 :grins: :) #KorAP https://korap.ids-mannheim.de mail@example.org @handle <3
+1	[_EMOJI:{{cool}}_]	_	_	EMOWIKI	_	_	_	_	_
+2	😂	😂	_	EMOIMG	g=smileys_&_emotion|s=face_smiling|q=fully_qualified|v=E0.6|n=face_with_tears_of_joy	_	_	_	_
+3	:grins:	_	_	AKW	_	_	_	_	_
+4	:)	_	_	EMOASC	_	_	_	_	_
+5	#KorAP	_	_	HST	_	_	_	_	_
+6	https://korap.ids-mannheim.de	_	_	URL	_	_	_	_	_
+7	mail@example.org	_	_	EML	_	_	_	_	_
+8	@handle	_	_	ADR	_	_	_	_	_
+9	<3	_	_	EMOASC	_	_	_	_	_
+```
+<!-- markdownlint-enable MD010 -->
+
+For compound emojis with modifiers or zero-width joiners, the tagger still writes `EMOIMG` and reduces LEMMA to the base emoji. For example, `✊🏿` is normalized to lemma `✊`, and `👨‍👨‍👦` is normalized to lemma `👨`.
+
+## Current Limitations
+
+- The tagger is purely pattern-based. It does not consider sentential, pragmatic, or discourse context.
+- The matching strategy is intentionally recall-oriented rather than precision-oriented. Ambiguous strings such as `<3` may therefore produce false positives.
+- Annotation quality depends heavily on tokenization. Unicode emojis, grapheme clusters, zero-width joiners, modifiers, emoticons, and Wikipedia emoji templates should already be segmented into correct token units before tagging.
+- We recommend [KorAP-Tokenizer](https://github.com/KorAP/KorAP-Tokenizer), which supports Unicode 17.0, including grapheme clusters, zero-width joiners, modifiers, emoticons, and Wikipedia-template-based emojis.
+
+## Local Usage
+
+The npm package exposes the CLI under both `conllu-cmc` and `cmc-tagger`.
+
+### Using npm/node
+
+```shell
+cat ./test/data/ndy.conllu | npx conllu-cmc
+
+# Show version
+npx conllu-cmc -V
+
+# Alternative CLI name
+npm exec --yes --package=. cmc-tagger -- -V
+```
+
+### Using standalone binary
+
+```shell
+cat ./test/data/ndy.conllu | ./bin/linux/cmc-tagger
+
+# Show version
+./bin/linux/cmc-tagger -V
+```
+
+### Generate KorAP-XML zip with CMC annotations
+
+```shell
+korapxml2conllu kyc.zip | cmc-tagger -s | conllu2korapxml > kyc.cmc.zip
+```
+
+## Docker Usage
+
+```shell
+# Annotate CoNLL-U input
+cat ./test/data/ndy.conllu | docker run --rm -i korap/conllu-cmc
+
+# With sparse output (only annotated lines)
+cat ./test/data/ndy.conllu | docker run --rm -i korap/conllu-cmc -s
+
+# Generate KorAP-XML zip with CMC annotations
+# For korapxmltool see <https://github.com/KorAP/korapxmltool>
+korapxmltool -A "docker run --rm -i korap/conllu-cmc -s" -t zip ndy.zip
+
+# Show help
+docker run --rm korap/conllu-cmc --help
+
+# Show version
+docker run --rm korap/conllu-cmc -V
+```
+
+## Performance
+
+The tagger is implemented in Node.js because the runtime provides efficient regular-expression execution, which is central to this regex-based annotation workflow.
+
+On CMC corpora with many matches, throughput is above 10 MB/s. This includes dense CMC material such as the NottDeuYTSch corpus.
+
+## Applications
+
+The tagger is already used in corpus analysis scenarios with the corpus analysis platform [KorAP](https://github.com/KorAP/).
+
+### German Wikipedia Talk Pages
+
+The German Wikipedia Talk Pages corpus is available at <https://korap.ids-mannheim.de/instance/wiki>. A query for an `EMOWIKI`, an `EMOASC`, and an `EMOIMG` sequence in one posting with up to 12 intervening tokens between each match is:
+
+```cqp
+[cmc/p=EMOWIKI] []{0,12} [cmc/p=EMOASC] []{0,12} [cmc/p=EMOIMG]
+```
+
+You can run this query directly here: <https://korap.ids-mannheim.de/instance/wiki?q=[cmc%2Fp%3DEMOWIKI]+[]{0%2C12}+[cmc%2Fp%3DEMOASC]+[]{0%2C12}+[cmc%2Fp%3DEMOIMG]>.
+
+### NottDeuYTSch
+
+The NottDeuYTSch corpus (Cotgrove 2023) is accessible on request via <https://korap.ids-mannheim.de/instance/nottdeuytsch>.
+
+## Installation
+
+### Pre-built Binaries
+
+Download pre-built executables from the [Releases](https://github.com/KorAP/KorAP-CoNLL-U-CMC/releases) page:
+
+- `cmc-tagger` - Linux x64
+- `cmc-tagger` - macOS x64
+- `cmc-tagger.exe` - Windows x64
+
+### npm
+
+```shell
+npm install 'git+https://gitlab.ids-mannheim.de/KorAP/conllu-cmc-docker.git'
+```
+
+### Build from source
+
+```shell
+npm install
+```
+
+#### Build standalone executables
+
+```shell
+# Build for all platforms
+npm run pkg-all
+
+# Or build for specific platforms
+npm run pkg-linux   # Linux x64
+npm run pkg-macos   # macOS x64
+npm run pkg-win     # Windows x64
+```
+
+Executables are created as `cmc-tagger` or `cmc-tagger.exe` in `bin/linux/`, `bin/macos/`, and `bin/win/`.
+
+### Docker
+
+```shell
+docker pull korap/conllu-cmc
+```
+
+## References
+
+- Beißwenger, Michael/Bartsch, Sabine/Evert, Stefan/Würzner, Kay-Michael (2016): EmpiriST 2015: A Shared Task on the Automatic Linguistic Annotation of Computer-Mediated Communication and Web Corpora. In: Proceedings of the 10th Web as Corpus Workshop. Berlin: Association for Computational Linguistics, S. 44–56. <https://doi.org/10.18653/v1/W16-2606>.
+- Cotgrove, Louis (2023): New opportunities for researching digital youth language: The NottDeuYTSch corpus. In: Kupietz, Marc/Schmidt, Thomas (Hrsg.): Neue Entwicklungen in der Korpuslandschaft der Germanistik. Beiträge zur IDS-Methodenmesse 2022. (= Korpuslinguistik und interdisziplinäre Perspektiven auf Sprache (CLIP) 11). Tübingen: Narr, S. 102-115.
+- Margaretha, Eliza/Lüngen, Harald/Diewald, Nils/Kupietz, Marc/Yaddehige, Rameela (2025): Building and querying Wikipedia discussion corpora using KorAP. In: Impulses and Approaches to Computer-Mediated Communication: Proceedings of the 12th International Conference on Computer Mediated Communication and Social Media Corpora for the Humanities (CMC 2025). Edited by Annamária Fábián/Igor Trost, S. 123-124.

package/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@korap/cmc-tagger",
+  "version": "1.1.0",
+  "publishConfig": {
+    "access": "public"
+  },
+  "description": "Reads CoNLL-U format from stdin and annotates emojis, emoticons, hashtags, URLs, email addresses, @addresses, and action words. Writes CoNLL-U format to stdout.",
+  "main": "src/index.js",
+  "files": [
+    "src/",
+    "Readme.md",
+    "CHANGELOG.md"
+  ],
+  "bin": {
+    "conllu-cmc": "src/index.js",
+    "cmc-tagger": "src/index.js"
+  },
+  "scripts": {
+    "pkg-linux": "rm -f bin/linux/conllu2cmc && pkg src/index.js --public -t node22-linux-x64 -o bin/linux/cmc-tagger && chmod +x bin/linux/cmc-tagger",
+    "pkg-macos": "rm -f bin/macos/conllu2cmc && pkg src/index.js --public -t node22-macos-x64 -o bin/macos/cmc-tagger",
+    "pkg-win": "rm -f bin/win/conllu2cmc.exe && pkg src/index.js --public -t node22-win-x64 -o bin/win/cmc-tagger.exe",
+    "pkg-all": "npm run pkg-linux && npm run pkg-macos && npm run pkg-win",
+    "pkg": "pkg",
+    "test": "jest",
+    "update-emoji-db": "node scripts/update_emoji_db.js",
+    "start": "node src/index.js"
+  },
+  "author": "Marc Kupietz",
+  "license": "BSD-2-Clause",
+  "dependencies": {
+    "command-line-args": "^5.2.1",
+    "command-line-usage": "^7.0.1",
+    "emoji-regex": "^10.3.0"
+  },
+  "pkg": {
+    "assets": [
+      "package.json"
+    ],
+    "scripts": [
+      "src/*.js"
+    ]
+  },
+  "jest": {
+    "modulePathIgnorePatterns": [
+      "/.gitlab-ci-local/"
+    ],
+    "testPathIgnorePatterns": [
+      "/node_modules/",
+      "/.gitlab-ci-local/"
+    ]
+  },
+  "devDependencies": {
+    "@yao-pkg/pkg": "^6.14.1",
+    "child_process": "^1.0.2",
+    "jest": "^30.3.0"
+  }
+}