twpipeline 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 79331304e3921fe1d8dcbd3b47154688dfcbea28315e5e4f90a4f8c8cde3f2be
+  data.tar.gz: f4d2e3acc42108b52edce6f74a98683f1fb27aeeaabee6a8adfb48a3a52fb0fd
+SHA512:
+  metadata.gz: bc121f3c3ec181db6939aaf78b758d17c50d3882eb1150a877988cd24576596d7387716fc358b0946a2d1a57e2817d4141ccbc080f915f274de4ee4daf2232a4
+  data.tar.gz: e59bfc90d8989ca058e222f0d1f3bbcfb391644809bb7fadbb0bd867b438ba675098d057b43fdfa9cc98370c598596d7b97527e91891be03694a525d5ddabbe2

data/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## 0.1.0 — 2026-08-03
+
+First release.
+
+- Nine JSONL stages from raw crawl to n-gram tables: source gating (`01_ingest`),
+  MOE punctuation normalization (`02_normalize`), sentence segmentation
+  (`03_segment`), character-inventory and lexical-origin filtering (`04_script`,
+  `05_lexicon`, both delegated to twfilter 0.1.0), block-level evidence judgment
+  (`06_evidence`), exact type deduplication with boilerplate flagging
+  (`07_dedup`), Viterbi word segmentation (`08_tokenize`) and external-sort
+  n-gram counting with a publication floor and `--once-per host` semantics
+  (`09_count`)
+- One record contract: stages add fields and typed findings, never remove them;
+  `--drop` is the only thing that discards a record
+- Resource envelope: `--cores` / `--memory` with `TWP_CORES` / `TWP_MEMORY` and
+  autodetection; byte-bounded chunking; counting bounded by `sort -S`, not by a
+  hash table
+- Per-stage measurement into `work/benchmarks.jsonl` via `getrusage(2)`, rendered
+  to BENCHMARKS.md
+- HPLT source adapter with the `.tw` ∪ `html_lang` gate, hard regional TLD
+  exclusion and an allowlist hook (shipped empty, with 300 documented candidates)

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Den Patin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,236 @@
+# twpipeline
+
+Corpus construction for Taiwan Mandarin. Nine stages, one JSONL record contract,
+`stdin` → `stdout`. Every stage is independently runnable and independently
+verifiable. No record is ever discarded silently: rejection sets `ok: false` and
+appends a typed finding; `--drop` is what removes it.
+
+Linguistic decisions live in [twfilter](https://github.com/taiwan-corpora/twfilter); this repository is the
+execution layer — sources, parallelism, external sorting, measurement.
+
+For corpus linguists and dataset engineers who need Taiwan Mandarin
+(`cmn-Hant-TW`) material with verifiable provenance. In: a web crawl or any
+JSONL of documents. Out: filtered sentence types, word segmentation, n-gram and
+frequency tables with per-host counting and a publication floor — the process
+behind [twngrams](https://huggingface.co/datasets/taiwan-corpora/twngrams).
+Single machine, no cluster: the full HPLT Taiwan slice (2.0 M documents)
+builds in under two hours on a laptop.
+
+## Pipeline
+
+```mermaid
+flowchart TD
+  subgraph acquire["acquisition"]
+    S1["<b>01_ingest</b> — source adapter<br/>HPLT gate: <code>.tw</code> ∪ <code>html_lang ∈ zh-TW</code> ∪ allowlist<br/>hard exclusion: <code>.cn .hk .mo .sg .my</code><br/><code>--host-cap N</code> bounds per-host concentration"]
+  end
+
+  subgraph shape["orthographic normalization — mandatory for publishable text"]
+    S2["<b>02_normalize</b> — MOE 《重訂標點符號手冊》<br/>“ ” → 「 」 · → ‧ … → …… — → ──<br/>fullwidth Latin → ASCII, invisibles stripped"]
+    S3["<b>03_segment</b> — sentence boundaries<br/>terminators <code>[。！？…；]+[」』）》〉]*</code>, <code>、</code> is not a boundary<br/>shape: han ∈ [6, 60], han ratio ≥ 0.65"]
+  end
+
+  subgraph origin["origin filter — mandatory"]
+    S4["<b>04_script</b> — character inventory<br/>MOE 常用 4 808 / 次常用 6 343 / 罕用 18 356 + 12 exceptions<br/>OpenCC STCharacters ∖ TSCharacters round-trip<br/>converted orthography 裏着衞爲説麽…"]
+    S5["<b>05_lexicon</b> — lexical origin<br/>149 hard + 38 soft mainland pairs, corpus-verified<br/>written Cantonese, Hong Kong and Singapore forms, foreign-topic tag, erhua two-window rule<br/>literary-Chinese density > 0.05 with ≥ 2 hits and ≥ 8 han"]
+  end
+
+  subgraph corpus["corpus hygiene — optional, source-dependent"]
+    S6["<b>06_evidence</b> — block judgment<br/>window 300 sentences; reject block if any member fails<br/>require ≥ 1.0 Taiwan evidence per 100 sentences"]
+    S7["<b>07_dedup</b> — exact type dedup<br/>SHA-256 → 62-bit fingerprint<br/>flags types occurring on ≥ 5 distinct hosts as boilerplate"]
+  end
+
+  subgraph measure["measurement"]
+    S8["<b>08_tokenize</b> — Viterbi over a unigram cost model<br/>EM re-estimation, max word length 8"]
+    S9["<b>09_count</b> — n-gram counting by external sort<br/><code>sort -S … --parallel …</code> ∘ <code>uniq -c</code><br/>frequency floor for publication (Google Ngram rule)"]
+  end
+
+  S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 --> S9
+  S9 --> OUT["frequency, dispersion (DP),<br/>collocations (G², logDice)"]
+
+  classDef must fill:#0b3d2e,stroke:#0b3d2e,color:#fff
+  classDef may fill:#3d3a0b,stroke:#3d3a0b,color:#fff
+  class S2,S3,S4,S5 must
+  class S6,S7 may
+```
+
+Green stages are mandatory for any Taiwan Mandarin corpus. Amber stages depend
+on the source: block judgment assumes documents are topically coherent, type
+deduplication assumes the source is web-scale.
+
+## Record contract
+
+One JSON object per line. Stages add fields; they never remove them.
+
+| field | stage | meaning |
+| --- | --- | --- |
+| `id`, `source` | 01 | stable identity |
+| `url`, `host`, `gate` | 01 | provenance and the gate that admitted it |
+| `register`, `doc_score` | 01 | HPLT web-register argmax, mean segment score |
+| `text` | 01–02 | document text, then punctuation-normalized |
+| `doc`, `index`, `han` | 03 | parent document, ordinal, han count |
+| `tier` | 04 | 0 常用, 1 次常用, 2 罕用 |
+| `evidence` | 06 | Taiwan evidence count over the block |
+| `tokens` | 08 | word segmentation |
+| `ok`, `findings[]` | all | verdict and every reason for it |
+
+A finding is `{check, code, severity, detail}` with `severity ∈ {reject, mark}`.
+
+## Install
+
+Ruby ≥ 3.4. The only runtime dependency is
+[twfilter](https://rubygems.org/gems/twfilter), pulled from RubyGems:
+
+```bash
+bundle install
+bin/twpipeline stages
+```
+
+External sorting uses `gsort` (GNU coreutils) when installed and the system
+`sort` otherwise; both understand `-S` and `--parallel`. Override with
+`TWP_SORT`.
+
+## Usage
+
+```bash
+twpipeline run --source hplt --input cmn_Hant/7_1.jsonl.zst --cores 16 --memory 8G
+cat raw.jsonl | twpipeline 04_script | twpipeline 05_lexicon > checked.jsonl
+echo '這個政策有沒有經過評估？' | twpipeline check
+twpipeline run --from 03_segment --to 07_dedup --drop --policy publishable
+twpipeline resources
+twpipeline stages
+```
+
+`--policy corpus` (default) keeps everything Taiwan-attested. `--policy
+publishable` restricts to MOE 常用 4 808, rejects soft mainland markers, and
+enforces the strict MOE punctuation inventory — the boundary of what may reach a
+commercial learner product.
+
+## Resources
+
+Cores and memory are read from `--cores` / `--memory`, then `TWP_CORES` /
+`TWP_MEMORY`, then autodetection (all cores, half of physical memory).
+`--ask` prompts interactively.
+
+Streaming stages fork `cores` workers per chunk; chunk size is bounded by bytes
+(`memory / 32`, capped at 1 GiB, overridable with `TWP_CHUNK_BYTES`), not by
+line count, so a source of 12 KiB documents and a source of 40-byte sentences
+both stay within the same envelope. Counting never holds a hash table: keys go
+through `sort -S <memory> --parallel <cores>`, so peak resident size is set by
+the flag, not by the corpus.
+
+Every stage writes one line to `work/benchmarks.jsonl` — wall time, CPU seconds,
+mean cores used, peak RSS, and the stage's own counters. The work directory
+defaults to `../work` beside the checkout; override with `TWP_WORK`. See
+[BENCHMARKS.md](BENCHMARKS.md).
+
+## Findings
+
+Measured on HPLT 3.0 `cmn_Hant`, quality bins 7–10 (4 116 754 documents). The
+admission gate kept 2 001 161 documents: 541 467 by `.tw` host, 1 459 694 by
+`html_lang`. All of them were carried through stages 02–07 in 28 minutes
+(twfilter 0.1.0, the published tables):
+
+| stage | in | out | lost |
+| --- | ---: | ---: | ---: |
+| 03_segment | 2 001 161 docs | 37 772 352 sentences | — |
+| 04_script | 37 772 352 | 37 122 017 | 1.72 % |
+| 05_lexicon | 37 122 017 | 36 455 533 | 1.80 % |
+| 06_evidence | 36 455 533 | 29 089 952 | 20.20 % |
+| 07_dedup | 29 089 952 | **12 339 761 types** | **57.58 %** |
+
+Three results worth stating plainly.
+
+**Deduplication dominates every linguistic filter.** More than half of all
+segmented sentences are exact repeats of another sentence — an order of magnitude
+more than the three origin filters remove between them. HPLT deduplicates
+documents, not sentences, and the Taiwanese long tail is largely search-optimized
+loan and hotel templates. Without stage 07 an n-gram table over this material
+measures template frequency, not language.
+
+A stratified 279 198-document sample drawn from the head of each bin file put
+duplication at 71.0 % and evidence loss at 5.6 %; the true figures are 57.6 % and
+20.2 %. Documents cluster by host inside a bin file, so a head sample overstates
+local repetition and understates topical variety. Sample to find the shape of a
+problem; measure on everything before publishing a number.
+
+**The two gates differ, but less than a head sample suggests.** Unique
+Taiwan-attested types produced per admitted document:
+
+| gate | documents | unique types | types per document |
+| --- | ---: | ---: | ---: |
+| `.tw` host | 541 467 | 4 230 061 | **7.81** |
+| `html_lang` on a generic TLD | 1 459 694 | 8 109 700 | **5.56** |
+
+A 1.41× gradient, not the 2.4× the sample showed. The `html_lang` gate supplies
+65.7 % of the final corpus at about two thirds of the per-document yield — worth keeping,
+not worth trusting blindly. The `gate` field is preserved on every record, so
+restricting to `.tw` afterwards costs nothing and remains available as a
+higher-precision subset.
+
+**Boilerplate is sub-sentential, so no amount of sentence deduplication removes
+it.** The phrase 提供相關細節的諮詢服務 occurs 263 210 times in the deduplicated
+corpus, inside 263 210 *distinct* sentences, across 222 900 documents — but only
+**1 948 hosts**. Sentence dedup cannot see it (every string differs), per-document
+counting cannot see it (every document differs), and `--host-cap` cannot see it
+(the network spans 11 691 throwaway domains, none of them large). 貸款 comes out
+as the 7th most frequent word in Taiwan Mandarin, which is false.
+
+The fix is to change what a count means. `--once-per host` counts each n-gram
+once per host instead of once per occurrence, so a frequency list measures *how
+many independent sites use a phrase* rather than *how many times a template was
+published*. On the phrase above that is a 135-fold correction. For web-derived
+frequency this is the correct semantics regardless of spam: a generated page is
+not an independent act of authorship.
+
+Occurrence counting remains the default, because for an edited corpus — statutes,
+news, dictionaries — repetition is evidence rather than noise.
+
+Measured effect on the full corpus. 貸款 falls from rank 7 to rank **1 299**;
+借錢, 包養, 二胎 and 信貸 drop below the 40-host floor and disappear entirely. The
+1-gram list becomes 的 是 有 一 在 了 我 會 也 要 不 個, the 4-gram list 越來越多,
+這是一個, 最重要的是, 更重要的是. In the run where both semantics were built,
+published rows fell by 7 % at order 1 and 28 % at order 4.
+
+**Known residual.** Eleven of the top hundred 4-grams are fragments of one
+string: `讓人們盡情分享，將這個世界變得更開闊，聯繫更緊密` — the Chinese Facebook
+tagline, carried by the social-plugin embed on roughly 3 900 hosts, about once
+per host, so per-host counting cannot suppress it: those hosts really are
+independent. This is third-party embedded interface text, a
+different problem from spam, and it needs either HTML-level boilerplate stripping
+upstream or an explicit stoplist. It contaminates 0.03 % of sentences but a
+visible share of the 4-gram head.
+
+The surviving corpus spans 149 022 hosts: top 10 hold 8.8 % of sentences, top 100
+20.8 %, top 1000 46.3 %, top 10 000 82.8 %. Diffuse enough to be a corpus rather
+than a scrape of a few sites, but the single largest host is `chinatimes.com` at
+411 005 sentences — 3.3 % of everything, one newspaper. Concentration of one
+rightsholder's expression, not total size, is what has drawn takedowns against
+comparable datasets; cap per host before publishing anything that carries text.
+
+## Layout
+
+```text
+lib/twpipeline/
+  resources.rb   cores and memory resolution
+  parallel.rb    fork-based map, byte-bounded streaming
+  sorting.rb     external sort, tally with a frequency floor
+  jsonl.rb       record contract
+  bench.rb       measurement, appended to work/benchmarks.jsonl
+  rusage.rb      getrusage(2) via Fiddle
+  segmenter.rb   Viterbi, EM re-estimation
+  frequency.rb   deviation of proportions
+  stage.rb       Stage, MapStage, registry
+  stages/        the nine stages
+  sources/       source adapters
+tools/
+  regression.rb  twfilter against every reference corpus
+```
+
+## Tests
+
+```bash
+bundle exec rake
+```
+
+RSpec suite plus RBS signature validation. Fixtures carry no corpus material;
+the suite needs no data and no credentials.

data/bin/twpipeline

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+lib = File.expand_path("../lib", __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+begin
+  require "twpipeline"
+rescue LoadError
+  $LOAD_PATH.unshift(File.expand_path("../../twfilter/lib", __dir__))
+  require "twpipeline"
+end
+require "twpipeline/cli"
+
+exit(TWPipeline::CLI.call(ARGV))

data/data/hplt/README.md

@@ -0,0 +1,26 @@
+# HPLT gate data
+
+`taiwan_domains.txt` — the curated allowlist read by `01_ingest --allowlist`.
+**Empty by default, and that is deliberate.**
+
+`taiwan_domains.candidates.txt` — 300 hosts derived automatically from the
+1 927 814 documents that failed both the `.tw` and the `html_lang` gates in HPLT
+3.0 `cmn_Hant` bins 7–10. A host qualified on: ≥ 5 documents sampled, zero
+origin-filter rejections, and ≥ 1.0 Taiwan evidence per 100 han characters.
+
+The list is published as *candidates*, not as an allowlist, because its
+precision is poor for the intended purpose. Ranked by evidence density, the top
+of the list is `downloadcenter.intel.com`, `powerapps.microsoft.com`,
+`developer.microsoft.com`, `dvdvideosoft.com`, `veed.io` — localized zh-TW
+product pages. They pass every origin test because localization vendors use
+correct Taiwanese IT vocabulary, but they are `production: translated`, not
+native Taiwan Mandarin. Admitting them would reintroduce exactly the
+translationese the provenance axes exist to exclude.
+
+The 300 hosts would have added 2 812 of 859 049 sampled documents — 0.3 % more
+volume for a measurable loss of precision. Curate by hand from the candidates
+file, or leave the allowlist empty.
+
+`tools/hplt_allowlist.rb` writes the full evidence table for every host
+examined — documents, han characters, evidence per 100 han, rejection rate — to
+`work/hplt/allowlist_report.tsv`.

data/data/hplt/taiwan_domains.candidates.txt

@@ -0,0 +1,300 @@
+173ng.com
+1989wolfe.com
+3dayseo.com
+5xruby.com
+9jgirl.com
+accessories.ap.dell.com
+adai69.blogspot.com
+adobe.com
+advertise.bingads.microsoft.com
+aiot.innodisk.com
+albertinformation.blogspot.com
+amedeephotography.com
+ancad.com
+android.lisisoft.com
+angela700422.pixnet.net
+animebase.org
+applewoods.org
+aquia.tian.yam.com
+arielku.tian.yam.com
+artchen.dev
+arubanetworks.com
+asusfoundation.org
+avast.com
+bakesmith.taipei
+barbachacha.com
+baristatotw.blogspot.ca
+beautyfashion-tw.com
+beckysfashionlifestyle.com
+best007.org
+blog.adahsu.net
+blog.alexw.net
+blog.asuscloud.com
+blog.balduran.cc
+blog.cashwu.com
+blog.changyy.org
+blog.gslin.org
+blog.jln.co
+blog.jw910731.me
+blog.miniasp.com
+blog.monkeypotion.net
+blog.poychang.net
+blog.roroiii.com
+blog.schokomi.com
+blog.steveyi.net
+blog.tedshd.io
+blog.ueplay.com
+blog.wabow.com
+blog.wing0826.com
+blog.yilang.org
+bonwawa.blogspot.com
+boringfreeware.blogspot.com
+bouken.space
+bt.cream123.com
+buy.hpe.com
+cadch.com
+car.u-3c.com
+casestudies.engenius.ai
+cchsu.com
+ccskin.com
+changeanddiffusion.blogspot.com
+cherstravel.com
+chia0302.tian.yam.com
+chinese.hu-hiu.xyz
+chnlnkuo.blogspot.com
+cloud-say.com
+cognex.com
+community.oppo.com
+cool007.org
+cyberformula.tian.yam.com
+ddselectivefashion.com
+delltechnologies.com
+desa713.tian.yam.com
+developer.microsoft.com
+developers.facebook.com
+diazue65.tian.yam.com
+digital.ni.com
+dinewithkirsty.com
+divakalife.blogspot.com
+docs.adobe.com
+docs.blackberry.com
+docs.plesk.com
+docs.vmware.com
+dokuroko.blog128.fc2.com
+donation-networks.savedogs.org
+downloadcenter.intel.com
+dualsnake.tian.yam.com
+duomaxwellr.blogspot.com
+dvdvideosoft.com
+easy.twmoneyloan.com
+easyloan.taipei
+enspyre.com
+ericsk.org
+estinet.com
+event.asus.com
+experience.dropbox.com
+f4mglendo.tian.yam.com
+fenixhsu.medium.com
+fitnfun.biz
+fortinet.com
+forum.macuknow.com
+fred-zone.blogspot.com
+freedom.support.tm
+fullstackladder.dev
+furnacedigital.blogspot.com
+gemstw.com
+girlstalk.cc
+go-traveler.com
+go-tribe-ycc.blogspot.com
+gomrcuriosity.com
+goodmommom.net
+grayliao.blogspot.com
+harry811016.medium.com
+help.mega.io
+help.ptc.com
+help.tableau.com
+helpx.adobe.com
+hitcon.org
+hlisfab2015.blogspot.com
+home.allproducts.com
+hotzsoft.com
+house-expert.com
+how-to-make-money12.blogspot.com
+huidu.io
+hungryowl.typepad.com
+hyperrate.com
+ifans.pixnet.net
+iiot.apacer.com
+imsean.pixnet.net
+indianic.com
+infinitiessoft.com
+ipoe.cc
+jaceju.net
+jerolin.tian.yam.com
+jingshouse.blogspot.com
+jjdtw.com
+julianshen.blogspot.com
+june-yu.com
+juniper.net
+kaiching.org
+key.chtouch.com
+khm-film.blogspot.com
+kirkchen.logdown.com
+kskb.eu.org
+ktrees.com
+labviewpro.net
+land.gov.taipei
+larrynote.com
+larrynung.github.io
+lbrwed.com
+leasonellis.com
+leonvhi.tian.yam.com
+leoyeh-me.gitbook.io
+letsgogonz.blogspot.co.nz
+librarythings.info
+lifewth.com
+linda.sopili.net
+line8.me
+linux.vbird.org
+logitechclub.com
+lookerideas.net
+luannukm1989.tian.yam.com
+m.california-companies.com
+magiclen.org
+mahjong.caca888.net
+mantoh247.tian.yam.com
+marleen741115.tian.yam.com
+mcafee.com
+meritlilin.com
+microsoftstore.com
+midnightcoder.blogspot.com
+mit-machining.com
+mlytics.com
+mobileapps.frederickwebb.com
+moeimo2016.blogspot.com
+moonpoet.com
+morrisa41.tian.yam.com
+movavi.com
+mrs-mo.com
+msdn.microsoft.com
+museumms.blogspot.com
+musewp.pro
+mysabeauty.com
+nctusdk.blogspot.com
+netapp.com
+nineblog.tian.yam.com
+normansem.com
+note.charlestw.com
+novell.com
+nubaby.cc
+origin2.logitech.com
+ouch1978.github.io
+oursong.com
+panypanla.blogspot.com
+pccentertwschoolkf.blogspot.com
+pdfshouce.com
+peachwaneversay.blogspot.com
+peterdavehello.org
+petnii.com
+pinhladysz.tian.yam.com
+pipedrive.com
+powerapps.microsoft.com
+protonmail.com
+pt2club.blogspot.com
+pvencs.blogspot.com
+radioecouteur.com
+rafaelu8mr.tian.yam.com
+rayaegis.com
+redhat.com
+richielin-mac.blogspot.com
+roroiii.coderbridge.io
+rpg2008.blogspot.com
+rtaiwanr.wordpress.com
+ruth7112.tian.yam.com
+salesforce.com
+sandrabb.pixnet.net
+saverinthebox.com
+sctaeaa.org
+sflowerpp.tian.yam.com
+shamitsu.com
+sharkjiang.com
+shingfay.tian.yam.com
+shopaholicwifes.blogspot.it
+shopifytaiwan.com
+shuanet.com
+shutterstock.com
+si402.tian.yam.com
+skype.com
+slsp.auo.com
+sonniekoenig.com
+splunk.com
+ssemble.com
+ssuns.tian.yam.com
+sszxfcxt.tian.yam.com
+stardouble.tian.yam.com
+stellahyc.pixnet.net
+strangerckbe.tian.yam.com
+strollmarket.com
+study-area.org
+support.norton.com
+support.ubisoft.com
+support.xbox.com
+synology.com
+synology.de
+tacolin.blogspot.com
+taiwan-photo-base.com
+taiwan.arcsight.com
+taiwan10000.com
+taiwanese-association-of-norway.org
+taiwanmooc.org
+taiwannextgenfoundation.org
+taiwantravelmap.com
+telco.vmware.com
+tgtf4s6bhrc.tian.yam.com
+theonevfx.com
+tos.ea.com
+tpweddinggift.com
+tsi.center
+tso-liang-wu.gitbook.io
+tukuyomimaid.blog126.fc2.com
+tw.andys.pro
+tw.aoc.com
+tw.cansonic.com
+tw.flux3dp.com
+tw.ipswitch.com
+tw.micron.com
+tw.websense.com
+twdn.net
+twosevenths.com
+twtybbs.com
+vaderisme.tian.yam.com
+vaemon.com
+veed.io
+vhbwxrlzghudo.tian.yam.com
+vielgluck.tian.yam.com
+visualstudio.com
+vmware.com
+w0.wetouch.me
+weatherrisk.com
+web.taiwanhot.net
+websecurity.symantec.com
+weebly.com
+wemoscooter.com
+windwithme.blogspot.com
+wooddiylonlon.blogspot.com
+wpbox.tips
+wznote.blogspot.com
+ya-boginhall-40.8rmi2.upcloudobjects.com
+ya-boginhall-5.s3.nl-ams.scw.cloud
+ya-boginhall-6.s3.pl-waw.scw.cloud
+ya-catering-14.sfo3.digitaloceanspaces.com
+yahootwtrip.tumblr.com
+yingzaiandtei.blogspot.com
+yiying.handysociality.com
+yllan.org
+youthfulcf.tian.yam.com
+yuchengcomuseum.benchurl.com
+yuhinas.blogspot.com
+yurenju.medium.com
+zh-tw.broadcom.com
+zh.unifab.ai

data/lib/twpipeline/bench.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require "json"
+require "pathname"
+require "time"
+
+module TWPipeline
+  module Bench
+    Sample = Data.define(:label, :seconds, :cpu_seconds, :peak_rss_bytes, :facts) do
+      def cores_used = seconds.zero? ? 0.0 : cpu_seconds / seconds
+
+      def to_h
+        {
+          label: label,
+          seconds: seconds.round(3),
+          cpu_seconds: cpu_seconds.round(3),
+          cores_used: cores_used.round(2),
+          peak_rss_bytes: peak_rss_bytes
+        }.merge(facts)
+      end
+    end
+
+    module_function
+
+    def path = Pathname(ENV.fetch("TWP_BENCHMARKS", TWPipeline.work.join("benchmarks.jsonl").to_s))
+
+    def measure(label, **facts)
+      started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      cpu_before = cpu_seconds
+      result = yield
+      sample = Sample.new(
+        label: label,
+        seconds: Process.clock_gettime(Process::CLOCK_MONOTONIC) - started,
+        cpu_seconds: cpu_seconds - cpu_before,
+        peak_rss_bytes: peak_rss,
+        facts: facts.merge(result.is_a?(Hash) ? result : {})
+      )
+      record(sample)
+      sample
+    end
+
+    def cpu_seconds
+      times = Process.times
+      times.utime + times.stime + times.cutime + times.cstime
+    end
+
+    def peak_rss = Rusage.peak
+
+    def record(sample)
+      path.dirname.mkpath
+      path.open("a") { |io| io.puts(JSON.generate({at: Time.now.utc.iso8601}.merge(sample.to_h))) }
+      sample
+    end
+  end
+end