uts58 0.2.4 → 0.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +15 -7
- data/lib/uts58/extractor.rb +12 -6
- data/lib/uts58/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 337c308523db3149f29705c2430457e99678eab43247a1bd09ac58b62dbaae35
|
|
4
|
+
data.tar.gz: a9c22dbc02f06985e701eaf294b51bb70a33cb818e82405d91a08bccb89f303f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a08a95051e7aca78c7ead8f6a47e0d2dd77c461d656b3392d073ae0bc0597b49bd557b2656324faa49bbd5af97e383af9161180cddca7fbb0b3886d35fa29133
|
|
7
|
+
data.tar.gz: 3915f09fb72db8c630132b4bb3986dbc167e2d6796ff2e4e9a6d5ae7e7b44ea27e6f5cd52c9f676b7dd150ae88cc0fa5441da87ce53136d7af7f00bc807e5973
|
data/README.md
CHANGED
|
@@ -5,8 +5,9 @@ the Unicode spec for finding links in running text. Given a chunk of text,
|
|
|
5
5
|
it returns the URLs and email addresses in it along with their character
|
|
6
6
|
offsets.
|
|
7
7
|
|
|
8
|
-
Both halves of UTS58 are covered:
|
|
9
|
-
The two are detected independently and can be combined.
|
|
8
|
+
Both halves of UTS58 are covered: web links and email addresses.
|
|
9
|
+
The two are detected independently and can be combined. You can also
|
|
10
|
+
combine them with detection of your own, e.g. usernames on your service.
|
|
10
11
|
|
|
11
12
|
Tested extensively on relevant OSes: [](https://github.com/arnt/uts58/actions/workflows/ruby.yml)
|
|
12
13
|
|
|
@@ -42,6 +43,10 @@ Uts58.extract_urls_with_indices("blogspot.com is still a thing")
|
|
|
42
43
|
# => [{ url: "https://blogspot.com", indices: [0, 12] }]
|
|
43
44
|
```
|
|
44
45
|
|
|
46
|
+
As in twitter-text, you can pass `extract_url_without_protocol: false` to
|
|
47
|
+
find only links that start with `http://` or `https://`. The default is
|
|
48
|
+
`true`, matching UTS58.
|
|
49
|
+
|
|
45
50
|
IDNs are decoded to use UTF8 in the output, for better readability:
|
|
46
51
|
|
|
47
52
|
```ruby
|
|
@@ -59,7 +64,7 @@ are handled per the spec.
|
|
|
59
64
|
## Email addresses
|
|
60
65
|
|
|
61
66
|
Email detection mirrors the URL methods. Each result carries the address
|
|
62
|
-
twice
|
|
67
|
+
twice (as a bare `:email` and as a `mailto:` `:url`) so it drops straight
|
|
63
68
|
into anything that already renders a `:url` entity:
|
|
64
69
|
|
|
65
70
|
```ruby
|
|
@@ -79,7 +84,7 @@ are recognised; the domain is IDN-decoded just like a URL host. A leading
|
|
|
79
84
|
## Combined extraction
|
|
80
85
|
|
|
81
86
|
`extract_entities_with_indices` runs both detectors, sorts by offset, and
|
|
82
|
-
strips overlaps
|
|
87
|
+
strips overlaps, very similar to `Twitter::TwitterText::Extractor#extract_entities_with_indices`.
|
|
83
88
|
The result is a mixed list of `:url` and email (`:email` + `:url`) hashes:
|
|
84
89
|
|
|
85
90
|
```ruby
|
|
@@ -100,7 +105,7 @@ you have two options, with different results for `contact info@example.com for p
|
|
|
100
105
|
1. **Extract both, then drop emails.** Take `extract_entities_with_indices`
|
|
101
106
|
(already overlap-stripped) and reject the hashes that have an `:email`
|
|
102
107
|
key. The address wins the overlap, so dropping it leaves that span
|
|
103
|
-
*unlinked* — `info@example.com` stays plain text. Choose this if an
|
|
108
|
+
*unlinked* — `contact info@example.com` stays plain text. Choose this if an
|
|
104
109
|
address shouldn't silently become a website link.
|
|
105
110
|
2. **Extract only URLs.** Call `extract_urls_with_indices` and skip email
|
|
106
111
|
detection entirely. The URL scan finds domain after the `@`, so the
|
|
@@ -110,7 +115,10 @@ you have two options, with different results for `contact info@example.com for p
|
|
|
110
115
|
## What's not here
|
|
111
116
|
|
|
112
117
|
- **Link validation.** Recognised URLs are not fetched, normalised beyond
|
|
113
|
-
IDN decoding, or their hostnames checked in the DNS.
|
|
118
|
+
IDN decoding, or their hostnames checked in the DNS. There is
|
|
119
|
+
no attempt at checking for possible attacks (`Мышκин.рф` and
|
|
120
|
+
`Мышкин.рф` are both detected, note the Greek kappa in the middle of
|
|
121
|
+
the prince's name). If you need this,
|
|
114
122
|
send me mail.
|
|
115
123
|
|
|
116
124
|
## Roadmap
|
|
@@ -118,7 +126,7 @@ you have two options, with different results for `contact info@example.com for p
|
|
|
118
126
|
My immediate need is UTS58 conformant link detection suitable for
|
|
119
127
|
public web pages. If you need something more, I rather think that an
|
|
120
128
|
item can be added to this roadmap, so long as the description in the
|
|
121
|
-
rdoc remains short and simple. Send mail
|
|
129
|
+
rdoc remains short and simple. Send me mail.
|
|
122
130
|
|
|
123
131
|
## License
|
|
124
132
|
|
data/lib/uts58/extractor.rb
CHANGED
|
@@ -60,9 +60,14 @@ module Uts58
|
|
|
60
60
|
# very common example is input that like "foo example.com bar",
|
|
61
61
|
# where the URL will be https://example.com, including "https://".
|
|
62
62
|
#
|
|
63
|
-
# Returns an empty array if +text+ contains no links.
|
|
64
|
-
#
|
|
63
|
+
# Returns an empty array if +text+ contains no links.
|
|
64
|
+
#
|
|
65
|
+
# +options+ follows twitter-text: pass
|
|
66
|
+
# <tt>extract_url_without_protocol: false</tt> to find only links
|
|
67
|
+
# that start with +http://+ or +https://+. UTS58 doesn't require a
|
|
68
|
+
# scheme, so the default is +true+.
|
|
65
69
|
def extract_urls_with_indices(text, options = {})
|
|
70
|
+
without_protocol = options.fetch(:extract_url_without_protocol, true)
|
|
66
71
|
result = []
|
|
67
72
|
# The '@' in the negative lookbehind is non-obvious part: UTS58
|
|
68
73
|
# has no userinfo, so a host that immediately follows an '@'
|
|
@@ -77,18 +82,19 @@ module Uts58
|
|
|
77
82
|
# nothing precedes it, and the actual link begins three
|
|
78
83
|
# codepoints later.
|
|
79
84
|
s = match.post_match
|
|
80
|
-
scheme_match =
|
|
85
|
+
scheme_match = /\A([\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}\p{Thai}\p{Lao}\p{Khmer}\p{Myanmar}]*?)(https?:\/\/)/i.match(s)
|
|
81
86
|
if scheme_match
|
|
82
87
|
scheme_offset = scheme_match[1].length
|
|
83
88
|
proto = scheme_match[2]
|
|
84
89
|
s = scheme_match.post_match
|
|
85
90
|
else
|
|
91
|
+
next unless without_protocol
|
|
86
92
|
scheme_offset = 0
|
|
87
93
|
proto = "https://"
|
|
88
94
|
end
|
|
89
95
|
# look for the prefix that might be a hostname or an IDN.
|
|
90
96
|
# this is a somewhat sloppy match, with a few false positives.
|
|
91
|
-
prefix =
|
|
97
|
+
prefix = /\A([-\p{L}\p{N}\p{M}\u00DF\u03C2\u06FD\u06FE\u0F0B\u3007]+[\.。]){1,4}[-\p{L}\p{N}\p{M}]+(?![-\p{L}\p{N}\p{M}])/.match(s)
|
|
92
98
|
if prefix && prefix[0].length < 254
|
|
93
99
|
host_raw = prefix.match(0).gsub(/。/, ".")
|
|
94
100
|
next unless valid_labels?(host_raw)
|
|
@@ -108,7 +114,7 @@ module Uts58
|
|
|
108
114
|
rest = rest[1..]
|
|
109
115
|
end
|
|
110
116
|
# a port number must be 1..65535
|
|
111
|
-
port =
|
|
117
|
+
port = /\A:(\d+)/.match(rest)
|
|
112
118
|
if port
|
|
113
119
|
n = port[1].to_i
|
|
114
120
|
next if n < 1 || n > 65535
|
|
@@ -180,7 +186,7 @@ module Uts58
|
|
|
180
186
|
local = lp_match[0]
|
|
181
187
|
next if local.start_with?('.') || local.end_with?('.') || local.include?('..')
|
|
182
188
|
s = match.post_match
|
|
183
|
-
prefix =
|
|
189
|
+
prefix = /\A([-\p{L}\p{N}\p{M}ßς۽۾་〇]+[\.。]){1,4}[-\p{L}\p{N}\p{M}]+(?![-\p{L}\p{N}\p{M}])/.match(s)
|
|
184
190
|
next unless prefix && prefix[0].length < 254
|
|
185
191
|
host_raw = prefix.match(0).gsub(/。/, ".")
|
|
186
192
|
next unless valid_labels?(host_raw)
|
data/lib/uts58/version.rb
CHANGED