twitter-text-kow 1.3.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 43c07d2316e4bce54cd0915b52cb5a86bb81366c2ec24aca5c3e618362c6dced
+  data.tar.gz: 5a163feb86e22059dd6f315422ce680c4c26c3afc22d50dace85e2d6683c837f
+SHA512:
+  metadata.gz: 707e1c2110e1fee3d76ab3a9f39bb799c755e6f33ce21093a4fef4ec7430ec8d25fb3de82e84fdbed4bec741d19a8cc6521161530cab2f14bdc2f719bf1e9b4c
+  data.tar.gz: 95fd66b4ee400291b77fce30b678a82a2745049ee30b5b8df88bab5f11e2fdfdfe104160e23c3f781ac47a277c0853ffdc5519047f588fe195a084e4c381822f

data/.gitignore

@@ -0,0 +1,40 @@
+*.gem
+*.rbc
+*.sw[a-p]
+*.tmproj
+*.tmproject
+*.un~
+*~
+.DS_Store
+.Spotlight-V100
+.Trashes
+._*
+.bundle
+.config
+.directory
+.elc
+.emacs.desktop
+.emacs.desktop.lock
+.redcar
+.yardoc
+Desktop.ini
+Gemfile.lock
+Icon?
+InstalledFiles
+Session.vim
+Thumbs.db
+\#*\#
+_yardoc
+auto-save-list
+coverage
+doc
+lib/bundler/man
+pkg
+pkg/*
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+tmtags
+tramp

data/.gitmodules

@@ -0,0 +1,3 @@
+[submodule "test/twitter-text-conformance"]
+	path = test/twitter-text-conformance
+	url = git://github.com/twitter/twitter-text-conformance.git

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format=documentation

data/CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [3.1.0]
+### Changed
+- Bump nokogiri version (#302)
+- Fix auto-link emoji parsing (#304)
+- Updates known gTLDs to recognize recent additions by IANA (#308)
+- Fix warning about has_rdoc usage (#309)
+
+## [3.0.0]
+### Added
+- New v3.json config file with emojiParsingEnabled config option. When
+  true, twitter-text will parse and discount emoji supported by the
+  twemoji library (see https://github.com/twitter/twemoji). The length
+  of these emoji will be the default weight (200 or two characters) even
+  if they contain multiple code points combined by zero-width
+  joiners. This means that emoji with skin tone and gender modifiers no
+  longer count as more characters than those without such modifiers.
+### Changed
+- Updates known gTLDs to recognize recent additions by IANA (#261)
+
+## [2.1] - 2017-12-20
+### Added
+- This CHANGELOG.md file
+
+### Changed
+- Top-level namespace changed from `Twitter` to `Twitter::TwitterText`. This
+  resolves a namespace collision with the popular
+  [twitter gem](https://github.com/sferik/twitter). This is considered
+  a breaking change, so the version has been bumped to 2.1. This fixes
+  issue [#221](https://github.com/twitter/twitter-text/issues/221),
+  "NoMethodError Exception: undefined method `[]' for nil:NilClasswhen
+  using gem in rails app"
+
+## [2.0.2] - 2017-12-18
+### Changed
+- Resolved issue
+  [#211](https://github.com/twitter/twitter-text/issues/211), "gem
+  breaks, asset file is a dangling symlink"
+- config files, tld_lib.yml files now copied into the right place
+- Rakefile now included `prebuild`, `clean` tasks

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify the gem's dependencies in twitter-text.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,188 @@
+Copyright 2011 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this work except in compliance with the License.
+You may obtain a copy of the License below, or at:
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+                             Apache License
+                       Version 2.0, January 2004
+                    http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent claims licensable
+  by such Contributor that are necessarily infringed by their
+  Contribution(s) alone or by combination of their Contribution(s)
+  with the Work to which such Contribution(s) was submitted. If You
+  institute patent litigation against any entity (including a
+  cross-claim or counterclaim in a lawsuit) alleging that the Work
+  or a Contribution incorporated within the Work constitutes direct
+  or contributory patent infringement, then any patent licenses
+  granted to You under this License for that Work shall terminate
+  as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+  Work or Derivative Works thereof in any medium, with or without
+  modifications, and in Source or Object form, provided that You
+  meet the following conditions:
+
+  (a) You must give any other recipients of the Work or
+      Derivative Works a copy of this License; and
+
+  (b) You must cause any modified files to carry prominent notices
+      stating that You changed the files; and
+
+  (c) You must retain, in the Source form of any Derivative Works
+      that You distribute, all copyright, patent, trademark, and
+      attribution notices from the Source form of the Work,
+      excluding those notices that do not pertain to any part of
+      the Derivative Works; and
+
+  (d) If the Work includes a "NOTICE" text file as part of its
+      distribution, then any Derivative Works that You distribute must
+      include a readable copy of the attribution notices contained
+      within such NOTICE file, excluding those notices that do not
+      pertain to any part of the Derivative Works, in at least one
+      of the following places: within a NOTICE text file distributed
+      as part of the Derivative Works; within the Source form or
+      documentation, if provided along with the Derivative Works; or,
+      within a display generated by the Derivative Works, if and
+      wherever such third-party notices normally appear. The contents
+      of the NOTICE file are for informational purposes only and
+      do not modify the License. You may add Your own attribution
+      notices within Derivative Works that You distribute, alongside
+      or as an addendum to the NOTICE text from the Work, provided
+      that such additional attribution notices cannot be construed
+      as modifying the License.
+
+  You may add Your own copyright statement to Your modifications and
+  may provide additional or different license terms and conditions
+  for use, reproduction, or distribution of Your modifications, or
+  for any such Derivative Works as a whole, provided Your use,
+  reproduction, and distribution of the Work otherwise complies with
+  the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+  any Contribution intentionally submitted for inclusion in the Work
+  by You to the Licensor shall be under the terms and conditions of
+  this License, without any additional terms or conditions.
+  Notwithstanding the above, nothing herein shall supersede or modify
+  the terms of any separate license agreement you may have executed
+  with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+  names, trademarks, service marks, or product names of the Licensor,
+  except as required for reasonable and customary use in describing the
+  origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+  agreed to in writing, Licensor provides the Work (and each
+  Contributor provides its Contributions) on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+  implied, including, without limitation, any warranties or conditions
+  of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+  PARTICULAR PURPOSE. You are solely responsible for determining the
+  appropriateness of using or redistributing the Work and assume any
+  risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+  whether in tort (including negligence), contract, or otherwise,
+  unless required by applicable law (such as deliberate and grossly
+  negligent acts) or agreed to in writing, shall any Contributor be
+  liable to You for damages, including any direct, indirect, special,
+  incidental, or consequential damages of any character arising as a
+  result of this License or out of the use or inability to use the
+  Work (including but not limited to damages for loss of goodwill,
+  work stoppage, computer failure or malfunction, or any and all
+  other commercial damages or losses), even if such Contributor
+  has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+  the Work or Derivative Works thereof, You may choose to offer,
+  and charge a fee for, acceptance of support, warranty, indemnity,
+  or other liability obligations and/or rights consistent with this
+  License. However, in accepting such obligations, You may act only
+  on Your own behalf and on Your sole responsibility, not on behalf
+  of any other Contributor, and only if You agree to indemnify,
+  defend, and hold each Contributor harmless for any liability
+  incurred by, or claims asserted against, such Contributor by reason
+  of your accepting any such warranty or additional liability.

data/README.md

@@ -0,0 +1,193 @@
+# twitter-text
+
+![](https://img.shields.io/gem/v/twitter-text.svg)
+
+This is the Ruby implementation of the twitter-text parsing
+library. The library has methods to parse Tweets and calculate length,
+validity, parse @mentions, #hashtags, URLs, and more.
+
+## Setup
+
+Installation uses bundler.
+
+```
+% gem install bundler
+% bundle install
+```
+
+## Conformance tests
+
+To run the Conformance test suite from the command line via rake:
+
+```
+% rake test:conformance:run
+```
+
+You can also run the rspec tests in the `spec` directory:
+
+```
+% rspec spec
+```
+
+# Length validation
+
+twitter-text 2.0 introduces configuration files that define how Tweets
+are parsed for length. This allows for backwards compatibility and
+flexibility going forward. Old-style traditional 140-character parsing
+is defined by the v1.json configuration file, whereas v2.json is
+updated for "weighted" Tweets where ranges of Unicode code points can
+have independent weights aside from the default weight. The sum of all
+code points, each weighted appropriately, should not exceed the max
+weighted length.
+
+Some old methods from twitter-text 1.0 have been marked deprecated,
+such as the `tweet_length()` method. The new API is based on the
+following method, `parse_tweet()`
+
+```ruby
+def parse_tweet(text, options = {}) { ... }
+```
+
+This method takes a string as input and returns a results object that
+contains information about the
+string. `Twitter::TwitterText::Validation::ParseResults` object includes:
+
+* `:weighted_length`: the overall length of the tweet with code points
+weighted per the ranges defined in the configuration file.
+
+* `:permillage`: indicates the proportion (per thousand) of the weighted
+length in comparison to the max weighted length. A value > 1000
+indicates input text that is longer than the allowable maximum.
+
+* `:valid`: indicates if input text length corresponds to a valid
+result.
+
+* `:display_range_start, :display_range_end`: An array of two unicode code point
+indices identifying the inclusive start and exclusive end of the
+displayable content of the Tweet. For more information, see
+the description of `display_text_range` here:
+[Tweet updates](https://developer.twitter.com/en/docs/tweets/tweet-updates)
+
+* `:valid_range_start, :valid_range_end`: An array of two unicode code point
+indices identifying the inclusive start and exclusive end of the valid
+content of the Tweet. For more information on the extended Tweet
+payload see [Tweet updates](https://developer.twitter.com/en/docs/tweets/tweet-updates)
+
+## Extraction Examples
+
+# Extraction
+```ruby
+class MyClass
+  include Twitter::TwitterText::Extractor
+  usernames = extract_mentioned_screen_names("Mentioning @twitter and @jack")
+  # usernames = ["twitter", "jack"]
+end
+```
+
+### Extraction with a block argument
+
+```ruby
+class MyClass
+  include Twitter::TwitterText::Extractor
+  extract_reply_screen_name("@twitter are you hiring?").do |username|
+    # username = "twitter"
+  end
+end
+```
+
+## Auto-linking Examples
+
+### Auto-link
+
+```ruby
+class MyClass
+  include Twitter::TwitterText::Autolink
+
+  html = auto_link("link @user, please #request")
+end
+```
+
+### For Ruby on Rails you want to add this to app/helpers/application_helper.rb
+```ruby
+module ApplicationHelper
+  include Twitter::TwitterText::Autolink
+end
+```
+
+### Now the auto_link function is available in every view. So in index.html.erb:
+```ruby
+<%= auto_link("link @user, please #request") %>
+```
+
+### Usernames
+
+Username extraction and linking matches all valid Twitter usernames but does
+not verify that the username is a valid Twitter account.
+
+### Lists
+
+Auto-link and extract list names when they are written in @user/list-name
+format.
+
+### Hashtags
+
+Auto-link and extract hashtags, where a hashtag can contain most letters or
+numbers but cannot be solely numbers and cannot contain punctuation.
+
+### URLs
+
+Asian languages like Chinese, Japanese or Korean may not use a delimiter such
+as a space to separate normal text from URLs making it difficult to identify
+where the URL ends and the text starts.
+
+For this reason twitter-text currently does not support extracting or
+auto-linking of URLs immediately followed by non-Latin characters.
+
+Example: "http://twitter.com/は素晴らしい" . The normal text is "は素晴らしい" and is not
+part of the URL even though it isn't space separated.
+
+### International
+
+Special care has been taken to be sure that auto-linking and extraction work
+in Tweets of all languages. This means that languages without spaces between
+words should work equally well.
+
+### Hit Highlighting
+
+Use to provide emphasis around the "hits" returned from the Search API, built
+to work against text that has been auto-linked already.
+
+## Issues
+
+Have a bug? Please create an issue here on GitHub!
+
+<https://github.com/twitter/twitter-text/issues>
+
+## Authors
+
+### V2.0
+
+* David LaMacchia (<https://github.com/dlamacchia>)
+* Yoshimasa Niwa (<https://github.com/niw>)
+* Sudheer Guntupalli (<https://github.com/sudhee>)
+* Kaushik Lakshmikanth (<https://github.com/kaushlakers>)
+* Jose Antonio Marquez Russo (<https://github.com/joseeight>)
+* Lee Adams (<https://github.com/leeaustinadams>)
+
+### Previous authors
+
+* Matt Sanford (<http://github.com/mzsanford>)
+* Raffi Krikorian (<http://github.com/r>)
+* Ben Cherry (<http://github.com/bcherry>)
+* Patrick Ewing (<http://github.com/hoverbird>)
+* Jeff Smick (<http://github.com/sprsquish>)
+* Kenneth Kufluk (<https://github.com/kennethkufluk>)
+* Keita Fujii (<https://github.com/keitaf>)
+* Jean-Philippe Bougie (<http://github.com/jpbougie>)
+* Erik Michaels-Ober (<https://github.com/sferik>)
+
+## License
+
+Copyright 2012-2020 Twitter, Inc and other contributors
+
+Licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)

data/Rakefile

@@ -0,0 +1,52 @@
+require 'bundler'
+include Rake::DSL
+Bundler::GemHelper.install_tasks
+
+task :build => ['prebuild']
+task :spec => ['prebuild']
+task :default => ['prebuild', 'spec', 'test:conformance']
+task :test => :spec
+
+directory "config"
+directory "lib/assets"
+
+desc "Prebuild task setup"
+task :prebuild => ["config", "lib/assets"] do
+  FileUtils.cp_r '../config/.', 'config', :verbose => true
+  FileUtils.cp_r '../conformance/tld_lib.yml', 'lib/assets', :verbose => true
+end
+
+require 'rubygems'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
+
+namespace :test do
+  namespace :conformance do
+    desc "Run conformance test suite"
+    task :run => ['prebuild'] do
+      ruby "test/conformance_test.rb"
+    end
+  end
+
+  desc "Run conformance test suite"
+  task :conformance => ['conformance:run'] do
+  end
+end
+
+require 'rdoc/task'
+namespace :doc do
+  RDoc::Task.new do |rd|
+    rd.main = "README.rdoc"
+    rd.rdoc_dir = 'doc'
+    rd.rdoc_files.include("README.rdoc", "lib/**/*.rb")
+  end
+end
+
+desc "Run cruise control build"
+task :cruise => [:spec, 'test:conformance'] do
+end
+
+desc "Clean build"
+task :clean do
+  rm_rf ["config", "pkg", "lib/assets", "Gemfile.lock"]
+end

data/config/README.md

@@ -0,0 +1,142 @@
+# twitter-text Configuration
+
+twitter-text 2.0 introduces a new configuration format as well as APIs
+for interpreting this configuration. The configuration is a JSON
+string (or file) and the parsing APIs have been provided in each of
+twitter-text’s four reference languages.
+
+## Format
+
+The configuration format is a JSON string. The JSON can have the following properties:
+
+* `version` (required, integer, min value 0)
+* `maxWeightedTweetLength` (required, integer, min value 0)
+* `scale` (required, integer, min value 1)
+* `defaultWeight` (required, integer, min value 0)
+* `emojiParsingEnabled` (optional, boolean)
+* `transformedURLLength` (integer, min value 0)
+* `ranges` (array of range items)
+
+A `range item` has the following properties:
+
+* `start` (required, integer, min value 0)
+* `end` (required, integer, min value 0)
+* `weight` (required, integer, min value 0)
+
+## Parameters
+
+### version
+
+The version for the configuration string. This is an integer that will
+monotonically increase in future releases. The legacy version of the
+string is version 1; weighted code point ranges and 280-character
+“long” tweets are supported in version 2.
+
+### maxWeightedTweetLength
+
+The maximum length of the tweet, weighted. Legacy v1 tweets had a
+maximum weighted length of 140 and all characters were weighted the
+same. In the new configuration format, this is represented as a
+maxWeightedTweetLength of 140 and a defaultWeight of 1 for all code
+points.
+
+### scale
+
+The Tweet length is the (`weighted length` / `scale`).
+
+### defaultWeight
+
+The default weight applied to all code points. This is overridden in
+one or more range items.
+
+### emojiParsingEnabled
+
+When set to true, the weighted Tweet length considers all emoji as a
+single code point (with a default weight of 200), including longer
+grapheme clusters combined by zero-width joiners. When set to false,
+Tweet length is calculated by weighing individual Unicode code points.
+
+### transformedURLLength
+
+The length counted for URLs against the total weight of the Tweet. In
+previous versions of twitter-text, which was the “shortened URL
+length.” Differentiating between the http and https shortened length
+for URLs has been deprecated (https is used for all t.co URLs). The
+default value is 23.
+
+### ranges
+
+An array of range items that describe ranges of Unicode code points
+and the weight to apply to each code point. Each range is defined by
+its start, end, and weight. Surrogate pairs have a length that is
+equivalent to the length of the first code unit in the surrogate
+pair. Note that certain graphemes are the result of joining code
+points together, such as by a zero-width joiner; unlike a surrogate
+pair, the length of such a grapheme will be the sum of the weighted
+length of all included code points.
+
+## API
+
+Each of the four reference language implementations provides a way to
+read the JSON configuration.
+
+## Java
+
+```java
+public static TwitterTextConfiguration configurationFromJson(@Nonnull String json, boolean isResource)
+```
+
+`json`: the configuration string or file name in the config directory (see `isResource`)
+`isResource`: if true, json refers to a file name for the configuration.
+
+## JavaScript
+
+Configurations are accessed via `twttr.text.configs` (example:
+`twttr.text.configs.version2`). This config is passed as an argument
+to `parseTweet:`
+
+```js
+twttr.txt.parseTweet(inputText, configVersion2)
+```
+
+## Objective-C
+
+The Objective-C implementation provides two methods for reading the
+input, either from a string or a file resource.
+
+```objective-c
++ (instancetype)configurationFromJSONResource:(NSString *)jsonResource;
++ (instancetype)configurationFromJSONString:(NSString *)jsonString;
+```
+
+The default configuration can also be set:
+
+```objective-c
++ (void)setDefaultParserConfiguration:(TwitterTextConfiguration *)configuration
+```
+
+The resource string refers to the two included configuration files
+(which are referenced in the Xcode project).
+
+## Ruby
+
+Ruby provides the `Twitter::Configuration` class and means to read
+from a file or string.
+
+```ruby
+def self.parse_string(string, options = {})
+def self.parse_file(filename)
+```
+
+You can use `configuration_from_file()` or initialize a configuration
+using `Twitter::Configuration.new(config)`, where `config` is the
+output of one of the two above methods.
+
+
+
+
+
+
+
+
+