ruby-spacy 0.1.5.4 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2d3ff404ae86e8089b01832121e6c87fef12d864bf5e873a9b62ce3df7b26448
-  data.tar.gz: 14060665f835b8932fe78b60bb3dcea7adee22388488de11031e6510e24724f3
+  metadata.gz: 9e9edb55398e8926b4fd9c06d65b49538129e34a960098b1ad20535d64a2b787
+  data.tar.gz: 639fa3186d563480d0eb268fa948d8b97428fcdf37887dfff64954fcfc86c1f0
 SHA512:
-  metadata.gz: d4c62ac434fbc2c2056b9225e4c74a9b6731c3a8a63697cf02d44094f4c235c2493aa4c898b55bf3b5e1d8303ed6d7036c9d61db37d941c69d09dc7f2a5b7d26
-  data.tar.gz: 753d212d51660e085740c8cf8836545983d8443e5acba6f213e57269558ab22a0c0917b67083157cf21cf4ce2261af4ba4113d0f16540a97b549729cc74c5601
+  metadata.gz: 74367e0cd67a3537b20f73427baf626ada1f123d9c34da1a55795a905c3cfd8239c5cc1a04e6cf92c8312c6338a6300ce95b837d24c642c3dbb77733a25060ed
+  data.tar.gz: 4723555e09a6416ec8cb5727b3344756be36a26f65429759508aaee697b245960065cb74699b7f619d552a67b574dbc70da1b53a0ddc434ade45901b0ca72dd7

data/.gitignore

@@ -57,3 +57,8 @@ Gemfile.lock
 
 .DS_Store
 tags
+
+.python-version
+.rubocop.yml
+.solargraph.yml
+.yardopts

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Change Log
 
+## 0.2.0 - 2022-10-02
+- spaCy 3.7.0 supported
+
+## 0.2.0 - 2022-10-02
+### Added
+- `Doc::openai_query`
+- `Doc::openai_completion`
+- `Doc::openai_embeddings`
+
 ## 0.1.4.1 - 2021-07-06
 - Test code refined
 - `Spacy::Language::most_similar` returns an array of hash-based objects that accepts method calls

data/Gemfile

@@ -6,7 +6,8 @@ source "https://rubygems.org"
 gemspec
 
 gem "numpy"
-gem "pycall"
+gem "pycall", "~> 1.5.1"
+gem "ruby-openai"
 gem "terminal-table"
 
 group :development do

data/README.md

@@ -11,6 +11,11 @@
 | ✅ | Named entity recognition                           |
 | ✅ | Syntactic dependency visualization                 |
 | ✅ | Access to pre-trained word vectors                 |
+| ✅ | OpenAI Chat/Completion/Embeddings API integration  |
+
+Current Version: `0.2.2`
+
+- Addressed installation issues in some environments
 
 ## Installation of prerequisites
 
@@ -32,6 +37,7 @@ Then, install [spaCy](https://spacy.io/). If you use `pip`, the following comman
 $ pip install spacy
 ```
 
+
 Install trained language models. For a starter, `en_core_web_sm` will be the most useful to conduct basic text processing in English. However, if you want to use advanced features of spaCy, such as named entity recognition or document similarity calculation, you should also install a larger model like `en_core_web_lg`.
 
 
@@ -469,9 +475,6 @@ Output:
 | 10   | marseille   | 0.6370999813079834 |
 
 
-
-
-
 ### Word vector calculation (Japanese)
 
 **東京 - 日本 + フランス = パリ ?**
@@ -518,6 +521,169 @@ Output:
 | 9    | アルザス       | 0.5644999742507935 |
 | 10   | 南仏           | 0.5547999739646912 |
 
+
+## OpenAI API Integration
+
+Easily leverage GPT models within ruby-spacy by using an OpenAI API key. When constructing prompts for the `Doc::openai_query` method, you can incorporate various token properties from the document. These properties are retrieved through function calls and seamlessly integrated into your prompt (`gpt-3.5-turbo-0613` or greater is needed). The available properties include:
+
+- `surface`
+- `lemma`
+- `tag`
+- `pos` (part of speech)
+- `dep` (dependency)
+- `ent_type` (entity type)
+- `morphology`
+
+### GPT Prompting 1
+
+Ruby code:
+
+```ruby
+
+require "ruby-spacy"
+
+api_key = ENV["OPENAI_API_KEY"]
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("The Beatles released 12 studio albums")
+
+# default parameter values
+# max_tokens: 1000
+# temperature: 0.7
+# model: "gpt-3.5-turbo-0613"
+res1 = doc.openai_query(
+  access_token: api_key,
+  prompt: "Translate the text to Japanese."
+)
+puts res1
+```
+
+Output:
+
+> ビートルズは12枚のスタジオアルバムをリリースしました。
+
+### GPT Prompting 2
+
+Ruby code: 
+
+```ruby
+require "ruby-spacy"
+
+api_key = ENV["OPENAI_API_KEY"]
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("The Beatles were an English rock band formed in Liverpool in 1960.")
+
+res = doc.openai_query(
+  access_token: api_key,
+  prompt: "Extract the topic of the document and list 10 entities (names, concepts, locations, etc.) that are relevant to the topic."
+)
+```
+
+Output:
+
+> Topic: The Beatles
+> 
+> Entities:
+> 1. The Beatles (band)
+> 2. English (nationality)
+> 3. Rock band
+> 4. Liverpool (city)
+> 5. 1960 (year)
+> 6. John Lennon (member)
+> 7. Paul McCartney (member)
+> 8. George Harrison (member)
+> 9. Ringo Starr (member)
+> 10. Music
+
+### GPT Prompting 3
+
+Ruby code: 
+
+```ruby
+require "ruby-spacy"
+
+api_key = ENV["OPENAI_API_KEY"]
+nlp = Spacy::Language.new("en_core_web_sm")
+
+res = doc.openai_query(
+  access_token: api_key,
+  model: "gpt-4",
+  prompt: "Generate a tree diagram from the text in the following style: [S [NP [Det the] [N cat]] [VP [V sat] [PP [P on] [NP the mat]]]"
+)
+puts res
+```
+
+Output:
+
+```
+[S
+  [NP
+    [Det The]
+    [N Beatles]
+  ]
+  [VP
+    [V released]
+    [NP
+      [Num 12]
+      [N
+        [N studio]
+        [N albums]
+      ]
+    ]
+  ]
+]
+```
+
+### GPT Text Completion
+
+Ruby code:
+
+```ruby
+require "ruby-spacy"
+
+api_key = ENV["OPENAI_API_KEY"]
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("Vladimir Nabokov was a")
+
+# default parameter values
+# max_tokens: 1000
+# temperature: 0.7
+# model: "gpt-3.5-turbo-0613"
+res = doc.openai_completion(access_token: api_key)
+puts res
+```
+
+Output:
+
+> Russian-American novelist and lepidopterist. He was born in 1899 in St. Petersburg, Russia, and later emigrated to the United States in 1940. Nabokov is best known for his novel "Lolita," which was published in 1955 and caused much controversy due to its controversial subject matter. Throughout his career, Nabokov wrote many other notable works, including "Pale Fire" and "Ada or Ardor: A Family Chronicle." In addition to his writing, Nabokov was also a passionate butterfly collector and taxonomist, publishing several scientific papers on the subject. He passed away in 1977, leaving behind a rich literary legacy.
+
+### Text Embeddings
+
+Ruby code:
+
+```ruby
+require "ruby-spacy"
+
+api_key = ENV["OPENAI_API_KEY"]
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("Vladimir Nabokov was a Russian-American novelist, poet, translator and entomologist.")
+
+# default model: text-embedding-ada-002
+res = doc.openai_embeddings(access_token: api_key)
+
+puts res
+```
+
+Output:
+
+```
+-0.00208362
+-0.01645165
+ 0.0110955965
+ 0.012802119
+ 0.0012175755
+ ...
+```
+
 ## Author
 
 Yoichiro Hasebe [<yohasebe@gmail.com>]

data/examples/get_started/lexeme.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/linguistic_annotations.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/morphology.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/most_similar.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/named_entities.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/pos_tags_and_dependencies.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/similarity.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("en_core_web_lg")

data/examples/get_started/tokenization.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/get_started/visualizing_dependencies.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("en_core_web_sm")

data/examples/get_started/visualizing_dependencies_compact.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("en_core_web_sm")

data/examples/get_started/visualizing_named_entities.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("en_core_web_sm")

data/examples/get_started/vocab.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("en_core_web_sm")

data/examples/get_started/word_vectors.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/ancestors.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/entity_annotations_and_labels.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/information_extraction.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require("ruby-spacy")
 require "terminal-table"
 

data/examples/japanese/lemmatization.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/most_similar.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/named_entity_recognition.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/navigating_parse_tree.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/noun_chunks.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/pos_tagging.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/sentence_segmentation.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("ja_core_news_sm")

data/examples/japanese/similarity.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 
 nlp = Spacy::Language.new("ja_core_news_lg")

data/examples/japanese/tokenization.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/visualizing_dependencies.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/japanese/visualizing_named_entities.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/ancestors.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/entity_annotations_and_labels.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/finding_a_verb_with_a_subject.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/information_extraction.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/iterating_children.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 
@@ -17,6 +20,6 @@ doc.each do |token|
   end
 end
 
-puts results.to_s
+puts results
 
 # ["shift"]

data/examples/linguistic_features/iterating_lefts_and_rights.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/lemmatization.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/named_entity_recognition.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/navigating_parse_tree.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/noun_chunks.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/pos_tagging.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"
 

data/examples/linguistic_features/retokenize_1.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
 require "ruby-spacy"
 require "terminal-table"