ricgraph 3.1__tar.gz → 3.3__tar.gz

{ricgraph-3.1/ricgraph.egg-info → ricgraph-3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ricgraph
-Version: 3.1
+Version: 3.3
 Summary: Ricgraph - Research in context graph
 Author: Rik D.T. Janssen
 License-Expression: MIT

{ricgraph-3.1 → ricgraph-3.3}/README.md

@@ -23,7 +23,9 @@ We use color=#4c1 (green), that color is used by the 'repo status' badge.
 Documentation about the Podman badges can be found here: https://github.com/eggplants/ghcr-badge.
 --->
 
-> Read the [Tutorial](docs/ricgraph_tutorial.md#tutorial-ricgraph---research-in-context-graph).
+> [Use Ricgraph](https://www.ricgraph.eu/pilot-project-open-ricgraph-demo-server.html).
+> [Read news about Ricgraph](https://www.ricgraph.eu/ricgraph-news.html).
+> Read the [Tutorial](https://docs.ricgraph.eu/docs/ricgraph_tutorial.html).
 > Go to the [Documentation website](https://docs.ricgraph.eu).
 > Consult the [Table of contents](docs/ricgraph_toc_documentation.md#table-of-contents-ricgraph-documentation), 
 > or the [Index](docs/ricgraph_index_documentation.md#index-ricgraph-documentation).
@@ -57,57 +59,32 @@ adapted to new application areas.
 Throughout this documentation, we illustrate how Ricgraph works by applying
 it to the application area research information.
 
-## Ricgraph news
-
-Read my recent preprint:
-
-* Rik D.T. Janssen (2025).
-  *Utilizing Ricgraph to gain insights into research collaborations across institutions,
-  at every organizational level*. Submitted to SoftwareX [preprint].
-  [https://doi.org/10.2139/ssrn.5524439](https://doi.org/10.2139/ssrn.5524439).
-
-See Ricgraph in action:
-
-* Use the Open Ricgraph demo server 
-  via [https://explorer.ricgraph.eu](https://explorer.ricgraph.eu).
-  Read more (and join) at
-  [pilot project Open Ricgraph demo 
-  server](https://www.ricgraph.eu/pilot-project-open-ricgraph-demo-server.html).
-
-
-Currently, there are people working on the following extensions for Ricgraph:
-
-* [BackToPure](https://github.com/UtrechtUniversity/backtopure)
-  is a tool designed to enhance an organization's
-  [Research Information System Pure](https://www.elsevier.com/solutions/pure)
-  by enriching its content. BackToPure can identify items
-  (such as publications, data sets or software)
-  that exist in other external sources but are missing from the organization's Pure,
-  and then insert (enrich) those items into Pure.
-  The result is a more complete overview of research at that organization.
-  Status: experimental stage (beta).
-* A chatbot that allows you to "talk" to Ricgraph. You can formulate questions in
-  plain English, such as "Please give me the research results of the Geosciences
-  faculty of Utrecht University?"
-  or "With what organizations does that faculty collaborate?".
-  It uses a local Large Language Model.
-  Status: planning stage (pre-alpha).
-* A project with six students from the University of Applied Sciences, Utrecht, that aims to
-  [create a general harvester for
-  Ricgraph](docs/ricgraph_outreach.md#ricgraph-projects-with-students).
-  It will remove a lot of code duplication in the harvest scripts, and will
-  make it easier to add new harvest sources.
-  Status: planning stage (pre-alpha).
-* A project with a student from the University of Applied Sciences, Utrecht, that aims to use
-  [AI and Large Language Models to find topics and visualize large amounts of research
-  information](docs/ricgraph_outreach.md#ricgraph-projects-with-students).
-  It assigns *topics* to publications,
-  data sets, and software. By selecting a number of topics, research results are
-  grouped, and possibly experts on these topics can be found.
-  Status: planning stage (pre-alpha).
-* A tool that helps Pure administrators to clean up "external organizations" in Pure.
-  Status: planning stage (pre-alpha).
-
+## Ricgraph introduction videos
+
+You may want to watch the following videos about Ricgraph. The left one is a
+short video, only including the Ricgraph use cases. The right one is a more
+extensive one (also including the use cases).
+
+<table>
+  <tr>
+    <td style="width:50%;"><b>Ricgraph use cases video (1m30s)</b></td>
+    <td style="width:50%;"><b>Ricgraph introduction video including use cases (2m28s)</b></td>
+  </tr>
+  <tr>
+    <td><video src="docs/videos/ricgraph_usecases.mp4" controls width="90%">Ricgraph use cases video (1m30s)</video></td>
+    <td><video src="docs/videos/ricgraph_intro_usecases.mp4" controls width="90%">Ricgraph introduction video including use cases (2m28s)</video></td>
+  </tr>
+  <tr>
+   <td>
+     <a href="https://docs.ricgraph.eu/docs/videos/ricgraph_usecases.mp4" target="_blank">
+     Click to play or download the Ricgraph use cases video</a>.
+   </td>
+   <td>
+     <a href="https://docs.ricgraph.eu/docs/videos/ricgraph_intro_usecases.mp4" target="_blank">
+     Click to play or download the Ricgraph introduction video</a>.
+   </td>
+  </tr>
+</table>
 
 ## Motivation
 
@@ -321,6 +298,8 @@ The following figure shows how three persons have contributed to one research ou
 
 * Explore the full documentation of Ricgraph on
   [https://docs.ricgraph.eu](https://docs.ricgraph.eu).
+* Play the [Ricgraph use cases video](docs/videos/ricgraph_usecases.mp4), or the
+  [Ricgraph introduction video](docs/videos/ricgraph_intro_usecases.mp4).
 * For a gentle introduction in Ricgraph, read the reference publication:
   Rik D.T. Janssen (2024). *Ricgraph: A flexible and extensible graph to explore research in
   context from various systems*. SoftwareX, 26(101736).
@@ -374,7 +353,8 @@ The following figure shows how three persons have contributed to one research ou
 
 ### Steps to take if you would like to use Ricgraph
 
-* Use [Ricgraph Explorer](docs/ricgraph_explorer.md#ricgraph-explorer),
+* [Use Ricgraph](https://www.ricgraph.eu/pilot-project-open-ricgraph-demo-server.html).
+* Read more about [Ricgraph Explorer](docs/ricgraph_explorer.md#ricgraph-explorer),
   the exploration tool for Ricgraph. 
 * Use the [Ricgraph REST API](docs/ricgraph_restapi.md#ricgraph-rest-api),
   the REST API for Ricgraph.
@@ -420,4 +400,4 @@ please let me know by creating a GitHub *Issue* at the top left of that page.
 <!-- Do not change the lines below. The version number will be
      modified by script maintenance/update_version.bash -->
 This README.md file is part of the 
-documentation for version 3.1 of Ricgraph - Research in context graph.
+documentation for version 3.3 of Ricgraph - Research in context graph.

{ricgraph-3.1 → ricgraph-3.3}/pyproject.toml

@@ -7,7 +7,7 @@ packages = ["ricgraph"]
 
 [project]
 name = "ricgraph"
-version = "3.1"
+version = "3.3"
 authors = [
   { name="Rik D.T. Janssen" },
 ]

{ricgraph-3.1 → ricgraph-3.3}/ricgraph/ricgraph.py

@@ -6,7 +6,7 @@
 #
 # MIT License
 # 
-# Copyright (c) 2023, 2024, 2025 Rik D.T. Janssen
+# Copyright (c) 2023 - 2026 Rik D.T. Janssen
 # 
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -90,7 +90,7 @@ In the default configuration of Ricgraph, the following properties are included:
 
 - url_other: other URL for a node, pointing to e.g. the originating record in a source system.
 
-- year: year of a research output.
+- year: year of a research result.
 
 - source_event: an event to be added to the _source list.
 
@@ -100,15 +100,19 @@ In the default configuration of Ricgraph, the following properties are included:
 
 from numpy import nan
 from pandas import DataFrame, concat, set_option
-from typing import Union
-from .ricgraph_utils import timestamp
+
+from .ricgraph_constants import PERSON_CATEGORY_PERSON
+from .ricgraph_utils import (timestamp, timestamp_posix,
+                             print_records_per_minute,
+                             print_progress)
 from .ricgraph_graphdb import (create_update_node, create_two_nodes_and_edge,
-                               RICGRAPH_NODEADD_MODE,
-                               RICGRAPH_PROPERTIES_STANDARD, RICGRAPH_PROPERTIES_ADDITIONAL)
+                               get_ricgraph_nodeadd_mode,
+                               get_ricgraph_properties_standard,
+                               get_ricgraph_properties_additional)
 
 
 __author__ = 'Rik D.T. Janssen'
-__copyright__ = 'Copyright (c) 2023, 2024, 2025 Rik D.T. Janssen'
+__copyright__ = 'Copyright (c) 2023 - 2026 Rik D.T. Janssen'
 __email__ = ''
 __license__ = 'MIT License'
 __package__ = 'Ricgraph'
@@ -116,7 +120,7 @@ __package__ = 'Ricgraph'
 # __version__ should be placed here instead of in __init__.py.
 # If not here, it cannot be found when executing something like
 # PYTHONPATH=../ricgraph ../bin/python ricgraph_explorer.py
-__version__ = '3.1'
+__version__ = '3.3'
 
 
 # ##############################################################################
@@ -148,30 +152,31 @@ def update_nodes_df(nodes: DataFrame) -> None:
     nodes_clean.drop_duplicates(keep='first', inplace=True, ignore_index=True)
 
     print('There are ' + str(len(nodes_clean)) + ' nodes ('
-          + timestamp() + '), updating node: 0  ', end='')
+          + timestamp() + '), updating node:')
     count = 0
+    start_ts = timestamp_posix()
     columns = nodes_clean.columns
     for row in nodes_clean.itertuples():
-        count += 1
-        if count % 250 == 0:
-            print(count, ' ', end='', flush=True)
-        if count % 2500 == 0:
-            print('(' + timestamp() + ')\n', end='', flush=True)
-
+        count = print_progress(count=count, interval=250)
         node_properties = {}
-        for prop_name in RICGRAPH_PROPERTIES_ADDITIONAL:
+        for prop_name in get_ricgraph_properties_additional():
             for other_name in columns:
                 if prop_name == other_name:
                     node_properties[prop_name] = getattr(row, other_name)
 
-        create_update_node(name=row.name, category=row.category, value=row.value,
+        create_update_node(name=str(row.name), category=str(row.category),
+                           value=str(row.value),
                            other_properties=node_properties)
 
-    print(count, '(' + timestamp() + ')\n', end='', flush=True)
+    print_progress(count=count, now=True)
+    end_ts = timestamp_posix()
+    print_records_per_minute(start_ts=start_ts, end_ts=end_ts,
+                             nr_records=count,
+                             what='Updated')
     return
 
 
-def create_nodepairs_and_edges_df(left_and_right_nodepairs: DataFrame) -> None:
+def create_nodepairs_and_edges_df(left_and_right_nodepairs: DataFrame | None) -> None:
     """Create two nodes (called 'node pairs') and two directed edges between those nodes.
     This is done for all the nodes passed. The nodes are specified in a DataFrame.
     One row in the DataFrame contains two nodes which are to be connected.
@@ -183,19 +188,17 @@ def create_nodepairs_and_edges_df(left_and_right_nodepairs: DataFrame) -> None:
     :param left_and_right_nodepairs: the node pairs in a DataFrame.
     :return: None.
     """
+    if left_and_right_nodepairs is None or left_and_right_nodepairs.empty:
+        return
     print('There are ' + str(len(left_and_right_nodepairs)) + ' rows ('
-          + timestamp() + '), creating nodes and edges for row: 0  ', end='')
+          + timestamp() + '), creating nodes and edges for row:')
     count = 0
+    start_ts = timestamp_posix()
     columns = left_and_right_nodepairs.columns
     for row in left_and_right_nodepairs.itertuples():
-        count += 1
-        if count % 250 == 0:
-            print(count, ' ', end='', flush=True)
-        if count % 2500 == 0:
-            print('(' + timestamp() + ')\n', end='', flush=True)
-
+        count = print_progress(count=count, interval=250)
         node_properties = {}
-        for prop_name in RICGRAPH_PROPERTIES_ADDITIONAL:
+        for prop_name in get_ricgraph_properties_additional():
             for other_name in columns:
                 if prop_name + '1' == other_name:
                     node_properties[prop_name + '1'] = getattr(row, other_name)
@@ -203,16 +206,22 @@ def create_nodepairs_and_edges_df(left_and_right_nodepairs: DataFrame) -> None:
                 if prop_name + '2' == other_name:
                     node_properties[prop_name + '2'] = getattr(row, other_name)
 
-        create_two_nodes_and_edge(name1=row.name1, category1=row.category1, value1=row.value1,
-                                  name2=row.name2, category2=row.category2, value2=row.value2,
+        create_two_nodes_and_edge(name1=str(row.name1), category1=str(row.category1),
+                                  value1=str(row.value1),
+                                  name2=str(row.name2), category2=str(row.category2),
+                                  value2=str(row.value2),
                                   **node_properties)
-    print(count, '(' + timestamp() + ')\n', end='', flush=True)
+    print_progress(count=count, now=True)
+    end_ts = timestamp_posix()
+    print_records_per_minute(start_ts=start_ts, end_ts=end_ts,
+                             nr_records=count,
+                             what='Created')
     return
 
 
-def create_nodepairs_and_edges_params(name1: Union[dict, str], category1: Union[dict, str], value1: Union[dict, str],
-                                      name2: Union[dict, str], category2: Union[dict, str], value2: Union[dict, str],
-                                      **other_properties: Union[dict, str]) -> None:
+def create_nodepairs_and_edges_params(name1: dict | str, category1: dict | str, value1: dict | str,
+                                      name2: dict | str, category2: dict | str, value2: dict | str,
+                                      **other_properties: dict | str) -> None:
     """Create two nodes (called 'node pairs') and two directed edges between those nodes.
     This is done for all the nodes passed. The nodes are specified by values passed to this function.
     Each value can be either a string or a dict containing strings.
@@ -242,7 +251,7 @@ def create_nodepairs_and_edges_params(name1: Union[dict, str], category1: Union[
         left_and_right_nodepairs.rename(columns={value1.name: 'value1'}, inplace=True)
 
     # This makes programming more efficient
-    allowed_properties = RICGRAPH_PROPERTIES_ADDITIONAL + RICGRAPH_PROPERTIES_STANDARD
+    allowed_properties = get_ricgraph_properties_additional() + get_ricgraph_properties_standard()
     ext_other_properties = other_properties.copy()
     ext_other_properties['name1'] = name1
     ext_other_properties['name2'] = name2
@@ -281,7 +290,7 @@ def create_nodepairs_and_edges_params(name1: Union[dict, str], category1: Union[
 
 
 def unify_personal_identifiers(personal_identifiers: DataFrame,
-                               source_event: Union[dict, str], history_event: Union[dict, str]) -> None:
+                               source_event: dict | str, history_event: dict | str) -> None:
     """Unify a collection of personal identifiers (e.g. ORCID, ISNI, etc.).
     That means that every column is unified to every other column.
     The identifiers are specified as columns in a DataFrame.
@@ -310,7 +319,7 @@ def unify_personal_identifiers(personal_identifiers: DataFrame,
             identifiers.dropna(axis=0, how='any', inplace=True)
             identifiers.drop_duplicates(keep='first', inplace=True, ignore_index=True)
 
-            if RICGRAPH_NODEADD_MODE == 'strict':
+            if get_ricgraph_nodeadd_mode() == 'strict':
                 # Remove rows which have a cell value that occurs more than once.
                 # There are two situations: (1) nodes with unique value; (2) nodes with non-unique value.
                 # (1) We assume that in one source system, always the same ID is used to connect to
@@ -339,9 +348,9 @@ def unify_personal_identifiers(personal_identifiers: DataFrame,
             identifiers.rename(columns={column1: 'value1',
                                         column2: 'value2'}, inplace=True)
             new_identifiers_columns = {'name1': column1,
-                                       'category1': 'person',
+                                       'category1': PERSON_CATEGORY_PERSON,
                                        'name2': column2,
-                                       'category2': 'person',
+                                       'category2': PERSON_CATEGORY_PERSON,
                                        'source_event1': source_event,
                                        'history_event1': history_event,
                                        'source_event2': source_event,

{ricgraph-3.1 → ricgraph-3.3}/ricgraph/ricgraph_cache.py

@@ -43,10 +43,10 @@
 
 
 from sys import getsizeof
-from typing import Optional
+from typing import Optional, Tuple
 from pymemcache.client.base import Client
 from .ricgraph_constants import MAX_NODES_CACHE_KEY_ID
-from .ricgraph_utils import (get_configfile_key,
+from .ricgraph_utils import (get_configfile_key_memcached_parameters,
                              serialize_value, deserialize_value)
 
 
@@ -70,19 +70,21 @@ def memcached_open_connection() -> None:
     If Memcached is not available, then return None.
     Set the Memcached client in a global variable.
     """
-    global _MEMCACHED_HOST, _MEMCACHED_PORT
     global _memcached_available, _memcached_client
 
-    if _MEMCACHED_TO_BE_USED != 'True':
+    memcached_to_be_used, memcached_host, memcached_port = get_configfile_key_memcached_parameters()
+    if memcached_to_be_used:
+        _memcached_available = True
+    else:
         _memcached_available = False
         return
 
-    if _MEMCACHED_HOST == '' or _MEMCACHED_PORT == '':
+    if memcached_host == '':
         _memcached_available = False
         return
 
     try:
-        memcached_client = Client(server=(_MEMCACHED_HOST, int(_MEMCACHED_PORT)),
+        memcached_client = Client(server=(memcached_host, memcached_port),
                                   allow_unicode_keys=True)
 
         # Test connection by setting and deleting a test key.
@@ -93,6 +95,8 @@ def memcached_open_connection() -> None:
         _memcached_client = memcached_client
         return
     except:
+        print('memcached_open_connection(): Warning: could not reach Memcached cache daemon,')
+        print('     it is probably not running, continuing with local cache.')
         _memcached_available = False
         _memcached_client = None
         return
@@ -122,12 +126,14 @@ def nodes_cache_key_id_create(key: str, elementid: str) -> None:
     :param elementid: id of a node.
     :return: None.
     """
-    global _nodes_cache_key_id, _memcached_available
+    global _nodes_cache_key_id, _memcached_available, _memcached_client
 
     if key == '' or elementid == '':
         return
 
     if memcached_check_available():
+        if _memcached_client is None:
+            return
         # The docs are unclear whether I need to serialize the key if I
         # use flag allow_unicode_keys=True in Client() above.
         # So I do serialize to be sure. Since keys cannot have spaces,
@@ -163,12 +169,14 @@ def nodes_cache_key_id_read(key: str) -> str:
     :param key: _key of a node.
     :return: The id of the node, or '' if not present.
     """
-    global _nodes_cache_key_id, _memcached_available
+    global _nodes_cache_key_id, _memcached_available, _memcached_client
 
     if key == '':
         return ''
 
     if memcached_check_available():
+        if _memcached_client is None:
+            return ''
         key_serialized = serialize_value(value=key.replace(' ', '_'))
         try:
             elementid_serialized = _memcached_client.get(key=key_serialized)
@@ -197,12 +205,14 @@ def nodes_cache_key_id_delete_key(key: str) -> None:
     :param key: _key of a node.
     :return: None.
     """
-    global _nodes_cache_key_id, _memcached_available
+    global _nodes_cache_key_id, _memcached_available, _memcached_client
 
     if key == '':
         return
 
     if memcached_check_available():
+        if _memcached_client is None:
+            return
         key_serialized = serialize_value(value=key.replace(' ', '_'))
         try:
             # Works regardless whether the key is present or not.
@@ -226,9 +236,11 @@ def nodes_cache_key_id_empty() -> None:
 
     :return: None.
     """
-    global _nodes_cache_key_id, _memcached_available
+    global _nodes_cache_key_id, _memcached_available, _memcached_client
 
     if memcached_check_available():
+        if _memcached_client is None:
+            return
         try:
             _memcached_client.flush_all()
         except:
@@ -243,44 +255,45 @@ def nodes_cache_key_id_empty() -> None:
     return
 
 
-def nodes_cache_key_id_type_size() -> str:
+def nodes_cache_key_id_size() -> Tuple[int, float]:
     """Return the size of the cache for node id's.
     'id' in this sentence is the id assigned by de graph database.
 
-    :return: a sentence with the length of the cache and its size.
+    :return: a tuple with number of items and size in kB.
     """
-    global _nodes_cache_key_id
+    global _nodes_cache_key_id, _memcached_client
 
     if memcached_check_available():
-        result = 'Ricgraph uses Memcached as cache. '
+        if _memcached_client is None:
+            return 0, 0.0
         try:
             stats = _memcached_client.stats()
-            items = int(stats.get(b'curr_items', 0))
+            nr_items = int(stats.get(b'curr_items', 0))
             bytes_used = int(stats.get(b'bytes', 0))
             size_kb = round(bytes_used / 1000, 1)
         except:
             # In case the connection to Memcached is lost, we will also end up here,
             # and we just continue.
-            items = 0
+            nr_items = 0
             size_kb = 0.0
     else:
-        result = 'Ricgraph uses a local cache. '
-        items = len(_nodes_cache_key_id)
+        nr_items = len(_nodes_cache_key_id)
         size_kb = round(getsizeof(_nodes_cache_key_id) / 1000, 1)
+    return nr_items, size_kb
 
-    result += 'This cache has ' + str(items) + ' elements, and its size is '
-    result += str(size_kb) + ' kB.'
-    return result
 
+def nodes_cache_key_id_type_size() -> str:
+    """Return the size of the cache for node id's.
+    'id' in this sentence is the id assigned by de graph database.
+
+    :return: a sentence with the length of the cache and its size.
+    """
+    nr_items, size_kb = nodes_cache_key_id_size()
+    if memcached_check_available():
+        result = 'Ricgraph uses Memcached as cache. '
+    else:
+        result = 'Ricgraph uses a local cache. '
 
-# ############################################
-# ################### main ###################
-# ############################################
-# This will be executed on module initialization
-_MEMCACHED_HOST = get_configfile_key(section='Memcached',
-                                     key='memcached_host')
-# This returns a str, will be converted to int later on.
-_MEMCACHED_PORT = get_configfile_key(section='Memcached',
-                                     key='memcached_port')
-_MEMCACHED_TO_BE_USED = get_configfile_key(section='Memcached',
-                                           key='ricgraph_explorer_uses_memcached')
+    result += 'This cache has ' + str(nr_items) + ' elements, and its size is '
+    result += str(size_kb) + ' kB.'
+    return result