ricgraph 2.13__tar.gz → 3.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {ricgraph-2.13/ricgraph.egg-info → ricgraph-3.1}/PKG-INFO +4 -3
- {ricgraph-2.13 → ricgraph-3.1}/README-PyPI.md +3 -2
- {ricgraph-2.13 → ricgraph-3.1}/README.md +46 -16
- {ricgraph-2.13 → ricgraph-3.1}/pyproject.toml +1 -1
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph.py +23 -7
- ricgraph-3.1/ricgraph/ricgraph_cache.py +286 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_constants.py +5 -7
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_cypher.py +76 -29
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_file.py +121 -33
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_graphdb.py +5 -1
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_utils.py +200 -4
- {ricgraph-2.13 → ricgraph-3.1/ricgraph.egg-info}/PKG-INFO +4 -3
- ricgraph-2.13/ricgraph/ricgraph_cache.py +0 -197
- {ricgraph-2.13 → ricgraph-3.1}/LICENSE +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/__init__.py +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_harvest.py +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_researchinfo.py +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph/ricgraph_restapi.py +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph.egg-info/SOURCES.txt +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph.egg-info/dependency_links.txt +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph.egg-info/requires.txt +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/ricgraph.egg-info/top_level.txt +0 -0
- {ricgraph-2.13 → ricgraph-3.1}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: ricgraph
|
|
3
|
-
Version:
|
|
3
|
+
Version: 3.1
|
|
4
4
|
Summary: Ricgraph - Research in context graph
|
|
5
5
|
Author: Rik D.T. Janssen
|
|
6
6
|
License-Expression: MIT
|
|
@@ -25,6 +25,7 @@ Requires-Dist: unidecode
|
|
|
25
25
|
Dynamic: license-file
|
|
26
26
|
|
|
27
27
|
[](https://doi.org/10.5281/zenodo.7524314)
|
|
28
|
+
[](https://doi.org/10.2139/ssrn.5524439)
|
|
28
29
|
[](https://doi.org/10.1016/j.softx.2024.101736)
|
|
29
30
|
[](https://www.ricgraph.eu)
|
|
30
31
|
[](https://docs.ricgraph.eu)
|
|
@@ -134,8 +135,8 @@ The items surrounded by the red line are the solution to this use case.
|
|
|
134
135
|
## Read more about Ricgraph
|
|
135
136
|
|
|
136
137
|
For a gentle introduction in Ricgraph, read the reference publication:
|
|
137
|
-
Rik D.T. Janssen (2024). Ricgraph: A flexible and extensible graph to explore research in
|
|
138
|
-
context from various systems
|
|
138
|
+
Rik D.T. Janssen (2024). *Ricgraph: A flexible and extensible graph to explore research in
|
|
139
|
+
context from various systems*. SoftwareX, 26(101736).
|
|
139
140
|
[https://doi.org/10.1016/j.softx.2024.101736](https://doi.org/10.1016/j.softx.2024.101736).
|
|
140
141
|
|
|
141
142
|
You might also want to read the documentation on
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
[](https://doi.org/10.5281/zenodo.7524314)
|
|
2
|
+
[](https://doi.org/10.2139/ssrn.5524439)
|
|
2
3
|
[](https://doi.org/10.1016/j.softx.2024.101736)
|
|
3
4
|
[](https://www.ricgraph.eu)
|
|
4
5
|
[](https://docs.ricgraph.eu)
|
|
@@ -108,8 +109,8 @@ The items surrounded by the red line are the solution to this use case.
|
|
|
108
109
|
## Read more about Ricgraph
|
|
109
110
|
|
|
110
111
|
For a gentle introduction in Ricgraph, read the reference publication:
|
|
111
|
-
Rik D.T. Janssen (2024). Ricgraph: A flexible and extensible graph to explore research in
|
|
112
|
-
context from various systems
|
|
112
|
+
Rik D.T. Janssen (2024). *Ricgraph: A flexible and extensible graph to explore research in
|
|
113
|
+
context from various systems*. SoftwareX, 26(101736).
|
|
113
114
|
[https://doi.org/10.1016/j.softx.2024.101736](https://doi.org/10.1016/j.softx.2024.101736).
|
|
114
115
|
|
|
115
116
|
You might also want to read the documentation on
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
[](https://doi.org/10.5281/zenodo.7524314)
|
|
2
2
|
[](https://doi.org/10.1016/j.softx.2024.101736)
|
|
3
|
+
[](https://doi.org/10.2139/ssrn.5524439)
|
|
3
4
|
[](https://www.ricgraph.eu)
|
|
4
5
|
[](https://docs.ricgraph.eu)
|
|
5
6
|
[](https://github.com/UtrechtUniversity/ricgraph)
|
|
@@ -53,6 +54,27 @@ relations that are not present in any of the separate source systems.
|
|
|
53
54
|
It is flexible and extensible, and can be
|
|
54
55
|
adapted to new application areas.
|
|
55
56
|
|
|
57
|
+
Throughout this documentation, we illustrate how Ricgraph works by applying
|
|
58
|
+
it to the application area research information.
|
|
59
|
+
|
|
60
|
+
## Ricgraph news
|
|
61
|
+
|
|
62
|
+
Read my recent preprint:
|
|
63
|
+
|
|
64
|
+
* Rik D.T. Janssen (2025).
|
|
65
|
+
*Utilizing Ricgraph to gain insights into research collaborations across institutions,
|
|
66
|
+
at every organizational level*. Submitted to SoftwareX [preprint].
|
|
67
|
+
[https://doi.org/10.2139/ssrn.5524439](https://doi.org/10.2139/ssrn.5524439).
|
|
68
|
+
|
|
69
|
+
See Ricgraph in action:
|
|
70
|
+
|
|
71
|
+
* Use the Open Ricgraph demo server
|
|
72
|
+
via [https://explorer.ricgraph.eu](https://explorer.ricgraph.eu).
|
|
73
|
+
Read more (and join) at
|
|
74
|
+
[pilot project Open Ricgraph demo
|
|
75
|
+
server](https://www.ricgraph.eu/pilot-project-open-ricgraph-demo-server.html).
|
|
76
|
+
|
|
77
|
+
|
|
56
78
|
Currently, there are people working on the following extensions for Ricgraph:
|
|
57
79
|
|
|
58
80
|
* [BackToPure](https://github.com/UtrechtUniversity/backtopure)
|
|
@@ -70,9 +92,15 @@ Currently, there are people working on the following extensions for Ricgraph:
|
|
|
70
92
|
or "With what organizations does that faculty collaborate?".
|
|
71
93
|
It uses a local Large Language Model.
|
|
72
94
|
Status: planning stage (pre-alpha).
|
|
73
|
-
* A project that
|
|
74
|
-
[
|
|
75
|
-
|
|
95
|
+
* A project with six students from the University of Applied Sciences, Utrecht, that aims to
|
|
96
|
+
[create a general harvester for
|
|
97
|
+
Ricgraph](docs/ricgraph_outreach.md#ricgraph-projects-with-students).
|
|
98
|
+
It will remove a lot of code duplication in the harvest scripts, and will
|
|
99
|
+
make it easier to add new harvest sources.
|
|
100
|
+
Status: planning stage (pre-alpha).
|
|
101
|
+
* A project with a student from the University of Applied Sciences, Utrecht, that aims to use
|
|
102
|
+
[AI and Large Language Models to find topics and visualize large amounts of research
|
|
103
|
+
information](docs/ricgraph_outreach.md#ricgraph-projects-with-students).
|
|
76
104
|
It assigns *topics* to publications,
|
|
77
105
|
data sets, and software. By selecting a number of topics, research results are
|
|
78
106
|
grouped, and possibly experts on these topics can be found.
|
|
@@ -80,9 +108,6 @@ Currently, there are people working on the following extensions for Ricgraph:
|
|
|
80
108
|
* A tool that helps Pure administrators to clean up "external organizations" in Pure.
|
|
81
109
|
Status: planning stage (pre-alpha).
|
|
82
110
|
|
|
83
|
-
Throughout this documentation, we illustrate how Ricgraph works by applying
|
|
84
|
-
it to the application area research information.
|
|
85
|
-
|
|
86
111
|
|
|
87
112
|
## Motivation
|
|
88
113
|
|
|
@@ -297,9 +322,14 @@ The following figure shows how three persons have contributed to one research ou
|
|
|
297
322
|
* Explore the full documentation of Ricgraph on
|
|
298
323
|
[https://docs.ricgraph.eu](https://docs.ricgraph.eu).
|
|
299
324
|
* For a gentle introduction in Ricgraph, read the reference publication:
|
|
300
|
-
Rik D.T. Janssen (2024). Ricgraph: A flexible and extensible graph to explore research in
|
|
301
|
-
context from various systems
|
|
325
|
+
Rik D.T. Janssen (2024). *Ricgraph: A flexible and extensible graph to explore research in
|
|
326
|
+
context from various systems*. SoftwareX, 26(101736).
|
|
302
327
|
[https://doi.org/10.1016/j.softx.2024.101736](https://doi.org/10.1016/j.softx.2024.101736).
|
|
328
|
+
* We also have an article about collaborations and Ricgraph:
|
|
329
|
+
Rik D.T. Janssen (2025).
|
|
330
|
+
*Utilizing Ricgraph to gain insights into research collaborations across institutions,
|
|
331
|
+
at every organizational level*. Submitted to SoftwareX [preprint].
|
|
332
|
+
[https://doi.org/10.2139/ssrn.5524439](https://doi.org/10.2139/ssrn.5524439).
|
|
303
333
|
* For a further introduction, you might want to read these two presentations:
|
|
304
334
|
* A general presentation about Ricgraph, presenting Ricgraph in a visual manner:
|
|
305
335
|
[https://doi.org/10.5281/zenodo.12634234](https://doi.org/10.5281/zenodo.12634234).
|
|
@@ -315,13 +345,13 @@ The following figure shows how three persons have contributed to one research ou
|
|
|
315
345
|
systems, not present in one's own Pure _A_.
|
|
316
346
|
|
|
317
347
|
* Read more about
|
|
318
|
-
[publications](docs/
|
|
319
|
-
[presentations](docs/
|
|
320
|
-
[newsletters](docs/
|
|
348
|
+
[publications](docs/ricgraph_outreach.md#ricgraph-publications),
|
|
349
|
+
[presentations](docs/ricgraph_outreach.md#ricgraph-presentations),
|
|
350
|
+
[newsletters](docs/ricgraph_outreach.md#ricgraph-newsletters)
|
|
321
351
|
(to subscribe, go to [Ricgraph Contact](#contact)),
|
|
322
|
-
[projects with students](docs/
|
|
323
|
-
[use](docs/
|
|
324
|
-
[mentions](docs/
|
|
352
|
+
[projects with students](docs/ricgraph_outreach.md#ricgraph-projects-with-students),
|
|
353
|
+
[use](docs/ricgraph_outreach.md#ricgraph-use), and
|
|
354
|
+
[mentions](docs/ricgraph_outreach.md#ricgraph-mentions)
|
|
325
355
|
of Ricgraph.
|
|
326
356
|
* The Ricgraph main website is www.ricgraph.eu.
|
|
327
357
|
Alternative sites are www.ricgraph.nl and www.ricgraph.com.
|
|
@@ -379,7 +409,7 @@ newsletter.
|
|
|
379
409
|
a subscribe email text. Please complete it
|
|
380
410
|
by filling in your name and email address](mailto:r.d.t.janssen@uu.nl?subject=Please%20add%20me%20to%20the%20Ricgraph%20newsletter%20email%20list&body=Hi%20Rik%2C%0D%0A%0D%0ACan%20you%20please%20add%20me%20to%20your%20Ricgraph%20newsletter%20email%20list%3F%20Thanks!%0D%0A%0D%0ABest%20regards%2C%0D%0A%0D%0A%5Bplease%20replace%20with%20your%20name%5D%0D%0A%5Bplease%20replace%20with%20your%20email%20address%5D).
|
|
381
411
|
We also have a
|
|
382
|
-
[newsletter archive](docs/
|
|
412
|
+
[newsletter archive](docs/ricgraph_outreach.md#ricgraph-newsletters).
|
|
383
413
|
|
|
384
414
|
If you like Ricgraph, please give it a GitHub Star by clicking on the top right
|
|
385
415
|
*Star* button on the Ricgraph GitHub page
|
|
@@ -390,4 +420,4 @@ please let me know by creating a GitHub *Issue* at the top left of that page.
|
|
|
390
420
|
<!-- Do not change the lines below. The version number will be
|
|
391
421
|
modified by script maintenance/update_version.bash -->
|
|
392
422
|
This README.md file is part of the
|
|
393
|
-
documentation for version
|
|
423
|
+
documentation for version 3.1 of Ricgraph - Research in context graph.
|
|
@@ -101,8 +101,8 @@ In the default configuration of Ricgraph, the following properties are included:
|
|
|
101
101
|
from numpy import nan
|
|
102
102
|
from pandas import DataFrame, concat, set_option
|
|
103
103
|
from typing import Union
|
|
104
|
-
from .
|
|
105
|
-
|
|
104
|
+
from .ricgraph_utils import timestamp
|
|
105
|
+
from .ricgraph_graphdb import (create_update_node, create_two_nodes_and_edge,
|
|
106
106
|
RICGRAPH_NODEADD_MODE,
|
|
107
107
|
RICGRAPH_PROPERTIES_STANDARD, RICGRAPH_PROPERTIES_ADDITIONAL)
|
|
108
108
|
|
|
@@ -116,7 +116,7 @@ __package__ = 'Ricgraph'
|
|
|
116
116
|
# __version__ should be placed here instead of in __init__.py.
|
|
117
117
|
# If not here, it cannot be found when executing something like
|
|
118
118
|
# PYTHONPATH=../ricgraph ../bin/python ricgraph_explorer.py
|
|
119
|
-
__version__ = '
|
|
119
|
+
__version__ = '3.1'
|
|
120
120
|
|
|
121
121
|
|
|
122
122
|
# ##############################################################################
|
|
@@ -336,10 +336,21 @@ def unify_personal_identifiers(personal_identifiers: DataFrame,
|
|
|
336
336
|
identifiers.drop_duplicates(subset=column1, keep=False, inplace=True, ignore_index=True)
|
|
337
337
|
identifiers.drop_duplicates(subset=column2, keep=False, inplace=True, ignore_index=True)
|
|
338
338
|
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
339
|
+
identifiers.rename(columns={column1: 'value1',
|
|
340
|
+
column2: 'value2'}, inplace=True)
|
|
341
|
+
new_identifiers_columns = {'name1': column1,
|
|
342
|
+
'category1': 'person',
|
|
343
|
+
'name2': column2,
|
|
344
|
+
'category2': 'person',
|
|
345
|
+
'source_event1': source_event,
|
|
346
|
+
'history_event1': history_event,
|
|
347
|
+
'source_event2': source_event,
|
|
348
|
+
'history_event2': history_event}
|
|
349
|
+
identifiers = identifiers.assign(**new_identifiers_columns)
|
|
350
|
+
identifiers = identifiers[['name1', 'category1', 'value1', 'source_event1', 'history_event1',
|
|
351
|
+
'name2', 'category2', 'value2', 'source_event2', 'history_event2']]
|
|
352
|
+
create_nodepairs_and_edges_df(left_and_right_nodepairs=identifiers)
|
|
353
|
+
|
|
343
354
|
print('\nDone at ' + timestamp() + '.\n')
|
|
344
355
|
return
|
|
345
356
|
|
|
@@ -357,3 +368,8 @@ set_option('display.width', 500)
|
|
|
357
368
|
# in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`.
|
|
358
369
|
# To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`"
|
|
359
370
|
set_option('future.no_silent_downcasting', True)
|
|
371
|
+
|
|
372
|
+
print('\nThis is Ricgraph, Research in contect graph.')
|
|
373
|
+
print('To learn about Ricgraph, please go to www.ricgraph.eu.')
|
|
374
|
+
print('You are using Ricgraph version ' + get_ricgraph_version() + '.\n')
|
|
375
|
+
|
|
@@ -0,0 +1,286 @@
|
|
|
1
|
+
# ########################################################################
|
|
2
|
+
#
|
|
3
|
+
# Ricgraph - Research in context graph
|
|
4
|
+
#
|
|
5
|
+
# ########################################################################
|
|
6
|
+
#
|
|
7
|
+
# MIT License
|
|
8
|
+
#
|
|
9
|
+
# Copyright (c) 2023 - 2025 Rik D.T. Janssen
|
|
10
|
+
#
|
|
11
|
+
# Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
12
|
+
# of this software and associated documentation files (the "Software"), to deal
|
|
13
|
+
# in the Software without restriction, including without limitation the rights
|
|
14
|
+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
15
|
+
# copies of the Software, and to permit persons to whom the Software is
|
|
16
|
+
# furnished to do so, subject to the following conditions:
|
|
17
|
+
#
|
|
18
|
+
# The above copyright notice and this permission notice shall be included in all
|
|
19
|
+
# copies or substantial portions of the Software.
|
|
20
|
+
#
|
|
21
|
+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
22
|
+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
23
|
+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
24
|
+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
25
|
+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
26
|
+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
27
|
+
# SOFTWARE.
|
|
28
|
+
#
|
|
29
|
+
# ########################################################################
|
|
30
|
+
#
|
|
31
|
+
# Ricgraph cache functions.
|
|
32
|
+
# For more information about Ricgraph and Ricgraph Explorer,
|
|
33
|
+
# go to https://www.ricgraph.eu and https://docs.ricgraph.eu.
|
|
34
|
+
#
|
|
35
|
+
# ########################################################################
|
|
36
|
+
#
|
|
37
|
+
# Original version Rik D.T. Janssen, December 2022.
|
|
38
|
+
# Updated Rik D.T. Janssen, February, March, September to December 2023.
|
|
39
|
+
# Updated Rik D.T. Janssen, February to June, September to December 2024.
|
|
40
|
+
# Updated Rik D.T. Janssen, January to June, October 2025.
|
|
41
|
+
#
|
|
42
|
+
# ########################################################################
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
from sys import getsizeof
|
|
46
|
+
from typing import Optional
|
|
47
|
+
from pymemcache.client.base import Client
|
|
48
|
+
from .ricgraph_constants import MAX_NODES_CACHE_KEY_ID
|
|
49
|
+
from .ricgraph_utils import (get_configfile_key,
|
|
50
|
+
serialize_value, deserialize_value)
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
# This dict is used as a cache for node id's. If we have a node id, we can
|
|
54
|
+
# do a direct lookup for a node in O(1) in the graph database,
|
|
55
|
+
# instead of a search in O(log n).
|
|
56
|
+
# The dict has the format: [Ricgraph _key]: [Node element_id].
|
|
57
|
+
_nodes_cache_key_id = {}
|
|
58
|
+
|
|
59
|
+
# Global indicating whether Memcached is available.
|
|
60
|
+
_memcached_available = False
|
|
61
|
+
|
|
62
|
+
# Global for connection to Memcached.
|
|
63
|
+
# Type hint necessary to avoid PyCharm warning.
|
|
64
|
+
_memcached_client: Optional[Client] = None
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def memcached_open_connection() -> None:
|
|
68
|
+
"""Open a connection to Memcached, but only when it should
|
|
69
|
+
be used according to ricgraph.ini.
|
|
70
|
+
If Memcached is not available, then return None.
|
|
71
|
+
Set the Memcached client in a global variable.
|
|
72
|
+
"""
|
|
73
|
+
global _MEMCACHED_HOST, _MEMCACHED_PORT
|
|
74
|
+
global _memcached_available, _memcached_client
|
|
75
|
+
|
|
76
|
+
if _MEMCACHED_TO_BE_USED != 'True':
|
|
77
|
+
_memcached_available = False
|
|
78
|
+
return
|
|
79
|
+
|
|
80
|
+
if _MEMCACHED_HOST == '' or _MEMCACHED_PORT == '':
|
|
81
|
+
_memcached_available = False
|
|
82
|
+
return
|
|
83
|
+
|
|
84
|
+
try:
|
|
85
|
+
memcached_client = Client(server=(_MEMCACHED_HOST, int(_MEMCACHED_PORT)),
|
|
86
|
+
allow_unicode_keys=True)
|
|
87
|
+
|
|
88
|
+
# Test connection by setting and deleting a test key.
|
|
89
|
+
memcached_client.set(key='test_key', value=b'test', expire=1)
|
|
90
|
+
memcached_client.delete(key='test_key')
|
|
91
|
+
|
|
92
|
+
_memcached_available = True
|
|
93
|
+
_memcached_client = memcached_client
|
|
94
|
+
return
|
|
95
|
+
except:
|
|
96
|
+
_memcached_available = False
|
|
97
|
+
_memcached_client = None
|
|
98
|
+
return
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
def memcached_check_available() -> bool:
|
|
102
|
+
"""Check if Memcached is available.
|
|
103
|
+
|
|
104
|
+
:return: True if available, False otherwise.
|
|
105
|
+
"""
|
|
106
|
+
global _memcached_available, _memcached_client
|
|
107
|
+
|
|
108
|
+
if not _memcached_available:
|
|
109
|
+
return False
|
|
110
|
+
|
|
111
|
+
if _memcached_client is None:
|
|
112
|
+
return False
|
|
113
|
+
else:
|
|
114
|
+
return True
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def nodes_cache_key_id_create(key: str, elementid: str) -> None:
|
|
118
|
+
"""Create an entry in the cache for node id's.
|
|
119
|
+
'id' in this sentence is the id assigned by de graph database.
|
|
120
|
+
|
|
121
|
+
:param key: _key of a node.
|
|
122
|
+
:param elementid: id of a node.
|
|
123
|
+
:return: None.
|
|
124
|
+
"""
|
|
125
|
+
global _nodes_cache_key_id, _memcached_available
|
|
126
|
+
|
|
127
|
+
if key == '' or elementid == '':
|
|
128
|
+
return
|
|
129
|
+
|
|
130
|
+
if memcached_check_available():
|
|
131
|
+
# The docs are unclear whether I need to serialize the key if I
|
|
132
|
+
# use flag allow_unicode_keys=True in Client() above.
|
|
133
|
+
# So I do serialize to be sure. Since keys cannot have spaces,
|
|
134
|
+
# these are replaced first.
|
|
135
|
+
key_serialized = serialize_value(value=key.replace(' ', '_'))
|
|
136
|
+
elementid_serialized = serialize_value(value=elementid)
|
|
137
|
+
try:
|
|
138
|
+
_memcached_client.set(key=key_serialized,
|
|
139
|
+
value=elementid_serialized,
|
|
140
|
+
expire=0)
|
|
141
|
+
except:
|
|
142
|
+
print('nodes_cache_key_id_create(): Warning, connection to Memcached lost, continuing...')
|
|
143
|
+
# Continue, hopefully the connection will come back soon.
|
|
144
|
+
return
|
|
145
|
+
|
|
146
|
+
if len(_nodes_cache_key_id) > MAX_NODES_CACHE_KEY_ID:
|
|
147
|
+
nodes_cache_key_id_empty()
|
|
148
|
+
|
|
149
|
+
# We use a 'dict', which does not allow for duplicates, so we
|
|
150
|
+
# do not need to check for duplicates.
|
|
151
|
+
# https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
|
|
152
|
+
# tells that Python dict keys and values can have _almost_ any type.
|
|
153
|
+
# So we do not need to serialize and deserialize.
|
|
154
|
+
_nodes_cache_key_id[key] = elementid
|
|
155
|
+
# print('Create: ' + str(key) + ' -- ' + _nodes_cache_key_id[key])
|
|
156
|
+
return
|
|
157
|
+
|
|
158
|
+
|
|
159
|
+
def nodes_cache_key_id_read(key: str) -> str:
|
|
160
|
+
"""Read an entry from the cache for node id's.
|
|
161
|
+
'id' in this sentence is the id assigned by de graph database.
|
|
162
|
+
|
|
163
|
+
:param key: _key of a node.
|
|
164
|
+
:return: The id of the node, or '' if not present.
|
|
165
|
+
"""
|
|
166
|
+
global _nodes_cache_key_id, _memcached_available
|
|
167
|
+
|
|
168
|
+
if key == '':
|
|
169
|
+
return ''
|
|
170
|
+
|
|
171
|
+
if memcached_check_available():
|
|
172
|
+
key_serialized = serialize_value(value=key.replace(' ', '_'))
|
|
173
|
+
try:
|
|
174
|
+
elementid_serialized = _memcached_client.get(key=key_serialized)
|
|
175
|
+
except:
|
|
176
|
+
print('nodes_cache_key_id_read(): Warning, connection to Memcached lost, continuing...')
|
|
177
|
+
# Continue, hopefully the connection will come back soon.
|
|
178
|
+
return ''
|
|
179
|
+
|
|
180
|
+
if elementid_serialized is None:
|
|
181
|
+
# Key not found.
|
|
182
|
+
return ''
|
|
183
|
+
else:
|
|
184
|
+
elementid = deserialize_value(serialized=elementid_serialized)
|
|
185
|
+
return elementid
|
|
186
|
+
|
|
187
|
+
if key in _nodes_cache_key_id:
|
|
188
|
+
# print('Read: ' + str(key) + ' -- ' + _nodes_cache_key_id[key])
|
|
189
|
+
return _nodes_cache_key_id[key]
|
|
190
|
+
return ''
|
|
191
|
+
|
|
192
|
+
|
|
193
|
+
def nodes_cache_key_id_delete_key(key: str) -> None:
|
|
194
|
+
"""Delete a key 'key' from the cache for node id's.
|
|
195
|
+
'id' in this sentence is the id assigned by de graph database.
|
|
196
|
+
|
|
197
|
+
:param key: _key of a node.
|
|
198
|
+
:return: None.
|
|
199
|
+
"""
|
|
200
|
+
global _nodes_cache_key_id, _memcached_available
|
|
201
|
+
|
|
202
|
+
if key == '':
|
|
203
|
+
return
|
|
204
|
+
|
|
205
|
+
if memcached_check_available():
|
|
206
|
+
key_serialized = serialize_value(value=key.replace(' ', '_'))
|
|
207
|
+
try:
|
|
208
|
+
# Works regardless whether the key is present or not.
|
|
209
|
+
_memcached_client.delete(key=key_serialized)
|
|
210
|
+
except:
|
|
211
|
+
print('nodes_cache_key_id_delete(): Error, connection to Memcached lost, exiting...')
|
|
212
|
+
# Exit because the cache gets in an unexpected state:
|
|
213
|
+
# an element remains in the cache while it should not.
|
|
214
|
+
exit(1)
|
|
215
|
+
return
|
|
216
|
+
|
|
217
|
+
if key in _nodes_cache_key_id:
|
|
218
|
+
# print('Delete: ' + str(key) + ' -- ' + _nodes_cache_key_id[key])
|
|
219
|
+
_nodes_cache_key_id.pop(key)
|
|
220
|
+
return
|
|
221
|
+
|
|
222
|
+
|
|
223
|
+
def nodes_cache_key_id_empty() -> None:
|
|
224
|
+
"""Empty the cache for node id's.
|
|
225
|
+
'id' in this sentence is the id assigned by de graph database.
|
|
226
|
+
|
|
227
|
+
:return: None.
|
|
228
|
+
"""
|
|
229
|
+
global _nodes_cache_key_id, _memcached_available
|
|
230
|
+
|
|
231
|
+
if memcached_check_available():
|
|
232
|
+
try:
|
|
233
|
+
_memcached_client.flush_all()
|
|
234
|
+
except:
|
|
235
|
+
print('nodes_cache_key_id_empty(): Error, connection to Memcached lost, exiting...')
|
|
236
|
+
# Exit because the cache gets in an unexpected state:
|
|
237
|
+
# it is not emptied while it should have been.
|
|
238
|
+
exit(1)
|
|
239
|
+
return
|
|
240
|
+
|
|
241
|
+
# print('Clear cache.')
|
|
242
|
+
_nodes_cache_key_id.clear()
|
|
243
|
+
return
|
|
244
|
+
|
|
245
|
+
|
|
246
|
+
def nodes_cache_key_id_type_size() -> str:
|
|
247
|
+
"""Return the size of the cache for node id's.
|
|
248
|
+
'id' in this sentence is the id assigned by de graph database.
|
|
249
|
+
|
|
250
|
+
:return: a sentence with the length of the cache and its size.
|
|
251
|
+
"""
|
|
252
|
+
global _nodes_cache_key_id
|
|
253
|
+
|
|
254
|
+
if memcached_check_available():
|
|
255
|
+
result = 'Ricgraph uses Memcached as cache. '
|
|
256
|
+
try:
|
|
257
|
+
stats = _memcached_client.stats()
|
|
258
|
+
items = int(stats.get(b'curr_items', 0))
|
|
259
|
+
bytes_used = int(stats.get(b'bytes', 0))
|
|
260
|
+
size_kb = round(bytes_used / 1000, 1)
|
|
261
|
+
except:
|
|
262
|
+
# In case the connection to Memcached is lost, we will also end up here,
|
|
263
|
+
# and we just continue.
|
|
264
|
+
items = 0
|
|
265
|
+
size_kb = 0.0
|
|
266
|
+
else:
|
|
267
|
+
result = 'Ricgraph uses a local cache. '
|
|
268
|
+
items = len(_nodes_cache_key_id)
|
|
269
|
+
size_kb = round(getsizeof(_nodes_cache_key_id) / 1000, 1)
|
|
270
|
+
|
|
271
|
+
result += 'This cache has ' + str(items) + ' elements, and its size is '
|
|
272
|
+
result += str(size_kb) + ' kB.'
|
|
273
|
+
return result
|
|
274
|
+
|
|
275
|
+
|
|
276
|
+
# ############################################
|
|
277
|
+
# ################### main ###################
|
|
278
|
+
# ############################################
|
|
279
|
+
# This will be executed on module initialization
|
|
280
|
+
_MEMCACHED_HOST = get_configfile_key(section='Memcached',
|
|
281
|
+
key='memcached_host')
|
|
282
|
+
# This returns a str, will be converted to int later on.
|
|
283
|
+
_MEMCACHED_PORT = get_configfile_key(section='Memcached',
|
|
284
|
+
key='memcached_port')
|
|
285
|
+
_MEMCACHED_TO_BE_USED = get_configfile_key(section='Memcached',
|
|
286
|
+
key='ricgraph_explorer_uses_memcached')
|
|
@@ -72,13 +72,8 @@ HTTP_RESPONSE_INVALID_SEARCH = 251
|
|
|
72
72
|
# The dict '_nodes_cache_key_id' is used to cache IDs to nodes. This is the cache size.
|
|
73
73
|
# The dict itself is defined in ricgraph_cache.py.
|
|
74
74
|
# This dict will be emptied if it reaches this number of elements.
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
# The dict '_nodes_cache_nodelink' is used to be able to pass nodes (i.e. links to Node)
|
|
78
|
-
# between the pages of Ricgraph Explorer.
|
|
79
|
-
# The dict itself is defined in ricgraph_cache.py.
|
|
80
|
-
# This dict will be emptied if it reaches this number of elements.
|
|
81
|
-
MAX_NODES_CACHE_NODELINK = 100000
|
|
75
|
+
# A cache entry is approx. 20 - 30 bytes. 30 x 2000000 ~ 60MB.
|
|
76
|
+
MAX_NODES_CACHE_KEY_ID = 2000000
|
|
82
77
|
|
|
83
78
|
# In merge_two_nodes(node_merge_from, node_merge_to), we are constructing a list of
|
|
84
79
|
# all the nodes that are to be merged with node_merge_to, to add to property _history
|
|
@@ -86,6 +81,9 @@ MAX_NODES_CACHE_NODELINK = 100000
|
|
|
86
81
|
# values that are too long).
|
|
87
82
|
MAX_NR_HISTORYITEMS_TO_ADD = 50
|
|
88
83
|
|
|
84
|
+
# The maximum length of the organization abbreviation.
|
|
85
|
+
MAX_ORG_ABBREVIATION_LENGTH = 3
|
|
86
|
+
|
|
89
87
|
|
|
90
88
|
# ########################################################################
|
|
91
89
|
# Research output types used in Ricgraph.
|