notion2pandas 1.4.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- notion2pandas-1.4.2/LICENSE +21 -0
- notion2pandas-1.4.2/PKG-INFO +274 -0
- notion2pandas-1.4.2/README.md +250 -0
- notion2pandas-1.4.2/notion2pandas/__init__.py +10 -0
- notion2pandas-1.4.2/notion2pandas/n2p_read_write.py +807 -0
- notion2pandas-1.4.2/notion2pandas/notion2pandas.py +506 -0
- notion2pandas-1.4.2/notion2pandas.egg-info/PKG-INFO +274 -0
- notion2pandas-1.4.2/notion2pandas.egg-info/SOURCES.txt +11 -0
- notion2pandas-1.4.2/notion2pandas.egg-info/dependency_links.txt +1 -0
- notion2pandas-1.4.2/notion2pandas.egg-info/requires.txt +2 -0
- notion2pandas-1.4.2/notion2pandas.egg-info/top_level.txt +1 -0
- notion2pandas-1.4.2/setup.cfg +4 -0
- notion2pandas-1.4.2/setup.py +36 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2024 Andrea Rosati
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,274 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: notion2pandas
|
|
3
|
+
Version: 1.4.2
|
|
4
|
+
Summary: Notion Client extension to import notion Database into pandas Dataframe
|
|
5
|
+
Home-page: https://gitlab.com/Jaeger87/notion2pandas
|
|
6
|
+
Author: Andrea Rosati
|
|
7
|
+
Author-email: rosati.1595834@gmail.com
|
|
8
|
+
License: MIT
|
|
9
|
+
Classifier: Development Status :: 4 - Beta
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Requires-Python: >=3.7, <4
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
Requires-Dist: notion-client
|
|
23
|
+
Requires-Dist: pandas
|
|
24
|
+
|
|
25
|
+
# Notion2Pandas
|
|
26
|
+
<p align="center">
|
|
27
|
+
<img src="https://gitlab.com/Jaeger87/notion2pandas/-/raw/main/readme_assets/logo.png?ref_type=heads" class="center">
|
|
28
|
+
</p> <p align="center">
|
|
29
|
+
|
|
30
|
+
<div align="center">
|
|
31
|
+
<p>
|
|
32
|
+
<a href="https://pypi.org/project/notion2pandas/"><img src="https://gitlab.com/Jaeger87/notion2pandas/-/badges/release.svg" alt="Latest Release"></a>
|
|
33
|
+
</p>
|
|
34
|
+
</div>
|
|
35
|
+
|
|
36
|
+
Notion2Pandas is a Python 3 package that extends the capabilities of the excellent [notion-sdk-py](https://ramnes.github.io/notion-sdk-py/) by [Ramnes](https://github.com/ramnes), It enables the seamless import of a Notion database into a pandas dataframe and vice versa, requiring just a single line of code.
|
|
37
|
+
|
|
38
|
+
## Installation
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
pip install notion2pandas
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
## Usage
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="https://gitlab.com/Jaeger87/notion2pandas/-/raw/main/readme_assets/notiondb2pandas.gif?ref_type=heads" class="center">
|
|
48
|
+
</p> <p align="left">
|
|
49
|
+
|
|
50
|
+
* As shown in the gif, you just need to import the Notion2PandasClient class.
|
|
51
|
+
```python
|
|
52
|
+
from notion2pandas import Notion2PandasClient
|
|
53
|
+
```
|
|
54
|
+
* Create an instance by passing your authentication token.
|
|
55
|
+
```python
|
|
56
|
+
n2p = Notion2PandasClient(auth=os.environ["NOTION_TOKEN"])
|
|
57
|
+
```
|
|
58
|
+
* Use the 'from_notion_DB_to_dataframe' method to get the data into a dataframe.
|
|
59
|
+
```python
|
|
60
|
+
df = n2p.from_notion_DB_to_dataframe(os.environ["DATABASE_ID"])
|
|
61
|
+
```
|
|
62
|
+
* When you're done working with your dataframe, use the 'update_notion_DB_from_dataframe' method to save the data back to Notion.
|
|
63
|
+
```python
|
|
64
|
+
n2p.update_notion_DB_from_dataframe(os.environ["DATABASE_ID"], df)
|
|
65
|
+
```
|
|
66
|
+
* If you need a queried or sorted database, you can create your filter / sort object [with this structure](https://developers.notion.com/reference/post-database-query) and pass it to the from_notion_DB_to_dataframe method:
|
|
67
|
+
|
|
68
|
+
```python
|
|
69
|
+
published_filter = {"filter": {
|
|
70
|
+
"property": "Published",
|
|
71
|
+
"checkbox": {
|
|
72
|
+
"equals": True
|
|
73
|
+
}
|
|
74
|
+
}}
|
|
75
|
+
|
|
76
|
+
df = n2p.from_notion_DB_to_dataframe(os.environ["DATABASE_ID"], published_filter)
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## PageID and Row_Hash
|
|
80
|
+
|
|
81
|
+
As you can see, in the pandas dataframe there are two additional columns compared to those in the original database, PageID and Row_Hash. As you can imagine, PageID it's the ID related to the page of that entry in Notion. Row_Hash is a value calculated based on the fields' values of the entry, this value is used by the update_notion_DB_from_dataframe function to determine if a row in the dataframe has been modified, and if not, it avoids making the API call to Notion for that row. Any change to those functions can lead to malfunctions, so please do not change them!
|
|
82
|
+
|
|
83
|
+
## Utility functions
|
|
84
|
+
|
|
85
|
+
Notion2Pandas is a class that extend [Client](https://github.com/ramnes/notion-sdk-py/blob/main/notion_client/client.py) from notion_client, so you can find every feature present in notion_client. In addition to the functions for importing and exporting dataframes, I've added some other convenient functions that wrap the usage of the notion_client functionality and allow them to be used more directly. These are:
|
|
86
|
+
|
|
87
|
+
* get_database_columns(database_ID)
|
|
88
|
+
* create_page(page_ID)
|
|
89
|
+
* update_page(page_ID, **kwargs)
|
|
90
|
+
* retrieve_page(page_ID)
|
|
91
|
+
* delete_page(page_ID)
|
|
92
|
+
* delete_rows_and_pages(dataframe, rows_to_delete_indexes)
|
|
93
|
+
* retrieve_block(block_ID)
|
|
94
|
+
* retrieve_block_children_list(block_ID)
|
|
95
|
+
* update_block(block_ID, field, field_value_updated)
|
|
96
|
+
|
|
97
|
+
## Read Write Functions
|
|
98
|
+
|
|
99
|
+
Notion2Pandas has the ability to transform a Notion database into a Pandas dataframe without having to specify how to parse the data. However, in some cases, the default parsing may not be what you want to achieve. Therefore, it's possible to specify how to parse the data. In Notion2Pandas, each data type in Notion is associated with a tuple consisting of two functions: one for reading the data and the other for writing it.
|
|
100
|
+
|
|
101
|
+
In this example, I'm changing the functions for reading and writing dates so that I can work only with the start date.
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
def date_read_only_start(notion_property):
|
|
105
|
+
return notion_property.get('date').get('start') if notion_property.get('date') is not None else ''
|
|
106
|
+
def date_write_only_start(row_value):
|
|
107
|
+
return {'date': {'start': row_value} if row_value != '' else None}
|
|
108
|
+
|
|
109
|
+
n2p.set_lambdas('date',date_read_only_start, date_write_only_start)
|
|
110
|
+
```
|
|
111
|
+
These function can accept up to three input arguments:
|
|
112
|
+
- `notion_property` for read functions, `row_value` for write functions: the data being read or written.
|
|
113
|
+
- `column_name`: the name of the column, used to identify exactly where to apply the function.
|
|
114
|
+
- `switcher`: the data structure containing the read/write functions, useful for parsing recursive data structures, such as formulas.
|
|
115
|
+
|
|
116
|
+
To override functions for a specific data type but apply them only to certain columns, you can use the `column_name` argument to determine which logic to apply for each field. For example:
|
|
117
|
+
|
|
118
|
+
```python
|
|
119
|
+
"""
|
|
120
|
+
Since 'Test table 1' is a relation field limited to just one relation;
|
|
121
|
+
we want a string instead of a list with only one element
|
|
122
|
+
"""
|
|
123
|
+
import ast
|
|
124
|
+
|
|
125
|
+
def relation_read(notion_property: dict, column_name: str):
|
|
126
|
+
relations = notion_property.get('relation', [])
|
|
127
|
+
relation_ids = [relation.get('id') for relation in relations]
|
|
128
|
+
if column_name == 'Test table 1':
|
|
129
|
+
if len(relation_ids) > 0:
|
|
130
|
+
return relation_ids[0]
|
|
131
|
+
return ''
|
|
132
|
+
return str(relation_ids)
|
|
133
|
+
|
|
134
|
+
def relation_write(row_value: str, column_name: str):
|
|
135
|
+
if row_value == '':
|
|
136
|
+
return {"relation": []}
|
|
137
|
+
if column_name == 'Test table 1':
|
|
138
|
+
return {"relation": [{"id": row_value}]}
|
|
139
|
+
notion_relations = ast.literal_eval(row_value)
|
|
140
|
+
relation_ids = [{"id": notion_relation} for
|
|
141
|
+
notion_relation in notion_relations]
|
|
142
|
+
|
|
143
|
+
return {"relation": relation_ids}
|
|
144
|
+
```
|
|
145
|
+
**Note:** Ensure that the three input arguments are always in the order (`notion_property`, `column_name`, `switcher`) and that no additional arguments are included beyond these three.
|
|
146
|
+
|
|
147
|
+
My suggestion for changing the read and write functions is to take the original function directly from the [n2p_read_write.py](https://gitlab.com/Jaeger87/notion2pandas/-/blob/main/notion2pandas/n2p_read_write.py) code and modify it until the desired result is achieved. These are the names of the keys associated to each kind of Notion Data:
|
|
148
|
+
|
|
149
|
+
| NotionData | Functions key |
|
|
150
|
+
|------------------|---------------------------------|
|
|
151
|
+
| Title | title |
|
|
152
|
+
| Rich Text | rich_text |
|
|
153
|
+
| Check box | checkbox |
|
|
154
|
+
| Number | number |
|
|
155
|
+
| Date | date |
|
|
156
|
+
| Date Range | date_range |
|
|
157
|
+
| Select | select |
|
|
158
|
+
| Multi Select | multi_select |
|
|
159
|
+
| Status | status |
|
|
160
|
+
| Email | email |
|
|
161
|
+
| People | people |
|
|
162
|
+
| Phone number | phone_number |
|
|
163
|
+
| URL | url |
|
|
164
|
+
| Relation | relation |
|
|
165
|
+
| Roll Up | rollup |
|
|
166
|
+
| Files | files |
|
|
167
|
+
| Formula | formula |
|
|
168
|
+
| String | string |
|
|
169
|
+
| Unique ID | unique_id |
|
|
170
|
+
| Button | button |
|
|
171
|
+
| Created by | created_by |
|
|
172
|
+
| Created time | created_time_read_write_lambdas |
|
|
173
|
+
| Last edited by | last_edited_by |
|
|
174
|
+
| Last edited time | last_edited_time |
|
|
175
|
+
|
|
176
|
+
## Adding and removes rows
|
|
177
|
+
|
|
178
|
+
If you add a row to the dataframe and then update the Notion database from it, Notion2Pandas is capable of adding the new row to the database.
|
|
179
|
+
|
|
180
|
+
(⚠) When adding a new row to the pandas DataFrame, specify an *empty string* as the default value for **PageID** and *zero* for **Row_Hash**
|
|
181
|
+
|
|
182
|
+
If a row is removed, Notion2Pandas will not automatically delete the row during the update. In this case, you can use the method **delete_rows_and_pages** by passing to it the notion2pandas dataframe and the list of indexes of the pages you want to delete; the method will delete the rows in the dataframe and the pages in the notion database.
|
|
183
|
+
|
|
184
|
+
## Adding page data to the dataframe
|
|
185
|
+
|
|
186
|
+
Sometimes, you may want to add data to the dataframe related to the Notion page or even from within the page itself. This data is not directly accessible from the page's entry in the database, but notion2pandas provides a way to include it during the dataframe creation process. For each column you want to add to the dataframe, you can define a function that retrieves the desired data and inserts it into the corresponding column.
|
|
187
|
+
|
|
188
|
+
```python
|
|
189
|
+
from notion2pandas import Notion2PandasClient
|
|
190
|
+
|
|
191
|
+
def get_cover_page(notion_page):
|
|
192
|
+
cover_obj = notion_page.get('cover')
|
|
193
|
+
if cover_obj == None:
|
|
194
|
+
return ''
|
|
195
|
+
cover_type = cover_obj.get('type')
|
|
196
|
+
if cover_type == 'external':
|
|
197
|
+
return cover_obj.get('external').get('url')
|
|
198
|
+
if cover_type == 'file':
|
|
199
|
+
return cover_obj.get('file').get('url')
|
|
200
|
+
return ''
|
|
201
|
+
|
|
202
|
+
def get_icon_page(notion_page):
|
|
203
|
+
icon_obj = notion_page.get('icon')
|
|
204
|
+
if icon_obj == None:
|
|
205
|
+
return ''
|
|
206
|
+
icon_type = icon_obj.get('type')
|
|
207
|
+
if icon_type == 'external':
|
|
208
|
+
return icon_obj.get('external').get('url')
|
|
209
|
+
if icon_type == 'file':
|
|
210
|
+
return icon_obj.get('file').get('url')
|
|
211
|
+
if icon_type == 'emoji':
|
|
212
|
+
return icon_obj.get('emoji')
|
|
213
|
+
return ''
|
|
214
|
+
|
|
215
|
+
def get_image_url(notion_blocks):
|
|
216
|
+
if notion_blocks == None:
|
|
217
|
+
return ''
|
|
218
|
+
for block in notion_blocks.get('results'):
|
|
219
|
+
if block.get('type') == 'image':
|
|
220
|
+
image = block.get('image')
|
|
221
|
+
image_type = image.get('type')
|
|
222
|
+
if image_type == 'file':
|
|
223
|
+
return image.get('file').get('url')
|
|
224
|
+
return ''
|
|
225
|
+
|
|
226
|
+
custom_page_prop = {
|
|
227
|
+
'icon': get_icon_page,
|
|
228
|
+
'cover': get_cover_page
|
|
229
|
+
}
|
|
230
|
+
|
|
231
|
+
custom_block_prop = {
|
|
232
|
+
'inside_image': get_image_url
|
|
233
|
+
}
|
|
234
|
+
|
|
235
|
+
n2p = Notion2PandasClient(auth= 'token')
|
|
236
|
+
df = n2p.from_notion_DB_to_dataframe_kwargs('page_id', columns_from_page = custom_page_prop, columns_from_blocks = custom_block_prop)
|
|
237
|
+
```
|
|
238
|
+
In this example, we add three columns to the dataframe: the icon, the cover image, and the first image found within the page in a block.
|
|
239
|
+
As you can see, the first step is to define the functions that extract the necessary data from the Notion page. Then, we create two dictionaries: one for the columns that retrieve data from the page itself, and another for the columns that pull data from the blocks within the page.
|
|
240
|
+
Afterward, we use the **from_notion_DB_to_dataframe_kwargs** method, which returns the dataframe containing both the database data and the additional data we've specified.
|
|
241
|
+
|
|
242
|
+
Be aware that using either of these parameters will result in **one API call per row** (so using both means **two API calls per row**).
|
|
243
|
+
This can be particularly slow when dealing with very large tables.
|
|
244
|
+
|
|
245
|
+
These columns are considered **read-only**, meaning that changing their values in the dataframe **will not update** them on Notion when using the update_notion_DB_from_dataframe method.
|
|
246
|
+
So if you want to change the values of this data, **use the appropriate methods**.
|
|
247
|
+
|
|
248
|
+
## Notion Executor
|
|
249
|
+
|
|
250
|
+
When notion2pandas needs to execute a method that uses the Notion API, it uses a method called _notionExecutor.
|
|
251
|
+
This method is designed to retry the Notion API call at regular intervals if something goes wrong (network issues, rate limits reached, internal server errors, etc.) until a maximum number of attempts is reached.
|
|
252
|
+
You can set the maximum number of attempts and the interval between attempts through the notion2pandas class constructor as shown in this example.
|
|
253
|
+
|
|
254
|
+
```python
|
|
255
|
+
n2p = Notion2PandasClient(auth= token, secondsToRetry= 20, maxAttemptsExecutioner= 10)
|
|
256
|
+
```
|
|
257
|
+
These arguments are optional and their default values are **30** for **secondsToRetry** and **3** for **maxAttemptsExecutioner**
|
|
258
|
+
|
|
259
|
+
|
|
260
|
+
# Roadmap
|
|
261
|
+
For the upcoming releases, I plan to release:
|
|
262
|
+
|
|
263
|
+
* A test suite with CI/CD for testing and deployment
|
|
264
|
+
* Managing the limit of 2700 API calls in 15 minutes
|
|
265
|
+
* Asynchronous client version of notion2pandas
|
|
266
|
+
* Custom Dataframe
|
|
267
|
+
|
|
268
|
+
# Changelog history
|
|
269
|
+
|
|
270
|
+
You can view the version changelog on the [changelog page](https://gitlab.com/Jaeger87/notion2pandas/-/blob/main/CHANGELOG.md?ref_type=heads).
|
|
271
|
+
|
|
272
|
+
# Support
|
|
273
|
+
Notion2Pandas is an open-source project; anyone can contribute to the project by reporting issues or proposing merge requests. I will commit to evaluating every proposal and responding to all. If you disagree with the decisions made and the direction the project may take, you are free to fork the project, and you will have my blessing!
|
|
274
|
+
|
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
# Notion2Pandas
|
|
2
|
+
<p align="center">
|
|
3
|
+
<img src="https://gitlab.com/Jaeger87/notion2pandas/-/raw/main/readme_assets/logo.png?ref_type=heads" class="center">
|
|
4
|
+
</p> <p align="center">
|
|
5
|
+
|
|
6
|
+
<div align="center">
|
|
7
|
+
<p>
|
|
8
|
+
<a href="https://pypi.org/project/notion2pandas/"><img src="https://gitlab.com/Jaeger87/notion2pandas/-/badges/release.svg" alt="Latest Release"></a>
|
|
9
|
+
</p>
|
|
10
|
+
</div>
|
|
11
|
+
|
|
12
|
+
Notion2Pandas is a Python 3 package that extends the capabilities of the excellent [notion-sdk-py](https://ramnes.github.io/notion-sdk-py/) by [Ramnes](https://github.com/ramnes), It enables the seamless import of a Notion database into a pandas dataframe and vice versa, requiring just a single line of code.
|
|
13
|
+
|
|
14
|
+
## Installation
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
pip install notion2pandas
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Usage
|
|
21
|
+
|
|
22
|
+
<p align="center">
|
|
23
|
+
<img src="https://gitlab.com/Jaeger87/notion2pandas/-/raw/main/readme_assets/notiondb2pandas.gif?ref_type=heads" class="center">
|
|
24
|
+
</p> <p align="left">
|
|
25
|
+
|
|
26
|
+
* As shown in the gif, you just need to import the Notion2PandasClient class.
|
|
27
|
+
```python
|
|
28
|
+
from notion2pandas import Notion2PandasClient
|
|
29
|
+
```
|
|
30
|
+
* Create an instance by passing your authentication token.
|
|
31
|
+
```python
|
|
32
|
+
n2p = Notion2PandasClient(auth=os.environ["NOTION_TOKEN"])
|
|
33
|
+
```
|
|
34
|
+
* Use the 'from_notion_DB_to_dataframe' method to get the data into a dataframe.
|
|
35
|
+
```python
|
|
36
|
+
df = n2p.from_notion_DB_to_dataframe(os.environ["DATABASE_ID"])
|
|
37
|
+
```
|
|
38
|
+
* When you're done working with your dataframe, use the 'update_notion_DB_from_dataframe' method to save the data back to Notion.
|
|
39
|
+
```python
|
|
40
|
+
n2p.update_notion_DB_from_dataframe(os.environ["DATABASE_ID"], df)
|
|
41
|
+
```
|
|
42
|
+
* If you need a queried or sorted database, you can create your filter / sort object [with this structure](https://developers.notion.com/reference/post-database-query) and pass it to the from_notion_DB_to_dataframe method:
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
published_filter = {"filter": {
|
|
46
|
+
"property": "Published",
|
|
47
|
+
"checkbox": {
|
|
48
|
+
"equals": True
|
|
49
|
+
}
|
|
50
|
+
}}
|
|
51
|
+
|
|
52
|
+
df = n2p.from_notion_DB_to_dataframe(os.environ["DATABASE_ID"], published_filter)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## PageID and Row_Hash
|
|
56
|
+
|
|
57
|
+
As you can see, in the pandas dataframe there are two additional columns compared to those in the original database, PageID and Row_Hash. As you can imagine, PageID it's the ID related to the page of that entry in Notion. Row_Hash is a value calculated based on the fields' values of the entry, this value is used by the update_notion_DB_from_dataframe function to determine if a row in the dataframe has been modified, and if not, it avoids making the API call to Notion for that row. Any change to those functions can lead to malfunctions, so please do not change them!
|
|
58
|
+
|
|
59
|
+
## Utility functions
|
|
60
|
+
|
|
61
|
+
Notion2Pandas is a class that extend [Client](https://github.com/ramnes/notion-sdk-py/blob/main/notion_client/client.py) from notion_client, so you can find every feature present in notion_client. In addition to the functions for importing and exporting dataframes, I've added some other convenient functions that wrap the usage of the notion_client functionality and allow them to be used more directly. These are:
|
|
62
|
+
|
|
63
|
+
* get_database_columns(database_ID)
|
|
64
|
+
* create_page(page_ID)
|
|
65
|
+
* update_page(page_ID, **kwargs)
|
|
66
|
+
* retrieve_page(page_ID)
|
|
67
|
+
* delete_page(page_ID)
|
|
68
|
+
* delete_rows_and_pages(dataframe, rows_to_delete_indexes)
|
|
69
|
+
* retrieve_block(block_ID)
|
|
70
|
+
* retrieve_block_children_list(block_ID)
|
|
71
|
+
* update_block(block_ID, field, field_value_updated)
|
|
72
|
+
|
|
73
|
+
## Read Write Functions
|
|
74
|
+
|
|
75
|
+
Notion2Pandas has the ability to transform a Notion database into a Pandas dataframe without having to specify how to parse the data. However, in some cases, the default parsing may not be what you want to achieve. Therefore, it's possible to specify how to parse the data. In Notion2Pandas, each data type in Notion is associated with a tuple consisting of two functions: one for reading the data and the other for writing it.
|
|
76
|
+
|
|
77
|
+
In this example, I'm changing the functions for reading and writing dates so that I can work only with the start date.
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
def date_read_only_start(notion_property):
|
|
81
|
+
return notion_property.get('date').get('start') if notion_property.get('date') is not None else ''
|
|
82
|
+
def date_write_only_start(row_value):
|
|
83
|
+
return {'date': {'start': row_value} if row_value != '' else None}
|
|
84
|
+
|
|
85
|
+
n2p.set_lambdas('date',date_read_only_start, date_write_only_start)
|
|
86
|
+
```
|
|
87
|
+
These function can accept up to three input arguments:
|
|
88
|
+
- `notion_property` for read functions, `row_value` for write functions: the data being read or written.
|
|
89
|
+
- `column_name`: the name of the column, used to identify exactly where to apply the function.
|
|
90
|
+
- `switcher`: the data structure containing the read/write functions, useful for parsing recursive data structures, such as formulas.
|
|
91
|
+
|
|
92
|
+
To override functions for a specific data type but apply them only to certain columns, you can use the `column_name` argument to determine which logic to apply for each field. For example:
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
"""
|
|
96
|
+
Since 'Test table 1' is a relation field limited to just one relation;
|
|
97
|
+
we want a string instead of a list with only one element
|
|
98
|
+
"""
|
|
99
|
+
import ast
|
|
100
|
+
|
|
101
|
+
def relation_read(notion_property: dict, column_name: str):
|
|
102
|
+
relations = notion_property.get('relation', [])
|
|
103
|
+
relation_ids = [relation.get('id') for relation in relations]
|
|
104
|
+
if column_name == 'Test table 1':
|
|
105
|
+
if len(relation_ids) > 0:
|
|
106
|
+
return relation_ids[0]
|
|
107
|
+
return ''
|
|
108
|
+
return str(relation_ids)
|
|
109
|
+
|
|
110
|
+
def relation_write(row_value: str, column_name: str):
|
|
111
|
+
if row_value == '':
|
|
112
|
+
return {"relation": []}
|
|
113
|
+
if column_name == 'Test table 1':
|
|
114
|
+
return {"relation": [{"id": row_value}]}
|
|
115
|
+
notion_relations = ast.literal_eval(row_value)
|
|
116
|
+
relation_ids = [{"id": notion_relation} for
|
|
117
|
+
notion_relation in notion_relations]
|
|
118
|
+
|
|
119
|
+
return {"relation": relation_ids}
|
|
120
|
+
```
|
|
121
|
+
**Note:** Ensure that the three input arguments are always in the order (`notion_property`, `column_name`, `switcher`) and that no additional arguments are included beyond these three.
|
|
122
|
+
|
|
123
|
+
My suggestion for changing the read and write functions is to take the original function directly from the [n2p_read_write.py](https://gitlab.com/Jaeger87/notion2pandas/-/blob/main/notion2pandas/n2p_read_write.py) code and modify it until the desired result is achieved. These are the names of the keys associated to each kind of Notion Data:
|
|
124
|
+
|
|
125
|
+
| NotionData | Functions key |
|
|
126
|
+
|------------------|---------------------------------|
|
|
127
|
+
| Title | title |
|
|
128
|
+
| Rich Text | rich_text |
|
|
129
|
+
| Check box | checkbox |
|
|
130
|
+
| Number | number |
|
|
131
|
+
| Date | date |
|
|
132
|
+
| Date Range | date_range |
|
|
133
|
+
| Select | select |
|
|
134
|
+
| Multi Select | multi_select |
|
|
135
|
+
| Status | status |
|
|
136
|
+
| Email | email |
|
|
137
|
+
| People | people |
|
|
138
|
+
| Phone number | phone_number |
|
|
139
|
+
| URL | url |
|
|
140
|
+
| Relation | relation |
|
|
141
|
+
| Roll Up | rollup |
|
|
142
|
+
| Files | files |
|
|
143
|
+
| Formula | formula |
|
|
144
|
+
| String | string |
|
|
145
|
+
| Unique ID | unique_id |
|
|
146
|
+
| Button | button |
|
|
147
|
+
| Created by | created_by |
|
|
148
|
+
| Created time | created_time_read_write_lambdas |
|
|
149
|
+
| Last edited by | last_edited_by |
|
|
150
|
+
| Last edited time | last_edited_time |
|
|
151
|
+
|
|
152
|
+
## Adding and removes rows
|
|
153
|
+
|
|
154
|
+
If you add a row to the dataframe and then update the Notion database from it, Notion2Pandas is capable of adding the new row to the database.
|
|
155
|
+
|
|
156
|
+
(⚠) When adding a new row to the pandas DataFrame, specify an *empty string* as the default value for **PageID** and *zero* for **Row_Hash**
|
|
157
|
+
|
|
158
|
+
If a row is removed, Notion2Pandas will not automatically delete the row during the update. In this case, you can use the method **delete_rows_and_pages** by passing to it the notion2pandas dataframe and the list of indexes of the pages you want to delete; the method will delete the rows in the dataframe and the pages in the notion database.
|
|
159
|
+
|
|
160
|
+
## Adding page data to the dataframe
|
|
161
|
+
|
|
162
|
+
Sometimes, you may want to add data to the dataframe related to the Notion page or even from within the page itself. This data is not directly accessible from the page's entry in the database, but notion2pandas provides a way to include it during the dataframe creation process. For each column you want to add to the dataframe, you can define a function that retrieves the desired data and inserts it into the corresponding column.
|
|
163
|
+
|
|
164
|
+
```python
|
|
165
|
+
from notion2pandas import Notion2PandasClient
|
|
166
|
+
|
|
167
|
+
def get_cover_page(notion_page):
|
|
168
|
+
cover_obj = notion_page.get('cover')
|
|
169
|
+
if cover_obj == None:
|
|
170
|
+
return ''
|
|
171
|
+
cover_type = cover_obj.get('type')
|
|
172
|
+
if cover_type == 'external':
|
|
173
|
+
return cover_obj.get('external').get('url')
|
|
174
|
+
if cover_type == 'file':
|
|
175
|
+
return cover_obj.get('file').get('url')
|
|
176
|
+
return ''
|
|
177
|
+
|
|
178
|
+
def get_icon_page(notion_page):
|
|
179
|
+
icon_obj = notion_page.get('icon')
|
|
180
|
+
if icon_obj == None:
|
|
181
|
+
return ''
|
|
182
|
+
icon_type = icon_obj.get('type')
|
|
183
|
+
if icon_type == 'external':
|
|
184
|
+
return icon_obj.get('external').get('url')
|
|
185
|
+
if icon_type == 'file':
|
|
186
|
+
return icon_obj.get('file').get('url')
|
|
187
|
+
if icon_type == 'emoji':
|
|
188
|
+
return icon_obj.get('emoji')
|
|
189
|
+
return ''
|
|
190
|
+
|
|
191
|
+
def get_image_url(notion_blocks):
|
|
192
|
+
if notion_blocks == None:
|
|
193
|
+
return ''
|
|
194
|
+
for block in notion_blocks.get('results'):
|
|
195
|
+
if block.get('type') == 'image':
|
|
196
|
+
image = block.get('image')
|
|
197
|
+
image_type = image.get('type')
|
|
198
|
+
if image_type == 'file':
|
|
199
|
+
return image.get('file').get('url')
|
|
200
|
+
return ''
|
|
201
|
+
|
|
202
|
+
custom_page_prop = {
|
|
203
|
+
'icon': get_icon_page,
|
|
204
|
+
'cover': get_cover_page
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
custom_block_prop = {
|
|
208
|
+
'inside_image': get_image_url
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
n2p = Notion2PandasClient(auth= 'token')
|
|
212
|
+
df = n2p.from_notion_DB_to_dataframe_kwargs('page_id', columns_from_page = custom_page_prop, columns_from_blocks = custom_block_prop)
|
|
213
|
+
```
|
|
214
|
+
In this example, we add three columns to the dataframe: the icon, the cover image, and the first image found within the page in a block.
|
|
215
|
+
As you can see, the first step is to define the functions that extract the necessary data from the Notion page. Then, we create two dictionaries: one for the columns that retrieve data from the page itself, and another for the columns that pull data from the blocks within the page.
|
|
216
|
+
Afterward, we use the **from_notion_DB_to_dataframe_kwargs** method, which returns the dataframe containing both the database data and the additional data we've specified.
|
|
217
|
+
|
|
218
|
+
Be aware that using either of these parameters will result in **one API call per row** (so using both means **two API calls per row**).
|
|
219
|
+
This can be particularly slow when dealing with very large tables.
|
|
220
|
+
|
|
221
|
+
These columns are considered **read-only**, meaning that changing their values in the dataframe **will not update** them on Notion when using the update_notion_DB_from_dataframe method.
|
|
222
|
+
So if you want to change the values of this data, **use the appropriate methods**.
|
|
223
|
+
|
|
224
|
+
## Notion Executor
|
|
225
|
+
|
|
226
|
+
When notion2pandas needs to execute a method that uses the Notion API, it uses a method called _notionExecutor.
|
|
227
|
+
This method is designed to retry the Notion API call at regular intervals if something goes wrong (network issues, rate limits reached, internal server errors, etc.) until a maximum number of attempts is reached.
|
|
228
|
+
You can set the maximum number of attempts and the interval between attempts through the notion2pandas class constructor as shown in this example.
|
|
229
|
+
|
|
230
|
+
```python
|
|
231
|
+
n2p = Notion2PandasClient(auth= token, secondsToRetry= 20, maxAttemptsExecutioner= 10)
|
|
232
|
+
```
|
|
233
|
+
These arguments are optional and their default values are **30** for **secondsToRetry** and **3** for **maxAttemptsExecutioner**
|
|
234
|
+
|
|
235
|
+
|
|
236
|
+
# Roadmap
|
|
237
|
+
For the upcoming releases, I plan to release:
|
|
238
|
+
|
|
239
|
+
* A test suite with CI/CD for testing and deployment
|
|
240
|
+
* Managing the limit of 2700 API calls in 15 minutes
|
|
241
|
+
* Asynchronous client version of notion2pandas
|
|
242
|
+
* Custom Dataframe
|
|
243
|
+
|
|
244
|
+
# Changelog history
|
|
245
|
+
|
|
246
|
+
You can view the version changelog on the [changelog page](https://gitlab.com/Jaeger87/notion2pandas/-/blob/main/CHANGELOG.md?ref_type=heads).
|
|
247
|
+
|
|
248
|
+
# Support
|
|
249
|
+
Notion2Pandas is an open-source project; anyone can contribute to the project by reporting issues or proposing merge requests. I will commit to evaluating every proposal and responding to all. If you disagree with the decisions made and the direction the project may take, you are free to fork the project, and you will have my blessing!
|
|
250
|
+
|