anta-database 0.1__tar.gz

anta_database-0.1/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2025 Antoine Hermant
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

anta_database-0.1/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.4
+Name: anta_database
+Version: 0.1
+Summary: SQLite database for the AntArchitecture Community Data
+Home-page: https://github.com/antoinehermant/anta_database
+Author: Antoine Hermant
+Author-email: antoine.hermant@etik.com
+License: MIT
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: pyproj
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Dynamic: author
+Dynamic: author-email
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary

anta_database-0.1/README.md

@@ -0,0 +1,240 @@
+
+# Table of Contents
+
+1.  [SQLite Database](#org7c02cbe)
+    1.  [Datasets currently included:](#orge88c16b)
+2.  [Key Features](#orgda9a069)
+3.  [Installation](#org0ae2129)
+4.  [How to use with examples](#org94cd2fd)
+    1.  [Browsing Metadata](#org89a166a)
+    2.  [Visualization](#orgf9cf6be)
+    3.  [Generate Data](#org6e8f9a4)
+5.  [Advanced: Managing the database](#org213a1d3)
+    1.  [Reindexing](#orge03e052)
+    2.  [(Re)compile the database](#org841514e)
+6.  [Support and contact](#org87c7675)
+7.  [Contribution](#org2896907)
+8.  [Acknowledgments](#org7ea1cbb)
+
+This Python module provides an efficient SQLite database for browsing, visualizing and processing Internal Reflecting Horizons (isochrones) across Antarctica, curated by the AntArchitecture action group. It is specifically designed for ice dynamic modelers who need a fast, memory-efficient data structure to constrain their models.
+
+
+<a id="org7c02cbe"></a>
+
+# SQLite Database
+
+The database uses SQLite for efficient indexing. Data is sourced from the associated DOIs and stored as binary DataFrame files for each layer (IRH) and trace ID. This structure enables:
+
+-   Browsing by author (region), layer age, or trace ID.
+-   Memory efficiency: Only relevant data is read when needed.
+-   Fast read performance: Lightweight and optimized for speed.
+
+
+<a id="orge88c16b"></a>
+
+## Datasets currently included:
+
+-   Winter et al. 2018, (<https://doi.org/10.1594/PANGAEA.895528>)
+-   Cavitte et al. 2020, (<https://doi.org/10.15784/601411>)
+-   Beem et al. 2021, (<https://doi.org/10.15784/601437>)
+-   Wang et al. 2023, (<https://doi.org/10.1594/PANGAEA.958462>)
+-   Sanderson et al. 2024, (<https://doi.org/10.5285/cfafb639-991a-422f-9caa-7793c195d316>)
+-   Franke et al. 2025, (<https://doi.org/10.1594/PANGAEA.973266>)
+
+
+<a id="orgda9a069"></a>
+
+# Key Features
+
+-   Efficient SQLite indexing
+-   Quick visualization on Antarctica map
+-   Generate lazy data for later use
+
+
+<a id="org0ae2129"></a>
+
+# Installation
+
+1- Clone this repository:
+
+    git clone git@github:antoinehermant/anta_database.git
+    cd anta_database
+
+2- Create a specific python environment (optional but recommended):
+
+    '''Using conda'''
+    conda create -n anta_database
+    conda activate anta_database
+    conda install pip
+    
+    '''Using venv'''
+    python -m venv /path/to/environment/anta_database # Replace the path with your desired environment path
+    source /path/to/environment/anta_database
+
+3- Install the Python module:
+
+    pip install .
+
+Note: The database is not included in this repository. Contact me for access.
+
+
+<a id="org94cd2fd"></a>
+
+# How to use with examples
+
+Also see the Jupyter Notebook &rsquo;example.ipynb&rsquo; for a more integrated tutorial.
+
+
+<a id="org89a166a"></a>
+
+## Browsing Metadata
+
+-   First, initialize the Database class:
+
+    from anta_database import Database
+    database = Database('/path/to/AntADatabase/') # insert the path of the downloaded database, absolute path is recommanded
+
+Then use the query() function to browse the database. query() without argument will return all metadata from the database:
+
+    database.query()
+
+By default, a dictionary with all available trace IDs is included in the query results. This can seem a lot, but it is actually useful for grepping traces with descriptive names such as &rsquo;DC<sub>LDC</sub><sub>DIVIDE</sub>&rsquo;.
+
+-   Refine your search either by author, age, or trace<sub>id</sub>, or by combining arguments. Lists are also allowed:
+
+    database.query(author='Cavitte_2020') # all data from Cavitte et al. 2020
+    database.query(age='38100') # all datasets with the 38.1ka isochrone
+    database.query(var='IceThk') # all datasets with IceThk variable
+    database.query(trace_id='DC_LDC_DIVIDE') # all layers with the trace ID DC_LDC_DIVIDE
+    database.query(author=['Franke_2025', 'Winter_2018'], age='38100') # example of multiple criteria
+
+Note: &rsquo;IceThk&rsquo;, &rsquo;SurfElev&rsquo; and &rsquo;BedElev&rsquo; are possible extra variables included in the database when the datasets contain the relevant data.
+
+
+<a id="orgf9cf6be"></a>
+
+## Visualization
+
+Use the results of the query in the plotting functions:
+
+    results = database.query(age='38100')
+    database.plotXY(results,
+                    downscale_factor=1000, # downscale the datasets n times, which makes no visual difference but it is much lighter
+                    title='AntArchitecture 38.1ka layer',
+                    xlim=(-500, 2400), ylim=(-2200, 2200), # set the plot extent in km
+                    scale_factor=0.5, # adjust the size of the plot
+                    latex=True, # use latex compilers for plotting if you have them installed on your system
+                    # save='AntA_38ka.pdf', # Uncomment to save the figure, otherwise it we visualize with pyplot
+                    )
+
+![img](./anta_database/figures/AntA_38ka.png "Example figure")
+
+
+<a id="org6e8f9a4"></a>
+
+## Generate Data
+
+Note: This part could be developed further in the future if there is the need. But for now, I am designing a separate Python module for constraining my ice sheet model of use, which is tailored to this database and other parallel processing libraries.
+
+The data<sub>generator</sub>() function reads the query and &rsquo;yield&rsquo; the dataframes for later use.
+Here is a quick example of how this can be used for computing the mean layer depth:
+
+    results = database.query(age='38100')
+    lazy_dfs = database.data_generator(results)
+    
+    mean_depth_trs = []
+    for df, md in lazy_dfs:
+        mean_depth_trs.append(np.mean(df['38100']))
+    
+    mean_depth = np.mean(mean_depth_trs)
+    print('The mean depth of the 38ka isochrone across East Antarctica is', mean_depth, 'm')
+
+The downscale<sub>factor</sub> argument in the data<sub>generator</sub> reduces the size of the generated data by n times. This is useful when dealing with large data where the downscaling has a neglectable influence on the results:
+
+    lazy_dfs = database.data_generator(results, downscale_factor=10)
+
+Note: In the database, the file paths are relative to the database path. When initiating the database, you provide the absolute path to the database. So the data<sub>generator</sub> function uses the relative path given in the results from the query and the absolute path to the database to read the files. The second argument returned by data<sub>generator</sub> (&rsquo;md&rsquo; in the example above) is the unique metadata for the current df.
+
+
+<a id="org213a1d3"></a>
+
+# Advanced: Managing the database
+
+
+<a id="orge03e052"></a>
+
+## Reindexing
+
+You may want for example to update the age of the layers in a particular dataset.
+For this, you just need to modify the ages in the file called &rsquo;IRH<sub>ages.tab</sub>&rsquo; located under a dataset directory. Then, reindex with the IndexDatabase class:
+
+    from anta_database import IndexDatabase
+    
+    db_path = '/path/to/root/of/the/database/' # path to 'AntADatabase/'
+    indexing = IndexDatabase(db_path)
+    indexing.index_database()
+
+
+<a id="org841514e"></a>
+
+## (Re)compile the database
+
+You can (re)compile the database, if for example you modify some data in the raw directories or if you add a dataset.
+For this, make sure to follow the structure:
+
+    AntADatabase/
+    ├── AntADatabase.db
+    ├── database_index.csv #List of directories to index: Author_YYYY,Author et al. YYYY,doi
+    ├── Author_YYYY
+        ├── IRH_ages.tab #IRH file names without .ext followed by there respective age in years
+        ├── original_new_column_names.csv #first row: names of columns to keep from raw files, second row: how the columns should be renamed
+        ├── raw/
+        └── pkl/
+
+Then use the CompileDatabase class to compile the database.
+
+    from anta_database import CompileDatabase
+    
+    dir_path_list = [ # list of the dataset subdirectories to compile
+        './Winter_2018',
+        './Sanderson_2024',
+        './Franke_2025',
+        './Cavitte_2020',
+        './Beem_2021',
+    ]
+    
+    compiler = CompileDatabase(dir_path_list)
+    compiler.compile()
+
+Then reindex (see above).
+By default, it assumes that the files in raw/ are sorted by IRH (one file = one layer and multiple traces). If the files are sorted the other way around (one file = one trace and multiple layers), you can set file<sub>type</sub>=&rsquo;trace&rsquo; in CompileDatabase(). Furthermore, if the depth is not given in meters but TWT, you should set the wave<sub>speed</sub> (units should match values in the file) for conversion and firn<sub>correction</sub> (meters):
+
+    dir_path = './Wang_2023'
+    compiler = CompileDatabase(dir_path, file_type='trace', wave_speed=0.1685, firn_correction=15.5)
+    compiler.compile()
+
+
+<a id="org87c7675"></a>
+
+# Support and contact
+
+You can email me for downloading the database: antoine.hermant@unibe.ch
+
+Feel free to raise an issue on the GitHub if you find any bug or if you would like a feature added.
+
+
+<a id="org2896907"></a>
+
+# Contribution
+
+If you like this database and wish to help me develop this module, do not hesitate to contact me. You should then fork the repo, build feature branches and pull request. That would be much appreciated!
+
+
+<a id="org7ea1cbb"></a>
+
+# Acknowledgments
+
+I am developing this tool as part of my PhD project, which is funded by the Swiss National Science Foundation (grant no. 211542, Project CHARIBDIS)
+Any data used through this database should be cited at source. For this, use the DOI provided in the metadata.
+If you used this tool for your work and this was useful, please cite this repo, so other people get to know that it exists.
+

anta_database-0.1/anta_database/__init__.py

@@ -0,0 +1,3 @@
+from .database import *
+from .indexing import *
+from .compiling import *

anta_database-0.1/anta_database/anta_database.py

@@ -0,0 +1,3 @@
+from database import *
+from indexing import *
+from compiling import *

anta_database-0.1/anta_database/compiling/__init__.py

@@ -0,0 +1 @@
+from .compile import *

anta_database-0.1/anta_database/compiling/compile.py

@@ -0,0 +1,192 @@
+import os
+import sys
+import pandas as pd
+import numpy as np
+import glob
+from pyproj import Transformer
+from typing import Union
+from multiprocessing import Pool, cpu_count
+
+class CompileDatabase:
+    def __init__(self, dir_list: Union[str, list[str]], file_type: str = 'layer', wave_speed: Union[None, float] = None, firn_correction: Union[None, float] = None) -> None:
+        self.dir_list = dir_list
+        self.wave_speed = wave_speed
+        self.firn_correction = firn_correction
+        self.file_type = file_type
+
+    def get_dict_ages(self, tab_file) -> dict:
+        ages = pd.read_csv(tab_file, header=None, sep='\t', names=['file', 'age'])
+        return dict(zip(ages['file'], ages['age']))
+
+    def compile(self, cpus: int = cpu_count()-1) -> None:
+        if not isinstance(self.dir_list, list):
+            self.dir_list = [self.dir_list]
+
+        num_tasks = len(self.dir_list)
+        num_workers = min(num_tasks, cpus)
+
+        print('\n',
+                'Will start compiling', len(self.dir_list), 'directories\n'
+                '\n   ', num_workers, 'worker(s) allocated out of', cpu_count(), 'available cpus\n')
+
+        if num_workers > 1:
+            with Pool(num_workers) as pool:
+                pool.map(self._compile, self.dir_list)
+        else:
+            for dir_path in self.dir_list:
+                self._compile(dir_path=dir_path)
+
+    def _compile(self, dir_path: str) -> None:
+        raw_files = glob.glob(f'{dir_path}/raw/*.*')
+
+        _, ext = os.path.splitext(raw_files[0])
+
+        ages = self.get_dict_ages(f'{dir_path}/IRH_ages.tab')
+        original_new_columns = pd.read_csv(f'{dir_path}/original_new_column_names.csv')
+
+        if ext == '.tab':
+            sep='\t'
+        elif ext == '.csv':
+            sep=','
+        else:
+            print('File type not supported, exiting ...')
+            sys.exit()
+
+        for i, file in enumerate(raw_files):
+            print('Processing', file, f'({i}/{len(raw_files)})')
+            ds = pd.read_csv(file, comment="#", header=0, sep=sep)
+            _, file_name = os.path.split(file)
+            file_name_, ext = os.path.splitext(file_name)
+
+            ds = ds[ds.columns.intersection(original_new_columns.columns)]
+            ds.columns = original_new_columns[ds.columns].iloc[0].values  # renaming the columns
+
+            if 'IceThk' in ds.columns and 'SurfElev' in ds.columns and not 'BedElev' in ds.columns:
+                ds['BedElev'] = ds['SurfElev'] - ds['IceThk']
+            if 'IceThk' in ds.columns and 'BedElev' in ds.columns and not 'SurfElev' in ds.columns:
+                ds['SurfElev'] = ds['BedElev'] + ds['IceThk']
+            if 'SurfElev' in ds.columns and 'BedElev' in ds.columns and not 'IceThk' in ds.columns:
+                ds['IceThk'] = ds['SurfElev'] - ds['BedElev']
+
+            if self.wave_speed:
+                for var in ['IceThk', 'BedElev']:
+                    if var in ds.columns:
+                        ds[var] *= self.wave_speed
+            if self.firn_correction:
+                for var in ['IceThk', 'BedElev']:
+                    if var in ds.columns:
+                        ds[var] += self.firn_correction
+
+            if 'x' not in ds.columns and 'y' not in ds.columns:
+                if 'lon' in ds.columns and 'lat' in ds.columns:
+                    transformer = Transformer.from_proj(
+                        "EPSG:4326",  # source: WGS84 (lon/lat)
+                        "+proj=stere +lon_0=0 +lat_0=-90 +lat_ts=-71 +datum=WGS84 +units=m +no_defs",  # target: polar
+                        always_xy=True
+                    )
+                    ds['x'], ds['y'] = transformer.transform(ds['lon'].values, ds['lat'].values)
+            elif 'lon' not in ds.columns and 'lat' not in ds.columns:
+                if 'x' in ds.columns and 'y' in ds.columns:
+                    inverse_transformer = Transformer.from_proj(
+                        "+proj=stere +lon_0=0 +lat_0=-90 +lat_ts=-71 +datum=WGS84 +units=m +no_defs",  # source: polar
+                        "EPSG:4326",  # target: WGS84 (lon/lat)
+                        always_xy=True
+                    )
+                    ds['lon'], ds['lat'] = inverse_transformer.transform(ds['x'].values, ds['y'].values)
+            elif 'lon' in ds.columns and 'lat' in ds.columns and 'x' in ds.columns and 'y' in ds.columns:
+                pass
+            else:
+                print('No coordinates found in the dataset, exiting ....')
+                sys.exit()
+
+            if self.file_type == 'layer':
+                age = str(ages[file_name_])
+                ds = ds.rename(columns={'IRHdepth': age})
+                if self.wave_speed:
+                    ds[age] *= self.wave_speed
+                if self.firn_correction:
+                    ds[age] += self.firn_correction
+
+                ds['Trace_ID'] = ds['Trace_ID'].astype(str)
+                ds['Trace_ID'] = ds['Trace_ID'].str.replace(r'/\s+', '_') # Replace slashes with underscores, otherwise the paths can get messy
+                ds['Trace_ID'] = ds['Trace_ID'].str.replace('/', '_')
+
+                ds.set_index('Trace_ID', inplace=True)
+
+                for trace_id in np.unique(ds.index):
+                    ds_trace = ds.loc[trace_id].copy()
+                    if 'distance' not in ds_trace.columns:
+                        x = ds_trace[['x', 'y']]
+                        distances = np.sqrt(np.sum(np.diff(x, axis=0)**2, axis=1))
+                        cumulative_distance = np.concatenate([[0], np.cumsum(distances)])
+                        ds_trace['distance'] = cumulative_distance
+
+                    ds_trace_file = f'{dir_path}/pkl/{trace_id}/{file_name_}.pkl'
+                    os.makedirs(f'{dir_path}/pkl/{trace_id}' , exist_ok=True)
+                    ds_trace.to_pickle(ds_trace_file)
+                    print(ds_trace_file)
+
+                    for var in ['IceThk', 'BedElev', 'SurfElev']:
+                        if var in ds.columns:
+                            ds_var = ds_trace[['x', 'y', 'distance', var]]
+                            ds_var_file = f'{dir_path}/pkl/{trace_id}/{var}.pkl'
+                            if os.path.exists(ds_var_file):
+                                var_data = pd.read_pickle(ds_var_file)
+                                merged_data = pd.concat([var_data, ds_var]).drop_duplicates(subset=['x', 'y'])
+
+                                merged_data.to_pickle(ds_var_file)
+                                print(ds_var_file)
+                            else:
+                                ds_var.to_pickle(ds_var_file)
+                                print(ds_var_file)
+
+            elif self.file_type == 'trace':
+                if 'distance' not in ds.columns:
+                    x = ds[['x', 'y']]
+                    distances = np.sqrt(np.sum(np.diff(x, axis=0)**2, axis=1))
+                    cumulative_distance = np.concatenate([[0], np.cumsum(distances)])
+                    ds['distance'] = cumulative_distance
+
+                trace_id = file_name_
+                os.makedirs(f'{dir_path}/pkl/{trace_id}' , exist_ok=True)
+
+                for var in ['IceThk', 'BedElev', 'SurfElev']:
+                    if var in ds.columns:
+                        ds_var = ds[['x', 'y', 'distance', var]]
+                        ds_var_file = f'{dir_path}/pkl/{trace_id}/{var}.pkl'
+                        if os.path.exists(ds_var_file):
+                            var_data = pd.read_pickle(ds_var_file)
+                            merged_data = pd.concat([var_data, ds_var]).drop_duplicates(subset=['x', 'y'])
+
+                            merged_data.to_pickle(ds_var_file)
+                            print(ds_var_file)
+                        else:
+                            ds_var.to_pickle(ds_var_file)
+                            print(ds_var_file)
+
+                ages = {key: ages[key] for key in ds.columns if key in ages}
+
+                for IRH in ages:
+                    age = str(ages.get(IRH))
+                    ds_IRH = ds[IRH]
+                    ds_IRH = pd.DataFrame({
+                        'lon': ds['lon'],
+                        'lat': ds['lat'],
+                        'x': ds['x'],
+                        'y': ds['y'],
+                        'distance': ds['distance'],
+                        age: ds_IRH,
+                    })
+                    if self.wave_speed:
+                        ds_IRH[age] *= self.wave_speed
+                    if self.firn_correction:
+                        ds_IRH += self.firn_correction
+
+                    for var in ['IceThk', 'BedElev', 'SurfElev']:
+                        if var in ds.columns:
+                            ds_IRH[var] = ds[var]
+
+                    ds_trace_file = f'{dir_path}/pkl/{trace_id}/{IRH}.pkl'
+
+                    ds_IRH.to_pickle(ds_trace_file)
+                    print(ds_trace_file)

anta_database-0.1/anta_database/database/__init__.py

@@ -0,0 +1 @@
+from .database import *

anta_database-0.1/anta_database/database/database.py

@@ -0,0 +1,288 @@
+import os, sys
+import sqlite3
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+from matplotlib.colors import LinearSegmentedColormap
+import seaborn as sns
+from typing import Union, List, Dict, Tuple
+
+class Database:
+    def __init__(self, database_dir: str, file_db: str = 'AntADatabase.db'):
+        module_dir = os.path.dirname(os.path.abspath(__file__))
+        self.gl = pd.read_pickle(os.path.join(module_dir, 'plotting', 'GL.pkl'))
+        self.db_dir = database_dir
+        self.file_db = file_db
+        self.file_db_path = os.path.join(self.db_dir, file_db)
+        self.md = None
+
+    def _build_query_and_params(self, age: Union[None, str, List[str]]=None, var: Union[None, str, List[str]]=None, author: Union[None, str, List[str]]=None, line: Union[None, str, List[str]]=None, select_clause='') -> Tuple[str, List[Union[str, int]]]:
+        """
+        Helper method to build the SQL query and parameters for filtering.
+        Returns the query string and parameters list.
+        """
+        query = f'''
+            SELECT {select_clause}
+            FROM datasets d
+            JOIN authors a ON d.author = a.id
+        '''
+        conditions = []
+        params = []
+
+        for field, column in [
+            (age, 'd.age'),
+            (var, 'd.var'),
+            (author, 'a.name'),
+            (line, 'd.trace_id')
+        ]:
+            if field is not None:
+                if isinstance(field, list):
+                    placeholders = ','.join(['?'] * len(field))
+                    conditions.append(f'{column} IN ({placeholders})')
+                    params.extend(field)
+                else:
+                    conditions.append(f'{column} = ?')
+                    params.append(field)
+
+        if conditions:
+            query += ' WHERE ' + ' AND '.join(conditions)
+
+        query += 'ORDER BY CAST(d.age AS INTEGER) ASC'
+        return query, params
+
+    def _get_file_metadata(self, file_path) -> Dict:
+        """
+        Helper method to build the SQL query and parameters for filtering.
+        Returns the query string and parameters list.
+        """
+        select_clause = 'a.name, d.age, d.var, d.trace_id, d.file_path'
+        query = f'''
+            SELECT {select_clause}
+            FROM datasets d
+            JOIN authors a ON d.author = a.id
+        '''
+        conditions = []
+        params = []
+
+        if file_path is not None:
+            if isinstance(file_path, list):
+                placeholders = ','.join(['?'] * len(file_path))
+                conditions.append(f'd.file_path IN ({placeholders})')
+                params.extend(file_path)
+            else:
+                conditions.append(f'd.file_path = ?')
+                params.append(file_path)
+
+        if conditions:
+            query += ' WHERE ' + ' AND '.join(conditions)
+
+        conn = sqlite3.connect(self.file_db_path)
+        cursor = conn.cursor()
+        cursor.execute(query, params)
+        results = cursor.fetchall()
+        conn.close()
+
+        metadata = {
+            'author': results[0][0],
+            'age': results[0][1],
+            'var': results[0][2],
+            'trace_id': results[0][3],
+            'file_path': results[0][4],
+            'database_path': self.db_dir,
+            'file_db': self.file_db,
+        }
+        return metadata
+
+    def query(self, age: Union[None, str, List[str]]=None, var: Union[None, str, List[str]]=None,author: Union[None, str, List[str]]=None, trace_id: Union[None, str, List[str]]=None) -> Dict:
+        select_clause = 'a.name, a.citation, a.doi, d.age, d.var, d.trace_id'
+        query, params = self._build_query_and_params(age, var, author, trace_id, select_clause)
+
+        conn = sqlite3.connect(self.file_db_path)
+        cursor = conn.cursor()
+        cursor.execute(query, params)
+        results = cursor.fetchall()
+        conn.close()
+
+        metadata = {
+            'author': [],
+            'age': [],
+            'var': [],
+            'reference': [],
+            'doi': [],
+            'trace_id': [],
+            '_query_params': {'age': age, 'var': var, 'author': author, 'trace_id': trace_id},
+            'database_path': self.db_dir,
+            'file_db': self.file_db,
+        }
+        ages_list = []
+        vars_list = []
+        for author_name, citations, doi, ages, vars, trace_id in results:
+            metadata['author'].append(author_name)
+            metadata['reference'].append(citations)
+            metadata['doi'].append(doi)
+            metadata['trace_id'].append(trace_id)
+            # Check if the age is numeric
+            if ages is not None and ages.isdigit():
+                ages_list.append(int(ages))
+            if vars is not None:
+                vars_list.append(vars)
+        metadata['age'] = sorted({str(age) for age in set(ages_list)}, key=int)
+        metadata['var'] = sorted(set(vars_list))
+        metadata['author'] = list(dict.fromkeys(metadata['author']))
+        metadata['reference'] = list(dict.fromkeys(metadata['reference']))
+        metadata['doi'] = list(dict.fromkeys(metadata['doi']))
+        metadata['trace_id'] = list(set(metadata['trace_id']))
+
+        self.md = metadata
+        return metadata
+
+    def _get_file_paths_from_metadata(self, metadata) -> List:
+        query_params = metadata.get('_query_params', {})
+        age = query_params.get('age')
+        var = query_params.get('var')
+        author = query_params.get('author')
+        line = query_params.get('trace_id')
+
+        select_clause = 'd.file_path'
+        query, params = self._build_query_and_params(age, var, author, line, select_clause)
+
+        conn = sqlite3.connect(self.file_db_path)
+        cursor = conn.cursor()
+        cursor.execute(query, params)
+        file_paths = [row[0] for row in cursor.fetchall()]
+        conn.close()
+
+        return file_paths
+
+    def data_generator(self, metadata: Union[None, Dict] = None, data_dir: Union[None, str] = None, downscale_factor: Union[None, int] = None, downsample_distance: Union[None, float, int] = None):
+        """
+        Generates DataFrames and their associated author names from the database based on the provided metadata.
+
+        This method queries the database using the filter parameters stored in the metadata,
+        retrieves the file paths and author names, and yields each DataFrame along with its author.
+
+        Args:
+            metadata: the results from the query()
+        """
+        if metadata:
+            md = metadata
+        elif self.md:
+            md = self.md
+        else:
+            print('Please provide metadata of the files you want to generate the data from. Exiting ...')
+            sys.exit()
+
+        query_params = md.get('_query_params', {})
+        age = query_params.get('age')
+        var = query_params.get('var')
+        author = query_params.get('author')
+        trace_id = query_params.get('trace_id')
+
+        select_clause = 'DISTINCT d.file_path, d.age'
+        query, params = self._build_query_and_params(age, var, author, trace_id, select_clause)
+
+        conn = sqlite3.connect(self.file_db_path)
+        cursor = conn.cursor()
+        cursor.execute(query, params)
+        results = cursor.fetchall()
+        conn.close()
+
+        if data_dir:
+            data_dir = data_dir
+        elif self.db_dir:
+            data_dir = self.db_dir
+        else:
+            print('No data dir provided, do not know where to look for data, exiting ...')
+            sys.exit()
+
+        for file_path, age in results:
+            df = pd.read_pickle(os.path.join(data_dir, file_path))
+            if downscale_factor:
+                df = df[::downscale_factor]
+            if downsample_distance:
+                df['bin'] = np.floor(df['distance'] / downsample_distance) * downsample_distance
+                df = df.groupby('bin').mean().reset_index()
+                df.drop(columns=['bin'], inplace=True)
+            metadata = self._get_file_metadata(file_path)
+            yield df, metadata
+
+    def custom_cmap(self):
+
+        flare = plt.colormaps['magma_r']  # Sequential (yellow/orange/red)
+        crest = plt.colormaps['mako']  # Diverging (blue/white/red)
+
+        flare_colors = flare(np.linspace(0, 0.8, 256))  # Shape: (256, 4) RGBA
+        crest_colors = crest(np.linspace(0.2, 1, 256))  # Shape: (256, 4) RGBA
+        combined_colors = np.vstack((flare_colors, crest_colors))
+        custom_cmap = LinearSegmentedColormap.from_list(
+            'flare_crest',  # Name your colormap
+            combined_colors,
+            N=512  # Double the resolution (256 + 256)
+        )
+        return custom_cmap
+
+
+    def plotXY(self,
+               metadata: Union[None, Dict] = None,
+               downscale_factor: Union[None, int] = None,
+               title: str = '',
+               xlim: tuple = (None, None),
+               ylim: tuple = (None, None),
+               scale_factor: float = 1.0,
+               latex:bool = False,
+               save: Union[str, None] = None) -> None:
+
+        if metadata:
+            metadata = metadata
+        elif self.md:
+            metadata = self.md
+        else:
+            print('Please provide metadata of the files you want to generate the data from. Exiting ...')
+            sys.exit()
+
+        cmaps = self.custom_cmap()
+        authors = list(metadata['author'])
+        color_indices = np.linspace(0.1, 0.9, len(authors))
+        colors = {author: cmaps(i) for i, author in zip(color_indices, authors)}
+
+        if latex:
+            from matplotlib import rc
+            rc('font', **{'family': 'serif', 'serif': ['Computer Modern']})
+            rc('text', usetex=True)
+
+        plt.figure()
+        plt.plot(self.gl.x/1000, self.gl.y/1000, linewidth=1, color='k')
+        for df, md in self.data_generator(metadata, downscale_factor=downscale_factor):
+            author = md['author']
+            plt.plot(df.x/1000, df.y/1000, linewidth=0.8, color=colors[author])
+
+        ax = plt.gca()
+        x0, x1 = ax.get_xlim() if xlim == (None, None) else xlim
+        y0, y1 = ax.get_ylim() if ylim == (None, None) else ylim
+
+        for author in authors:
+            citation = self.query(author=author)['reference']
+            plt.plot([], [], color=colors[author], label=citation)
+        plt.legend(loc='lower left')
+
+        x_extent = x1 - x0
+        y_extent = y1 - y0
+
+        aspect_ratio = y_extent / x_extent
+
+        # Set the figure size based on the aspect ratio and a scale factor
+        plt.gcf().set_size_inches(8 * scale_factor, 8 * aspect_ratio * scale_factor)
+        ax.set_xlabel('x [km]')
+        ax.set_ylabel('y [km]')
+        ax.set_xlim(*xlim)
+        ax.set_ylim(*ylim)
+        # self.scale_axes(ax, factor=1000)
+        plt.title(title, fontsize=30*scale_factor)
+        plt.tight_layout()
+        if save:
+            plt.savefig(save, dpi=200)
+            plt.close()
+            print('Figure saved as', save)
+        else:
+            plt.show()
+            plt.close()

anta_database-0.1/anta_database/database/plotting/__init__.py

@@ -0,0 +1 @@
+

anta_database-0.1/anta_database/indexing/__init__.py

@@ -0,0 +1 @@
+from .index_database import *

anta_database-0.1/anta_database/indexing/index_database.py

@@ -0,0 +1,101 @@
+import os
+import glob
+import pandas as pd
+import sqlite3
+
+class IndexDatabase:
+    def __init__(self, database_dir: str, file_db: str = 'AntADatabase.db', index: str = 'database_index.csv'):
+        self.db_dir = database_dir
+        self.file_db = os.path.join(self.db_dir, file_db)
+        self.file_index = index
+        self.index = pd.read_csv(f'{self.db_dir}/{self.file_index}', header=0)
+
+    def get_dict_ages(self, tab_file) -> dict:
+        ages = pd.read_csv(tab_file, header=None, sep='\t', names=['file', 'age'])
+        return dict(zip(ages['file'], ages['age']))
+
+    def get_dict_vars(self, tab_file) -> dict:
+        ages = pd.read_csv(tab_file, header=None, sep='\t', names=['file', 'var'])
+        return dict(zip(ages['file'], ages['var']))
+
+    def index_database(self):
+        Authors_ages = {}
+        for _, row in self.index.iterrows():
+            ages = self.get_dict_ages(f"{self.db_dir}/{row.directory}/IRH_ages.tab")
+            Authors_ages.update({f"{row.directory}": ages})
+
+        Extra_vars = {}
+        for _, row in self.index.iterrows():
+            vars = self.get_dict_ages(f"{self.db_dir}/{row.directory}/vars.tab")
+            Extra_vars.update({f"{row.directory}": vars})
+
+        if os.path.exists(self.file_db):
+            os.remove(self.file_db)
+
+        conn = sqlite3.connect(self.file_db)
+        cursor = conn.cursor()
+        # Create a table for original reference to datasets
+        cursor.execute('''
+            CREATE TABLE IF NOT EXISTS authors (
+                id INTEGER PRIMARY KEY,
+                name TEXT UNIQUE,
+                citation TEXT,
+                doi TEXT
+            )
+        ''')
+
+        for _, row in self.index.iterrows():
+            try:
+                cursor.execute(
+                    'INSERT INTO authors (name, citation, doi) VALUES (?, ?, ?)',
+                    (row.directory, row.citation, row.doi)
+                )
+            except sqlite3.IntegrityError:
+                # Author already exists, skip
+                continue
+
+        # Create a table to store the metadata for each dataset
+        cursor.execute('''
+            CREATE TABLE IF NOT EXISTS datasets (
+                id INTEGER PRIMARY KEY,
+                file_path TEXT,
+                author TEXT,
+                age TEXT,
+                var TEXT,
+                trace_id TEXT,
+                FOREIGN KEY (author) REFERENCES authors (id)
+            )
+        ''')
+
+        # Insert the metadata for each dataset into the table
+        for file in glob.glob(f'{self.db_dir}/**/**/*.pkl', recursive=True):
+            dir_name, file_name = os.path.split(file)
+            pkl_dir, trace_id = os.path.split(dir_name)
+            author_dir, _ = os.path.split(pkl_dir)
+
+            author = os.path.basename(author_dir)
+            file_name_, ext = os.path.splitext(file_name)
+            relative_file_path = f'{author}/pkl/{trace_id}/{file_name}'
+            print(relative_file_path)
+
+            # Get the author's ID from the authors table
+            cursor.execute('SELECT id FROM authors WHERE name = ?', (author,))
+            author_id = cursor.fetchone()[0]
+
+            if file_name_ in Authors_ages[author]:
+                age = Authors_ages[author][file_name_]
+            else:
+                age = None
+
+            if file_name_ in Extra_vars[author]:
+                var = Extra_vars[author][file_name_]
+            else:
+                var = None
+
+            cursor.execute('''
+                INSERT INTO datasets (file_path, author, age, var, trace_id)
+                VALUES (?, ?, ?, ?, ?)
+            ''', (relative_file_path, author_id, age, var, trace_id))
+
+        conn.commit()
+        conn.close()

anta_database-0.1/anta_database.egg-info/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.4
+Name: anta_database
+Version: 0.1
+Summary: SQLite database for the AntArchitecture Community Data
+Home-page: https://github.com/antoinehermant/anta_database
+Author: Antoine Hermant
+Author-email: antoine.hermant@etik.com
+License: MIT
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: pyproj
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Dynamic: author
+Dynamic: author-email
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary

anta_database-0.1/anta_database.egg-info/SOURCES.txt

@@ -0,0 +1,18 @@
+LICENSE
+README.md
+setup.py
+anta_database/__init__.py
+anta_database/anta_database.py
+anta_database.egg-info/PKG-INFO
+anta_database.egg-info/SOURCES.txt
+anta_database.egg-info/dependency_links.txt
+anta_database.egg-info/requires.txt
+anta_database.egg-info/top_level.txt
+anta_database/compiling/__init__.py
+anta_database/compiling/compile.py
+anta_database/database/__init__.py
+anta_database/database/database.py
+anta_database/database/plotting/GL.pkl
+anta_database/database/plotting/__init__.py
+anta_database/indexing/__init__.py
+anta_database/indexing/index_database.py

anta_database-0.1/anta_database.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

anta_database-0.1/anta_database.egg-info/requires.txt

@@ -0,0 +1,4 @@
+pandas
+pyproj
+matplotlib
+seaborn

anta_database-0.1/anta_database.egg-info/top_level.txt

@@ -0,0 +1 @@
+anta_database

anta_database-0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

anta_database-0.1/setup.py

@@ -0,0 +1,22 @@
+from setuptools import setup, find_packages
+
+setup(
+    name='anta_database',
+    version="0.1",
+    author="Antoine Hermant",
+    author_email= "antoine.hermant@etik.com",
+    url="https://github.com/antoinehermant/anta_database",
+    description= "SQLite database for the AntArchitecture Community Data",
+    license="MIT",
+    long_description="""""",
+    packages=find_packages(),
+    package_data={
+        'anta_database': ['database/plotting/*.pkl'],
+    },
+    install_requires=[
+        'pandas',
+        'pyproj',
+        'matplotlib',
+        'seaborn',
+    ]
+)