PyPI - LCNE-patchseq-analysis - Versions diffs - 0.0.2__tar.gz → 0.2.0__tar.gz - Mend

LCNE-patchseq-analysis 0.0.2tar.gz → 0.2.0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (38) hide show

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/.flake8 RENAMED Viewed

@@ -4,6 +4,7 @@ exclude =
     __pycache__,
     build,
     .venv,
-    venv
+    venv,
+    code/
 max-complexity = 10
 max-line-length = 100

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/.gitignore RENAMED Viewed

@@ -139,4 +139,5 @@ dmypy.json
 **/.DS_Store
 .vscode
-metadata.yml
+metadata.yml
+data

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: LCNE-patchseq-analysis
-Version: 0.0.2
+Version: 0.2.0
 Summary: Generated from aind-library-template
 Author: Allen Institute for Neural Dynamics
 Author-email: Han Hou <han.hou@alleninstitute.org>
@@ -11,6 +11,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: pandas
+Requires-Dist: matplotlib
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: coverage; extra == "dev"
@@ -19,24 +20,27 @@ Requires-Dist: interrogate; extra == "dev"
 Requires-Dist: isort; extra == "dev"
 Requires-Dist: Sphinx; extra == "dev"
 Requires-Dist: furo; extra == "dev"
+Provides-Extra: pipeline
+Requires-Dist: black; extra == "pipeline"
+Requires-Dist: coverage; extra == "pipeline"
+Requires-Dist: flake8; extra == "pipeline"
+Requires-Dist: interrogate; extra == "pipeline"
+Requires-Dist: isort; extra == "pipeline"
+Requires-Dist: Sphinx; extra == "pipeline"
+Requires-Dist: furo; extra == "pipeline"
+Requires-Dist: openpyxl; extra == "pipeline"
+Requires-Dist: pg8000; extra == "pipeline"
 # LCNE-patchseq-analysis
 [![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
 ![Code Style](https://img.shields.io/badge/code%20style-black-black)
 [![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)
-![Interrogate](https://img.shields.io/badge/interrogate-100.0%25-brightgreen)
+![Interrogate](https://img.shields.io/badge/interrogate-81.2%25-yellow)
 ![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen?logo=codecov)
 ![Python](https://img.shields.io/badge/python->=3.9-blue?logo=python)
-## Usage
- - To use this template, click the green `Use this template` button and `Create new repository`.
- - After github initially creates the new repository, please wait an extra minute for the initialization scripts to finish organizing the repo.
- - To enable the automatic semantic version increments: in the repository go to `Settings` and `Collaborators and teams`. Click the green `Add people` button. Add `svc-aindscicomp` as an admin. Modify the file in `.github/workflows/tag_and_publish.yml` and remove the if statement in line 65. The semantic version will now be incremented every time a code is committed into the main branch.
- - To publish to PyPI, enable semantic versioning and uncomment the publish block in `.github/workflows/tag_and_publish.yml`. The code will now be published to PyPI every time the code is committed into the main branch.
- - The `.github/workflows/test_and_lint.yml` file will run automated tests and style checks every time a Pull Request is opened. If the checks are undesired, the `test_and_lint.yml` can be deleted. The strictness of the code coverage level, etc., can be modified by altering the configurations in the `pyproject.toml` file and the `.flake8` file.
+Library for LCNE-patchseq data analysis
 ## Installation
 To use the software, in the root directory, run

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/README.md RENAMED Viewed

@@ -3,18 +3,11 @@
 [![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
 ![Code Style](https://img.shields.io/badge/code%20style-black-black)
 [![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)
-![Interrogate](https://img.shields.io/badge/interrogate-100.0%25-brightgreen)
+![Interrogate](https://img.shields.io/badge/interrogate-81.2%25-yellow)
 ![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen?logo=codecov)
 ![Python](https://img.shields.io/badge/python->=3.9-blue?logo=python)
-## Usage
- - To use this template, click the green `Use this template` button and `Create new repository`.
- - After github initially creates the new repository, please wait an extra minute for the initialization scripts to finish organizing the repo.
- - To enable the automatic semantic version increments: in the repository go to `Settings` and `Collaborators and teams`. Click the green `Add people` button. Add `svc-aindscicomp` as an admin. Modify the file in `.github/workflows/tag_and_publish.yml` and remove the if statement in line 65. The semantic version will now be incremented every time a code is committed into the main branch.
- - To publish to PyPI, enable semantic versioning and uncomment the publish block in `.github/workflows/tag_and_publish.yml`. The code will now be published to PyPI every time the code is committed into the main branch.
- - The `.github/workflows/test_and_lint.yml` file will run automated tests and style checks every time a Pull Request is opened. If the checks are undesired, the `test_and_lint.yml` can be deleted. The strictness of the code coverage level, etc., can be modified by altering the configurations in the `pyproject.toml` file and the `.flake8` file.
+Library for LCNE-patchseq data analysis
 ## Installation
 To use the software, in the root directory, run

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/docs/source/conf.py RENAMED Viewed

@@ -1,4 +1,5 @@
 """Configuration file for the Sphinx documentation builder."""
 #
 # For the full list of built-in configuration values, see the documentation:
 # https://www.sphinx-doc.org/en/master/usage/configuration.html

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/environment/Dockerfile RENAMED Viewed

@@ -15,13 +15,5 @@ RUN cd /.code-server \
     && rm code-server.tar.gz \
     && ln -s /.code-server/code-server-4.95.3-linux-amd64/bin/code-server  /usr/bin/code-server
-RUN mkdir -p /.vscode/extensions \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension REditorSupport.R \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension continue.continue \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension ms-python.python \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension ms-toolsai.jupyter \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension reageyao.bioSyntax \
-    && code-server --extensions-dir="/.vscode/extensions" --install-extension saoudrizwan.claude-dev
 COPY postInstall /
 RUN /postInstall

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/environment/postInstall RENAMED Viewed

@@ -10,6 +10,7 @@ if code-server --disable-telemetry --version; then
 			 fi
        code-server --disable-telemetry --extensions-dir=/.vscode/extensions --install-extension ms-python.python
+       code-server --disable-telemetry --extensions-dir=/.vscode/extensions --install-extension ms-toolsai.jupyter
        code-server --disable-telemetry --extensions-dir=/.vscode/extensions --install-extension njpwerner.autodocstring
        code-server --disable-telemetry --extensions-dir=/.vscode/extensions --install-extension KevinRose.vsc-python-indent
        code-server --disable-telemetry --extensions-dir=/.vscode/extensions --install-extension mhutchie.git-graph

lcne_patchseq_analysis-0.2.0/notebook/demo.ipynb ADDED Viewed

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from LCNE_patchseq_analysis.data_util.metadata import read_brian_spreadsheet, cross_check_metadata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 63,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dfs = read_brian_spreadsheet(add_lims=True)\n",
+    "df = dfs[\"df_all\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##  Cross tab sanity check"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check overlapped columns across tabs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 64,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found 9 inconsistencies between tab_xyz and master tables:\n",
+      "          Date                       jem-id_cell_specimen  x_tab_master  \\\n",
+      "165 2023-09-01  Dbh-Cre_KH212;RCL-H2B-GFP-692026.10.10.02  10534.982420   \n",
+      "166 2023-08-25  Dbh-Cre_KH212;RCL-H2B-GFP-692022.09.06.01           NaN   \n",
+      "167 2023-08-20  Dbh-Cre_KH212;RCL-H2B-GFP-692023.08.06.01  10541.875980   \n",
+      "168 2023-08-20  Dbh-Cre_KH212;RCL-H2B-GFP-692023.08.06.02  10702.283200   \n",
+      "170 2023-06-02  Dbh-Cre_KH212;RCL-H2B-GFP-676766.10.06.03  10521.757810   \n",
+      "202 2023-03-15                    C57BL6J-665266.11.06.03  10451.809570   \n",
+      "217 2023-01-20      Ndnf-IRES2-dgCre;Ai14-659663.11.06.03  10391.497070   \n",
+      "219 2023-01-20      Ndnf-IRES2-dgCre;Ai14-659663.11.06.04   9531.198242   \n",
+      "220 2023-01-20      Ndnf-IRES2-dgCre;Ai14-659663.11.06.01           NaN   \n",
+      "\n",
+      "     y_tab_master  z_tab_master Annotated structure_tab_master  \\\n",
+      "165   4183.531250        4984.0                            PAG   \n",
+      "166           NaN           NaN                           SCiw   \n",
+      "167   4110.681641        5034.0                             PB   \n",
+      "168   3840.954834        4727.0                             LC   \n",
+      "170   4256.657715        4889.0                            LDT   \n",
+      "202   4402.110352        4889.0                            LDT   \n",
+      "217   4161.165039        4889.0                            PCG   \n",
+      "219   2449.594727        4265.0                            PCG   \n",
+      "220           NaN           NaN                            LDT   \n",
+      "\n",
+      "    notes_tab_master     x_tab_xyz    y_tab_xyz  z_tab_xyz  \\\n",
+      "165              NaN  10151.019530  3701.974609     4824.0   \n",
+      "166              NaN   9531.198242  2449.594727     4265.0   \n",
+      "167              NaN  10702.283200  3840.954834     4727.0   \n",
+      "168              NaN  10761.001950  4288.832031     4727.0   \n",
+      "170              NaN  10541.875980  4110.681641     5034.0   \n",
+      "202              NaN  10534.982420  4183.531250     4984.0   \n",
+      "217              NaN  10521.757810  4256.657715     4889.0   \n",
+      "219              NaN  10451.809570  4402.110352     4889.0   \n",
+      "220              NaN  10391.497070  4161.165039     4889.0   \n",
+      "\n",
+      "    Annotated structure_tab_xyz notes_tab_xyz  \n",
+      "165                         PAG           NaN  \n",
+      "166                        SCiw           NaN  \n",
+      "167                          PB           NaN  \n",
+      "168                          LC           NaN  \n",
+      "170                         LDT           NaN  \n",
+      "202                         LDT           NaN  \n",
+      "217                         PCG           NaN  \n",
+      "219                         PCG           NaN  \n",
+      "220                         LDT           NaN  \n",
+      "\n",
+      "\n",
+      "Found 103 inconsistencies between tab_ephys_fx and master tables:\n",
+      "          Date                             jem-id_cell_specimen  \\\n",
+      "0   2025-02-06                          C57BL6J-785653.03.02.02   \n",
+      "1   2025-02-06                          C57BL6J-785653.04.02.02   \n",
+      "2   2025-02-06                          C57BL6J-785653.03.02.01   \n",
+      "3   2025-02-06                          C57BL6J-785653.04.02.01   \n",
+      "4   2025-02-05                          C57BL6J-785652.03.02.02   \n",
+      "..         ...                                              ...   \n",
+      "187 2023-04-19            Slc17a6-IRES-Cre;Ai14-670829.11.06.02   \n",
+      "243 2022-11-17            Slc17a6-IRES-Cre;Ai14-651168.10.06.03   \n",
+      "251 2022-11-15  Dbh-Cre_KH212;RCL-Sun1sfGFP-neo-650884.09.06.05   \n",
+      "257 2022-11-02              Rbp4-Cre_KL100;Ai14-650443.10.06.02   \n",
+      "258 2022-10-27                          C57BL6J-647687.09.06.01   \n",
+      "\n",
+      "     failed_electrode_0_tab_master  failed_no_seal_tab_master  \\\n",
+      "0                              NaN                        NaN   \n",
+      "1                              NaN                        NaN   \n",
+      "2                              NaN                        NaN   \n",
+      "3                              NaN                        NaN   \n",
+      "4                              NaN                        NaN   \n",
+      "..                             ...                        ...   \n",
+      "187                            0.0                        1.0   \n",
+      "243                            0.0                        1.0   \n",
+      "251                            0.0                        1.0   \n",
+      "257                            0.0                        1.0   \n",
+      "258                            0.0                        1.0   \n",
+      "\n",
+      "     failed_bad_rs_tab_master  failed_electrode_0_tab_ephys_fx  \\\n",
+      "0                         NaN                              0.0   \n",
+      "1                         NaN                              0.0   \n",
+      "2                         NaN                              0.0   \n",
+      "3                         NaN                              0.0   \n",
+      "4                         NaN                              0.0   \n",
+      "..                        ...                              ...   \n",
+      "187                       0.0                              0.0   \n",
+      "243                       0.0                              0.0   \n",
+      "251                       0.0                              0.0   \n",
+      "257                       0.0                              0.0   \n",
+      "258                       0.0                              0.0   \n",
+      "\n",
+      "     failed_no_seal_tab_ephys_fx  failed_bad_rs_tab_ephys_fx  \n",
+      "0                            0.0                         0.0  \n",
+      "1                            0.0                         0.0  \n",
+      "2                            0.0                         0.0  \n",
+      "3                            0.0                         0.0  \n",
+      "4                            0.0                         0.0  \n",
+      "..                           ...                         ...  \n",
+      "187                          0.0                         0.0  \n",
+      "243                          0.0                         0.0  \n",
+      "251                          0.0                         0.0  \n",
+      "257                          0.0                         0.0  \n",
+      "258                          0.0                         0.0  \n",
+      "\n",
+      "[103 rows x 8 columns]\n",
+      "\n",
+      "\n",
+      "Found 15 inconsistencies between lims and master tables:\n",
+      "         Date             jem-id_cell_specimen  ephys_roi_id_tab_master  \\\n",
+      "0  2025-02-06          C57BL6J-785653.03.02.02               1418804349   \n",
+      "1  2025-02-06          C57BL6J-785653.04.02.02               1418799012   \n",
+      "2  2025-02-06          C57BL6J-785653.03.02.01               1418797120   \n",
+      "3  2025-02-06          C57BL6J-785653.04.02.01               1418784590   \n",
+      "4  2025-02-05          C57BL6J-785652.03.02.02               1418553949   \n",
+      "5  2025-02-05          C57BL6J-785652.03.02.01               1418549638   \n",
+      "6  2025-02-05          C57BL6J-785652.03.01.01               1418547172   \n",
+      "7  2025-02-05          C57BL6J-785652.04.02.01               1418555572   \n",
+      "8  2025-02-05          C57BL6J-785652.04.02.02               1418561975   \n",
+      "9  2025-01-30  Dbh-Cre-KI;Ai65-780952.04.02.01               1417392272   \n",
+      "10 2025-01-30  Dbh-Cre-KI;Ai65-780952.03.01.01               1417382638   \n",
+      "11 2025-01-30  Dbh-Cre-KI;Ai65-780952.04.01.02               1417380803   \n",
+      "12 2025-01-30  Dbh-Cre-KI;Ai65-780952.03.02.01               1417375160   \n",
+      "13 2025-01-30  Dbh-Cre-KI;Ai65-780952.04.01.01               1417373093   \n",
+      "14 2025-01-29  Dbh-Cre-KI;Ai65-780955.03.01.01               1417138763   \n",
+      "\n",
+      "   ephys_qc_tab_master storage_directory_tab_master  ephys_roi_id_lims  \\\n",
+      "0          auto_passed                          NaN       1.418804e+09   \n",
+      "1          auto_passed                          NaN       1.418799e+09   \n",
+      "2          auto_passed                          NaN       1.418797e+09   \n",
+      "3          auto_passed                          NaN       1.418785e+09   \n",
+      "4          auto_passed                          NaN       1.418554e+09   \n",
+      "5          auto_passed                          NaN       1.418550e+09   \n",
+      "6          auto_passed                          NaN       1.418547e+09   \n",
+      "7          auto_passed                          NaN       1.418556e+09   \n",
+      "8          auto_passed                          NaN       1.418562e+09   \n",
+      "9          auto_passed                          NaN       1.417392e+09   \n",
+      "10         auto_failed                          NaN       1.417383e+09   \n",
+      "11         auto_passed                          NaN       1.417381e+09   \n",
+      "12         auto_passed                          NaN       1.417375e+09   \n",
+      "13         auto_passed                          NaN       1.417373e+09   \n",
+      "14         auto_passed                          NaN       1.417139e+09   \n",
+      "\n",
+      "   ephys_qc_lims                             storage_directory_lims  \n",
+      "0    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "1    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "2    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "3    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "4    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "5    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "6    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "7    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "8    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "9    auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "10   auto_failed  /allen/programs/celltypes/production/mousecell...  \n",
+      "11   auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "12   auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "13   auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "14   auto_passed  /allen/programs/celltypes/production/mousecell...  \n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "dfs = read_brian_spreadsheet()\n",
+    "for source in [\"tab_xyz\", \"tab_ephys_fx\", \"lims\"]:\n",
+    "    df_inconsistencies = cross_check_metadata(dfs[\"df_all\"], source)\n",
+    "    \n",
+    "    if len(df_inconsistencies) == 0:\n",
+    "        print(\"All good!\")\n",
+    "        continue\n",
+    "    \n",
+    "    print(f\"Found {len(df_inconsistencies)} inconsistencies between {source} and master tables:\")\n",
+    "    print(df_inconsistencies)\n",
+    "    print(\"\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### ❌ Oh no! These inconsistencies must be caused by manually copying and pasting across the tabs!!!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Quick overview using pygwalker"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!pip install pygwalker --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pygwalker as pyg\n",
+    "walker = pyg.walk(df)\n",
+    "walker"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "patchseq_pipeline",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.21"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/pyproject.toml RENAMED Viewed

@@ -19,7 +19,8 @@ dynamic = ["version"]
 dependencies = [
     'numpy',
-    'pandas'
+    'pandas',
+    'matplotlib',
 ]
 [project.optional-dependencies]
@@ -33,6 +34,18 @@ dev = [
     'furo'
 ]
+pipeline = [
+    'black',
+    'coverage',
+    'flake8',
+    'interrogate',
+    'isort',
+    'Sphinx',
+    'furo',
+    'openpyxl',
+    'pg8000',
+]
 [tool.setuptools.packages.find]
 where = ["src"]
@@ -55,6 +68,7 @@ exclude = '''
     | _build
     | build
     | dist
+    | code
   )/
   | .gitignore
 )

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis/__init__.py ADDED Viewed

	@@ -0,0 +1,2 @@
1	+ """Init package"""
2	+ __version__ = "0.2.0"

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis/data_util/__init__.py ADDED Viewed

	@@ -0,0 +1 @@
1	+ """Utils for accessing data"""

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis/data_util/ephys.py ADDED Viewed

	@@ -0,0 +1 @@
1	+ """Get ephys data"""

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis/data_util/lims.py ADDED Viewed

@@ -0,0 +1,73 @@
+"""Utilities for querying the LIMS database.
+From Brian
+"""
+import pandas as pd  # pandas will be needed to work in a dataframe
+import pg8000  # pg8000 access SQL databases
+# code from Agata
+# these are nice functions to open LIMS, make a query and then close LIMS after
+def _connect(user="limsreader", host="limsdb2", database="lims2", password="limsro", port=5432):
+    conn = pg8000.connect(user=user, host=host, database=database, password=password, port=port)
+    return conn, conn.cursor()
+def _select(cursor, query):
+    cursor.execute(query)
+    columns = [d[0] for d in cursor.description]
+    return [dict(zip(columns, c)) for c in cursor.fetchall()]
+def limsquery(
+    query, user="limsreader", host="limsdb2", database="lims2", password="limsro", port=5432
+):
+    """A function that takes a string containing a SQL query, connects to the LIMS database
+    and outputs the result."""
+    conn, cursor = _connect(user, host, database, password, port)
+    try:
+        results = _select(cursor, query)
+    finally:
+        cursor.close()
+        conn.close()
+    return results
+# this last function will take our query results and put them in a dataframe
+# so that they are easy to work with
+def get_lims_dataframe(query):
+    """Return a dataframe with lims query"""
+    result = limsquery(query)
+    try:
+        data_df = pd.DataFrame(data=result, columns=result[0].keys())
+    except IndexError:
+        print("Could not find results for your query.")
+        data_df = pd.DataFrame()
+    return data_df
+# Query for LCNE patchseq experiments
+def get_lims_LCNE_patchseq():
+    lims_query = """
+        SELECT
+            s.id AS specimen_id,
+            s.name AS specimen_name,
+            proj.code,
+            err.id AS ephys_roi_id,
+            err.workflow_state AS Ephys_QC,
+            s.patched_cell_container,
+            err.storage_directory
+        FROM ephys_roi_results AS err
+        JOIN specimens AS s ON s.ephys_roi_result_id = err.id
+        JOIN projects AS proj ON s.project_id = proj.id
+        WHERE proj.code = 'mIVSCC-MET-R01_LC';
+    """
+    lims_df = get_lims_dataframe(lims_query)
+    return lims_df
+if __name__ == "__main__":
+    lims_df = get_lims_LCNE_patchseq()
+    print(lims_df.head())

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis/data_util/metadata.py ADDED Viewed

@@ -0,0 +1,129 @@
+"""Get metadata"""
+import logging
+import os
+import pandas as pd
+from LCNE_patchseq_analysis.data_util.lims import get_lims_LCNE_patchseq
+metadata_path = os.path.expanduser(R"~\Downloads\IVSCC_LC_summary.xlsx")
+logger = logging.getLogger(__name__)
+def read_brian_spreadsheet(file_path=metadata_path, add_lims=True):
+    """Read metadata, cell xyz coordinates, and ephys features from Brian's spreadsheet
+    Assuming IVSCC_LC_summary.xlsx is downloaded at file_path
+    Args:
+        file_path (str): Path to the metadata spreadsheet
+        add_lims (bool): Whether to add LIMS data
+    """
+    if not os.path.exists(file_path):
+        raise FileNotFoundError(f"File not found at {file_path}")
+    logger.info(f"Reading metadata from {file_path}...")
+    tab_names = pd.ExcelFile(file_path).sheet_names
+    # Get the master table
+    tab_master = [name for name in tab_names if "updated" in name.lower()][0]
+    df_master = pd.read_excel(file_path, sheet_name=tab_master)
+    # Get xyz coordinates
+    tab_xyz = [name for name in tab_names if "xyz" in name.lower()][0]
+    df_xyz = pd.read_excel(file_path, sheet_name=tab_xyz)
+    # Get ephys features
+    tab_ephys_fx = [name for name in tab_names if "ephys_fx" in name.lower()][0]
+    df_ephys_fx = pd.read_excel(file_path, sheet_name=tab_ephys_fx)
+    # Merge the tables
+    df_all = (
+        df_master.merge(
+            df_xyz.rename(
+                columns={
+                    "specimen_name": "jem-id_cell_specimen",
+                    "structure_acronym": "Annotated structure",
+                }
+            ),
+            on="jem-id_cell_specimen",
+            how="outer",
+            suffixes=("_tab_master", "_tab_xyz"),
+        )
+        .merge(
+            df_ephys_fx.rename(
+                columns={
+                    "failed_seal": "failed_no_seal",
+                    "failed_input_access_resistance": "failed_bad_rs",
+                }
+            ),
+            on="cell_specimen_id",
+            how="outer",
+            suffixes=("_tab_master", "_tab_ephys_fx"),
+        )
+        .sort_values("Date", ascending=False)
+    )
+    if add_lims:
+        logger.info("Querying and adding LIMS data...")
+        df_lims = get_lims_LCNE_patchseq()
+        df_all = df_all.merge(
+            df_lims,
+            left_on="jem-id_cell_specimen",
+            right_on="specimen_name",
+            how="left",
+            suffixes=("_tab_master", "_lims"),
+        )
+    return {
+        "df_all": df_all,
+        "df_master": df_master,
+        "df_xyz": df_xyz,
+        "df_ephys_fx": df_ephys_fx,
+        **({"df_lims": df_lims} if add_lims else {}),
+    }
+def cross_check_metadata(df, source):
+    """Cross-check metadata between source and master tables
+    source in ["tab_xyz", "tab_ephys_fx", "lims]
+    """
+    source_columns = [col for col in df.columns if source in col]
+    master_columns = [col.replace(source, "tab_master") for col in source_columns]
+    logger.info(f"Cross-checking metadata between {source} and master tables...")
+    logger.info(f"Source columns: {source_columns}")
+    logger.info(f"Master columns: {master_columns}")
+    # Find out inconsistencies between source and master, if both of them are not null
+    df_inconsistencies = df.loc[
+        (
+            df[source_columns].notnull()
+            & df[source_columns].notnull()
+            & (df[source_columns].to_numpy() != df[master_columns].to_numpy())
+        ).any(axis=1),
+        ["Date", "jem-id_cell_specimen"] + master_columns + source_columns,
+    ]
+    return df_inconsistencies
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    dfs = read_brian_spreadsheet()
+    for source in ["tab_xyz", "tab_ephys_fx", "lims"]:
+        df_inconsistencies = cross_check_metadata(dfs["df_all"], source)
+        if len(df_inconsistencies) == 0:
+            print("All good!")
+            continue
+        print(
+            f"Found {len(df_inconsistencies)} inconsistencies between {source} and master tables:"
+        )
+        print(df_inconsistencies)
+        print("\n")

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/src/LCNE_patchseq_analysis.egg-info/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: LCNE-patchseq-analysis
-Version: 0.0.2
+Version: 0.2.0
 Summary: Generated from aind-library-template
 Author: Allen Institute for Neural Dynamics
 Author-email: Han Hou <han.hou@alleninstitute.org>
@@ -11,6 +11,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: pandas
+Requires-Dist: matplotlib
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: coverage; extra == "dev"
@@ -19,24 +20,27 @@ Requires-Dist: interrogate; extra == "dev"
 Requires-Dist: isort; extra == "dev"
 Requires-Dist: Sphinx; extra == "dev"
 Requires-Dist: furo; extra == "dev"
+Provides-Extra: pipeline
+Requires-Dist: black; extra == "pipeline"
+Requires-Dist: coverage; extra == "pipeline"
+Requires-Dist: flake8; extra == "pipeline"
+Requires-Dist: interrogate; extra == "pipeline"
+Requires-Dist: isort; extra == "pipeline"
+Requires-Dist: Sphinx; extra == "pipeline"
+Requires-Dist: furo; extra == "pipeline"
+Requires-Dist: openpyxl; extra == "pipeline"
+Requires-Dist: pg8000; extra == "pipeline"
 # LCNE-patchseq-analysis
 [![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
 ![Code Style](https://img.shields.io/badge/code%20style-black-black)
 [![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)
-![Interrogate](https://img.shields.io/badge/interrogate-100.0%25-brightgreen)
+![Interrogate](https://img.shields.io/badge/interrogate-81.2%25-yellow)
 ![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen?logo=codecov)
 ![Python](https://img.shields.io/badge/python->=3.9-blue?logo=python)
-## Usage
- - To use this template, click the green `Use this template` button and `Create new repository`.
- - After github initially creates the new repository, please wait an extra minute for the initialization scripts to finish organizing the repo.
- - To enable the automatic semantic version increments: in the repository go to `Settings` and `Collaborators and teams`. Click the green `Add people` button. Add `svc-aindscicomp` as an admin. Modify the file in `.github/workflows/tag_and_publish.yml` and remove the if statement in line 65. The semantic version will now be incremented every time a code is committed into the main branch.
- - To publish to PyPI, enable semantic versioning and uncomment the publish block in `.github/workflows/tag_and_publish.yml`. The code will now be published to PyPI every time the code is committed into the main branch.
- - The `.github/workflows/test_and_lint.yml` file will run automated tests and style checks every time a Pull Request is opened. If the checks are undesired, the `test_and_lint.yml` can be deleted. The strictness of the code coverage level, etc., can be modified by altering the configurations in the `pyproject.toml` file and the `.flake8` file.
+Library for LCNE-patchseq data analysis
 ## Installation
 To use the software, in the root directory, run

{lcne_patchseq_analysis-0.0.2 → lcne_patchseq_analysis-0.2.0}/src/LCNE_patchseq_analysis.egg-info/SOURCES.txt RENAMED Viewed

@@ -19,11 +19,16 @@ docs/source/_static/favicon.ico
 docs/source/_static/light-logo.svg
 environment/Dockerfile
 environment/postInstall
+notebook/demo.ipynb
 src/LCNE_patchseq_analysis/__init__.py
 src/LCNE_patchseq_analysis.egg-info/PKG-INFO
 src/LCNE_patchseq_analysis.egg-info/SOURCES.txt
 src/LCNE_patchseq_analysis.egg-info/dependency_links.txt
 src/LCNE_patchseq_analysis.egg-info/requires.txt
 src/LCNE_patchseq_analysis.egg-info/top_level.txt
+src/LCNE_patchseq_analysis/data_util/__init__.py
+src/LCNE_patchseq_analysis/data_util/ephys.py
+src/LCNE_patchseq_analysis/data_util/lims.py
+src/LCNE_patchseq_analysis/data_util/metadata.py
 tests/__init__.py
 tests/test_example.py

lcne_patchseq_analysis-0.2.0/src/LCNE_patchseq_analysis.egg-info/requires.txt ADDED Viewed

@@ -0,0 +1,23 @@
+numpy
+pandas
+matplotlib
+[dev]
+black
+coverage
+flake8
+interrogate
+isort
+Sphinx
+furo
+[pipeline]
+black
+coverage
+flake8
+interrogate
+isort
+Sphinx
+furo
+openpyxl
+pg8000