pyrekordbox 0.2.2__py3-none-any.whl → 0.3.0__py3-none-any.whl

docs/source/index.md

@@ -11,7 +11,6 @@ sphinx-quickstart on Thu Apr  7 15:06:50 2022.
 [![Python][python-badge+]][pypi-link]
 [![Platform][platform-badge]][pypi-link]
 [![license: MIT][license-badge]][license-link]
-[![style: black][black-badge]][black-link]
 
 ```{admonition} Disclaimer
 This project is not affiliated with Pioneer Corp. or its related companies
@@ -45,6 +44,7 @@ caption: User Guide
 
 quickstart
 installation
+key
 tutorial/index
 api
 ```
@@ -90,16 +90,14 @@ PyPI <https://pypi.org/project/pyrekordbox/>
 [tests-badge]: https://img.shields.io/github/actions/workflow/status/dylanljones/pyrekordbox/tests.yml?branch=master&label=tests&logo=github&style=flat
 [codecov-badge]: https://codecov.io/gh/dylanljones/pyrekordbox/branch/master/graph/badge.svg?token=5Z2KVGL7N3
 [python-badge]: https://img.shields.io/pypi/pyversions/pyrekordbox?style=flat
-[python-badge+]: https://img.shields.io/badge/python-3.7+-blue.svg
+[python-badge+]: https://img.shields.io/badge/python-3.8+-blue.svg
 [platform-badge]: https://img.shields.io/badge/platform-win%20%7C%20osx-blue?style=flat
 [pypi-badge]: https://img.shields.io/pypi/v/pyrekordbox?style=flat
 [license-badge]: https://img.shields.io/pypi/l/pyrekordbox?color=lightgrey
-[black-badge]: https://img.shields.io/badge/code%20style-black-000000?style=flat
 
 [pypi-link]: https://pypi.org/project/pyrekordbox/
 [license-link]: https://github.com/dylanljones/pyrekordbox/blob/master/LICENSE
 [tests-link]: https://github.com/dylanljones/pyrekordbox/actions/workflows/tests.yml
-[black-link]: https://github.com/psf/black
 [codecov-link]: https://app.codecov.io/gh/dylanljones/pyrekordbox/tree/master
 
 [issue]: https://github.com/dylanljones/pyrekordbox/issues/64

docs/source/installation.md

@@ -1,3 +1,271 @@
-```{include} ../../INSTALLATION.md
+# Installation
 
+Pyrekordbox is available on [PyPI]:
+
+```sh
+$ pip install pyrekordbox
+```
+
+Alternatively, it can be installed via [GitHub]:
+
+```sh
+$ pip install git+https://github.com/dylanljones/pyrekordbox.git@VERSION
+```
+
+where `VERSION` is a branch, tag or release. The project can also be cloned/forked
+and installed via
+
+```sh
+$ pip install .
+```
+
+
+## Installing SQLCipher
+
+Unlocking the new Rekordbox 6 `master.db` database file requires [SQLCipher][sqlcipher].
+Pyrekordbox makes no attempt to download/install SQLCipher, as it is a
+pure Python package - whereas the SQLCipher/sqlcipher3 installation is
+platform-dependent and can not be installed easily via ``pip``.
+
+[sqlcipher3] can either be built with the system [SQLCipher][sqlcipher] or built against
+a statically linked amalgamation of the SQLite3 source code.
+
+### Windows
+
+#### SQLCipher Amalagamation
+
+The easiest method to install SQLCipher on Windows is to build [sqlcipher3]
+against an amalgamation of the SQLite3 source code.
+
+1. **Install [Visual Studio Community Edition][VS]**
+
+   Make sure to select all the GCC options (VC++, C++, etc) in the installation process.
+   The following workloads under ``Desktop & Mobile`` should be sufficient:
+   - Desktop Development with C++
+   - .NET desktop development
+
+
+2. **Install a prebuilt [OpenSSL binary][OpenSSL]**
+
+   Choose the latest Win32/Win64 version. Make sure to download the full version,
+   not the light version.
+
+
+3. **Confirm that the `OPENSSL_CONF` environment variable is set properly in environment variables**
+
+   This should not be root openssl path (ex: `C:/Program Files/openssl-Win64`),
+   but instead should be the path to the config file, for example:
+   - 32-bit: ``C:/Program Files (x86)/openssl-Win32/bin/openssl.cfg``
+   - 64-bit: ``C:/Program Files/openssl-Win64/bin/openssl.cfg``
+
+   ```{note}
+   The library names of OpenSSL have changed in version 1.1.0 (see [this](https://stackoverflow.com/questions/65345077/unable-to-build-sqlcipher3-on-windows) discussion).
+   If you are using a newer version, you can set an environment variable
+   ``OPENSSL_LIBNAME`` to the name of the library, e.g. ``libcrypto.lib``.
+   Alternatively, you can modify the ``setup.py`` script (see step 8 below).
+
+   You might have to restart Windows for the changes to take effect.
+   ```
+
+4. **Copy the openssl folder to the Microsoft Visual Studio 14 VC include directory**
+
+   The openssl folder can be found here:
+   - 32-bit: `C:/Program Files (x86)/OpenSSL-Win32/include/openssl`
+   - 64-bit: `C:/Program Files/OpenSSL-Win64/include/openssl`
+
+   The VC include directory can be found in the
+   Visual Studio 14 installation directory:
+
+   > C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/include
+
+   In newer versions it might aso be in
+
+   > C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/[version]/include
+
+   Confirm the following path exists `.../include/openssl/aes.h`
+
+
+5. **Download / compile the SQLCipher 3 amalgamation files**
+
+   Pre-built SQLCipher amalgamation files can be downloaded from [this repo][amalgamation]:
+   ````shell
+   git clone https://github.com/geekbrother/sqlcipher-amalgamation
+   ````
+   To compile the amalgamation files on your own, follow [this tutorial](http://www.jerryrw.com/howtocompile.php).
+
+
+6. **Clone [sqlcipher3] into any directory**
+
+   ````shell
+   git clone https://github.com/coleifer/sqlcipher3
+   ````
+
+
+7. **Copy amalgamation files to the `sqlcipher3` directory**
+
+   Copy files ``sqlite3.c`` and ``sqlite3.h`` from the amalgamation directory from step 5
+   to the root of the ``sqlcipher3`` directory from step 6.
+   ````shell
+   Copy-Item -Path 'sqlcipher-amalgamation/src/sqlite3.c' -Destination "sqlcipher3/"
+   Copy-Item -Path 'sqlcipher-amalgamation/src/sqlite3.h' -Destination "sqlcipher3/"
+   ````
+
+
+8. **Modify the ``sqlcipher3/setup.py`` script (optional)**
+
+   If building the amalgamation fails and you haven't set the ``OPENSSL_LIBNAME``
+   environment variable in step 3, you have to modify the ``setup.py`` script. Change
+   ````python
+   openssl_libname = os.environ.get('OPENSSL_LIBNAME') or 'libeay32.lib'
+   ````
+   to
+   ````python
+   openssl_libname = os.environ.get('OPENSSL_LIBNAME') or 'libcrypto.lib'
+   ````
+
+
+9. **Build using the amalgamation and install**
+
+    ``cd`` into the ``sqlcipher3`` directory and run
+    ````shell
+    python setup.py build_static build
+    python setup.py install
+    ````
+
+
+You now should have a working ``sqlcipher3`` installation! The directory of the
+cloned ``sqlcipher3`` repo can be deleted after installing the package.
+
+Steps 5-9 can be automated using the CLI of ``pyrekordbox``:
+
+    > python3 -m pyrekordbox install-sqlcipher --help
+    usage: pyrekordbox install-sqlcipher [-h] [-t TMPDIR] [-l CRYPTOLIB] [-q] [-b]
+
+      -t TMPDIR, --tmpdir TMPDIR
+                            Path for storing temporary data (default: '.tmp')
+      -l CRYPTOLIB, --cryptolib CRYPTOLIB
+                            The name of the OpenSSl crypto libary (default: 'libcrypto.lib')
+      -b, --buildonly       Don't install sqlcipher3, only build the amalgamation
+
+
+##### Troubleshooting
+
+- **Microsoft Visual C++ error**
+
+  If you are getting an error like
+  ````shell
+  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools"``
+  ````
+  and have Visual Studio installed, you might not have all the necessary C/C++ components.
+
+- **LINK error**
+
+  If you are getting an error like
+  ````shell
+  LINK : fatal error LNK1158: cannot run 'rc.exe'
+  ````
+  or
+  ````shell
+  LINK : fatal error LNK1327: failure during running rc.exe
+  ````
+  make sure all the necessary C/C++ components are installed and that you have selected
+  the latest Win 10/11 SDK in the Visual Studio installer. If you are still getting the error,
+  follow the suggestions in this [StackOverflow post](https://stackoverflow.com/questions/14372706/visual-studio-cant-build-due-to-rc-exe).
+
+
+### MacOS
+
+
+#### System SQLCipher
+
+For building [sqlcipher3] against the system SQLCipher installation on MacOS follow these steps:
+
+1) Install [Homebrew](https://brew.sh) if you do not have it on your machine.
+2) Install SQLCipher with `brew install SQLCipher`.
+3) With the python environment you are using to run pyrekordbox active execute the following:
+```shell
+git clone https://github.com/coleifer/sqlcipher3
+cd sqlcipher3
+SQLCIPHER_PATH=$(brew info sqlcipher | awk 'NR==4 {print $1; exit}'); C_INCLUDE_PATH="$SQLCIPHER_PATH"/include LIBRARY_PATH="$SQLCIPHER_PATH"/lib python setup.py build
+SQLCIPHER_PATH=$(brew info sqlcipher | awk 'NR==4 {print $1; exit}'); C_INCLUDE_PATH="$SQLCIPHER_PATH"/include LIBRARY_PATH="$SQLCIPHER_PATH"/lib python setup.py install
+```
+Make sure the `C_INCLUDE` and `LIBRARY_PATH` point to the installed SQLCipher path. It may differ on your machine.
+
+````{note}
+If you are having issues building sqlcipher on M1 Macs you might have to add some symlinks:
+```shell
+ln -s /opt/homebrew/lib/libsqlcipher.a /usr/local/lib/libsqlcipher.a
+ln -s /opt/homebrew/include/sqlcipher /usr/local/include/sqlcipher
 ```
+````
+#### SQLCipher Amalagamation
+
+You can also build [sqlcipher3] against an amalgamation on MacOS.
+
+1. **Download / compile the SQLCipher amalgamation files**
+
+   Pre-built SQLCipher amalgamation files can be downloaded from [this repo][amalgamation]:
+   ````shell
+   git clone https://github.com/geekbrother/sqlcipher-amalgamation
+   ````
+   You can also build the amalagamtion files on your own.
+
+
+2. **Clone [sqlcipher3] into any directory**
+
+   ````shell
+   git clone https://github.com/coleifer/sqlcipher3
+   ````
+
+
+3. **Copy amalgamation files to the `sqlcipher3` directory**
+
+   Copy files ``sqlite3.c`` and ``sqlite3.h`` from the amalgamation directory from step 1
+   to the root of the ``sqlcipher3`` directory from step 2.
+   ````shell
+   cp sqlcipher-amalgamation/src/sqlite3.[ch] sqlcipher3/
+   ````
+
+4. **Build using the amalgamation and install**
+
+    ``cd`` into the ``sqlcipher3`` directory and run
+    ````shell
+    python setup.py build_static build
+    python setup.py install
+    ````
+
+The steps above can be automated using the CLI of ``pyrekordbox``
+
+    > python3 -m pyrekordbox install-sqlcipher --help
+    usage: pyrekordbox install-sqlcipher [-h] [-t TMPDIR] [-l CRYPTOLIB] [-q] [-b]
+
+      -t TMPDIR, --tmpdir TMPDIR
+                            Path for storing temporary data (default: '.tmp')
+      -b, --buildonly       Don't install sqlcipher3, only build the amalgamation
+
+
+```{note}
+The `CRYPTOLIB` argument is only used on Windows
+```
+
+
+### Using SQLCipher
+
+After the installation SQLCipher-databases can be unlocked via the `sqlcipher3` package:
+````python
+from sqlcipher3 import dbapi2 as sqlite3
+
+conn = sqlite3.connect('test.db')
+c = conn.cursor()
+c.execute("PRAGMA key='password'")
+````
+
+
+
+[VS]: https://visualstudio.microsoft.com/de/vs/community/
+[OpenSSL]: https://slproweb.com/products/Win32OpenSSL.html
+[sqlcipher3]: https://github.com/coleifer/sqlcipher3
+[amalgamation]: https://github.com/geekbrother/sqlcipher-amalgamation
+[Pypi]: https://pypi.org/project/pyrekordbox/
+[GitHub]: https://github.com/dylanljones/pyrekordbox
+[sqlcipher]: https://www.zetetic.net/sqlcipher/open-source/

docs/source/key.md

@@ -0,0 +1,104 @@
+# Database key
+
+If you are using Rekordbox v6.6.5 or later and have no cached key from a previous
+Rekordbox version, the database can not be unlocked automatically.
+However, the command line interface of ``pyrekordbox`` provides a command for downloading
+the key from known sources and writing it to the cache file:
+````shell
+python -m pyrekordbox download-key
+````
+Once the key is cached the database can be opened without providing the key.
+if you obtained the key from another source, you can also pass it to the database handler
+````python
+db = Rekordbox6Database(key="<insert key here>")
+````
+or write it to the cache file manually:
+````python
+from pyrekordbox.config import write_db6_key_cache
+
+write_db6_key_cache("<insert key here>")  # call once
+db = Rekordbox6Database()
+````
+
+## Alternative methods
+
+The key can be extracted manually from the users machine. After the key is obtained
+it can be writen to the cache file as described above.
+The method varies depending on the operating system.
+
+### Windows
+
+On Windows, the key can be extracted from the Rekordbox executable using a debugger:
+
+1. Download [x64dbg] and run it.
+
+2. Options -> Preferences. Make sure "Entry Breakpoint" is set in the Events tab.
+
+3. File -> Open... `rekordbox.exe` (the main application executable)
+
+4. Look at the status bar. It should have a yellow "Paused" icon followed by some status text.
+   Right now it should say "System breakpoint reached!"
+
+5. Hit F9 or press the Run button in the top bar. The status text should change to
+   "INT3 breakpoint 'entry breakpoint' at <rekordbox.EntryPoint>".
+
+6. Click in the disassembly window, then press Ctrl+G to open the Go To Expression box,
+   and search for `sqlite3_key_v2` and press OK. This should jump you to the code for
+   that function, which typically starts with `mov dword ptr ss:[rsp+xx], r9d` or similar.
+
+7. Without clicking anywhere on the disassembly window, press F2 to toggle breakpoint.
+   The top instruction's address should turn red.
+
+8. Hit F9 or press the Run button in the top bar. The status text will start changing
+   a bunch, while the program starts up. Wait until the status bar goes back to "Paused"
+   in yellow. If the status text says something like "First chance exception on..."
+   press F9 again.
+
+9. The status bar should go to "Paused" in yellow again, this time with status text
+   that says "INT3 breakpoint at <sqlite3.sqlite3_key_v2>". This means our breakpoint
+   has been hit.
+
+10. Click the register panel (top right, where RAX, RBX, RCX, etc. are listed) so
+   it updates. Right click the red address next to R8, and click "Follow in dump".
+
+11. The dump at the bottom left will move to that address. Right click the dump panel
+   and select Text -> ASCII at the bottom. You should now see the key as a string.
+   You can drag-select it, then right click to copy selected line.
+
+12. Go to Debug -> Close to close the process, then close x64dbg.
+
+
+```{figure} /_static/images/x64dbg_rb_key.png
+:align: center
+:scale: 60
+```
+
+
+### MacOS
+
+On MacOS, the key can be extracted using the [RekordLocksmith] tool:
+
+1. Install LLDB:
+   - LLDB can be installed with Xcode on macOS via the App Store or xcode-cli-commands.
+   - Ensure LLDB is accessible from the terminal by running `lldb` in the terminal.
+
+2. Disable SIP:
+   - Restart your Mac and hold down `Command-R` as it boots to enter Recovery Mode.
+   - Open the Terminal from the Utilities menu.
+   - Type `csrutil disable` and press Enter.
+   - Restart your Mac.
+
+3. Download RekordLocksmith:
+   - Clone or download the [RekordLocksmith] repository from GitHub.
+
+4. Run RekordLocksmith:
+   - Use the terminal to navigate to the folder containing `rekordlocksmith.py`.
+   - Run the script:
+     ````shell
+     python3 rekordlocksmith.py /Applications/rekordbox\ 6/rekordbox.app/Contents/MacOS/rekordbox
+     ````
+     The tool will output the database key to the terminal and save it to a file named `rekordbox_db_pass.txt` in the current directory.
+
+
+[x64dbg]: https://x64dbg.com/
+[RekordLocksmith]: https://github.com/Bide-UK/rekordlocksmith#rekordlocksmith

docs/source/quickstart.md

@@ -101,7 +101,7 @@ between two databases.
 Pyrekordbox can read and write Rekordbox XML databases.
 
 ````python
-from pyrekordbox.xml import RekordboxXml
+from pyrekordbox.rbxml import RekordboxXml
 
 xml = RekordboxXml("database.xml")
 

pyrekordbox/__init__.py

@@ -4,7 +4,7 @@
 
 from .logger import logger
 from .config import show_config, get_config, update_config
-from .xml import RekordboxXml, XmlDuplicateError, XmlAttributeKeyError
+from .rbxml import RekordboxXml, XmlDuplicateError, XmlAttributeKeyError
 from .anlz import get_anlz_paths, walk_anlz_paths, read_anlz_files, AnlzFile
 from .mysettings import (
     get_mysetting_paths,

pyrekordbox/__main__.py

@@ -91,10 +91,6 @@ def install_pysqlcipher(
     install=True,
     cleanup=True,
 ):
-    if sys.platform != "win32":
-        print("Not on Windows, aborting...")
-        return
-
     tmpdir = Path(tmpdir)
     # Download pysqlcipher3 and prepare amalgamation build
     with WorkingDir(tmpdir):
@@ -103,7 +99,7 @@ def install_pysqlcipher(
         amalgamation_src = amalgamation_dir / "src"
 
         prepare_pysqlcipher(pysqlcipher_dir, amalgamation_src)
-        if os.getenv("OPENSSL_LIBNAME") is None:
+        if sys.platform == "win32" and os.getenv("OPENSSL_LIBNAME") is None:
             print("No OPENSSL_LIBNAME environment variable found, updating `setup.py`!")
             patch_pysqlcipher_setup(pysqlcipher_dir, crypto_lib)
 

pyrekordbox/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.2.2'
-__version_tuple__ = version_tuple = (0, 2, 2)
+__version__ = version = '0.3.0'
+__version_tuple__ = version_tuple = (0, 3, 0)

pyrekordbox/anlz/file.py

@@ -114,10 +114,10 @@ class AnlzFile(abc.Mapping):
                 # Check if the file is garbled (only on exported files)
                 # For this we check the validity of mood and bank
                 # Mood: High=1, Mid=2, Low=3
-                # Bank: 1-8
+                # Bank: 0-8
                 mood = Int16ub.parse(tag_data[18:20])
                 bank = Int16ub.parse(tag_data[28:30])
-                if 1 <= mood <= 3 and 1 <= bank <= 8:
+                if 1 <= mood <= 3 and 0 <= bank <= 8:
                     logger.debug("PSSI is not garbled!")
                 else:
                     logger.debug("PSSI is garbled!")

pyrekordbox/config.py

@@ -223,9 +223,10 @@ def read_rekordbox6_asar(rb6_install_dir: Union[str, Path]) -> str:
         location = rb6_install_dir / "rekordboxAgent-win32-x64" / "resources"
         encoding = "ANSI"
     elif sys.platform == "darwin":
+        if not str(rb6_install_dir).endswith(".app"):
+            rb6_install_dir = rb6_install_dir / "rekordbox.app"
         location = (
             rb6_install_dir
-            / "rekordbox.app"
             / "Contents"
             / "MacOS"
             / "rekordboxAgent.app"
@@ -244,6 +245,14 @@ def read_rekordbox6_asar(rb6_install_dir: Union[str, Path]) -> str:
     return data
 
 
+def _extract_version(name, major_version):
+    name = name.replace(".app", "")  # Needed for MacOS
+    ver_str = name.replace("rekordbox", "").strip()
+    if not ver_str:
+        ver_str = str(major_version)
+    return ver_str
+
+
 def _get_rb_config(
     pioneer_install_dir: Path,
     pioneer_app_dir: Path,
@@ -272,43 +281,38 @@ def _get_rb_config(
     if application_dirname:
         # Applitcation dirname is given, only extract version from it
         # `major_version` is compared to the version string
-        rb_version = application_dirname.replace("rekordbox", "").strip()
-        rb_version = rb_version.replace(".app", "")
-        if not rb_version.startswith(str(major_version)):
-            raise ValueError(
-                f"Major version is {major_version}, but the supplied application "
-                f"dirname is '{application_dirname}'"
-            )
         rb_prog_dir = pioneer_install_dir / application_dirname
         if not rb_prog_dir.exists():
             raise InvalidApplicationDirname(
                 f"The supplied application dirname '{application_dirname}' does not "
                 f"exist in '{pioneer_install_dir}'"
             )
+        rb_version = _extract_version(application_dirname, major_version)
     else:
         # Get latest Rekordbox installation directory for major release `major_version`
 
         # Find all 'V.x.x' version strings in dir names
-        versions = list()
+        installations = {}
         for p in pioneer_install_dir.iterdir():
             name = p.name
             if name.startswith("rekordbox"):
-                ver_str = name.replace("rekordbox", "").strip()
-                ver_str = ver_str.replace(".app", "")
+                ver_str = _extract_version(name, major_version)
                 if ver_str.startswith(str(major_version)):
-                    v = packaging.version.parse(ver_str)
-                    versions.append(v)
+                    version = packaging.version.parse(ver_str)
+                    installations[version] = name
         # Get latest 'V.x.x' version string and assure there is one
-        versions.sort()  # key=lambda s: list(map(int, s.split("."))))
+        versions = list(installations.keys())
+        versions.sort()
         try:
-            rb_version = str(versions[-1])
+            version = versions[-1]
         except IndexError:
             raise FileNotFoundError(
                 f"No Rekordbox {major_version} folder found in installation "
                 f"directory '{pioneer_install_dir}'"
             )
         # Name of the Rekordbox application directory in `pioneer_install_dir`
-        rb_prog_dir = pioneer_install_dir / f"rekordbox {rb_version}"
+        rb_version = str(versions[-1])
+        rb_prog_dir = pioneer_install_dir / installations[version]
 
     # Check installation directory
     if not rb_prog_dir.exists():