dfindexeddb 20240417__tar.gz → 20240519__tar.gz

{dfindexeddb-20240417/dfindexeddb.egg-info → dfindexeddb-20240519}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dfindexeddb
-Version: 20240417
+Version: 20240519
 Summary: dfindexeddb is an experimental Python tool for performing digital forensic analysis of IndexedDB and leveldb files.
 Author-email: Syd Pleno <sydp@google.com>
 Maintainer-email: dfIndexeddb Developers <dfindexeddb-dev@googlegroups.com>
@@ -219,16 +219,19 @@ License-File: LICENSE
 License-File: AUTHORS
 Requires-Dist: python-snappy==0.6.1
 Requires-Dist: zstd==1.5.5.1
+Provides-Extra: plugins
+Requires-Dist: protobuf; extra == "plugins"
+Requires-Dist: dfdatetime; extra == "plugins"
 
 # dfIndexeddb
 
 dfindexeddb is an experimental Python tool for performing digital forensic
-analysis of IndexedDB and leveldb files.
+analysis of IndexedDB and LevelDB files.
 
-It parses leveldb, IndexedDB and javascript structures from these files without
+It parses LevelDB, IndexedDB and JavaScript structures from these files without
 requiring native libraries.  (Note: only a subset of IndexedDB key types and
-Javascript types for Chromium-based browsers are currently supported.  Safari
-and Firefox are under development).
+JavaScript types for Safari and Chromium-based browsers are currently supported.
+Firefox is under development).
 
 The content of IndexedDB files is dependent on what a web application stores
 locally/offline using the web browser's
@@ -255,6 +258,12 @@ include:
     $ pip install dfindexeddb
 ```
 
+To also install the dependencies for leveldb/indexeddb plugins, run
+```
+    $ pip install 'dfindexeddb[plugins]'
+```
+
+
 ## Installation from source
 
 1. [Linux] Install the snappy compression development package
@@ -273,9 +282,14 @@ include:
     $ pip install .
 ```
 
+To also install the dependencies for leveldb/indexeddb plugins, run
+```
+    $ pip install '.[plugins]'
+```
+
 ## Usage
 
-Two CLI tools for parsing IndexedDB/leveldb files are available after
+Two CLI tools for parsing IndexedDB/LevelDB files are available after
 installation:
 
 
@@ -297,49 +311,42 @@ options:
   -h, --help    show this help message and exit
 ```
 
-To parse Indexeddb records from a LevelDB folder, use the following command:
+#### Examples:
 
-```
-dfindexeddb db -h
-usage: dfindexeddb db [-h] -s SOURCE [--use_manifest] [-o {json,jsonl,repr}]
+To parse IndexedDB records from an sqlite file for Safari and output the
+results as JSON-L, use the following command:
 
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source leveldb folder
-  --use_manifest        Use manifest file to determine active/recovered records.
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
+```
+dfindexeddb db -s SOURCE --format safari -o jsonl
 ```
 
-To parse Indexeddb records from a LevelDB ldb (.ldb) file, use the following 
-command:
+To parse IndexedDB records from a LevelDB folder for Chrome/Chromium, using the
+manifest file to determine recovered records and output as JSON, use the
+following command:
 
 ```
-dfindexeddb ldb -h
-usage: dfindexeddb ldb [-h] -s SOURCE [-o {json,jsonl,repr}]
+dfindexeddb db -s SOURCE --format chrome --use_manifest
+```
+
+To parse IndexedDB records from a LevelDB ldb (.ldb) file and output the
+results as JSON-L, use the following command:
 
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source .ldb file.
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
+```
+dfindexeddb ldb -s SOURCE -o jsonl
 ```
 
-To parse Indexeddb records from a LevelDB log (.log) file, use the following 
-command:
+To parse IndexedDB records from a LevelDB log (.log) file and output the
+results as the Python printable representation, use the following command:
 
 ```
-dfindexeddb log -h
-usage: dfindexeddb log [-h] -s SOURCE [-o {json,jsonl,repr}]
+dfindexeddb log -s SOURCE -o repr
+```
+
+To parse a file as a Chrome/Chromium IndexedDB blink value and output the
+results as JSON:
 
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source .log file.
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
+```
+dfindexeddb blink -s SOURCE
 ```
 
 ### LevelDB
@@ -361,64 +368,51 @@ options:
   -h, --help            show this help message and exit
 ```
 
+#### Examples
+
 To parse records from a LevelDB folder, use the following command:
 
 ```
-dfindexeddb db -h
-usage: dfindexeddb db [-h] -s SOURCE [--use_manifest] [-o {json,jsonl,repr}]
-
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source leveldb folder
-  --use_manifest        Use manifest file to determine active/recovered records.
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
+dfleveldb db -s SOURCE
 ```
 
-To parse records from a LevelDB log (.log) file, use the following command:
+To parse records from a LevelDB folder, and use the sequence number to 
+determine recovered records and output as JSON, use the
+following command:
 
 ```
-$ dfleveldb log  -s SOURCE [-o {json,jsonl,repr}] [-t {blocks,physical_records,write_batches,parsed_internal_key}]
-
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source leveldb file
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
-  -t {blocks,physical_records,write_batches,parsed_internal_key}, --structure_type {blocks,physical_records,write_batches,parsed_internal_key}
-                        Parses the specified structure. Default is parsed_internal_key.
+dfleveldb db -s SOURCE --use_sequence_number
 ```
 
-To parse records from a LevelDB table (.ldb) file, use the following command:
+To parse blocks / physical records/ write batches / internal key records from a
+LevelDB log (.log) file, use the following command, specifying the type (block,
+physical_records, etc) via the `-t` option.  By default, internal key records are parsed:
 
 ```
-$ dfleveldb ldb -s SOURCE [-o {json,jsonl,repr}] [-t {blocks,records}]
+$ dfleveldb log  -s SOURCE [-t {blocks,physical_records,write_batches,parsed_internal_key}]
+```
 
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source leveldb file
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
-  -t {blocks,records}, --structure_type {blocks,records}
-                        Parses the specified structure. Default is records.
+To parse blocks / records from a LevelDB table (.ldb) file, use the following
+command, specifying the type (blocks, records) via the `-t` option.  By
+default, records are parsed:
+
+```
+$ dfleveldb ldb -s SOURCE [-t {blocks,records}]
 ```
 
-To parse version edit records from a Descriptor (MANIFEST) file:
+To parse version edit records from a Descriptor (MANIFEST) file, use the
+following command:
 
 ```
 $ dfleveldb descriptor -s SOURCE [-o {json,jsonl,repr}] [-t {blocks,physical_records,versionedit} | -v]
-
-options:
-  -h, --help            show this help message and exit
-  -s SOURCE, --source SOURCE
-                        The source leveldb file
-  -o {json,jsonl,repr}, --output {json,jsonl,repr}
-                        Output format. Default is json
-  -t {blocks,physical_records,versionedit}, --structure_type {blocks,physical_records,versionedit}
-                        Parses the specified structure. Default is versionedit.
-  -v, --version_history
-                        Parses the leveldb version history.
 ```
+
+#### Plugins
+
+To apply a plugin parser for a leveldb file/folder, add the 
+`--plugin [Plugin Name]` argument.  Currently, there is support for the 
+following artifacts:
+
+| Plugin Name | Artifact Name |
+| -------- | ------- |
+| `ChromeNotificationRecord` | Chrome/Chromium Notifications |

dfindexeddb-20240519/README.md

@@ -0,0 +1,193 @@
+# dfIndexeddb
+
+dfindexeddb is an experimental Python tool for performing digital forensic
+analysis of IndexedDB and LevelDB files.
+
+It parses LevelDB, IndexedDB and JavaScript structures from these files without
+requiring native libraries.  (Note: only a subset of IndexedDB key types and
+JavaScript types for Safari and Chromium-based browsers are currently supported.
+Firefox is under development).
+
+The content of IndexedDB files is dependent on what a web application stores
+locally/offline using the web browser's
+[IndexedDB API](https://www.w3.org/TR/IndexedDB/).  Examples of content might
+include:
+* text from a text/source-code editor application,
+* emails and contact information from an e-mail application,
+* images and metadata from a photo gallery application
+
+
+## Installation
+
+1. [Linux] Install the snappy compression development package
+
+```
+    $ sudo apt install libsnappy-dev
+```
+
+2. Create a virtual environment and install the package
+
+```
+    $ python3 -m venv .venv
+    $ source .venv/bin/activate
+    $ pip install dfindexeddb
+```
+
+To also install the dependencies for leveldb/indexeddb plugins, run
+```
+    $ pip install 'dfindexeddb[plugins]'
+```
+
+
+## Installation from source
+
+1. [Linux] Install the snappy compression development package
+
+```
+    $ sudo apt install libsnappy-dev
+```
+
+2. Clone or download/unzip the repository to your local machine.
+
+3. Create a virtual environment and install the package
+
+```
+    $ python3 -m venv .venv
+    $ source .venv/bin/activate
+    $ pip install .
+```
+
+To also install the dependencies for leveldb/indexeddb plugins, run
+```
+    $ pip install '.[plugins]'
+```
+
+## Usage
+
+Two CLI tools for parsing IndexedDB/LevelDB files are available after
+installation:
+
+
+### IndexedDB
+
+```
+$ dfindexeddb -h
+usage: dfindexeddb [-h] {db,ldb,log} ...
+
+A cli tool for parsing indexeddb files
+
+positional arguments:
+  {db,ldb,log}
+    db          Parse a directory as indexeddb.
+    ldb         Parse a ldb file as indexeddb.
+    log         Parse a log file as indexeddb.
+
+options:
+  -h, --help    show this help message and exit
+```
+
+#### Examples:
+
+To parse IndexedDB records from an sqlite file for Safari and output the
+results as JSON-L, use the following command:
+
+```
+dfindexeddb db -s SOURCE --format safari -o jsonl
+```
+
+To parse IndexedDB records from a LevelDB folder for Chrome/Chromium, using the
+manifest file to determine recovered records and output as JSON, use the
+following command:
+
+```
+dfindexeddb db -s SOURCE --format chrome --use_manifest
+```
+
+To parse IndexedDB records from a LevelDB ldb (.ldb) file and output the
+results as JSON-L, use the following command:
+
+```
+dfindexeddb ldb -s SOURCE -o jsonl
+```
+
+To parse IndexedDB records from a LevelDB log (.log) file and output the
+results as the Python printable representation, use the following command:
+
+```
+dfindexeddb log -s SOURCE -o repr
+```
+
+To parse a file as a Chrome/Chromium IndexedDB blink value and output the
+results as JSON:
+
+```
+dfindexeddb blink -s SOURCE
+```
+
+### LevelDB
+
+```
+$ dfleveldb -h
+usage: dfleveldb [-h] {db,log,ldb,descriptor} ...
+
+A cli tool for parsing leveldb files
+
+positional arguments:
+  {db,log,ldb,descriptor}
+    db                  Parse a directory as leveldb.
+    log                 Parse a leveldb log file.
+    ldb                 Parse a leveldb table (.ldb) file.
+    descriptor          Parse a leveldb descriptor (MANIFEST) file.
+
+options:
+  -h, --help            show this help message and exit
+```
+
+#### Examples
+
+To parse records from a LevelDB folder, use the following command:
+
+```
+dfleveldb db -s SOURCE
+```
+
+To parse records from a LevelDB folder, and use the sequence number to 
+determine recovered records and output as JSON, use the
+following command:
+
+```
+dfleveldb db -s SOURCE --use_sequence_number
+```
+
+To parse blocks / physical records/ write batches / internal key records from a
+LevelDB log (.log) file, use the following command, specifying the type (block,
+physical_records, etc) via the `-t` option.  By default, internal key records are parsed:
+
+```
+$ dfleveldb log  -s SOURCE [-t {blocks,physical_records,write_batches,parsed_internal_key}]
+```
+
+To parse blocks / records from a LevelDB table (.ldb) file, use the following
+command, specifying the type (blocks, records) via the `-t` option.  By
+default, records are parsed:
+
+```
+$ dfleveldb ldb -s SOURCE [-t {blocks,records}]
+```
+
+To parse version edit records from a Descriptor (MANIFEST) file, use the
+following command:
+
+```
+$ dfleveldb descriptor -s SOURCE [-o {json,jsonl,repr}] [-t {blocks,physical_records,versionedit} | -v]
+```
+
+#### Plugins
+
+To apply a plugin parser for a leveldb file/folder, add the 
+`--plugin [Plugin Name]` argument.  Currently, there is support for the 
+following artifacts:
+
+| Plugin Name | Artifact Name |
+| -------- | ------- |
+| `ChromeNotificationRecord` | Chrome/Chromium Notifications |

{dfindexeddb-20240417 → dfindexeddb-20240519}/dfindexeddb/indexeddb/chromium/blink.py

@@ -780,6 +780,9 @@ class V8ScriptValueDecoder:
 
     Returns:
       A parsed CryptoKey.
+
+    Raises:
+      ParserError: if there is an unexpected CryptoKeySubTag.
     """
     _, raw_key_byte = self.deserializer.decoder.DecodeUint8()
     key_byte = definitions.CryptoKeySubTag(raw_key_byte)
@@ -795,6 +798,8 @@ class V8ScriptValueDecoder:
       key_type, algorithm_parameters = self._ReadED25519Key()
     elif key_byte == definitions.CryptoKeySubTag.NO_PARAMS_KEY:
       key_type, algorithm_parameters = self.ReadNoParamsKey()
+    else:
+      raise errors.ParserError('Unexpected CryptoKeySubTag')
 
     _, raw_usages = self.deserializer.decoder.DecodeUint32Varint()
     usages = definitions.CryptoKeyUsage(raw_usages)

{dfindexeddb-20240417 → dfindexeddb-20240519}/dfindexeddb/indexeddb/chromium/record.py

@@ -17,7 +17,11 @@ from __future__ import annotations
 from dataclasses import dataclass, field
 from datetime import datetime
 import io
-from typing import Any, BinaryIO, Optional, Tuple, Type, TypeVar, Union
+import pathlib
+import sys
+import traceback
+from typing import Any, BinaryIO, Generator, Optional, Tuple, Type, TypeVar, \
+    Union
 
 from dfindexeddb import errors
 from dfindexeddb.indexeddb.chromium import blink
@@ -456,7 +460,7 @@ class MaxDatabaseIdKey(BaseIndexedDBKey):
       cls, decoder: utils.LevelDBDecoder, key_prefix: KeyPrefix,
       base_offset: int = 0
   ) -> MaxDatabaseIdKey:
-    """Decodes the maximum databse key."""
+    """Decodes the maximum database key."""
     offset, key_type = decoder.DecodeUint8()
     if key_type != definitions.GlobalMetadataKeyType.MAX_DATABASE_ID:
       raise errors.ParserError('Not a MaxDatabaseIdKey')
@@ -1271,14 +1275,17 @@ class ExternalObjectEntry(utils.FromDecoderMixin):
         filename = None
         last_modified = None
       token = None
-    elif (object_type ==
-        definitions.ExternalObjectType.FILE_SYSTEM_ACCESS_HANDLE):
+    else:
+      if (object_type ==
+          definitions.ExternalObjectType.FILE_SYSTEM_ACCESS_HANDLE):
+        _, token = decoder.DecodeBlobWithLength()
+      else:
+        token = None
       blob_number = None
       mime_type = None
       size = None
       filename = None
       last_modified = None
-      _, token = decoder.DecodeBlobWithLength()
 
     return cls(offset=base_offset + offset, object_type=object_type,
         blob_number=blob_number, mime_type=mime_type, size=size,
@@ -1331,12 +1338,14 @@ class IndexedDBRecord:
   """An IndexedDB Record.
 
   Attributes:
+    path: the source file path
     offset: the offset of the record.
     key: the key of the record.
     value: the value of the record.
     sequence_number: if available, the sequence number of the record.
     type: the type of the record.
-    level: the leveldb level, None indicates the record came from a log file.
+    level: the leveldb level, if applicable, None can indicate the record
+        originated from a log file or the level could not be determined.
     recovered: True if the record is a recovered record.
   """
   path: str
@@ -1350,7 +1359,8 @@ class IndexedDBRecord:
 
   @classmethod
   def FromLevelDBRecord(
-      cls, db_record: record.LevelDBRecord
+      cls,
+      db_record: record.LevelDBRecord
   ) -> IndexedDBRecord:
     """Returns an IndexedDBRecord from a ParsedInternalKey."""
     idb_key = IndexedDbKey.FromBytes(
@@ -1366,3 +1376,76 @@ class IndexedDBRecord:
         type=db_record.record.record_type,
         level=db_record.level,
         recovered=db_record.recovered)
+
+  @classmethod
+  def FromFile(
+      cls,
+      file_path: pathlib.Path
+  ) -> Generator[IndexedDBRecord, None, None]:
+    """Yields IndexedDBRecords from a file."""
+    for db_record in record.LevelDBRecord.FromFile(file_path):
+      try:
+        yield cls.FromLevelDBRecord(db_record)
+      except(
+          errors.ParserError,
+          errors.DecoderError,
+          NotImplementedError) as err:
+        print((
+            'Error parsing Indexeddb record: '
+            f'{err} at offset {db_record.record.offset} in '
+            f'{db_record.path}'),
+            file=sys.stderr)
+        print(f'Traceback: {traceback.format_exc()}', file=sys.stderr)
+
+
+class FolderReader:
+  """A IndexedDB folder reader for Chrome/Chromium.
+
+  Attributes:
+    foldername (str): the source LevelDB folder.
+  """
+
+  def __init__(self, foldername: pathlib.Path):
+    """Initializes the FileReader.
+
+    Args:
+      foldername: the source IndexedDB folder.
+
+    Raises:
+      ValueError: if foldername is None or not a directory.
+    """
+    if not foldername or not foldername.is_dir():
+      raise ValueError(f'{foldername} is None or not a directory')
+    self.foldername = foldername
+
+  def GetRecords(
+      self,
+      use_manifest: bool = False,
+      use_sequence_number: bool = False
+  ) -> Generator[IndexedDBRecord, None, None]:
+    """Yield LevelDBRecords.
+
+    Args:
+      use_manifest: True to use the current manifest in the folder as a means to
+          find the active file set.
+      use_sequence_number: True to use the sequence number to determine the
+    Yields:
+      IndexedDBRecord.
+    """
+    leveldb_folder_reader = record.FolderReader(self.foldername)
+    for leveldb_record in leveldb_folder_reader.GetRecords(
+        use_manifest=use_manifest,
+        use_sequence_number=use_sequence_number):
+      try:
+        yield IndexedDBRecord.FromLevelDBRecord(
+            leveldb_record)
+      except(
+          errors.ParserError,
+          errors.DecoderError,
+          NotImplementedError) as err:
+        print((
+            'Error parsing Indexeddb record: '
+            f'{err} at offset {leveldb_record.record.offset} in '
+            f'{leveldb_record.path}'),
+            file=sys.stderr)
+        print(f'Traceback: {traceback.format_exc()}', file=sys.stderr)