listpick 0.1.15.20__tar.gz → 0.1.16.1__tar.gz

{listpick-0.1.15.20 → listpick-0.1.16.1}/CHANGELOG.md

@@ -1,5 +1,24 @@
 # CHANGELOG.md
 Note that the changes between 0.1.11.0 and 1.1.12.0 are listed under 0.1.11
+
+## [0.1.16.1] 2025-09-07
+ - Massive improvements to Picker data generation.
+   - Data generation is now multithreaded.
+     - ~5x quicker when getting information on video files
+   - Data generation is now done **asynchronously**.
+     - generate_picker_data sets the header and items of the Picker with only the Files column filled. The rest of the cells have "..." as a placeholder. Multiple threads are created and they generate data for each of the cells.
+     - The Picker can receive user interaction while the data loads.
+   - Implemented a task priority queue which the threads utilise to determine their next task.
+     - Cells that are in view are prioritised in the queue and are generated first.
+   - Created generate_picker_data_from_file() function and separated the generation-specific code into the original generate_picker_data() function.
+     - This allows one to utilise the new generation capabilities by simply passing a list of functions and filenames to the generate_picker_data() function--no toml file necessary.
+       - This has been implemented in lpfman to display file attributes.
+ - Bug fixes:
+    - Static footer string was not displayed in some cases.
+
+## [0.1.16.1] 2025-09-05
+ - Data generation is now multithreaded.
+
 ## [0.1.15.20] 2025-08-31
  - Fixed screen refresh function for default options selector.
 

{listpick-0.1.15.20/src/listpick.egg-info → listpick-0.1.16.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: listpick
-Version: 0.1.15.20
+Version: 0.1.16.1
 Summary: Listpick is a powerful TUI data tool for creating TUI apps or viewing/comparing tabulated data.
 Home-page: https://github.com/grimandgreedy/listpick
 Author: Grim
@@ -45,7 +45,7 @@ Dynamic: summary
 
 listpick is a TUI tool which displays a tabulated list of rows and allows the user to operate upon these rows--select, copy, pipe. A very simple concept but also, I hope, a powerful tool that will make it easier for people to develop TUI apps.
 
-Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr, items:list[list[str]])` and then the menu will be displayed by running `menu.run()`.
+Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr: curses.window, items: list[list[str]])` and then the menu will be displayed by running `menu.run()`.
 
 It works great as the backend for a TUI application and can also be used as a standalone data viewer.
 
@@ -53,13 +53,13 @@ It works great as the backend for a TUI application and can also be used as a st
 
 # Quickstart
 
-Install listpick
+Install listpick:
 
 ```
 python -m pip installl "listpick[full]"
 ```
 
-Create a picker object to display rows:
+Create a Picker:
 
 ```
 from listpick.listpick_app import Picker, start_curses, close_curses
@@ -79,7 +79,7 @@ x.run()
 close_curses(stdscr)
 
 ```
-Use the listpick binary to generate and display rows based on a list of commands:
+Or use the listpick binary to generate and display rows based on a list of commands:
 
 ```
 wget https://raw.githubusercontent.com/grimandgreedy/listpick/refs/heads/master/examples/data_generation/list_files.toml

{listpick-0.1.15.20 → listpick-0.1.16.1}/README.md

@@ -2,7 +2,7 @@
 
 listpick is a TUI tool which displays a tabulated list of rows and allows the user to operate upon these rows--select, copy, pipe. A very simple concept but also, I hope, a powerful tool that will make it easier for people to develop TUI apps.
 
-Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr, items:list[list[str]])` and then the menu will be displayed by running `menu.run()`.
+Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr: curses.window, items: list[list[str]])` and then the menu will be displayed by running `menu.run()`.
 
 It works great as the backend for a TUI application and can also be used as a standalone data viewer.
 
@@ -10,13 +10,13 @@ It works great as the backend for a TUI application and can also be used as a st
 
 # Quickstart
 
-Install listpick
+Install listpick:
 
 ```
 python -m pip installl "listpick[full]"
 ```
 
-Create a picker object to display rows:
+Create a Picker:
 
 ```
 from listpick.listpick_app import Picker, start_curses, close_curses
@@ -36,7 +36,7 @@ x.run()
 close_curses(stdscr)
 
 ```
-Use the listpick binary to generate and display rows based on a list of commands:
+Or use the listpick binary to generate and display rows based on a list of commands:
 
 ```
 wget https://raw.githubusercontent.com/grimandgreedy/listpick/refs/heads/master/examples/data_generation/list_files.toml

{listpick-0.1.15.20 → listpick-0.1.16.1}/TODO.md

@@ -28,6 +28,12 @@
 > - [x] Selected column isn't highlighted properly when cell beginning is offscreen
 > - [x] Header rows not aligned properly sometimes...
 > - [ ] Check leftmost_char after resizing
+> - [ ] Kill threads when we exit the picker.
+> - [ ] Data locks when data is still being generated. No modifications.
+> - [ ] (!!!) Cell cursor value with hidden columns
+>   - [ ] The selected column can be a hidden column
+>   - [ ] Hide all columns and toggle cell cursor mode. >> Hangs
+>   - [ ] Toggle last visible column in cell cursor mode. >> Hangs
 
 
 
@@ -388,6 +394,7 @@
 >   - [x] Update footer height when there is a search query.
 > - [x] Header columns are not aligned with long header values.
 >   - [x] Done: 2025-08-27
+> - [ ] Pressing page forward/page back in an empty picker changes the index in the footer from 0 to 1...
 
 
 

listpick-0.1.16.1/examples/data_generation/video_duplicates.toml

@@ -0,0 +1,50 @@
+#
+# video_duplicates.toml
+# Generate list of mp4 files along with their: sha1, duration, width, height, and size.
+# 
+# Author: GrimAndGreedy
+# License: MIT
+
+
+[environment]
+cwd="~/Videos/"
+
+[data]
+# files_command = "eza -1 --no-quotes | grep -E mp4$"
+
+commands = [
+  # """find . -iname '*.m4v' -or -iname '*.webm'""",
+  """eza -1 --no-quotes | grep -E mp4$""",
+  """du -hs {} | awk '{{ print $1 }}'""",
+
+  # """sha1sum {} | awk '{{print $1}}'""",
+
+  ## Duration, width, height methods
+  """mediainfo --Inform="Video;%Duration/String%" {}""",
+  """mediainfo --Inform="Video;%Width%" {}""",
+  """mediainfo --Inform="Video;%Height%" {}""",
+  """mediainfo --Inform="Video;%BitRate/String%" {}""",
+  """mediainfo --Inform="Video;%CodecID%" {}""",
+  """mediainfo --Inform="Video;%BitDepth%" {}""",
+
+  ## Alternate duration, width, height methods
+  # """ffprobe -show_entries format=duration -v quiet -of csv="p=0" -i {}""",
+  # """ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=s=x:p=0 {}""",
+  # """ffprobe -v error -select_streams v:0 -show_entries stream=height -of csv=s=x:p=0 {}""",
+]
+
+header = [
+  "File",
+  "Size",
+  # "sha1",
+  "Duration",
+  "Width",
+  "Height",
+  "Bit Rate",
+  "Codec",
+  "Bit Depth",
+]
+
+# run = [
+#   "mpv",
+# ]

{listpick-0.1.15.20 → listpick-0.1.16.1}/setup.py

@@ -16,7 +16,7 @@ with open("README.md", "r", encoding = "utf-8") as fh:
 
 setuptools.setup(
     name = "listpick",
-    version = "0.1.15.20",
+    version = "0.1.16.1",
     author = "Grim",
     author_email = "grimandgreedy@protonmail.com",
     description = "Listpick is a powerful TUI data tool for creating TUI apps or viewing/comparing tabulated data.",

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick/listpick_app.py

@@ -36,13 +36,14 @@ from listpick.utils.paste_operations import *
 from listpick.utils.searching import search
 from listpick.ui.help_screen import help_lines
 from listpick.ui.keys import picker_keys, notification_keys, options_keys, help_keys
-from listpick.utils.generate_data import generate_picker_data
+# from listpick.utils.generate_data import generate_picker_data
+from listpick.utils.generate_data_multithreaded import generate_picker_data_from_file
 from listpick.utils.dump import dump_state, load_state, dump_data
 from listpick.ui.build_help import build_help_rows
 from listpick.ui.footer import StandardFooter, CompactFooter, NoFooter
 from listpick.utils.picker_log import setup_logger
 from listpick.utils.user_input import get_char, open_tty
-from listpick.pane.pane_functions import right_split_file_attributes, right_split_graph, right_split_display_list
+from listpick.pane.pane_functions import right_split_file_attributes, right_split_file_attributes_dynamic, right_split_graph, right_split_display_list
 from listpick.pane.get_data import *
 
 
@@ -77,7 +78,7 @@ class Picker:
         timer: float = 5,
 
         get_new_data: bool =False,                          # Whether we can get new data
-        refresh_function: Optional[Callable] = lambda: [],  # The function with which we get new data
+        refresh_function: Optional[Callable] = lambda items, header, visible_rows_indices, getting_data: None,  # The function with which we get new data
         get_data_startup: bool =False,                      # Whether we should get data at statrup
         track_entries_upon_refresh: bool = True,
         pin_cursor: bool = False,
@@ -127,17 +128,18 @@ class Picker:
         columns_sort_method: list[int] = [0],
         key_chain: str = "",
         last_key: Optional[str] = None,
+        disabled_keys: list=[],
 
         paginate: bool =False,
         cancel_is_back: bool = False,
         mode_index: int = 0,
         modes: list[dict] = [],
-        display_modes: bool =False,
-        require_option: list=[],
-        require_option_default: list=[],
+        display_modes: bool = False,
+        require_option: list[bool] = [],
+        require_option_default: bool = False,
         option_functions: list[Callable[..., Tuple[bool, str]]] = [],
         default_option_function: Callable[..., Tuple[bool, str]] = default_option_input,
-        disabled_keys: list=[],
+
 
         show_header: bool = True,
         show_row_header: bool = False,
@@ -145,7 +147,7 @@ class Picker:
         footer_style: int = 0,
         footer_string: str="",
         footer_string_auto_refresh: bool=False,
-        footer_string_refresh_function: Optional[Callable] = None,
+        footer_string_refresh_function: Optional[Callable] = lambda : None,
         footer_timer: float=1,
         get_footer_string_startup=False,
         unicode_char_width: bool = True,
@@ -195,6 +197,9 @@ class Picker:
         split_right: bool = False,
         right_panes: list = [],
         right_pane_index: int = 0,
+
+        # getting_data: threading.Event = threading.Event(),
+
     ):
         self.stdscr = stdscr
         self.items = items
@@ -280,7 +285,7 @@ class Picker:
         self.footer_string_auto_refresh = footer_string_auto_refresh
         self.footer_string_refresh_function = footer_string_refresh_function
         self.footer_timer = footer_timer
-        self.get_footer_string_startup = get_footer_string_startup,
+        self.get_footer_string_startup = get_footer_string_startup
         self.unicode_char_width = unicode_char_width
 
 
@@ -312,7 +317,6 @@ class Picker:
 
 
         # Refresh function variables
-        self.data_refreshed = False
         self.refreshing_data = False
         self.data_lock = threading.Lock()
         self.data_ready = False
@@ -347,12 +351,22 @@ class Picker:
         self.split_right = split_right
         self.right_panes = right_panes
         self.right_pane_index = right_pane_index
+        self.visible_rows_indices = []
+
+
+
+
         self.initialise_picker_state(reset_colours=self.reset_colours)
 
         # Note: We have to set the footer after initialising the picker state so that the footer can use the get_function_data method
         self.footer_options = [StandardFooter(self.stdscr, colours_start, self.get_function_data), CompactFooter(self.stdscr, colours_start, self.get_function_data), NoFooter(self.stdscr, colours_start, self.get_function_data)]
         self.footer = self.footer_options[self.footer_style]
-        self.__version__ = "1.0"
+
+
+
+        # getting_data.is_set() is True when we are getting data
+        self.getting_data = threading.Event()
+        self.getting_data.set()
 
     def __sizeof__(self):
 
@@ -461,8 +475,11 @@ class Picker:
             end_index = min(start_index + self.items_per_page, len(self.indexed_items))
         if len(self.indexed_items) == 0: start_index, end_index = 0, 0
 
-        rows = [v[1] for v in self.indexed_items[start_index:end_index]] if len(self.indexed_items) else self.items
-        return rows
+        self.visible_rows = [v[1] for v in self.indexed_items[start_index:end_index]] if len(self.indexed_items) else self.items
+        # self.visible_rows_indices = [v[0] for v in self.indexed_items[start_index:end_index]] if len(self.indexed_items) else []
+        self.visible_rows_indices.clear()
+        self.visible_rows_indices.extend([v[0] for v in self.indexed_items[start_index:end_index]])
+        return self.visible_rows
 
     def initialise_picker_state(self, reset_colours=False) -> None:
         """ Initialise state variables for the picker. These are: debugging and colours. """
@@ -556,7 +573,10 @@ class Picker:
                     self.cursor_pos_id = self.indexed_items[self.cursor_pos][1][self.id_column]
                     self.cursor_pos_prev = self.cursor_pos
         
-            self.items, self.header = self.refresh_function()
+
+
+            self.getting_data.clear()
+            self.refresh_function(self.items, self.header, self.visible_rows_indices, self.getting_data)
 
         self.items = pad_lists_to_same_length(self.items)
                     
@@ -564,7 +584,7 @@ class Picker:
         ## Ensure that items is a List[List[Str]] object
         if len(self.items) > 0 and not isinstance(self.items[0], list):
             self.items = [[item] for item in self.items]
-        self.items = [[str(cell) for cell in row] for row in self.items]
+        # self.items = [[str(cell) for cell in row] for row in self.items]
 
 
         # Ensure that header is of the same length as the rows
@@ -818,8 +838,9 @@ class Picker:
         # Determine widths based only on the currently indexed rows
         # rows = [v[1] for v in self.indexed_items] if len(self.indexed_items) else self.items
         # Determine widths based only on the currently displayed indexed rows
-        rows = [v[1] for v in self.indexed_items[start_index:end_index]] if len(self.indexed_items) else self.items
-        self.column_widths = get_column_widths(rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
+        # rows = [v[1] for v in self.indexed_items[start_index:end_index]] if len(self.indexed_items) else self.items
+        self.get_visible_rows()
+        self.column_widths = get_column_widths(self.visible_rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
         visible_column_widths = [c for i,c in enumerate(self.column_widths) if i not in self.hidden_columns]
         visible_columns_total_width = sum(visible_column_widths) + len(self.separator)*(len(visible_column_widths)-1)
 
@@ -1197,6 +1218,11 @@ class Picker:
             else:
                 self.stdscr.addstr(0,self.term_w-3,"  ", curses.color_pair(self.colours_start+23) | curses.A_BOLD)
 
+        # Display data fetch symbol
+        if not self.getting_data.is_set():
+            self.stdscr.addstr(0,self.term_w-3,"  ", curses.color_pair(self.colours_start+21) | curses.A_BOLD)
+            # self.stdscr.addstr(0,self.term_w-6,"⏳", curses.color_pair(self.colours_start+21) | curses.A_BOLD)
+
         ## Display footer
         if self.show_footer:
             # self.footer = NoFooter(self.stdscr, self.colours_start, self.get_function_data)
@@ -2121,7 +2147,7 @@ class Picker:
 
                 self.stdscr = tmp
 
-                self.notification(self.stdscr, f"{repr(file_to_load)} has been loaded!")
+                # self.notification(self.stdscr, f"{repr(file_to_load)} has been loaded!")
 
                 self.set_function_data({}, reset_absent_variables=True)
                 self.load_file(self.loaded_file)
@@ -2144,7 +2170,9 @@ class Picker:
     def fetch_data(self) -> None:
         """ Refesh data asynchronously. When data has been fetched self.data_ready is set to True. """
         self.logger.info(f"function: fetch_data()")
-        tmp_items, tmp_header = self.refresh_function()
+        tmp_items, tmp_header = [], []
+        self.getting_data.clear()
+        self.refresh_function(tmp_items, tmp_header, self.visible_rows_indices, self.getting_data)
         if self.track_entries_upon_refresh:
             selected_indices = get_selected_indices(self.selections)
             self.ids = [item[self.id_column] for i, item in enumerate(self.items) if i in selected_indices]
@@ -2422,6 +2450,10 @@ class Picker:
             else: return False
 
         COLS, LINES = os.get_terminal_size()
+
+
+        getting_data_prev = False
+
         # Main loop
         while True:
             # key = self.stdscr.getch()
@@ -2430,6 +2462,16 @@ class Picker:
             if key != -1:
                 self.logger.info(f"key={key}")
 
+            # Ensure that
+
+            if not self.getting_data.is_set():
+                self.initialise_variables()
+                getting_data_prev = True
+            elif getting_data_prev:
+                ## Ensure that we reinitialise one final time after all data is retrieved.
+                self.initialise_variables()
+                getting_data_prev = False
+
             self.term_resize_event = terminal_resized(COLS, LINES)
             COLS, LINES = os.get_terminal_size()
             if self.term_resize_event: 
@@ -2948,8 +2990,8 @@ class Picker:
 
 
                 ## Scroll with column select
-                rows = self.get_visible_rows()
-                self.column_widths = get_column_widths(rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
+                self.get_visible_rows()
+                self.column_widths = get_column_widths(self.visible_rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
                 visible_column_widths = [c for i,c in enumerate(self.column_widths) if i not in self.hidden_columns]
                 column_set_width = sum(visible_column_widths)+len(self.separator)*len(visible_column_widths)
                 start_of_cell = sum(visible_column_widths[:self.selected_column])+len(self.separator)*self.selected_column
@@ -2975,8 +3017,8 @@ class Picker:
                 #     curses.flash()
 
                 ## Scroll with column select
-                rows = self.get_visible_rows()
-                self.column_widths = get_column_widths(rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
+                self.get_visible_rows()
+                self.column_widths = get_column_widths(self.visible_rows, header=self.header, max_column_width=self.max_column_width, number_columns=self.number_columns, max_total_width=self.rows_w, unicode_char_width=self.unicode_char_width)
                 visible_column_widths = [c for i,c in enumerate(self.column_widths) if i not in self.hidden_columns]
                 column_set_width = sum(visible_column_widths)+len(self.separator)*len(visible_column_widths)
                 start_of_cell = sum(visible_column_widths[:self.selected_column])+len(self.separator)*self.selected_column
@@ -3706,7 +3748,7 @@ def parse_arguments() -> Tuple[argparse.Namespace, dict]:
     #     input_arg = args.filename
 
     elif args.generate:
-        function_data["refresh_function"] = lambda : generate_picker_data(args.generate)
+        function_data["refresh_function"] = lambda items, header, visible_rows_indices, getting_data: generate_picker_data_from_file(args.generate, items, header, visible_rows_indices, getting_data)
         function_data["get_data_startup"] = True
         function_data["get_new_data"] = True
         return args, function_data
@@ -3975,7 +4017,7 @@ def main() -> None:
     # function_data["debug_level"] = 1
 
     function_data["split_right"] = False
-    function_data["right_pane_index"] = 3
+    function_data["right_pane_index"] = 2
 
     function_data["right_panes"] = [
         # Graph or random numbers generated each second
@@ -3996,15 +4038,24 @@ def main() -> None:
             "data": ["Files", [str(x) for x in range(100)]],
             "refresh_time": 1.0,
         },
-        # File attribures
+        # File attributes
         {
             "proportion": 2/3,
             "auto_refresh": False,
             "get_data": lambda data, state: [],
             "display": right_split_file_attributes,
-            "data": ["Files", [str(x) for x in range(100)]],
+            "data": [],
             "refresh_time": 1.0,
         },
+        # File attributes dynamic
+        {
+            "proportion": 2/3,
+            "auto_refresh": True,
+            "get_data": update_file_attributes,
+            "display": right_split_file_attributes_dynamic,
+            "data": [],
+            "refresh_time": 2.0,
+        },
         # List of random numbers generated each second
         {
             "proportion": 1/2,

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick/pane/get_data.py

@@ -8,6 +8,8 @@ Author: GrimAndGreedy
 License: MIT
 """
 
+from listpick.pane.pane_utils import get_file_attributes
+
 def data_refresh_randint_by_row(data, state):
     """
     Add a random number to the data if row id is the same.
@@ -93,3 +95,21 @@ def get_dl(data, state):
         data[0].append(data[0][-1]+1)
         data[1].append(dl)
     return data
+
+
+def update_file_attributes(data, state):
+    """
+    Get file attributes
+
+    data[0]: ["size: {}", filetype: {}, last modified: {}]
+    ]
+    data[1]: id
+    """
+    if state["indexed_items"]:
+        # id = state["indexed_items"][state["cursor_pos"]][1][state["id_column"]]
+        id = state["indexed_items"][state["cursor_pos"]][1][0]
+    else:
+        return [[], -1]
+    return [get_file_attributes(id), id]
+    
+

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick/pane/pane_functions.py

@@ -11,15 +11,14 @@ License: MIT
 import curses
 import os
 from listpick.pane.pane_utils import get_file_attributes, get_graph_string, escape_ansi
+from listpick.pane.get_data import update_file_attributes
 
-def right_split_file_attributes(stdscr, x, y, w, h, state, row, cell, past_data: list = [], data: list = [], test: bool = False):
+def right_split_file_attributes(stdscr, x, y, w, h, state, row, cell, data: list = [], test: bool = False):
     """
     Display file attributes in right pane.
     """
     if test: return True
 
-    os.chdir(os.path.expanduser("~/Downloads/new"))
-
     # Title
     title = "File attributes"
     if len(title) < w: title = f"{title:^{w}}"
@@ -45,7 +44,48 @@ def right_split_file_attributes(stdscr, x, y, w, h, state, row, cell, past_data:
 
     return []
 
-def right_split_graph(stdscr, x, y, w, h, state, row, cell, past_data: list = [], data: list = [], test: bool = False):
+
+def right_split_file_attributes_dynamic(stdscr, x, y, w, h, state, row, cell, data: list = [], test: bool = False):
+    """
+    Display file attributes in right pane.
+    """
+    if test: return True
+
+    # Title
+    title = "File attributes"
+    if len(title) < w: title = f"{title:^{w}}"
+    stdscr.addstr(y, x,title[:w], curses.color_pair(state["colours_start"]+4) | curses.A_BOLD)
+
+    # Separator
+    for j in range(h):
+        stdscr.addstr(j+y, x, ' ', curses.color_pair(state["colours_start"]+16))
+
+    # Display pane count
+    pane_count = len(state["right_panes"])
+    pane_index = state["right_pane_index"]
+    if pane_count > 1:
+        s = f" {pane_index+1}/{pane_count} "
+        stdscr.addstr(y+h-1, x+w-len(s)-1, s, curses.color_pair(state["colours_start"]+20))
+
+    if len(state["indexed_items"]) == 0:
+        return []
+
+    # Filename/cursor cell value
+    stdscr.addstr(y+2, x+2, cell[:w-3])
+
+    # If the cursor-hovered file is different then reload the data
+    if data[1] != cell:
+        data[:] = update_file_attributes(data, state)
+
+    # attributes = get_file_attributes(cell)
+    if len(data)  == 0: return []
+    attributes = data[0]
+    for i, attr in enumerate(attributes):
+        stdscr.addstr(y+3+i, x+4, attr[:w-5])
+
+    return []
+
+def right_split_graph(stdscr, x, y, w, h, state, row, cell, data: list = [], test: bool = False):
     """
     Display a graph of the data in right pane.
 
@@ -95,7 +135,7 @@ def right_split_graph(stdscr, x, y, w, h, state, row, cell, past_data: list = []
 
 
 
-def right_split_display_list(stdscr, x, y, w, h, state, row, cell, past_data: list = [], data: list = [], test: bool = False):
+def right_split_display_list(stdscr, x, y, w, h, state, row, cell, data: list = [], test: bool = False):
     """
     data[0]:str = title
     data[1]:list[str] = list of strings to display

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick/pane/pane_utils.py

@@ -83,6 +83,6 @@ def get_file_attributes(filename):
         return attributes
     
     except Exception as e:
-        print(f"An error occurred: {e}")
+        # print(f"An error occurred: {e}")
         return []
 

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick/utils/generate_data.py

@@ -15,8 +15,44 @@ import toml
 import logging
 
 logger = logging.getLogger('picker_log')
+import concurrent.futures
 
 def generate_columns(funcs: list, files: list) -> list[list[str]]:
+    """
+    Takes a list of functions and a list of files. 
+    Tasks are run in parallel using concurrent.futures.
+    """
+    logger.info("function: generate_columns (generate_data.py)")
+    
+    results = []
+    # Create a future object for each combination of func and file
+    with concurrent.futures.ThreadPoolExecutor() as executor:
+        futures = [[executor.submit(func, file) for func in funcs] for file in files]
+        
+        for file_futures in futures:
+            result = [future.result() for future in file_futures]
+            results.append(result)
+    return results
+
+
+def generate_columns_multithread(funcs: list, files: list) -> list[list[str]]:
+    """
+    Takes a list of functions and a list of files. 
+    Tasks are run in parallel using concurrent.futures.
+    """
+    logger.info("function: generate_columns (generate_data.py)")
+    
+    results = []
+    # Create a future object for each combination of func and file
+    with concurrent.futures.ThreadPoolExecutor() as executor:
+        futures = [[executor.submit(func, file) for func in funcs] for file in files]
+        
+        for file_futures in futures:
+            result = [future.result() for future in file_futures]
+            results.append(result)
+    return results
+
+def generate_columns_single_thread(funcs: list, files: list) -> list[list[str]]:
     """
     Takes a list of functions and a list of files. Each function is run for each file and a list of lists is returned.
     """
@@ -30,7 +66,7 @@ def generate_columns(funcs: list, files: list) -> list[list[str]]:
             except:
                 item.append("")
         items.append(item)
-        
+
     return items
 
 def command_to_func(command: str) -> Callable:

listpick-0.1.16.1/src/listpick/utils/generate_data_multithreaded.py

@@ -0,0 +1,202 @@
+#!/bin/python
+# -*- coding: utf-8 -*-
+"""
+generate_data_multithreaded.py
+Generate data for listpick Picker.
+
+
+1. Read toml file.
+2. Set environment variables.
+3. Get files from first command.
+4. Create items with "..." for cells to be filled.
+5. Create a priority queue which determines which cells are to be filled first.
+6. Create a queue updater which increases the priorty of cells which are on screen which does so each second.
+7. Create threads to start generating data for cells.
+
+Author: GrimAndGreedy
+License: MIT
+"""
+
+import subprocess
+import os
+from typing import Tuple, Callable
+import toml
+import logging
+import threading
+from queue import PriorityQueue
+import time
+
+logger = logging.getLogger('picker_log')
+
+def generate_columns_worker(
+    funcs: list,
+    files: list,
+    items: list[list[str]],
+    getting_data: threading.Event,
+    task_queue: PriorityQueue,
+    completed_cells: set,
+) -> None:
+    """ Get a task from the priorty queue and fill the data for that cell."""
+    while task_queue.qsize() > 0:
+        _, (i, j) = task_queue.get()
+
+        if (i, j) in completed_cells:
+            task_queue.task_done()
+            continue
+            
+        generate_cell(
+            func=funcs[j],
+            file=files[i],
+            items=items,
+            row=i,
+            col=j+1,
+        )
+        completed_cells.add((i, j))
+        task_queue.task_done()
+    getting_data.set()
+
+def generate_cell(func: Callable, file: str, items: list[list[str]], row: int, col: int) -> None:
+    """
+    Takes a function, file and a file and then sets items[row][col] to the result.
+    """
+    
+    items[row][col] = func(file).strip()
+
+def update_queue(task_queue: PriorityQueue, visible_rows_indices: list[int], rows: int, cols: int):
+    """ Increase the priority of getting the data for the cells that are currently visible. """
+    while task_queue.qsize() > 0:
+        time.sleep(0.1)
+        for row in visible_rows_indices:
+            for col in range(cols):
+                if 0 <= row < rows:
+                    task_queue.put((1, (row, col)))
+
+        # Delay
+        time.sleep(0.9)
+
+def command_to_func(command: str) -> Callable:
+    """
+    Convert a command string to a function that will run the command.
+
+    E.g.,
+        mediainfo {} | grep -i format
+        mediainfo {} | grep -i format | head -n 1 | awk '{{print $3}}'
+    """
+    logger.info("function: command_to_func (generate_data.py)")
+    
+    func = lambda arg: subprocess.run(command.format(repr(arg)), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.decode("utf-8").strip()
+    return func
+
+def load_environment(envs:dict):
+    """
+    Load environment variables from an envs dict.
+    """
+    logger.info("function: load_environment (generate_data.py)")
+
+    if "cwd" in envs:
+        os.chdir(os.path.expandvars(os.path.expanduser(envs["cwd"])))
+
+
+def read_toml(file_path) -> Tuple[dict, list, list]:
+    """
+    Read toml file and return the environment, commands and header sections.
+    """
+    logger.info("function: read_toml (generate_data.py)")
+    with open(file_path, 'r') as file:
+        config = toml.load(file)
+
+    environment = config['environment'] if 'environment' in config else {}
+    data = config['data'] if 'data' in config else {}
+    commands = [command.strip() for command in data['commands']] if 'commands' in data else []
+    header = [header for header in data['header']]  if 'header' in data else []
+    return environment, commands, header
+    
+    
+def generate_picker_data_from_file(
+    file_path: str,
+    items,
+    header,
+    visible_rows_indices,
+    getting_data
+) -> None:
+    """
+    Generate data for Picker based upon the toml file commands.
+    """
+    logger.info("function: generate_picker_data (generate_data.py)")
+
+
+    environment, lines, hdr = read_toml(file_path)
+
+    # Load any environment variables from the toml file
+    if environment:
+        load_environment(environment)
+
+
+    # Get list of files to be displayed in the first column.
+    get_files_command = lines[0]
+    files = subprocess.run(get_files_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.decode("utf-8").strip().split("\n")
+    files = [file.strip() for file in files if files]
+    
+    commands_list = [line.strip() for line in lines[1:]]
+    command_funcs = [command_to_func(command) for command in commands_list]
+
+    generate_picker_data(
+        files = files,
+        column_functions = command_funcs,
+        data_header = hdr,
+        items = items,
+        picker_header = header,
+        visible_rows_indices = visible_rows_indices,
+        getting_data = getting_data,
+    )
+
+def generate_picker_data(
+    files: list[str],
+    column_functions: list[Callable],
+    data_header,
+    items,
+    picker_header,
+    visible_rows_indices,
+    getting_data
+) -> None:
+    """
+    Generate data from a list of files and a list of column functions which will be used to 
+        generate subsequent columns.
+
+    This function is performed asynchronously with os.cpu_count() threads.
+
+    data_header: header list to be set for the picker
+    picker_header: the picker header will be passed in so that it can be set for the class
+
+    """
+    logger.info("function: generate_picker_data (generate_data.py)")
+
+    items.clear()
+    items.extend([[file] + ["..." for _ in column_functions] for file in files])
+    # items[:] = [[file] + ["..." for _ in column_functions] for file in files]
+    picker_header[:] = data_header
+
+
+    task_queue = PriorityQueue()
+    for i in range(len(files)):
+        for j in range(len(column_functions)):
+            task_queue.put((10, (i, j)))
+
+    num_workers = os.cpu_count()
+    if num_workers in [None, -1]: num_workers = 4
+    completed_cells = set()
+
+    for _ in range(num_workers):
+        gen_items_thread = threading.Thread(
+            target=generate_columns_worker,
+            args=(column_functions, files, items, getting_data, task_queue, completed_cells),
+        )
+        gen_items_thread.daemon = True
+        gen_items_thread.start()
+
+    update_queue_thread = threading.Thread(
+        target=update_queue,
+        args=(task_queue, visible_rows_indices, len(files), len(column_functions)),
+    )
+    update_queue_thread.daemon = True
+    update_queue_thread.start()

{listpick-0.1.15.20 → listpick-0.1.16.1/src/listpick.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: listpick
-Version: 0.1.15.20
+Version: 0.1.16.1
 Summary: Listpick is a powerful TUI data tool for creating TUI apps or viewing/comparing tabulated data.
 Home-page: https://github.com/grimandgreedy/listpick
 Author: Grim
@@ -45,7 +45,7 @@ Dynamic: summary
 
 listpick is a TUI tool which displays a tabulated list of rows and allows the user to operate upon these rows--select, copy, pipe. A very simple concept but also, I hope, a powerful tool that will make it easier for people to develop TUI apps.
 
-Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr, items:list[list[str]])` and then the menu will be displayed by running `menu.run()`.
+Rows of data can be viewed, selected, generated, saved, loaded, refreshed, modified or copied to the clipboard. Easy to integrate into your project by creating a `menu = Picker(stdscr: curses.window, items: list[list[str]])` and then the menu will be displayed by running `menu.run()`.
 
 It works great as the backend for a TUI application and can also be used as a standalone data viewer.
 
@@ -53,13 +53,13 @@ It works great as the backend for a TUI application and can also be used as a st
 
 # Quickstart
 
-Install listpick
+Install listpick:
 
 ```
 python -m pip installl "listpick[full]"
 ```
 
-Create a picker object to display rows:
+Create a Picker:
 
 ```
 from listpick.listpick_app import Picker, start_curses, close_curses
@@ -79,7 +79,7 @@ x.run()
 close_curses(stdscr)
 
 ```
-Use the listpick binary to generate and display rows based on a list of commands:
+Or use the listpick binary to generate and display rows based on a list of commands:
 
 ```
 wget https://raw.githubusercontent.com/grimandgreedy/listpick/refs/heads/master/examples/data_generation/list_files.toml

{listpick-0.1.15.20 → listpick-0.1.16.1}/src/listpick.egg-info/SOURCES.txt

@@ -46,6 +46,7 @@ src/listpick/utils/config.py
 src/listpick/utils/dump.py
 src/listpick/utils/filtering.py
 src/listpick/utils/generate_data.py
+src/listpick/utils/generate_data_multithreaded.py
 src/listpick/utils/graphing.py
 src/listpick/utils/keycodes.py
 src/listpick/utils/options_selectors.py

listpick-0.1.15.20/examples/data_generation/video_duplicates.toml

@@ -1,36 +0,0 @@
-#
-# video_duplicates.toml
-# Generate list of mp4 files along with their: sha1, duration, width, height, and size.
-# 
-# Author: GrimAndGreedy
-# License: MIT
-
-
-[environment]
-cwd="~/Downloads/new/creators/ashla/"
-# cwd="~/Videos/"
-
-[data]
-files_command = "eza -1 --no-quotes | grep -E mp4$"
-
-commands = [
-  """eza -1 --no-quotes | grep -E mp4$""",
-  # """sha1sum {} | awk '{{print $1}}'""",
-  """ffprobe -show_entries format=duration -v quiet -of csv="p=0" -i {}""",
-  """ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=s=x:p=0 {}""",
-  """ffprobe -v error -select_streams v:0 -show_entries stream=height -of csv=s=x:p=0 {}""",
-  """du -hs {} | awk '{{ print $1 }}'"""
-]
-
-header = [
-  "file",
-  # "sha1",
-  "duration",
-  "width",
-  "height",
-  "size",
-]
-
-# run = [
-#   "mpv",
-# ]