fuse4dbricks 0.7.3__tar.gz → 0.7.5__tar.gz

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/CHANGELOG.md

@@ -1,3 +1,45 @@
+# 0.7.5 (2026-08-06)
+
+- Fix fatal startup/runtime errors exiting with status 0 instead of 1.
+- Fix a race in Databricks config file validation where ownership was
+  checked on a path and the file was read from that same path separately.
+  Ownership is now checked on the open file descriptor that is read.
+- Fix prefetch reading already-downloaded chunks back from disk on every
+  read, instead of just checking they exist. Reading a large file
+  sequentially could pull roughly an order of magnitude more bytes off
+  local disk than the file actually contains.
+
+# 0.7.4 (2026-06-20)
+
+- Make writes durable and immediately visible on `close()`. The upload ran only
+  in `release()`, which the kernel does not wait for and whose errors it
+  discards, so `close()` returned before the data was uploaded — a file written
+  through the mount was invisible (and write errors were silent) for the
+  duration of the asynchronous upload. The upload now runs in `flush()`, which
+  handles the `close()` syscall: the kernel blocks on it, the upload completes
+  before `close()` returns, and a failed upload surfaces as a `close()` error.
+  `release()` keeps the upload as a fallback for the rare close-without-flush.
+- Fix a stale read after overwriting a file within the same second. Cached
+  chunks are keyed by the server `mtime`, but the Files API `Last-Modified`
+  header has only 1-second resolution (verified against the live API: no
+  sub-second timestamp and no `ETag`), so two writes in the same second shared a
+  key and a read could return the previous content. The chunk cache key now
+  carries a per-path generation counter (RAM key and on-disk chunk filename)
+  that `DataManager.invalidate_path()` bumps O(1) on every local write (wired
+  into write/flush, rename, truncate and unlink); old chunks become unreachable
+  and age out via the existing LRUs. Exposed by the durable-`close()` fix above,
+  which made same-second overwrites reachable.
+- Fix cache invalidation not reaching the kernel. Mutating ops route through
+  `invalidate()`, which evicted the in-process attribute/dir/negative caches but
+  never told the FUSE kernel module, so the kernel kept serving stale `stat`
+  results from its own attribute and dentry caches for up to its
+  `attr_timeout` / `entry_timeout` (the pyfuse3 default, 300s) — a deleted file
+  kept stat-ing OK and a recreated one stayed hidden. `invalidate()` now also
+  resolves the path through the inode manager and calls
+  `pyfuse3.invalidate_entry_async()` (drop the parent dentry; the async variant
+  avoids deadlocking against the unlink/rename/create request in flight) and
+  `pyfuse3.invalidate_inode()` (drop the inode's cached attributes and data).
+
 # 0.7.3 (2026-06-19)
 
 - Fix `mv` of a freshly written file failing with an I/O error. `rename()`

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fuse4dbricks
-Version: 0.7.3
+Version: 0.7.5
 Summary: FUSE driver for Databricks Unity Catalog Volumes.
 Project-URL: Homepage, https://github.com/zeehio/fuse4dbricks
 Author-email: Sergio Oller <sergioller@gmail.com>
@@ -322,6 +322,12 @@ discussing them, so feel free to open an issue if any of them is a problem for y
   appear until this short TTL expires. Unlike file metadata, these negative results are *not*
   shared between users.
 
+- **Revoked permissions can stay effective for a short time.** Catalog and
+  schema access permissions are cached using `--metadata-cache-ttl-catalog-sec`,
+  set to 10 minutes (600 seconds) by default. Volume access permissions are
+  cached using `--metadata-cache-ttl-sec`, set to 30 seconds by default. This
+  reduces the request load on the Unity Catalog API, improving performance.
+
 - **Writes are object-store semantics, not POSIX semantics.** The Databricks Files API is a
   full-replace store: a file is not visible to other processes until the writing process closes
   it (`release`). Concurrent writers follow last-write-wins — there is no locking. Deleting a

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/README.md

@@ -299,6 +299,12 @@ discussing them, so feel free to open an issue if any of them is a problem for y
   appear until this short TTL expires. Unlike file metadata, these negative results are *not*
   shared between users.
 
+- **Revoked permissions can stay effective for a short time.** Catalog and
+  schema access permissions are cached using `--metadata-cache-ttl-catalog-sec`,
+  set to 10 minutes (600 seconds) by default. Volume access permissions are
+  cached using `--metadata-cache-ttl-sec`, set to 30 seconds by default. This
+  reduces the request load on the Unity Catalog API, improving performance.
+
 - **Writes are object-store semantics, not POSIX semantics.** The Databricks Files API is a
   full-replace store: a file is not visible to other processes until the writing process closes
   it (`release`). Concurrent writers follow last-write-wins — there is no locking. Deleting a

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/fuse4dbricks/auth/provider.py

@@ -156,114 +156,111 @@ class DatabricksUnifiedAuthProvider:
             logger.error("Failed to read environ file %s for pid %s. Exception: %s", environ_file, pid, exc)
             return None
 
-    def _is_safe_config_file(self, config_file: str, uid: int) -> bool:
-        """Return True only if ``config_file`` is safe to read on behalf of ``uid``.
-
-        When fuse4dbricks runs as root (the allow_other deployment), it reads
-        config files out of arbitrary users' home directories. A user who
-        controls their own ``~/.databrickscfg`` could symlink it at another
-        user's config file; root would follow the link and cache the *victim's*
-        token under the *attacker's* uid. ``os.stat`` follows symlinks, so the
-        ``st_uid == uid`` check is what defeats that: a symlinked-to victim file
-        is owned by the victim, not the requesting uid, and is rejected. This is
-        applied to *every* config file path, env-supplied or default, never just
-        one branch.
+    def _read_token_from_path(
+        self, config_file: str, uid: int, profile: str
+    ) -> tuple[bool, Optional[str]]:
+        """Reads ``profile``'s token from ``config_file`` on behalf of ``uid``.
+
+        Returns ``(True, token)`` if ``config_file`` exists and passes the
+        ownership/type checks below (``token`` is ``None`` if the profile or
+        its token key is missing -- a content error, not a trust failure, so
+        it is not retried against another location). Returns ``(False,
+        None)`` if the path does not exist or fails validation, so the
+        caller knows to fall back to the next candidate location.
+
+        fuse4dbricks may run as root and read config files out of arbitrary
+        users' home directories (the allow_other deployment), so a path is
+        trusted only if it is owned by ``uid``. The file is opened once;
+        ownership is checked with ``fstat`` on that descriptor, and the token
+        is read from the same descriptor -- never a second, independent
+        lookup of ``config_file`` by name -- so the ownership check and the
+        read always refer to the same file.
         """
         try:
-            st = os.stat(config_file)
+            fd = os.open(config_file, os.O_RDONLY)
         except FileNotFoundError:
             # A missing config file is the normal state for any uid without a
             # Databricks setup (e.g. root or a system daemon probing the mount).
             # Logging it at error level floods the journal with one line per
             # request, so keep it at debug.
             logger.debug("Config file %s does not exist, ignoring", config_file)
-            return False
+            return False, None
         except OSError as exc:
-            # Other stat failures (permission denied, etc.) are unexpected and
+            # Other open failures (permission denied, etc.) are unexpected and
             # worth surfacing, but are not fatal to the mount.
+            logger.warning("Cannot open config file %s: %s", config_file, exc)
+            return False, None
+
+        try:
+            st = os.fstat(fd)
+        except OSError as exc:
             logger.warning("Cannot stat config file %s: %s", config_file, exc)
-            return False
+            os.close(fd)
+            return False, None
         if not stat.S_ISREG(st.st_mode):
             logger.error("Config file %s is not a regular file, ignoring", config_file)
-            return False
+            os.close(fd)
+            return False, None
         if st.st_uid != uid or not (st.st_mode & stat.S_IRUSR):
             logger.error(
                 "Config file %s is not owned and readable by uid %s, ignoring",
                 config_file, uid,
             )
-            return False
-        return True
+            os.close(fd)
+            return False, None
 
-    def _get_profile_and_config_file_name(self, env: dict[str, str]|None, uid: int) -> tuple[str, str] | tuple[None, None]:
-        """ Gets the databricks profile and config file name.
-        1. Check if env defines DATABRICKS_CONFIG_PROFILE and DATABRICKS_CONFIG_FILE
-        2. If not, default to DATABRICKS_CONFIG_PROFILE=DEFAULT and DATABRICKS_CONFIG_FILE=~/.databrickscfg (in the user's home directory)
-
-        Whichever path is selected must be owned by and readable for ``uid``
-        (see ``_is_safe_config_file``); an env-supplied path that fails the
-        check falls through to the default location.
-        """
-        if env is not None:
-            profile = env.get("DATABRICKS_CONFIG_PROFILE", "DEFAULT")
-            config_file = env.get("DATABRICKS_CONFIG_FILE")
-        else:
-            profile = "DEFAULT"
-            config_file = None
-
-        # An env-supplied config file that fails validation is ignored, falling
-        # through to the default location below.
-        if config_file is not None and not self._is_safe_config_file(config_file, uid):
-            config_file = None
-
-        if config_file is None:
-            # Default to ~/.databrickscfg in the user's home directory. This path
-            # is validated identically: under root it may live in another user's
-            # home and must still be owned by the requesting uid.
-            home_dir = self._home_for_uid(uid)
-            default_config = os.path.join(home_dir, ".databrickscfg")
-            if self._is_safe_config_file(default_config, uid):
-                config_file = default_config
-
-        if config_file is None:
-            # Expected for any uid without a Databricks config; debug-level so a
-            # repeatedly-probing uid (e.g. root) does not flood the journal.
-            logger.debug("No databricks config file found for uid %s. Checked environment variable DATABRICKS_CONFIG_FILE and default location", uid)
-            return None, None
-        return profile, config_file
-
-    def _get_token_from_config(self, profile: str, config_file: str) -> Optional[str]:
-        """ Reads the databricks token from the config file for the given profile. """
         config = configparser.ConfigParser()
         try:
-            config.read(config_file)
+            with os.fdopen(fd, "r") as f:
+                config.read_file(f)
         except Exception as exc:
             logger.error("Failed to read databricks config file at %s for profile %s. Exception: %s\n%s", config_file, profile, exc, traceback.format_exc())
-            return None
+            return True, None
+
         if profile not in config:
             logger.error("Profile '%s' not found in %s", profile, config_file)
-            return None
+            return True, None
         try:
-            token = config[profile]["token"]
-            return token
+            return True, config[profile]["token"]
         except KeyError:
             logger.error(f"'token' not found in profile '{profile}' of {config_file}")
-            return None
+            return True, None
 
     async def get_access_token(self, ctx: pyfuse3.RequestContext) -> str|None:
-        """ Gets the databricks token for the given request context by checking the environment variables of the requesting process and then the config file. """
+        """ Gets the databricks token for the given request context by checking the environment variables of the requesting process and then the config file.
+        1. Check if the process defines DATABRICKS_TOKEN in its environment.
+        2. Otherwise, look up a config file: DATABRICKS_CONFIG_PROFILE / DATABRICKS_CONFIG_FILE
+           from the process environment if set, falling back to profile DEFAULT and
+           ~/.databrickscfg in the uid's home directory. An env-supplied config file that
+           fails validation (see ``_read_token_from_path``) falls through to the default.
+        """
         env = self._get_env_for_pid(ctx.pid)
         # The process defines an access token:
         if env is not None and "DATABRICKS_TOKEN" in env:
             return env["DATABRICKS_TOKEN"]
-        # The process does not define a token, but defines a profile or config file:
-        profile, config_file = self._get_profile_and_config_file_name(env, ctx.uid)
-        if profile is None or config_file is None:
-            # Routine when a uid has neither a token in its environment nor a
-            # config file; the caller turns this into EACCES. Debug-level to
-            # avoid one error line per request from a probing uid.
-            logger.debug("No databricks token found in environment variables for pid %s, and no profile or config file found for uid %s", ctx.pid, ctx.uid)
-            return None
-        return self._get_token_from_config(profile, config_file)
+
+        if env is not None:
+            profile = env.get("DATABRICKS_CONFIG_PROFILE", "DEFAULT")
+            config_file = env.get("DATABRICKS_CONFIG_FILE")
+        else:
+            profile = "DEFAULT"
+            config_file = None
+
+        if config_file is not None:
+            found, token = self._read_token_from_path(config_file, ctx.uid, profile)
+            if found:
+                return token
+
+        home_dir = self._home_for_uid(ctx.uid)
+        default_config = os.path.join(home_dir, ".databrickscfg")
+        found, token = self._read_token_from_path(default_config, ctx.uid, profile)
+        if found:
+            return token
+
+        # Expected for any uid without a Databricks config; debug-level so a
+        # repeatedly-probing uid (e.g. root) does not flood the journal.
+        logger.debug("No databricks token found in environment variables for pid %s, and no profile or config file found for uid %s", ctx.pid, ctx.uid)
+        return None
 
 class AuthProvider:
     # Key for the shared token in single-principal mode. -1 is never a valid uid,

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/fuse4dbricks/fs/data_manager.py

@@ -24,16 +24,24 @@ class _ChunkRequest:
     "piece to download"
     mtime: float
     "last modified, to ensure chunks are not altered between reads"
+    gen: int
+    "cache generation: bumped on a local write so a same-second overwrite (mtime has 1s resolution) is not served stale"
     chunk_size: int
     "size of this chunk. It may be smaller than the typical chunk_size on the last chunk of o a file"
     ctx: pyfuse3.RequestContext
     "request context, used for auth when downloading"
 
+
+# Chunk cache key: (fs_path, chunk_id, mtime, gen). mtime guards against the
+# remote file changing under us; gen is a local epoch bumped on every write so a
+# stale chunk is unreachable even when the server mtime (1s resolution) collides.
+_CacheKey = Tuple[str, int, float, int]
+
 class DownloadScheduler:
     def __init__(self, process_request, num_workers=10):
         self._requests_lock = trio.Lock()
-        self._requests_priority: OrderedDict[Tuple[str, int, float], _ChunkRequest] = OrderedDict()
-        self._requests_regular: OrderedDict[Tuple[str, int, float], _ChunkRequest] = OrderedDict()
+        self._requests_priority: OrderedDict[_CacheKey, _ChunkRequest] = OrderedDict()
+        self._requests_regular: OrderedDict[_CacheKey, _ChunkRequest] = OrderedDict()
         self._requests_num_workers = num_workers
         self._process_request = process_request
 
@@ -43,7 +51,7 @@ class DownloadScheduler:
         for i in range(self._requests_num_workers):
             nursery.start_soon(self._downloader)
 
-    async def _dequeue_one(self) -> tuple[Tuple[str, int, float] | None, _ChunkRequest | None, str]:
+    async def _dequeue_one(self) -> tuple[_CacheKey | None, _ChunkRequest | None, str]:
         async with self._requests_lock:
             try:
                 cache_key, chunk_request = self._requests_priority.popitem(last=False)
@@ -92,7 +100,7 @@ class DownloadScheduler:
             pass
 
     async def enqueue_request(self, chunk_request: _ChunkRequest, high_priority: bool):
-        cache_key = (chunk_request.fs_path, chunk_request.chunk_id, chunk_request.mtime)
+        cache_key = (chunk_request.fs_path, chunk_request.chunk_id, chunk_request.mtime, chunk_request.gen)
         async with self._requests_lock:
             if high_priority:
                 # If request for that chunk was present as a regular request, promote it:
@@ -118,29 +126,54 @@ class DataManager:
         self.uc_client = uc_client
         self.persistence = persistence
         self.chunk_size = 8 * 1024 * 1024
-        self._inflight_coalescer: InflightCoalescer[Tuple[str, int, float]] = InflightCoalescer()
-        "Inflight coalescer for downloads identified by (file_id, chunk_id, mtime)"
+        self._inflight_coalescer: InflightCoalescer[_CacheKey] = InflightCoalescer()
+        "Inflight coalescer for downloads identified by (fs_path, chunk_id, mtime, gen)"
 
         self._download_scheduler = DownloadScheduler(process_request=self._process_request, num_workers=num_workers)
 
+        # Per-path cache generation. Bumped on a local write so previously cached
+        # chunks become unreachable without scanning. Absent path => generation 0.
+        self._write_generation: dict[str, int] = {}
+
         # max number of chunks to cache in memory
         max_ram_chunks = int(ram_cache_mb*1024*1024 / self.chunk_size)
-        self._ram_cache: RamCache[Tuple[str, int, float]] = RamCache(max_entries = max_ram_chunks)
+        self._ram_cache: RamCache[_CacheKey] = RamCache(max_entries = max_ram_chunks)
 
     def close(self) -> None:
         self._download_scheduler.close()
 
+    def _generation(self, fs_path: str) -> int:
+        return self._write_generation.get(fs_path, 0)
+
+    async def invalidate_path(self, fs_path: str) -> None:
+        """Drop the cached content of ``fs_path`` after a local write.
+
+        O(1): bumps the path's cache generation rather than scanning RAM or
+        globbing disk. Subsequent reads compute a new key, so the old chunks are
+        unreachable and age out via the RAM/disk LRUs. Necessary because chunks
+        are keyed by the server ``mtime``, which has only 1-second resolution and
+        so cannot distinguish two writes made within the same second.
+        """
+        self._write_generation[fs_path] = self._generation(fs_path) + 1
+
     async def _process_request(self, chunk_request: _ChunkRequest, priority: str):
         try:
-            cache_key = (chunk_request.fs_path, chunk_request.chunk_id, chunk_request.mtime)
-            # We have a chunk_request to download
-            # check if we already downloaded it:
-            chunk = await self.persistence.retrieve_chunk(fs_path=chunk_request.fs_path, chunk_index=chunk_request.chunk_id, mtime=chunk_request.mtime)
-            if chunk is None:
-                # download it.
-                chunk = await self._fetch_chunk(chunk_request)
+            cache_key = (chunk_request.fs_path, chunk_request.chunk_id, chunk_request.mtime, chunk_request.gen)
             if priority == "high":
+                # A real read needs the bytes now, so fetch them (from disk,
+                # falling back to network) and cache in RAM.
+                chunk = await self.persistence.retrieve_chunk(fs_path=chunk_request.fs_path, chunk_index=chunk_request.chunk_id, mtime=chunk_request.mtime, gen=chunk_request.gen)
+                if chunk is None:
+                    chunk = await self._fetch_chunk(chunk_request)
                 await self._ram_cache.put(cache_key, chunk)
+            else:
+                # A prefetch only needs the chunk to end up on disk. Reading
+                # it back (as retrieve_chunk does) would cost as much disk
+                # I/O as the read it's trying to save, for a chunk that may
+                # not even be read; a plain existence check is enough to
+                # decide whether a download is needed.
+                if not await self.persistence.chunk_exists(fs_path=chunk_request.fs_path, chunk_index=chunk_request.chunk_id, mtime=chunk_request.mtime, gen=chunk_request.gen):
+                    await self._fetch_chunk(chunk_request)
         finally:
             await self._inflight_coalescer.notify_done(cache_key)
 
@@ -164,39 +197,40 @@ class DataManager:
             fs_path=chunk_request.fs_path,
             chunk_index=chunk_request.chunk_id,
             mtime=chunk_request.mtime,
+            gen=chunk_request.gen,
             stream=stream,
         )
         return chunk
 
-    async def _request_fetch_ahead_chunks(self, fs_path: str, chunks_to_prefetch: list[Tuple[int, int]], mtime: float, ctx: pyfuse3.RequestContext):
+    async def _request_fetch_ahead_chunks(self, fs_path: str, chunks_to_prefetch: list[Tuple[int, int]], mtime: float, gen: int, ctx: pyfuse3.RequestContext):
         for (chunk_id, chunk_size) in chunks_to_prefetch:
-            cache_key = (fs_path, chunk_id, mtime)
+            cache_key = (fs_path, chunk_id, mtime, gen)
             (wait_event, leader) = await self._inflight_coalescer.join_or_lead(cache_key)
             if not leader:
                 continue  # already being fetched, we don't care about it
-            chunk_request = _ChunkRequest(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, chunk_size=chunk_size, ctx=ctx)
+            chunk_request = _ChunkRequest(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, gen=gen, chunk_size=chunk_size, ctx=ctx)
             await self._download_scheduler.enqueue_request(chunk_request, high_priority=False)
         return None
 
-    async def _get_chunk_from_cache_or_disk(self, fs_path: str, chunk_id: int, mtime: float) -> bytes | None:
-        cache_key = (fs_path, chunk_id, mtime)
+    async def _get_chunk_from_cache_or_disk(self, fs_path: str, chunk_id: int, mtime: float, gen: int) -> bytes | None:
+        cache_key = (fs_path, chunk_id, mtime, gen)
         # get chunk from ram
         cached = await self._ram_cache.get(cache_key)
         if cached is not None:
             return cached
         # get chunk from disk
-        chunk = await self.persistence.retrieve_chunk(fs_path, chunk_id, mtime)
+        chunk = await self.persistence.retrieve_chunk(fs_path, chunk_id, mtime, gen)
         if chunk is not None:
             await self._ram_cache.put(cache_key, chunk)
         return chunk
 
 
     async def _read_chunk(
-        self, fs_path: str, chunk_id: int, mtime: float, chunk_size: int, 
+        self, fs_path: str, chunk_id: int, mtime: float, gen: int, chunk_size: int,
         ctx: pyfuse3.RequestContext, out_dict: dict[int, bytes|None]
     ):
-        cache_key = (fs_path, chunk_id, mtime)
-        chunk = await self._get_chunk_from_cache_or_disk(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime)
+        cache_key = (fs_path, chunk_id, mtime, gen)
+        chunk = await self._get_chunk_from_cache_or_disk(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, gen=gen)
         if chunk is not None:
             out_dict[chunk_id] = chunk
             return
@@ -206,12 +240,12 @@ class DataManager:
 
         if leader:
             # get chunk from network
-            chunk_request = _ChunkRequest(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, chunk_size=chunk_size, ctx=ctx)
+            chunk_request = _ChunkRequest(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, gen=gen, chunk_size=chunk_size, ctx=ctx)
             await self._download_scheduler.enqueue_request(chunk_request, high_priority=True)
 
         # Now we wait for the download to complete and fetch it from RAM or Disk
         await wait_event.wait()
-        chunk = await self._get_chunk_from_cache_or_disk(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime)
+        chunk = await self._get_chunk_from_cache_or_disk(fs_path=fs_path, chunk_id=chunk_id, mtime=mtime, gen=gen)
         # here if chunk is none, we will return None and let the caller decide what to do (probably return EIO)
         out_dict[chunk_id] = chunk
         return
@@ -224,6 +258,9 @@ class DataManager:
         Reads file contents
         Raises on error raise pyfuse3.FUSEError(errno.EIO)
         """
+        # Resolve the cache generation once for the whole read so every chunk
+        # (and the prefetch below) shares a consistent key.
+        gen = self._generation(fs_path)
         # Determine path and chunks that need to be read
         if offset >= file_size:
             return bytes()
@@ -246,7 +283,7 @@ class DataManager:
                 chunk_size = last_chunk_size
             else:
                 chunk_size = self.chunk_size
-            await self._read_chunk(fs_path, chunk_id, mtime, chunk_size, ctx, chunks)
+            await self._read_chunk(fs_path, chunk_id, mtime, gen, chunk_size, ctx, chunks)
         else:
             # dictionary: chunk_id -> bytes
             async with trio.open_nursery() as nursery:
@@ -256,7 +293,7 @@ class DataManager:
                     else:
                         chunk_size = self.chunk_size
                     nursery.start_soon(
-                        self._read_chunk, fs_path, chunk_id, mtime, chunk_size, ctx, chunks
+                        self._read_chunk, fs_path, chunk_id, mtime, gen, chunk_size, ctx, chunks
                     )
         # fetch ahead next 10 chunks once we are beyond chunk 0 (avoid prefetching too much on `head *` command)
         num_chunks_to_prefetch = 10 if end_chunk > 0 else 1
@@ -273,7 +310,7 @@ class DataManager:
                 chunk_size = self.chunk_size
             # Add to list to fetch:
             chunks_to_prefetch.append((chunk_to_prefetch, chunk_size))
-        await self._request_fetch_ahead_chunks(fs_path, chunks_to_prefetch, mtime, ctx)
+        await self._request_fetch_ahead_chunks(fs_path, chunks_to_prefetch, mtime, gen, ctx)
         # Assemble chunks
         result = bytearray()
         for chunk_id in chunks_to_read:
@@ -287,6 +324,3 @@ class DataManager:
             )
             result.extend(chunk_data[chunk_start:chunk_end])
         return bytes(result)
-
-    async def write(self, fs_path: str, offset: int, buffer: bytes, ctx: pyfuse3.RequestContext) -> int:
-        raise pyfuse3.FUSEError(errno.EACCES)

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/fuse4dbricks/fs/inode_manager.py

@@ -134,6 +134,19 @@ class InodeManager:
         if entry.ref_count <= 0:
             self._delete_inode_internal(inode)
 
+    def detach_path(self, fs_path: str) -> None:
+        """Detach ``fs_path`` -- and, if it is a directory, everything
+        beneath it -- from the tree after it has been deleted remotely.
+
+        A no-op if ``fs_path`` has no known inode. This only touches the
+        in-process tree; it does not tell the kernel to drop its own cached
+        attributes/dentries for the path (see
+        ``MetadataManager._invalidate_kernel_cache`` for that).
+        """
+        inode = self.get_inode_by_path(fs_path)
+        if inode is not None:
+            self._prune_subtree(inode)
+
     def add_entry(
         self, parent_inode: int, name: str, attr: InodeEntryAttr
     ):

{fuse4dbricks-0.7.3 → fuse4dbricks-0.7.5}/fuse4dbricks/fs/metadata_manager.py

@@ -15,7 +15,7 @@ import pyfuse3
 
 from fuse4dbricks.api.uc_client import (UcNodeType, UnityCatalogClient,
                                         UnityCatalogEntry)
-from fuse4dbricks.fs.inode_manager import InodeEntry, InodeEntryAttr
+from fuse4dbricks.fs.inode_manager import InodeEntry, InodeEntryAttr, InodeManager
 from fuse4dbricks.fs.utils import (fs_to_uc_path,  fs_to_securable,
                                    InflightCoalescer,
                                    uc_to_fs_path)
@@ -54,6 +54,7 @@ class MetadataManager:
         max_entries=20000,
         ttl_catalog=600.0,
         ttl_negative: float=5.0,
+        inode_manager: InodeManager | None = None,
     ):
         """
         :param uc_client: The Unity Catalog API client.
@@ -61,8 +62,15 @@ class MetadataManager:
         :param max_entries: Maximum number of metadata entries to keep in RAM.
         :param ttl_negative: Time-to-live in seconds for negative (not-found)
             cache entries. Kept short because it hides newly-created paths.
+        :param inode_manager: Used by ``invalidate`` to resolve an fs_path to an
+            inode so the kernel's own attribute/dentry cache can be dropped too.
+            Optional: when ``None`` only the in-process caches are evicted (the
+            kernel still expires its entries on its own ``attr_timeout`` /
+            ``entry_timeout``). Always wired in production; left unset in unit
+            tests that exercise the in-process caches in isolation.
         """
         self.uc_client = uc_client
+        self._inode_manager = inode_manager
         self._ttl: float = ttl
         self._ttl_catalog = ttl_catalog
         self._ttl_negative = ttl_negative
@@ -496,7 +504,60 @@ class MetadataManager:
             if parent_path in self._dir_cache:
                 del self._dir_cache[parent_path]
         except Exception:
-            pass
+            logger.warning("Cache invalidation failed for %s", fs_path, exc_info=True)
+
+        # Evicting our own caches is not enough: the kernel keeps its own
+        # attribute and dentry caches (for up to attr_timeout / entry_timeout),
+        # so without this a deleted file would keep stat-ing OK and a fresh one
+        # would stay hidden until the kernel's timeout expired. Tell the kernel
+        # to forget this path too.
+        self._invalidate_kernel_cache(fs_path)
+
+    def _invalidate_kernel_cache(self, fs_path: str):
+        """Drop the kernel's cached attributes, data and dentry for ``fs_path``.
+
+        Resolves the path through the inode manager and notifies the FUSE
+        kernel module. A no-op when no inode manager is wired (unit tests) or
+        when neither the path nor its parent is currently known to the kernel.
+        """
+        if self._inode_manager is None:
+            return
+        norm = fs_path.rstrip("/") or "/"
+        if norm == "/":
+            return
+        parent_path, _, name = norm.rpartition("/")
+        parent_path = parent_path or "/"
+
+        # Forget the directory entry in the parent so a deleted name stops
+        # resolving and a recreated one is re-looked-up. Must use the *_async
+        # variant: invalidate() runs inside the very unlink/rename/create
+        # request that touched this entry, and the synchronous invalidate_entry
+        # would deadlock against it.
+        parent_inode = self._inode_manager.get_inode_by_path(parent_path)
+        if parent_inode is not None:
+            try:
+                pyfuse3.invalidate_entry_async(
+                    parent_inode, name.encode("utf-8"), ignore_enoent=True
+                )
+            except Exception:
+                logger.debug(
+                    "invalidate_entry_async failed for %s", fs_path, exc_info=True
+                )
+
+        # Forget cached attributes and page data for the inode itself, so a
+        # changed size/mtime or stale contents are re-fetched on next access.
+        inode = self._inode_manager.get_inode_by_path(norm)
+        if inode is not None:
+            try:
+                pyfuse3.invalidate_inode(inode)
+            except OSError:
+                # ENOSYS (kernel lacks support) or a transient failure: the
+                # kernel's own attr_timeout still bounds how long it stays stale.
+                pass
+            except Exception:
+                logger.debug(
+                    "invalidate_inode failed for %s", fs_path, exc_info=True
+                )
 
     # =========================================================================
     # Internal Logic & Helpers