apache-airflow-providers-sftp 5.3.4rc1__tar.gz → 5.4.0rc1__tar.gz

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-sftp
-Version: 5.3.4rc1
+Version: 5.4.0rc1
 Summary: Provider package apache-airflow-providers-sftp for Apache Airflow
 Keywords: airflow-provider,sftp,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -27,8 +27,8 @@ Requires-Dist: asyncssh>=2.12.0
 Requires-Dist: apache-airflow-providers-common-compat ; extra == "common-compat"
 Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.3.4/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.3.4
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.4.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.4.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -61,9 +61,8 @@ Provides-Extra: openlineage
 
 Package ``apache-airflow-providers-sftp``
 
-Release: ``5.3.4``
+Release: ``5.4.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `SSH File Transfer Protocol (SFTP) <https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/>`__
 
@@ -75,12 +74,12 @@ This is a provider package for ``sftp`` provider. All classes for this provider
 are in ``airflow.providers.sftp`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.3.4/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.4.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-sftp``
 
@@ -120,5 +119,5 @@ Dependent package
 ==================================================================================================================  =================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.3.4/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.4.0/changelog.html>`_.
 

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/README.rst

@@ -23,9 +23,8 @@
 
 Package ``apache-airflow-providers-sftp``
 
-Release: ``5.3.4``
+Release: ``5.4.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `SSH File Transfer Protocol (SFTP) <https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/>`__
 
@@ -37,12 +36,12 @@ This is a provider package for ``sftp`` provider. All classes for this provider
 are in ``airflow.providers.sftp`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.3.4/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.4.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-sftp``
 
@@ -82,4 +81,4 @@ Dependent package
 ==================================================================================================================  =================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.3.4/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-sftp/5.4.0/changelog.html>`_.

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/docs/changelog.rst

@@ -27,6 +27,30 @@
 Changelog
 ---------
 
+5.4.0
+.....
+
+
+Release Date: ``|PypiReleaseDate|``
+
+Features
+~~~~~~~~
+
+* ``Feature: add optional managed connection (#52700)``
+* ``Add file_pattern to template fields (#54562)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix sftp async hoook (#54763)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Switch pre-commit to prek (#54258)``
+
+.. Review and move the new changes to one of the sections above:
+   * ``Fix Airflow 2 reference in README/index of providers (#55240)``
+
 5.3.4
 .....
 

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/docs/index.rst

@@ -70,9 +70,7 @@ apache-airflow-providers-sftp package
 `SSH File Transfer Protocol (SFTP) <https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/>`__
 
 
-Release: 5.3.4
-
-Release Date: ``|PypiReleaseDate|``
+Release: 5.4.0
 
 Provider package
 ----------------
@@ -83,7 +81,7 @@ All classes for this package are included in the ``airflow.providers.sftp`` pyth
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation via
+You can install this package on top of an existing Airflow installation via
 ``pip install apache-airflow-providers-sftp``.
 For the minimum Airflow version supported, see ``Requirements`` below.
 
@@ -128,5 +126,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-sftp 5.3.4 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4.tar.gz.sha512>`__)
-* `The apache-airflow-providers-sftp 5.3.4 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.3.4-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-sftp 5.4.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-sftp 5.4.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_sftp-5.4.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/provider.yaml

@@ -22,12 +22,13 @@ description: |
     `SSH File Transfer Protocol (SFTP) <https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/>`__
 
 state: ready
-source-date-epoch: 1754503469
+source-date-epoch: 1756877505
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 5.4.0
   - 5.3.4
   - 5.3.3
   - 5.3.2

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-sftp"
-version = "5.3.4rc1"
+version = "5.4.0rc1"
 description = "Provider package apache-airflow-providers-sftp for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -54,7 +54,7 @@ requires-python = ">=3.10"
 
 # The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
-# Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
+# Make sure to run ``prek update-providers-dependencies --all-files``
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
     "apache-airflow>=2.10.0rc1",
@@ -111,8 +111,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.3.4"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.3.4/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.4.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/5.4.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/src/airflow/providers/sftp/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "5.3.4"
+__version__ = "5.4.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

apache_airflow_providers_sftp-5.4.0rc1/src/airflow/providers/sftp/exceptions.py

@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow.exceptions import AirflowException
+
+
+class ConnectionNotOpenedException(AirflowException):
+    """Thrown when a connection has not been opened and has been tried to be used."""

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/src/airflow/providers/sftp/hooks/sftp.py

@@ -21,6 +21,7 @@ from __future__ import annotations
 
 import concurrent.futures
 import datetime
+import functools
 import os
 import stat
 import warnings
@@ -34,7 +35,11 @@ from typing import IO, TYPE_CHECKING, Any, cast
 import asyncssh
 from asgiref.sync import sync_to_async
 
-from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
+from airflow.exceptions import (
+    AirflowException,
+    AirflowProviderDeprecationWarning,
+)
+from airflow.providers.sftp.exceptions import ConnectionNotOpenedException
 from airflow.providers.sftp.version_compat import BaseHook
 from airflow.providers.ssh.hooks.ssh import SSHHook
 
@@ -46,6 +51,25 @@ if TYPE_CHECKING:
     from airflow.models.connection import Connection
 
 
+def handle_connection_management(func: Callable) -> Callable:
+    @functools.wraps(func)
+    def handle_connection_management_wrapper(self, *args: Any, **kwargs: dict[str, Any]) -> Any:
+        if not self.use_managed_conn:
+            if self.conn is None:
+                raise ConnectionNotOpenedException(
+                    "Connection not open, use with hook.get_managed_conn() Managed Connection in order to create and open the connection"
+                )
+
+            return func(self, *args, **kwargs)
+
+        with self.get_managed_conn() as conn:
+            self.conn = conn
+            result = func(self, *args, **kwargs)
+            return result
+
+    return handle_connection_management_wrapper
+
+
 class SFTPHook(SSHHook):
     """
     Interact with SFTP.
@@ -86,10 +110,12 @@ class SFTPHook(SSHHook):
         self,
         ssh_conn_id: str | None = "sftp_default",
         host_proxy_cmd: str | None = None,
+        use_managed_conn: bool = True,
         *args,
         **kwargs,
     ) -> None:
         self.conn: SFTPClient | None = None
+        self.use_managed_conn = use_managed_conn
 
         # TODO: remove support for ssh_hook when it is removed from SFTPOperator
         if kwargs.get("ssh_hook") is not None:
@@ -155,6 +181,7 @@ class SFTPHook(SSHHook):
         """Get the number of open connections."""
         return self._conn_count
 
+    @handle_connection_management
     def describe_directory(self, path: str) -> dict[str, dict[str, str | int | None]]:
         """
         Get file information in a directory on the remote system.
@@ -164,36 +191,34 @@ class SFTPHook(SSHHook):
 
         :param path: full path to the remote directory
         """
-        with self.get_managed_conn() as conn:  # type: SFTPClient
-            flist = sorted(conn.listdir_attr(path), key=lambda x: x.filename)
-            files = {}
-            for f in flist:
-                modify = datetime.datetime.fromtimestamp(f.st_mtime).strftime("%Y%m%d%H%M%S")  # type: ignore
-                files[f.filename] = {
-                    "size": f.st_size,
-                    "type": "dir" if stat.S_ISDIR(f.st_mode) else "file",  # type: ignore
-                    "modify": modify,
-                }
-            return files
+        return {
+            f.filename: {
+                "size": f.st_size,
+                "type": "dir" if stat.S_ISDIR(f.st_mode) else "file",  # type: ignore[union-attr]
+                "modify": datetime.datetime.fromtimestamp(f.st_mtime or 0).strftime("%Y%m%d%H%M%S"),
+            }
+            for f in sorted(self.conn.listdir_attr(path), key=lambda f: f.filename)  # type: ignore[union-attr]
+        }
 
+    @handle_connection_management
     def list_directory(self, path: str) -> list[str]:
         """
         List files in a directory on the remote system.
 
         :param path: full path to the remote directory to list
         """
-        with self.get_managed_conn() as conn:
-            return sorted(conn.listdir(path))
+        return sorted(self.conn.listdir(path))  # type: ignore[union-attr]
 
+    @handle_connection_management
     def list_directory_with_attr(self, path: str) -> list[SFTPAttributes]:
         """
         List files in a directory on the remote system including their SFTPAttributes.
 
         :param path: full path to the remote directory to list
         """
-        with self.get_managed_conn() as conn:
-            return [file for file in conn.listdir_attr(path)]
+        return [file for file in self.conn.listdir_attr(path)]  # type: ignore[union-attr]
 
+    @handle_connection_management
     def mkdir(self, path: str, mode: int = 0o777) -> None:
         """
         Create a directory on the remote system.
@@ -204,33 +229,33 @@ class SFTPHook(SSHHook):
         :param path: full path to the remote directory to create
         :param mode: int permissions of octal mode for directory
         """
-        with self.get_managed_conn() as conn:
-            conn.mkdir(path, mode=mode)
+        return self.conn.mkdir(path, mode)  # type: ignore[union-attr,return-value]
 
+    @handle_connection_management
     def isdir(self, path: str) -> bool:
         """
         Check if the path provided is a directory.
 
         :param path: full path to the remote directory to check
         """
-        with self.get_managed_conn() as conn:
-            try:
-                return stat.S_ISDIR(conn.stat(path).st_mode)  # type: ignore
-            except OSError:
-                return False
+        try:
+            return stat.S_ISDIR(self.conn.stat(path).st_mode)  # type: ignore[union-attr,arg-type]
+        except OSError:
+            return False
 
+    @handle_connection_management
     def isfile(self, path: str) -> bool:
         """
         Check if the path provided is a file.
 
         :param path: full path to the remote file to check
         """
-        with self.get_managed_conn() as conn:
-            try:
-                return stat.S_ISREG(conn.stat(path).st_mode)  # type: ignore
-            except OSError:
-                return False
+        try:
+            return stat.S_ISREG(self.conn.stat(path).st_mode)  # type: ignore[arg-type,union-attr]
+        except OSError:
+            return False
 
+    @handle_connection_management
     def create_directory(self, path: str, mode: int = 0o777) -> None:
         """
         Create a directory on the remote system.
@@ -253,9 +278,9 @@ class SFTPHook(SSHHook):
             self.create_directory(dirname, mode)
         if basename:
             self.log.info("Creating %s", path)
-            with self.get_managed_conn() as conn:
-                conn.mkdir(path, mode=mode)
+            self.conn.mkdir(path, mode=mode)  # type: ignore
 
+    @handle_connection_management
     def delete_directory(self, path: str, include_files: bool = False) -> None:
         """
         Delete a directory on the remote system.
@@ -269,13 +294,13 @@ class SFTPHook(SSHHook):
             files, dirs, _ = self.get_tree_map(path)
             dirs = dirs[::-1]  # reverse the order for deleting deepest directories first
 
-        with self.get_managed_conn() as conn:
-            for file_path in files:
-                conn.remove(file_path)
-            for dir_path in dirs:
-                conn.rmdir(dir_path)
-            conn.rmdir(path)
+        for file_path in files:
+            self.conn.remove(file_path)  # type: ignore
+        for dir_path in dirs:
+            self.conn.rmdir(dir_path)  # type: ignore
+        self.conn.rmdir(path)  # type: ignore
 
+    @handle_connection_management
     def retrieve_file(self, remote_full_path: str, local_full_path: str, prefetch: bool = True) -> None:
         """
         Transfer the remote file to a local location.
@@ -287,28 +312,28 @@ class SFTPHook(SSHHook):
         :param local_full_path: full path to the local file or a file-like buffer
         :param prefetch: controls whether prefetch is performed (default: True)
         """
-        with self.get_managed_conn() as conn:
-            if isinstance(local_full_path, BytesIO):
-                # It's a file-like object ( BytesIO), so use getfo().
-                self.log.info("Using streaming download for %s", remote_full_path)
-                conn.getfo(remote_full_path, local_full_path, prefetch=prefetch)
-            # We use hasattr checking for 'write' for cases like google.cloud.storage.fileio.BlobWriter
-            elif hasattr(local_full_path, "write"):
-                self.log.info("Using streaming download for %s", remote_full_path)
-                # We need to cast to pass pre-commit checks
-                stream_full_path = cast("IO[bytes]", local_full_path)
-                conn.getfo(remote_full_path, stream_full_path, prefetch=prefetch)
-            elif isinstance(local_full_path, (str, bytes, os.PathLike)):
-                # It's a string path, so use get().
-                self.log.info("Using standard file download for %s", remote_full_path)
-                conn.get(remote_full_path, local_full_path, prefetch=prefetch)
-            # If it's neither, it's an unsupported type.
-            else:
-                raise TypeError(
-                    f"Unsupported type for local_full_path: {type(local_full_path)}. "
-                    "Expected a stream-like object or a path-like object."
-                )
+        if isinstance(local_full_path, BytesIO):
+            # It's a file-like object ( BytesIO), so use getfo().
+            self.log.info("Using streaming download for %s", remote_full_path)
+            self.conn.getfo(remote_full_path, local_full_path, prefetch=prefetch)
+        # We use hasattr checking for 'write' for cases like google.cloud.storage.fileio.BlobWriter
+        elif hasattr(local_full_path, "write"):
+            self.log.info("Using streaming download for %s", remote_full_path)
+            # We need to cast to pass prek hook checks
+            stream_full_path = cast("IO[bytes]", local_full_path)
+            self.conn.getfo(remote_full_path, stream_full_path, prefetch=prefetch)  # type: ignore[union-attr]
+        elif isinstance(local_full_path, (str, bytes, os.PathLike)):
+            # It's a string path, so use get().
+            self.log.info("Using standard file download for %s", remote_full_path)
+            self.conn.get(remote_full_path, local_full_path, prefetch=prefetch)  # type: ignore[union-attr]
+        # If it's neither, it's an unsupported type.
+        else:
+            raise TypeError(
+                f"Unsupported type for local_full_path: {type(local_full_path)}. "
+                "Expected a stream-like object or a path-like object."
+            )
 
+    @handle_connection_management
     def store_file(self, remote_full_path: str, local_full_path: str, confirm: bool = True) -> None:
         """
         Transfer a local file to the remote location.
@@ -319,20 +344,19 @@ class SFTPHook(SSHHook):
         :param remote_full_path: full path to the remote file
         :param local_full_path: full path to the local file or a file-like buffer
         """
-        with self.get_managed_conn() as conn:
-            if isinstance(local_full_path, BytesIO):
-                conn.putfo(local_full_path, remote_full_path, confirm=confirm)
-            else:
-                conn.put(local_full_path, remote_full_path, confirm=confirm)
+        if isinstance(local_full_path, BytesIO):
+            self.conn.putfo(local_full_path, remote_full_path, confirm=confirm)  # type: ignore
+        else:
+            self.conn.put(local_full_path, remote_full_path, confirm=confirm)  # type: ignore
 
+    @handle_connection_management
     def delete_file(self, path: str) -> None:
         """
         Remove a file on the server.
 
         :param path: full path to the remote file
         """
-        with self.get_managed_conn() as conn:
-            conn.remove(path)
+        self.conn.remove(path)  # type: ignore[arg-type, union-attr]
 
     def retrieve_directory(self, remote_full_path: str, local_full_path: str, prefetch: bool = True) -> None:
         """
@@ -348,14 +372,13 @@ class SFTPHook(SSHHook):
         if Path(local_full_path).exists():
             raise AirflowException(f"{local_full_path} already exists")
         Path(local_full_path).mkdir(parents=True)
-        with self.get_managed_conn():
-            files, dirs, _ = self.get_tree_map(remote_full_path)
-            for dir_path in dirs:
-                new_local_path = os.path.join(local_full_path, os.path.relpath(dir_path, remote_full_path))
-                Path(new_local_path).mkdir(parents=True, exist_ok=True)
-            for file_path in files:
-                new_local_path = os.path.join(local_full_path, os.path.relpath(file_path, remote_full_path))
-                self.retrieve_file(file_path, new_local_path, prefetch)
+        files, dirs, _ = self.get_tree_map(remote_full_path)
+        for dir_path in dirs:
+            new_local_path = os.path.join(local_full_path, os.path.relpath(dir_path, remote_full_path))
+            Path(new_local_path).mkdir(parents=True, exist_ok=True)
+        for file_path in files:
+            new_local_path = os.path.join(local_full_path, os.path.relpath(file_path, remote_full_path))
+            self.retrieve_file(file_path, new_local_path, prefetch)
 
     def retrieve_directory_concurrently(
         self,
@@ -419,6 +442,7 @@ class SFTPHook(SSHHook):
             for conn in conns:
                 conn.close()
 
+    @handle_connection_management
     def store_directory(self, remote_full_path: str, local_full_path: str, confirm: bool = True) -> None:
         """
         Transfer a local directory to the remote location.
@@ -431,21 +455,16 @@ class SFTPHook(SSHHook):
         """
         if self.path_exists(remote_full_path):
             raise AirflowException(f"{remote_full_path} already exists")
-        with self.get_managed_conn():
-            self.create_directory(remote_full_path)
-            for root, dirs, files in os.walk(local_full_path):
-                for dir_name in dirs:
-                    dir_path = os.path.join(root, dir_name)
-                    new_remote_path = os.path.join(
-                        remote_full_path, os.path.relpath(dir_path, local_full_path)
-                    )
-                    self.create_directory(new_remote_path)
-                for file_name in files:
-                    file_path = os.path.join(root, file_name)
-                    new_remote_path = os.path.join(
-                        remote_full_path, os.path.relpath(file_path, local_full_path)
-                    )
-                    self.store_file(new_remote_path, file_path, confirm)
+        self.create_directory(remote_full_path)
+        for root, dirs, files in os.walk(local_full_path):
+            for dir_name in dirs:
+                dir_path = os.path.join(root, dir_name)
+                new_remote_path = os.path.join(remote_full_path, os.path.relpath(dir_path, local_full_path))
+                self.create_directory(new_remote_path)
+            for file_name in files:
+                file_path = os.path.join(root, file_name)
+                new_remote_path = os.path.join(remote_full_path, os.path.relpath(file_path, local_full_path))
+                self.store_file(new_remote_path, file_path, confirm)
 
     def store_directory_concurrently(
         self,
@@ -512,28 +531,28 @@ class SFTPHook(SSHHook):
             for conn in conns:
                 conn.close()
 
+    @handle_connection_management
     def get_mod_time(self, path: str) -> str:
         """
         Get an entry's modification time.
 
         :param path: full path to the remote file
         """
-        with self.get_managed_conn() as conn:
-            ftp_mdtm = conn.stat(path).st_mtime
-            return datetime.datetime.fromtimestamp(ftp_mdtm).strftime("%Y%m%d%H%M%S")  # type: ignore
+        ftp_mdtm = self.conn.stat(path).st_mtime  # type: ignore[union-attr]
+        return datetime.datetime.fromtimestamp(ftp_mdtm).strftime("%Y%m%d%H%M%S")  # type: ignore
 
+    @handle_connection_management
     def path_exists(self, path: str) -> bool:
         """
         Whether a remote entity exists.
 
         :param path: full path to the remote file or directory
         """
-        with self.get_managed_conn() as conn:
-            try:
-                conn.stat(path)
-            except OSError:
-                return False
-            return True
+        try:
+            self.conn.stat(path)  # type: ignore[union-attr]
+        except OSError:
+            return False
+        return True
 
     @staticmethod
     def _is_path_match(path: str, prefix: str | None = None, delimiter: str | None = None) -> bool:
@@ -718,19 +737,16 @@ class SFTPHookAsync(BaseHook):
             self.private_key = extra_options["private_key"]
 
         host_key = extra_options.get("host_key")
-        no_host_key_check = extra_options.get("no_host_key_check")
-
-        if no_host_key_check is not None:
-            no_host_key_check = str(no_host_key_check).lower() == "true"
-            if host_key is not None and no_host_key_check:
-                raise ValueError("Host key check was skipped, but `host_key` value was given")
-            if no_host_key_check:
-                self.log.warning(
-                    "No Host Key Verification. This won't protect against Man-In-The-Middle attacks"
-                )
-                self.known_hosts = "none"
+        nhkc_raw = extra_options.get("no_host_key_check")
+        no_host_key_check = True if nhkc_raw is None else (str(nhkc_raw).lower() == "true")
+
+        if host_key is not None and no_host_key_check:
+            raise ValueError("Host key check was skipped, but `host_key` value was given")
 
-        if host_key is not None:
+        if no_host_key_check:
+            self.log.warning("No Host Key Verification. This won't protect against Man-In-The-Middle attacks")
+            self.known_hosts = "none"
+        elif host_key is not None:
             self.known_hosts = f"{conn.host} {host_key}".encode()
 
     async def _get_conn(self) -> asyncssh.SSHClientConnection:

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/src/airflow/providers/sftp/sensors/sftp.py

@@ -54,6 +54,7 @@ class SFTPSensor(BaseSensorOperator):
 
     template_fields: Sequence[str] = (
         "path",
+        "file_pattern",
         "newer_than",
     )
 
@@ -67,6 +68,7 @@ class SFTPSensor(BaseSensorOperator):
         python_callable: Callable | None = None,
         op_args: list | None = None,
         op_kwargs: dict[str, Any] | None = None,
+        use_managed_conn: bool = True,
         deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
         **kwargs,
     ) -> None:
@@ -76,37 +78,37 @@ class SFTPSensor(BaseSensorOperator):
         self.hook: SFTPHook | None = None
         self.sftp_conn_id = sftp_conn_id
         self.newer_than: datetime | str | None = newer_than
+        self.use_managed_conn = use_managed_conn
         self.python_callable: Callable | None = python_callable
         self.op_args = op_args or []
         self.op_kwargs = op_kwargs or {}
         self.deferrable = deferrable
 
-    def poke(self, context: Context) -> PokeReturnValue | bool:
-        self.hook = SFTPHook(self.sftp_conn_id)
-        self.log.info("Poking for %s, with pattern %s", self.path, self.file_pattern)
-        files_found = []
+    def _get_files(self) -> list[str]:
+        files_from_pattern: list[str] = []
+        files_found: list[str] = []
 
         if self.file_pattern:
-            files_from_pattern = self.hook.get_files_by_pattern(self.path, self.file_pattern)
+            files_from_pattern = self.hook.get_files_by_pattern(self.path, self.file_pattern)  # type: ignore[union-attr]
             if files_from_pattern:
                 actual_files_present = [
                     os.path.join(self.path, file_from_pattern) for file_from_pattern in files_from_pattern
                 ]
             else:
-                return False
+                return files_found
         else:
             try:
                 # If a file is present, it is the single element added to the actual_files_present list to be
                 # processed. If the file is a directory, actual_file_present will be assigned an empty list,
                 # since SFTPHook.isfile(...) returns False
-                actual_files_present = [self.path] if self.hook.isfile(self.path) else []
+                actual_files_present = [self.path] if self.hook.isfile(self.path) else []  # type: ignore[union-attr]
             except Exception as e:
                 raise AirflowException from e
 
         if self.newer_than:
             for actual_file_present in actual_files_present:
                 try:
-                    mod_time = self.hook.get_mod_time(actual_file_present)
+                    mod_time = self.hook.get_mod_time(actual_file_present)  # type: ignore[union-attr]
                     self.log.info("Found File %s last modified: %s", actual_file_present, mod_time)
                 except OSError as e:
                     if e.errno != SFTP_NO_SUCH_FILE:
@@ -135,6 +137,19 @@ class SFTPSensor(BaseSensorOperator):
         else:
             files_found = actual_files_present
 
+        return files_found
+
+    def poke(self, context: Context) -> PokeReturnValue | bool:
+        self.hook = SFTPHook(self.sftp_conn_id, use_managed_conn=self.use_managed_conn)
+
+        self.log.info("Poking for %s, with pattern %s", self.path, self.file_pattern)
+
+        if self.use_managed_conn:
+            files_found = self._get_files()
+        else:
+            with self.hook.get_managed_conn():
+                files_found = self._get_files()
+
         if not len(files_found):
             return False
 

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/tests/unit/sftp/hooks/test_sftp.py

@@ -854,7 +854,7 @@ class TestSFTPHookAsync:
             "username": "username",
             "password": "password",
             "client_keys": "~/keys/my_key",
-            "known_hosts": "~/.ssh/known_hosts",
+            "known_hosts": None,
             "passphrase": "mypassphrase",
         }
 
@@ -882,6 +882,7 @@ class TestSFTPHookAsync:
             "username": "username",
             "password": "password",
             "client_keys": ["test"],
+            "known_hosts": None,
             "passphrase": "mypassphrase",
         }
 

{apache_airflow_providers_sftp-5.3.4rc1 → apache_airflow_providers_sftp-5.4.0rc1}/tests/unit/sftp/sensors/test_sftp.py

@@ -97,6 +97,14 @@ class TestSFTPSensor:
         sftp_hook_mock.return_value.close_conn.assert_not_called()
         assert output
 
+    @patch("airflow.providers.sftp.sensors.sftp.SFTPHook")
+    def test_only_creating_one_connection_with_unmanaged_conn(self, sftp_hook_mock):
+        sftp_hook_mock.return_value.isfile.return_value = True
+        sftp_hook_mock.return_value.get_mod_time.return_value = "19700101000000"
+        sftp_sensor = SFTPSensor(task_id="test", path="path/to/whatever/test", use_managed_conn=False)
+        sftp_sensor.poke({})
+        assert sftp_hook_mock.return_value.get_managed_conn.called is True
+
     @patch("airflow.providers.sftp.sensors.sftp.SFTPHook")
     def test_file_not_new_enough(self, sftp_hook_mock):
         sftp_hook_mock.return_value.isfile.return_value = True
@@ -193,9 +201,12 @@ class TestSFTPSensor:
         )
         context = {"ds": "1970-01-00"}
         output = sftp_sensor.poke(context)
-        sftp_hook_mock.return_value.get_mod_time.assert_has_calls(
-            [mock.call("/path/to/file/text_file1.txt"), mock.call("/path/to/file/text_file2.txt")]
-        )
+        assert (
+            [
+                mock.call("/path/to/file/text_file1.txt"),
+                mock.call("/path/to/file/text_file2.txt"),
+            ]
+        ) in sftp_hook_mock.return_value.get_mod_time.mock_calls
         sftp_hook_mock.return_value.close_conn.assert_not_called()
         assert output
 
@@ -220,7 +231,7 @@ class TestSFTPSensor:
         )
         context = {"ds": "1970-01-00"}
         output = sftp_sensor.poke(context)
-        sftp_hook_mock.return_value.get_mod_time.assert_has_calls(
+        sftp_hook_mock.return_value.get_mod_time.mock_call(
             [
                 mock.call("/path/to/file/text_file1.txt"),
                 mock.call("/path/to/file/text_file2.txt"),