airbyte-cdk 0.0.0.dev0__py3-none-any.whl

airbyte_cdk/sources/file_based/README.md

@@ -0,0 +1,152 @@
+## Behavior
+
+The Airbyte protocol defines the actions `spec`, `discover`, `check` and `read` for a source to be compliant. Here is the high-level description of the flow for a file-based source:
+
+- spec: calls AbstractFileBasedSpec.documentation_url and AbstractFileBasedSpec.schema to return a ConnectorSpecification.
+- discover: calls Source.streams, and subsequently Stream.get_json_schema; this uses Source.open_file to open files during schema discovery.
+- check: Source.check_connection is called from the entrypoint code (in the main CDK).
+- read: Stream.read_records calls Stream.list_files which calls Source.list_matching_files, and then also uses Source.open_file to parse records from the file handle.
+
+## How to Implement Your Own
+
+To create a file-based source a user must extend three classes – AbstractFileBasedSource, AbstractFileBasedSpec, and AbstractStreamReader – to create an implementation for the connector’s specific storage system. They then initialize a FileBasedSource with the instance of AbstractStreamReader specific to their storage system.
+
+The abstract classes house the vast majority of the logic required by file-based sources. For example, when extending AbstractStreamReader, users only have to implement three methods:
+
+- list_matching_files: lists files matching the glob pattern(s) provided in the config.
+- open_file: returns a file handle for reading.
+- config property setter: concrete implementations of AbstractFileBasedStreamReader's config setter should assert that `value` is of the correct config type for that type of StreamReader.
+
+The result is that an implementation of a source might look like this:
+
+```
+class CustomStreamReader(AbstractStreamReader):
+    def open_file(self, remote_file: RemoteFile) -> FileHandler:
+        <...>
+
+    def get_matching_files(
+        self,
+        globs: List[str],
+        logger: logging.Logger,
+    ) -> Iterable[RemoteFile]:
+        <...>
+
+    @config.setter
+    def config(self, value: Config):
+        assert isinstance(value, CustomConfig)
+        self._config = value
+
+
+class CustomConfig(AbstractFileBasedSpec):
+    @classmethod
+    def documentation_url(cls) -> AnyUrl:
+        return AnyUrl("https://docs.airbyte.com/integrations/sources/s3", scheme="https")
+
+    a_spec_field: str = Field(title="A Spec Field", description="This is where you describe the fields of the spec", order=0)
+    <...>
+```
+
+For more information, feel free to check the docstrings of each classes or check specific implementations (like source-s3).
+
+## Supported File Types
+
+### Avro
+
+Avro is a serialization format developed by [Apache](https://avro.apache.org/docs/). Avro configuration options for the file-based CDK:
+
+- `double_as_string`: Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.
+
+### CSV
+
+CSV is a format loosely described by [RFC 4180](https://www.rfc-editor.org/rfc/rfc4180). The format is quite flexible which leads to a ton of options to consider:
+
+- `delimiter`: The character delimiting individual cells in the CSV data. By name, CSV is comma separated so the default value is `,`
+- `quote_char`: When quoted fields are used, it is possible for a field to span multiple lines, even when line breaks appear within such field. The default quote character is `"`.
+- `escape_char`: The character used for escaping special characters.
+- `encoding`: The character encoding of the file. By default, `UTF-8`
+- `double_quote`: Whether two quotes in a quoted CSV value denote a single quote in the data.
+- `quoting_behavior`: The quoting behavior determines when a value in a row should have quote marks added around it.
+- `skip_rows_before_header`: The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field.
+- `skip_rows_after_header`: The number of rows to skip after the header row.
+- `autogenerate_column_names`: If your CSV does not have a header row, the file-based CDK will need this enable to generate column names.
+- `null_values`: As CSV does not explicitly define a value for null values, the user can specify a set of case-sensitive strings that should be interpreted as null values.
+- `true_values`: As CSV does not explicitly define a value for positive boolean, the user can specify a set of case-sensitive strings that should be interpreted as true values.
+- `false_values`: As CSV does not explicitly define a value for negative boolean, the user can specify a set of case-sensitive strings that should be interpreted as false values.
+
+### JSONL
+
+[JSONL](https://jsonlines.org/) (or JSON Lines) is a format where each row is a JSON object. There are no configuration option for this format. For backward compatibility reasons, the JSONL parser currently supports multiline objects even though this is not part of the JSONL standard. Following some data gathering, we reserve the right to remove the support for this. Given that files have multiline JSON objects, performances will be slow.
+
+### Parquet
+
+Parquet is a file format defined by [Apache](https://parquet.apache.org/). Configuration options are:
+
+- `decimal_as_float`: Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended.
+
+### Document file types (PDF, DOCX, Markdown)
+
+For file share source connectors, the `unstructured` parser can be used to parse document file types. The textual content of the whole file will be parsed as a single record with a `content` field containing the text encoded as markdown.
+
+To use the unstructured parser, the libraries `poppler` and `tesseract` need to be installed on the system running the connector. For example, on Ubuntu, you can install them with the following command:
+
+```
+apt-get install -y tesseract-ocr poppler-utils
+```
+
+on Mac, you can install these via brew:
+
+```
+brew install poppler
+brew install tesseract
+```
+
+## Schema
+
+Having a schema allows for the file-based CDK to take action when there is a discrepancy between a record and what are the expected types of the record fields.
+
+Schema can be either inferred or user provided.
+
+- If the user defines it a format using JSON types, inference will not apply. Input schemas are a key/value pair of strings describing column name and data type. Supported types are `["string", "number", "integer", "object", "array", "boolean", "null"]`. For example, `{"col1": "string", "col2": "boolean"}`.
+- If the user enables schemaless sync, schema will `{"data": "object"}` and therefore emitted records will look like `{"data": {"col1": val1, …}}`. This is recommended if the contents between files in the stream vary significantly, and/or if data is very nested.
+- Else, the file-based CDK will infer the schema depending on the file type. Some file formats defined the schema as part of their metadata (like Parquet), some do on the record-level (like Avro) and some don't have any explicit typing (like JSON or CSV). Note that all CSV values are inferred as strings except where we are supporting legacy configurations. Any file format that does not define their schema on a metadata level will require the file-based CDK to iterate to a number of records. There is a limit of bytes that will be consumed in order to infer the schema.
+
+### Validation Policies
+
+Users will be required to select one of 3 different options, in the event that records are encountered that don’t conform to the schema.
+
+- Skip nonconforming records: check each record to see if it conforms to the user-input or inferred schema; skip the record if it doesn't conform. We keep a count of the number of records in each file that do and do not conform and emit a log message with these counts once we’re done reading the file.
+- Emit all records: emit all records, even if they do not conform to the user-provided or inferred schema. Columns that don't exist in the configured catalog probably won't be available in the destination's table since that's the current behavior.
+  Only error if there are conflicting field types or malformed rows.
+- Stop the sync and wait for schema re-discovery: if a record is encountered that does not conform to the configured catalog’s schema, we log a message and stop the whole sync. Note: this option is not recommended if the files have very different columns or datatypes, because the inferred schema may vary significantly at discover time.
+
+When the `schemaless` is enabled, validation will be skipped.
+
+## Breaking Changes (compared to previous S3 implementation)
+
+- [CSV] Mapping of type `array` and `object`: before, they were mapped as `large_string` and hence casted as strings. Given the new changes, if `array` or `object` is specified, the value will be casted as `array` and `object` respectively.
+- [CSV] Before, a string value would not be considered as `null_values` if the column type was a string. We will now start to cast string columns with values matching `null_values` to null.
+- [CSV] `decimal_point` option is deprecated: It is not possible anymore to use another character than `.` to separate the integer part from non-integer part. Given that the float is format with another character than this, it will be considered as a string.
+- [Parquet] `columns` option is deprecated: You can use Airbyte column selection in order to have the same behavior. We don't expect it, but this could have impact on the performance as payload could be bigger.
+
+## Incremental syncs
+
+The file-based connectors supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes):
+
+| Feature                                        | Supported? |
+| :--------------------------------------------- | :--------- |
+| Full Refresh Sync                              | Yes        |
+| Incremental Sync                               | Yes        |
+| Replicate Incremental Deletes                  | No         |
+| Replicate Multiple Files \(pattern matching\)  | Yes        |
+| Replicate Multiple Streams \(distinct tables\) | Yes        |
+| Namespaces                                     | No         |
+
+We recommend you do not manually modify files that are already synced. The connector has file-level granularity, which means adding or modifying a row in a CSV file will trigger a re-sync of the content of that file.
+
+### Incremental sync
+
+After the initial sync, the connector only pulls files that were modified since the last sync.
+
+The connector checkpoints the connection states when it is done syncing all files for a given timestamp. The connection's state only keeps track of the last 10 000 files synced. If more than 10 000 files are synced, the connector won't be able to rely on the connection state to deduplicate files. In this case, the connector will initialize its cursor to the minimum between the earliest file in the history, or 3 days ago.
+
+Both the maximum number of files, and the time buffer can be configured by connector developers.

airbyte_cdk/sources/file_based/__init__.py

@@ -0,0 +1,24 @@
+from .config.abstract_file_based_spec import AbstractFileBasedSpec
+from .config.csv_format import CsvFormat
+from .config.file_based_stream_config import FileBasedStreamConfig
+from .config.jsonl_format import JsonlFormat
+from .exceptions import CustomFileBasedException, ErrorListingFiles, FileBasedSourceError
+from .file_based_source import DEFAULT_CONCURRENCY, FileBasedSource
+from .file_based_stream_reader import AbstractFileBasedStreamReader, FileReadMode
+from .remote_file import RemoteFile
+from .stream.cursor import DefaultFileBasedCursor
+
+__all__ = [
+    "AbstractFileBasedSpec",
+    "AbstractFileBasedStreamReader",
+    "CsvFormat",
+    "CustomFileBasedException",
+    "DefaultFileBasedCursor",
+    "ErrorListingFiles",
+    "FileBasedSource",
+    "FileBasedSourceError",
+    "FileBasedStreamConfig",
+    "FileReadMode",
+    "JsonlFormat",
+    "RemoteFile",
+]

airbyte_cdk/sources/file_based/availability_strategy/__init__.py

@@ -0,0 +1,11 @@
+from .abstract_file_based_availability_strategy import (
+    AbstractFileBasedAvailabilityStrategy,
+    AbstractFileBasedAvailabilityStrategyWrapper,
+)
+from .default_file_based_availability_strategy import DefaultFileBasedAvailabilityStrategy
+
+__all__ = [
+    "AbstractFileBasedAvailabilityStrategy",
+    "AbstractFileBasedAvailabilityStrategyWrapper",
+    "DefaultFileBasedAvailabilityStrategy",
+]

airbyte_cdk/sources/file_based/availability_strategy/abstract_file_based_availability_strategy.py

@@ -0,0 +1,73 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from __future__ import annotations
+
+import logging
+from abc import abstractmethod
+from typing import TYPE_CHECKING, Optional, Tuple
+
+from airbyte_cdk.sources import Source
+from airbyte_cdk.sources.streams.availability_strategy import AvailabilityStrategy
+from airbyte_cdk.sources.streams.concurrent.availability_strategy import (
+    AbstractAvailabilityStrategy,
+    StreamAvailability,
+    StreamAvailable,
+    StreamUnavailable,
+)
+from airbyte_cdk.sources.streams.core import Stream
+
+if TYPE_CHECKING:
+    from airbyte_cdk.sources.file_based.stream import AbstractFileBasedStream
+
+
+class AbstractFileBasedAvailabilityStrategy(AvailabilityStrategy):
+    @abstractmethod
+    def check_availability(  # type: ignore[override]  # Signature doesn't match base class
+        self,
+        stream: Stream,
+        logger: logging.Logger,
+        _: Optional[Source],
+    ) -> Tuple[bool, Optional[str]]:
+        """
+        Perform a connection check for the stream.
+
+        Returns (True, None) if successful, otherwise (False, <error message>).
+        """
+        ...
+
+    @abstractmethod
+    def check_availability_and_parsability(
+        self,
+        stream: AbstractFileBasedStream,
+        logger: logging.Logger,
+        _: Optional[Source],
+    ) -> Tuple[bool, Optional[str]]:
+        """
+        Performs a connection check for the stream, as well as additional checks that
+        verify that the connection is working as expected.
+
+        Returns (True, None) if successful, otherwise (False, <error message>).
+        """
+        ...
+
+
+class AbstractFileBasedAvailabilityStrategyWrapper(AbstractAvailabilityStrategy):
+    def __init__(self, stream: AbstractFileBasedStream) -> None:
+        self.stream = stream
+
+    def check_availability(self, logger: logging.Logger) -> StreamAvailability:
+        is_available, reason = self.stream.availability_strategy.check_availability(
+            self.stream, logger, None
+        )
+        if is_available:
+            return StreamAvailable()
+        return StreamUnavailable(reason or "")
+
+    def check_availability_and_parsability(
+        self, logger: logging.Logger
+    ) -> Tuple[bool, Optional[str]]:
+        return self.stream.availability_strategy.check_availability_and_parsability(
+            self.stream, logger, None
+        )

airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py

@@ -0,0 +1,149 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from __future__ import annotations
+
+import logging
+import traceback
+from typing import TYPE_CHECKING, Optional, Tuple
+
+from airbyte_cdk import AirbyteTracedException
+from airbyte_cdk.sources import Source
+from airbyte_cdk.sources.file_based.availability_strategy import (
+    AbstractFileBasedAvailabilityStrategy,
+)
+from airbyte_cdk.sources.file_based.exceptions import (
+    CheckAvailabilityError,
+    CustomFileBasedException,
+    FileBasedSourceError,
+)
+from airbyte_cdk.sources.file_based.file_based_stream_reader import AbstractFileBasedStreamReader
+from airbyte_cdk.sources.file_based.remote_file import RemoteFile
+from airbyte_cdk.sources.file_based.schema_helpers import conforms_to_schema
+
+if TYPE_CHECKING:
+    from airbyte_cdk.sources.file_based.stream import AbstractFileBasedStream
+
+
+class DefaultFileBasedAvailabilityStrategy(AbstractFileBasedAvailabilityStrategy):
+    def __init__(self, stream_reader: AbstractFileBasedStreamReader) -> None:
+        self.stream_reader = stream_reader
+
+    def check_availability(  # type: ignore[override]  # Signature doesn't match base class
+        self,
+        stream: AbstractFileBasedStream,
+        logger: logging.Logger,
+        _: Optional[Source],
+    ) -> Tuple[bool, Optional[str]]:
+        """
+        Perform a connection check for the stream (verify that we can list files from the stream).
+
+        Returns (True, None) if successful, otherwise (False, <error message>).
+        """
+        try:
+            self._check_list_files(stream)
+        except CheckAvailabilityError:
+            return False, "".join(traceback.format_exc())
+
+        return True, None
+
+    def check_availability_and_parsability(
+        self,
+        stream: AbstractFileBasedStream,
+        logger: logging.Logger,
+        _: Optional[Source],
+    ) -> Tuple[bool, Optional[str]]:
+        """
+        Perform a connection check for the stream.
+
+        Returns (True, None) if successful, otherwise (False, <error message>).
+
+        For the stream:
+        - Verify the parser config is valid per check_config method of the parser.
+        - Verify that we can list files from the stream using the configured globs.
+        - Verify that we can read one file from the stream as long as the stream parser is not setting parser_max_n_files_for_parsability to 0.
+
+        This method will also check that the files and their contents are consistent
+        with the configured options, as follows:
+        - If the files have extensions, verify that they don't disagree with the
+          configured file type.
+        - If the user provided a schema in the config, check that a subset of records in
+          one file conform to the schema via a call to stream.conforms_to_schema(schema).
+        """
+        parser = stream.get_parser()
+        config_check_result, config_check_error_message = parser.check_config(stream.config)
+        if config_check_result is False:
+            return False, config_check_error_message
+        try:
+            file = self._check_list_files(stream)
+            if not parser.parser_max_n_files_for_parsability == 0:
+                self._check_parse_record(stream, file, logger)
+            else:
+                # If the parser is set to not check parsability, we still want to check that we can open the file.
+                handle = stream.stream_reader.open_file(file, parser.file_read_mode, None, logger)
+                handle.close()
+        except AirbyteTracedException as ate:
+            raise ate
+        except CheckAvailabilityError:
+            return False, "".join(traceback.format_exc())
+
+        return True, None
+
+    def _check_list_files(self, stream: AbstractFileBasedStream) -> RemoteFile:
+        """
+        Check that we can list files from the stream.
+
+        Returns the first file if successful, otherwise raises a CheckAvailabilityError.
+        """
+        try:
+            file = next(iter(stream.get_files()))
+        except StopIteration:
+            raise CheckAvailabilityError(FileBasedSourceError.EMPTY_STREAM, stream=stream.name)
+        except CustomFileBasedException as exc:
+            raise CheckAvailabilityError(str(exc), stream=stream.name) from exc
+        except Exception as exc:
+            raise CheckAvailabilityError(
+                FileBasedSourceError.ERROR_LISTING_FILES, stream=stream.name
+            ) from exc
+
+        return file
+
+    def _check_parse_record(
+        self,
+        stream: AbstractFileBasedStream,
+        file: RemoteFile,
+        logger: logging.Logger,
+    ) -> None:
+        parser = stream.get_parser()
+
+        try:
+            record = next(
+                iter(
+                    parser.parse_records(
+                        stream.config, file, self.stream_reader, logger, discovered_schema=None
+                    )
+                )
+            )
+        except StopIteration:
+            # The file is empty. We've verified that we can open it, so will
+            # consider the connection check successful even though it means
+            # we skip the schema validation check.
+            return
+        except AirbyteTracedException as ate:
+            raise ate
+        except Exception as exc:
+            raise CheckAvailabilityError(
+                FileBasedSourceError.ERROR_READING_FILE, stream=stream.name, file=file.uri
+            ) from exc
+
+        schema = stream.catalog_schema or stream.config.input_schema
+        if schema and stream.validation_policy.validate_schema_before_sync:
+            if not conforms_to_schema(record, schema):  # type: ignore
+                raise CheckAvailabilityError(
+                    FileBasedSourceError.ERROR_VALIDATING_RECORD,
+                    stream=stream.name,
+                    file=file.uri,
+                )
+
+        return None

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

@@ -0,0 +1,153 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+import copy
+from abc import abstractmethod
+from typing import Any, Dict, List, Literal, Optional, Union
+
+import dpath
+from pydantic.v1 import AnyUrl, BaseModel, Field
+
+from airbyte_cdk import OneOfOptionConfig
+from airbyte_cdk.sources.file_based.config.file_based_stream_config import FileBasedStreamConfig
+from airbyte_cdk.sources.specs.transfer_modes import DeliverPermissions
+from airbyte_cdk.sources.utils import schema_helpers
+
+
+class DeliverRecords(BaseModel):
+    class Config(OneOfOptionConfig):
+        title = "Replicate Records"
+        description = "Recommended - Extract and load structured records into your destination of choice. This is the classic method of moving data in Airbyte. It allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed and deduped depending on the destination."
+        discriminator = "delivery_type"
+
+    delivery_type: Literal["use_records_transfer"] = Field("use_records_transfer", const=True)
+
+
+class DeliverRawFiles(BaseModel):
+    class Config(OneOfOptionConfig):
+        title = "Copy Raw Files"
+        description = "Copy raw files without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. Recommended for use with unstructured text data, non-text and compressed files."
+        discriminator = "delivery_type"
+
+    delivery_type: Literal["use_file_transfer"] = Field("use_file_transfer", const=True)
+
+    preserve_directory_structure: bool = Field(
+        title="Preserve Sub-Directories in File Paths",
+        description=(
+            "If enabled, sends subdirectory folder structure "
+            "along with source file names to the destination. "
+            "Otherwise, files will be synced by their names only. "
+            "This option is ignored when file-based replication is not enabled."
+        ),
+        default=True,
+    )
+
+
+class AbstractFileBasedSpec(BaseModel):
+    """
+    Used during spec; allows the developer to configure the cloud provider specific options
+    that are needed when users configure a file-based source.
+    """
+
+    start_date: Optional[str] = Field(
+        title="Start Date",
+        description="UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.",
+        examples=["2021-01-01T00:00:00.000000Z"],
+        format="date-time",
+        pattern="^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}Z$",
+        pattern_descriptor="YYYY-MM-DDTHH:mm:ss.SSSSSSZ",
+        order=1,
+    )
+
+    streams: List[FileBasedStreamConfig] = Field(
+        title="The list of streams to sync",
+        description='Each instance of this configuration defines a <a href="https://docs.airbyte.com/cloud/core-concepts#stream">stream</a>. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.',
+        order=10,
+    )
+
+    delivery_method: Union[DeliverRecords, DeliverRawFiles, DeliverPermissions] = Field(
+        title="Delivery Method",
+        discriminator="delivery_type",
+        type="object",
+        order=7,
+        display_type="radio",
+        group="advanced",
+        default="use_records_transfer",
+        airbyte_hidden=True,
+    )
+
+    @classmethod
+    @abstractmethod
+    def documentation_url(cls) -> AnyUrl:
+        """
+        :return: link to docs page for this source e.g. "https://docs.airbyte.com/integrations/sources/s3"
+        """
+
+    @classmethod
+    def schema(cls, *args: Any, **kwargs: Any) -> Dict[str, Any]:
+        """
+        Generates the mapping comprised of the config fields
+        """
+        schema = super().schema(*args, **kwargs)
+        transformed_schema: Dict[str, Any] = copy.deepcopy(schema)
+        schema_helpers.expand_refs(transformed_schema)
+        cls.replace_enum_allOf_and_anyOf(transformed_schema)
+        cls.remove_discriminator(transformed_schema)
+
+        return transformed_schema
+
+    @staticmethod
+    def remove_discriminator(schema: Dict[str, Any]) -> None:
+        """pydantic adds "discriminator" to the schema for oneOfs, which is not treated right by the platform as we inline all references"""
+        dpath.delete(schema, "properties/**/discriminator")
+
+    @staticmethod
+    def replace_enum_allOf_and_anyOf(schema: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        allOfs are not supported by the UI, but pydantic is automatically writing them for enums.
+        Unpacks the enums under allOf and moves them up a level under the enum key
+        anyOfs are also not supported by the UI, so we replace them with the similar oneOf, with the
+        additional validation that an incoming config only matches exactly one of a field's types.
+        """
+        objects_to_check = schema["properties"]["streams"]["items"]["properties"]["format"]
+        objects_to_check["type"] = "object"
+        objects_to_check["oneOf"] = objects_to_check.pop("anyOf", [])
+        for format in objects_to_check["oneOf"]:
+            for key in format["properties"]:
+                object_property = format["properties"][key]
+                AbstractFileBasedSpec.move_enum_to_root(object_property)
+
+        properties_to_change = ["validation_policy"]
+        for property_to_change in properties_to_change:
+            property_object = schema["properties"]["streams"]["items"]["properties"][
+                property_to_change
+            ]
+            if "anyOf" in property_object:
+                schema["properties"]["streams"]["items"]["properties"][property_to_change][
+                    "type"
+                ] = "object"
+                schema["properties"]["streams"]["items"]["properties"][property_to_change][
+                    "oneOf"
+                ] = property_object.pop("anyOf")
+            AbstractFileBasedSpec.move_enum_to_root(property_object)
+
+        csv_format_schemas = list(
+            filter(
+                lambda format: format["properties"]["filetype"]["default"] == "csv",
+                schema["properties"]["streams"]["items"]["properties"]["format"]["oneOf"],
+            )
+        )
+        if len(csv_format_schemas) != 1:
+            raise ValueError(f"Expecting only one CSV format but got {csv_format_schemas}")
+        csv_format_schemas[0]["properties"]["header_definition"]["oneOf"] = csv_format_schemas[0][
+            "properties"
+        ]["header_definition"].pop("anyOf", [])
+        csv_format_schemas[0]["properties"]["header_definition"]["type"] = "object"
+        return schema
+
+    @staticmethod
+    def move_enum_to_root(object_property: Dict[str, Any]) -> None:
+        if "allOf" in object_property and "enum" in object_property["allOf"][0]:
+            object_property["enum"] = object_property["allOf"][0]["enum"]
+            object_property.pop("allOf")

airbyte_cdk/sources/file_based/config/avro_format.py

@@ -0,0 +1,25 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+
+from pydantic.v1 import BaseModel, Field
+
+from airbyte_cdk.utils.oneof_option_config import OneOfOptionConfig
+
+
+class AvroFormat(BaseModel):
+    class Config(OneOfOptionConfig):
+        title = "Avro Format"
+        discriminator = "filetype"
+
+    filetype: str = Field(
+        "avro",
+        const=True,
+    )
+
+    double_as_string: bool = Field(
+        title="Convert Double Fields to Strings",
+        description="Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers.",
+        default=False,
+    )