fluent-plugin-openlineage 0.1.2 → 0.1.3

data/spec/Naming.md

@@ -0,0 +1,500 @@
+# Naming
+
+We define the unique name strategy per resource to ensure it is followed uniformly independently of who is producing
+metadata, so we can connect lineage from various sources.
+
+Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their schedulers. The namespace for a
+dataset is the unique name for its datasource.
+
+## Datasets
+
+The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)
+
+- Namespace = scheme:[//authority] (the datasource)
+- Name = path (the datasets)
+
+### Naming conventions for common data stores
+
+This list is not exhaustive, if a data store is missing, please create an issue or open a PR.
+
+### Data Warehouses
+
+- [Athena](#Athena)
+- [Azure Cosmos DB](#Azure-Cosmos-DB)
+- [Azure Data Explorer](#Azure-Data-Explorer)
+- [Azure Synapse](#Azure-Synapse)
+- [BigQuery](#BigQuery)
+- [Cassandra](#Cassandra)
+- [MySQL](#MySQL)
+- [Postgres](#Postgres)
+- [Redshift](#Redshift)
+- [Snowflake](#Snowflake)
+- [Trino](#Trino)
+
+### Distributed file systems/blob stores
+
+- [ABFSS (Azure Data Lake Gen2)](#ABFSS "Azure Data Lake Gen2")
+- [DBFS (Databricks File System)](#DBFS "Databricks File System")
+- [GCS](#GCS)
+- [HDFS](#HDFS)
+- [Kafka](#Kafka)
+- [Local file system](#Local-file-system)
+- [S3](#S3)
+- [WASBS (Azure Blob Storage)](#WASBS "Azure Blob Storage")
+
+Datasets are called tables. Tables are organized into databases and schemas.
+
+#### Athena:
+
+Datasource hierarchy:
+
+- Host: athena.{region_name}.amazonaws.com
+
+Naming hierarchy:
+
+- Catalog
+- Database
+- Table
+
+Identifier:
+
+- Namespace: awsathena://athena.{region_name}.amazonaws.com of the service instance.
+  - Scheme = awsathena
+  - Authority = athena.{region_name}.amazonaws.com
+- Unique name: {catalog}.{database}.{table}
+  - URI = awsathena://athena.{region_name}.amazonaws.com/{catalog}.{database}.{table}
+
+#### Azure Cosmos DB:
+
+Datasource hierarchy: azurecosmos://%s.documents.azure.com/dbs/%s
+
+- Host: \<XXXXXXXXXXXX>.documents.azure.com
+- Database
+
+Naming hierarchy:
+
+- Schema
+- Table
+
+Identifier:
+
+- Namespace: azurecosmos://{host}/dbs/{database}
+  - Scheme = azurecosmos
+  - Authority = {host}
+- Unique name: /colls/{table}
+  - URI = azurecosmos://{host}.documents.azure.com/dbs/{database}/colls/{table}
+
+#### Azure Data Explorer:
+
+Datasource hierarchy:
+
+- Host: \<clustername>.\<clusterlocation>
+- Database
+- Table
+
+Naming hierarchy:
+
+- Database
+- Table
+
+Identifier:
+
+- Namespace: azurekusto://{host}.kusto.windows.net/{database}
+  - Scheme = azurekusto
+- Unique name: {database}/{table}
+  - URI = azurekusto://{host}.kusto.windows.net/{database}/{table}
+
+#### Azure Synapse:
+
+Datasource hierarchy:
+
+- Host: \<XXXXXXXXXXXX>.sql.azuresynapse.net
+- Port: 1433
+- Database: SQLPool1
+
+Naming hierarchy:
+
+- Schema
+- Table
+
+Identifier:
+
+- Namespace: sqlserver://{host}:{port};database={database};
+  - Scheme = sqlserver
+  - Authority = {host}:{port}
+- Unique name: {database}.{schema}.{table}
+  - URI = sqlserver://{host}:{port}/{database}.{schema}.{table}
+
+#### BigQuery
+
+See:
+[Creating and managing projects | Resource Manager Documentation](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+[Introduction to datasets | BigQuery](https://cloud.google.com/bigquery/docs/datasets-intro)
+[Introduction to tables | BigQuery](https://cloud.google.com/bigquery/docs/tables-intro)
+
+Datasource hierarchy:
+
+- bigquery
+
+Naming hierarchy:
+
+- Project Name: {project name} => is not unique
+- Project number: {project number} => numeric: is unique across Google cloud
+- Project ID: {project id} => readable: is unique across Google cloud
+- dataset: {dataset name} => is unique within a project
+- table: {table name} => is unique within a dataset
+
+Identifier:
+
+- Namespace: bigquery
+  - Scheme = bigquery
+  - Authority =
+- Unique name: {project id}.{dataset name}.{table name}
+  - URI = bigquery:{project id}.{dataset name}.{table name}
+
+#### Cassandra:
+
+Datasource hierarchy:
+
+- Host
+- Port
+
+Naming hierarchy:
+
+- Keyspace
+- Table
+
+Identifier:
+
+- Namespace: cassandra://{host}:{port} of the service instance.
+  - Scheme = cassandra
+  - Authority = {host}:{port}
+- Unique name: {keyspace}.{table}
+  - URI = cassandra://{host}:{port}/{keyspace}.{table}
+
+#### MySQL:
+
+Datasource hierarchy:
+
+- Host
+- Port
+
+Naming hierarchy:
+
+- Database
+- Table
+
+Identifier:
+
+- Namespace: mysql://{host}:{port} of the service instance.
+  - Scheme = mysql
+  - Authority = {host}:{port}
+- Unique name: {database}.{table}
+  - URI = mysql://{host}:{port}/{database}.{table}
+
+#### Postgres:
+
+Datasource hierarchy:
+
+- Host
+- Port
+
+Naming hierarchy:
+
+- Database
+- Schema
+- Table
+
+Identifier:
+
+- Namespace: postgres://{host}:{port} of the service instance.
+  - Scheme = postgres
+  - Authority = {host}:{port}
+- Unique name: {database}.{schema}.{table}
+  - URI = postgres://{host}:{port}/{database}.{schema}.{table}
+
+#### Redshift:
+
+Datasource hierarchy:
+
+- Host: examplecluster.\<XXXXXXXXXXXX>.us-west-2.redshift.amazonaws.com
+- Port: 5439
+
+OR
+
+- Cluster identifier
+- Region name
+- Port (defaults to 5439)
+
+Naming hierarchy:
+
+- Database
+- Schema
+- Table
+
+One can interact with Redshift using SQL or Data API. The combination of cluster identifier and region name is the only
+common unique ID available to both.
+
+Identifier:
+
+- Namespace: redshift://{cluster_identifier}.{region_name}:{port} of the cluster instance.
+  - Scheme = redshift
+  - Authority = {cluster_identifier}.{region_name}:{port}
+- Unique name: {database}.{schema}.{table}
+  - URI = redshift://{cluster_identifier}.{region_name}:{port}/{database}.{schema}.{table}
+
+#### Snowflake
+
+See:
+
+- [Account Identifiers | Snowflake Documentation](https://docs.snowflake.com/en/user-guide/admin-account-identifier)
+- [Object Identifiers | Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/identifiers.html)
+
+Datasource hierarchy:
+
+- account identifier (composite of organization name and account name)
+
+Naming hierarchy:
+
+- Database: {database name} => unique across the account
+- Schema: {schema name} => unique within the database
+- Table: {table name} => unique within the schema
+
+Identifier:
+
+- Namespace: snowflake://{organization name}-{account name}
+  - Scheme = snowflake
+  - Authority = {organization name}-{account name}
+- Name: {database}.{schema}.{table}
+  - URI = snowflake://{organization name}-{account name}/{database}.{schema}.{table}
+
+Snowflake resolves and stores names for databases, schemas, tables and columns differently depending on how they are
+[expressed in statements](https://docs.snowflake.com/en/sql-reference/identifiers-syntax) (e.g. unquoted vs quoted). The
+representation of names in OpenLineage events should be based on the canonical name that Snowflake stores. Specifically:
+
+- For dataset names, each period-delimited part (database/schema/table) should be in the simplest form it would take in
+  a statement i.e. quoted only if necessary. For example, a table `My Table` in schema `MY_SCHEMA` and in database
+  `MY_DATABASE` would be represented as `MY_DATABASE.MY_SCHEMA."My Table"`. If in doubt, check
+  [Snowflake's `ACCESS_HISTORY` view](https://docs.snowflake.com/en/sql-reference/account-usage/access_history) to see
+  how `objectName` is formed for a given table.
+- For column names, the canonical name should always be used verbatim.
+
+#### Trino:
+
+Datasource hierarchy:
+
+- Host
+- Port
+
+Naming hierarchy:
+
+- Catalog
+- Schema
+- Table
+
+Identifier:
+
+- Namespace: trino://{host}:{port} of the service instance.
+  - Scheme = trino
+  - Authority = {host}:{port}
+- Unique name: {catalog}.{schema}.{table}
+  - URI = trino://{host}:{port}/{catalog}.{schema}.{table}
+
+### Distributed file systems/blob stores
+
+#### ABFSS (Azure Data Lake Gen2)
+
+Naming hierarchy:
+
+- service name => globally unique
+- Path
+
+Identifier :
+
+- Namespace: abfss://{container name}@{service name}
+  - Scheme = abfss
+  - Authority = service name
+- Unique name: {path}
+  - URI = abfss://{container name}@{service name}{path}
+
+#### DBFS (Databricks File System)
+
+Naming hierarchy:
+
+- workspace name: globally unique
+- Path
+
+Identifier :
+
+- Namespace: hdfs://{workspace name}
+  - Scheme = hdfs
+  - Authority = workspace name
+- Unique name: {path}
+  - URI = hdfs://{workspace name}{path}
+
+#### GCS
+
+Datasource hierarchy: none, naming is global
+
+Naming hierarchy:
+
+- bucket name => globally unique
+- Path
+
+Identifier :
+
+- Namespace: gs://{bucket name}
+  - Scheme = gs
+  - Authority = {bucket name}
+- Unique name: {path}
+  - URI = gs://{bucket name}{path}
+
+#### HDFS
+
+Naming hierarchy:
+
+- Namenode: host + port
+- Path
+
+Identifier :
+
+- Namespace: hdfs://{namenode host}:{namenode port}
+  - Scheme = hdfs
+  - Authority = {namenode host}:{namenode port}
+- Unique name: {path}
+  - URI = hdfs://{namenode host}:{namenode port}{path}
+
+### Kafka
+
+Naming hierarchy:
+
+- Kafka bootstrap server host + port
+- topic name
+
+Identifier :
+
+- Namespace: kafka://{bootstrap server host}:{port}
+  - Scheme = kafka
+  - Authority = bootstrap server
+- Unique name: {topic name}
+  - URI = kafka://{bootstrap server host}:{port}/{topic name}
+
+### Local file system
+
+Datasource hierarchy:
+
+- IP
+- Port
+
+Naming hierarchy:
+
+- Path
+
+Identifier :
+
+- Namespace: file://{IP}:{port}
+  - Scheme = file
+  - Authority = {IP}:{port}
+- Unique name: {path}
+  - URI = file://{IP}:{port}{path}
+
+#### S3
+
+Naming hierarchy:
+
+- bucket name => globally unique
+- Path
+
+Identifier :
+
+- Namespace: s3://{bucket name}
+  - Scheme = s3
+  - Authority = {bucket name}
+- Unique name: {path}
+  - URI = s3://{bucket name}{path}
+
+#### WASBS (Azure Blob Storage)
+
+Naming hierarchy:
+
+- service name => globally unique
+- Path
+
+Identifier :
+
+- Namespace: wasbs://{container name}@{service name}
+  - Scheme = wasbs
+  - Authority = service name
+- Unique name: {path}
+  - URI = wasbs://{container name}@{service name}{path}
+
+## Jobs
+
+### Context
+
+A `Job` is a recurring data transformation with inputs and outputs. Each execution is captured as a `Run` with
+corresponding metadata. A `Run` event identifies the `Job` it is an instance of by providing the job’s unique
+identifier. The `Job` identifier is composed of a `Namespace` and a `Name`. The job name is unique within that
+namespace.
+
+The core property we want to identify about a `Job` is how it changes over time. Different schedules of the same logic
+applied to different datasets (possibly with different parameters) are different jobs. The notion of a `job` is tied to
+a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a
+streaming job.
+
+If the same code artifact (for example a Spark jar or a templated SQL query) is used in the context of different
+schedules with different input or outputs, then they are different jobs. We are interested first in how they affect the
+datasets they produce.
+
+### Job Namespace and constructing job names
+
+Jobs have a `name` that is unique to them in their `namespace` by construction.
+
+The Namespace is the root of the naming hierarchy. The job name is constructed to identify the job within that
+namespace.
+
+Example:
+
+- Airflow:
+  - Namespace: the namespace is assigned to the Airflow instance. Ex: airflow-staging, airflow-prod
+  - Job: each task in a DAG is a job. name: {dag name}.{task name}
+- Spark:
+  - Namespace: the namespace is provided as a configuration parameter as in Airflow. If there's a parent job, we use the
+    same namespace, otherwise it is provided by configuration.
+  - Spark app job name: the spark.app.name
+  - Spark action job name: {spark.app.name}.{node.name}
+
+### Parent job run: a nested hierarchy of Jobs
+
+It is often the case that jobs are part of a nested hierarchy. For example, an Airflow DAG contains tasks. An instance
+of the DAG is finished when all of the tasks are finished. Similarly, a Spark job can spawn multiple actions, each of
+them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.
+
+Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection
+and make sure that we treat the same application logic triggered at different schedules as different jobs. For example:
+if an Airflow DAG runs individual tasks per partition (e.g., market segments) using the same underlying job logic, they
+will be tracked as separate jobs.
+
+To capture this, a run event provides
+[a `ParentRun` facet](https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331) referring to
+the parent `Job` and `Run`. This allows tracking a recurring job from the root of the schedule for which it is running.
+If there's a parent job, we use the same namespace, otherwise it is provided by configuration.
+
+Example:
+
+```json
+{
+  "run": {
+    "runId": "run_uuid"
+  },
+  "job": {
+    "namespace": "job_namespace",
+    "name": "job_name"
+  }
+}
+```
+
+---
+
+SPDX-License-Identifier: Apache-2.0\
+Copyright 2018-2024 contributors to the OpenLineage project