lineage-data-format 0.2.0__tar.gz → 0.3.0__tar.gz

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/CHANGELOG.md

@@ -7,6 +7,29 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [0.3.0] - 2026-11-03
+
+### Added
+
+- Alias compression, on by default (`aliases` option; `--no-aliases` on the CLI;
+  `aliases` argument of `convert_json_file_to_lineage`). New constructs:
+  `REF` section with `@Rn` references in `HP` / `ATTR` / `DSD`, `*` default entry in
+  `HP` / `DSD`, `IKP` section with `ikp=` node token, `NDEF` section with default node
+  tokens, `E` grouped by source (`N1>N2,N3`), default `EP` lines (`*>* etype=…`,
+  `* etype=…`).
+- The decoder reads all of the above and the older, expanded form.
+
+### Changed
+
+- **The default output is different.** Documents written with the default options are
+  smaller (e.g. 30% / 43% of the previous size on `out.json` / `big_query_lineage.json`)
+  and need a reader from this version. Use `aliases=False` / `--no-aliases` for the
+  0.2.0 form.
+- The order of the edge list is not preserved with `aliases` on. The set of edges and
+  their types is.
+
+---
+
 ## [0.2.0] - 2026-11-02
 
 ### Added

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: lineage-data-format
-Version: 0.2.0
+Version: 0.3.0
 Summary: Lineage Definition Format — token-optimized serialization for data lineage graphs
 Project-URL: Homepage, https://github.com/IBM/lineage-data-format
 Project-URL: Repository, https://github.com/IBM/lineage-data-format
@@ -129,6 +129,9 @@ convert_lineage_file_to_json('input.ldf', 'output.json')
 # Compact mode (single blank line between sections instead of two)
 ldf_compact = json_to_lineage_format(backend_response, {'compact': True})
 
+# Legacy, fully expanded output (no alias compression)
+ldf_expanded = json_to_lineage_format(backend_response, {'aliases': False})
+
 # Exclude fields from the output (applied to the input, before serialization)
 ldf_slim = json_to_lineage_format(
     backend_response,
@@ -136,6 +139,13 @@ ldf_slim = json_to_lineage_format(
 )
 ```
 
+#### `aliases`
+
+A `bool` in the options dict, `True` by default. It removes repeated values from
+the output (see [Alias compression](#alias-compression)). `False` writes the
+expanded form of version 0.2.0. The decoder reads both forms. Command line:
+`--no-aliases`.
+
 #### `exclude_fields`
 
 A list of `str` passed in the options dict. The default is an empty list, in which
@@ -167,6 +177,9 @@ ldf json-to-lineage input.json output.ldf
 # Convert with compact format (minimal blank lines)
 ldf json-to-lineage input.json output.ldf --compact
 
+# Expanded output without alias compression
+ldf json-to-lineage input.json output.ldf --no-aliases
+
 # Exclude fields (comma-separated and/or repeated; names, dotted paths, wildcards)
 ldf json-to-lineage input.json output.ldf --exclude children,descendant_statistics,is_favorite,source_code_snippets
 
@@ -187,8 +200,10 @@ The Lineage Definition Format is a compact, plain-text format that defines repea
 |---|---|---|
 | `GRAPH` | Structural | Marker line + optional `dt=` / `ts=` meta |
 | `START` | Structural | Alias of the first asset |
-| `E` | Structural | Directed edges: `N1>N2`; an edge without a target (end edge) is just the source alias: `N7` |
-| `EP` | Structural | Edge properties: `N1>N2 etype=direct` (end edge: `N7 etype=summary`), listed in `E` order |
+| `E` | Structural | Directed edges: `N1>N2`, grouped by source `N1>N2,N3`; an edge without a target (end edge) is just the source alias: `N7` |
+| `EP` | Structural | Edge properties: `N1>N2 etype=direct` (end edge: `N7 etype=summary`), listed in `E` order; `*>* etype=X` / `* etype=Y` are the defaults |
+| `NDEF` | Defaults | Node tokens shared by most nodes: `origin=design path=T1 flags=7` |
+| `IKP` | Lookup | `identity_key` prefixes: `P1=<encoded prefix>` |
 | `N` | Structural | Node definitions (see format below) |
 | `T` | Lookup | Technology names: `T1=PostgreSQL` |
 | `TID` | Lookup | Technology UUIDs: `T1=<uuid>` |
@@ -204,6 +219,25 @@ The Lineage Definition Format is a compact, plain-text format that defines repea
 | `PA` | Per-node | Project assignments (JSON): `N1=[…]` |
 | `SA` | Per-node | Space assignments (JSON): `N1=[…]` |
 | `DSD` | Per-node | Data source definition asset: `N1=id:name` |
+| `REF` | Lookup | Shared values of `HP` / `ATTR` / `DSD`: `R1=<value>`, used as `@R1` |
+
+### Alias compression
+
+With `aliases` on, repeated values are written once:
+
+| Construct | Example | Meaning |
+|---|---|---|
+| `REF` + `@Rn` | `R1=TRANSFORMATION_TAG=TRANSFORMING`, entry `N1=@R1\|@R2` | a whole entry value or one `\|`-separated element is replaced by the reference; a reference may refer to other references. An `@…` that is not defined in `REF` is a literal |
+| `*` entry | `HP:` `*=<value>`, `N3=<other>`, `N4=` | default value of `HP` / `DSD`. A node with its own entry keeps it; an empty entry (`N4=`) means none |
+| `IKP` | `P1=uuid/dsd%5BDatabase%5D/`, node `ikp=P1 ik=Asset%201…` | `identity_key` = prefix + `ik` |
+| `NDEF` | `origin=design flags=7` | tokens inherited by every node. A node's own token overrides it, an empty one (`origin=`) removes it; `flags=0` is written explicitly |
+| grouped `E` | `N1>N2,N3` | all targets of one source on one line |
+| default `EP` | `*>* etype=direct`, `* etype=summary` | type of edges with a target / of end edges; `N1>N2 etype=X` overrides it, `etype=` means no type |
+
+The text is smaller, the data is the same: assets and the set of edges (with their
+types) survive a roundtrip. The order of the edge list does not: sources keep their
+first-occurrence order and targets keep their original order. A section is written
+only when it saves space.
 
 ### Node Line Format
 

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/README.md

@@ -101,6 +101,9 @@ convert_lineage_file_to_json('input.ldf', 'output.json')
 # Compact mode (single blank line between sections instead of two)
 ldf_compact = json_to_lineage_format(backend_response, {'compact': True})
 
+# Legacy, fully expanded output (no alias compression)
+ldf_expanded = json_to_lineage_format(backend_response, {'aliases': False})
+
 # Exclude fields from the output (applied to the input, before serialization)
 ldf_slim = json_to_lineage_format(
     backend_response,
@@ -108,6 +111,13 @@ ldf_slim = json_to_lineage_format(
 )
 ```
 
+#### `aliases`
+
+A `bool` in the options dict, `True` by default. It removes repeated values from
+the output (see [Alias compression](#alias-compression)). `False` writes the
+expanded form of version 0.2.0. The decoder reads both forms. Command line:
+`--no-aliases`.
+
 #### `exclude_fields`
 
 A list of `str` passed in the options dict. The default is an empty list, in which
@@ -139,6 +149,9 @@ ldf json-to-lineage input.json output.ldf
 # Convert with compact format (minimal blank lines)
 ldf json-to-lineage input.json output.ldf --compact
 
+# Expanded output without alias compression
+ldf json-to-lineage input.json output.ldf --no-aliases
+
 # Exclude fields (comma-separated and/or repeated; names, dotted paths, wildcards)
 ldf json-to-lineage input.json output.ldf --exclude children,descendant_statistics,is_favorite,source_code_snippets
 
@@ -159,8 +172,10 @@ The Lineage Definition Format is a compact, plain-text format that defines repea
 |---|---|---|
 | `GRAPH` | Structural | Marker line + optional `dt=` / `ts=` meta |
 | `START` | Structural | Alias of the first asset |
-| `E` | Structural | Directed edges: `N1>N2`; an edge without a target (end edge) is just the source alias: `N7` |
-| `EP` | Structural | Edge properties: `N1>N2 etype=direct` (end edge: `N7 etype=summary`), listed in `E` order |
+| `E` | Structural | Directed edges: `N1>N2`, grouped by source `N1>N2,N3`; an edge without a target (end edge) is just the source alias: `N7` |
+| `EP` | Structural | Edge properties: `N1>N2 etype=direct` (end edge: `N7 etype=summary`), listed in `E` order; `*>* etype=X` / `* etype=Y` are the defaults |
+| `NDEF` | Defaults | Node tokens shared by most nodes: `origin=design path=T1 flags=7` |
+| `IKP` | Lookup | `identity_key` prefixes: `P1=<encoded prefix>` |
 | `N` | Structural | Node definitions (see format below) |
 | `T` | Lookup | Technology names: `T1=PostgreSQL` |
 | `TID` | Lookup | Technology UUIDs: `T1=<uuid>` |
@@ -176,6 +191,25 @@ The Lineage Definition Format is a compact, plain-text format that defines repea
 | `PA` | Per-node | Project assignments (JSON): `N1=[…]` |
 | `SA` | Per-node | Space assignments (JSON): `N1=[…]` |
 | `DSD` | Per-node | Data source definition asset: `N1=id:name` |
+| `REF` | Lookup | Shared values of `HP` / `ATTR` / `DSD`: `R1=<value>`, used as `@R1` |
+
+### Alias compression
+
+With `aliases` on, repeated values are written once:
+
+| Construct | Example | Meaning |
+|---|---|---|
+| `REF` + `@Rn` | `R1=TRANSFORMATION_TAG=TRANSFORMING`, entry `N1=@R1\|@R2` | a whole entry value or one `\|`-separated element is replaced by the reference; a reference may refer to other references. An `@…` that is not defined in `REF` is a literal |
+| `*` entry | `HP:` `*=<value>`, `N3=<other>`, `N4=` | default value of `HP` / `DSD`. A node with its own entry keeps it; an empty entry (`N4=`) means none |
+| `IKP` | `P1=uuid/dsd%5BDatabase%5D/`, node `ikp=P1 ik=Asset%201…` | `identity_key` = prefix + `ik` |
+| `NDEF` | `origin=design flags=7` | tokens inherited by every node. A node's own token overrides it, an empty one (`origin=`) removes it; `flags=0` is written explicitly |
+| grouped `E` | `N1>N2,N3` | all targets of one source on one line |
+| default `EP` | `*>* etype=direct`, `* etype=summary` | type of edges with a target / of end edges; `N1>N2 etype=X` overrides it, `etype=` means no type |
+
+The text is smaller, the data is the same: assets and the set of edges (with their
+types) survive a roundtrip. The order of the edge list does not: sources keep their
+first-occurrence order and targets keep their original order. A section is written
+only when it saves space.
 
 ### Node Line Format
 

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "lineage-data-format"
-version = "0.2.0"
+version = "0.3.0"
 description = "Lineage Definition Format — token-optimized serialization for data lineage graphs"
 readme = "README.md"
 license = "BSD-3-Clause"

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/src/ldf/__init__.py

@@ -12,7 +12,7 @@ from .converter import (
     convert_lineage_file_to_json,
 )
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     "json_to_lineage_format",

{lineage_data_format-0.2.0 → lineage_data_format-0.3.0}/src/ldf/cli.py

@@ -46,6 +46,11 @@ Examples:
         action='store_true',
         help='Use compact format (no extra whitespace)'
     )
+    json_parser.add_argument(
+        '--no-aliases',
+        action='store_true',
+        help='Disable alias compression (write the legacy, fully expanded form)'
+    )
     json_parser.add_argument(
         '--exclude',
         action='append',
@@ -84,7 +89,8 @@ Examples:
                 args.input,
                 args.output,
                 compact=args.compact,
-                exclude_fields=exclude_fields
+                exclude_fields=exclude_fields,
+                aliases=not args.no_aliases
             )
             print(f"✓ Converted {args.input} → {args.output}")