artesian-sdk 4.2.2.dev19__tar.gz → 4.2.3b2.dev54__tar.gz

{artesian_sdk-4.2.2.dev19 → artesian_sdk-4.2.3b2.dev54}/.github/workflows/dependency-review.yaml

@@ -9,9 +9,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: 'Checkout Repository'
-        uses: actions/checkout@v5
+        uses: actions/checkout@v7
       - name: 'Dependency Review'
-        uses: actions/dependency-review-action@v4
+        uses: actions/dependency-review-action@v5.0.0
         with:
           # Use comma-separated names to pass list arguments:
           allow-licenses: MIT,Apache-2.0

{artesian_sdk-4.2.2.dev19 → artesian_sdk-4.2.3b2.dev54}/.github/workflows/python-tests.yml

@@ -12,6 +12,32 @@ on:
     branches: [master]
 
 jobs:
+  pyright:
+    name: pyright
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v7
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run Pyright
+        uses: jordemort/action-pyright@v1
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          reporter: github-check
+          lib:
+          level: warning
+          filter_mode: file
+
   build:
     runs-on: ${{ matrix.os }}
 
@@ -21,9 +47,9 @@ jobs:
         os: ["ubuntu-latest", "windows-latest", "macos-latest"]
 
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v7
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v6
+        uses: actions/setup-python@v7
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
@@ -35,21 +61,12 @@ jobs:
         run: |
           flake8 .
 
-      - name: Run Pyright
-        uses: jordemort/action-pyright@v1
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          reporter: github-check
-          lib:
-          level: warning
-          filter_mode: file
-
       - name: Test Pytest
         run: |
           pytest --junitxml=junit/test-results-${{ matrix.python-version }}-${{ matrix.os }}.xml
 
       - name: Upload pytest test results
-        uses: actions/upload-artifact@v5
+        uses: actions/upload-artifact@v7
         with:
           name: pytest-results-${{ matrix.python-version }}-${{ matrix.os }}
           path: junit/test-results-${{ matrix.python-version }}-${{ matrix.os }}.xml
@@ -59,16 +76,16 @@ jobs:
   publish:
     if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/v'))
     runs-on: ubuntu-latest
-    needs: build
+    needs: [build, pyright]
     env:
       python-version: "3.11"
 
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v7
         with:
           fetch-depth: 0
       - name: Set up Python ${{ env.python-version }}
-        uses: actions/setup-python@v6
+        uses: actions/setup-python@v7
         with:
           python-version: ${{ env.python-version }}
       - name: Install dependencies

{artesian_sdk-4.2.2.dev19 → artesian_sdk-4.2.3b2.dev54}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: artesian-sdk
-Version: 4.2.2.dev19
+Version: 4.2.3b2.dev54
 Summary: Library provides read access to the Artesian API
 Author-email: ARK Lab <arklab@ark-energy.eu>
 Maintainer-email: ARK Lab <arklab@ark-energy.eu>
@@ -466,23 +466,53 @@ res = mds.searchFacet(page, pageSize, searchText, filters, sorts, doNotLoadAddit
 
 Artesian support Query over GME Public Offers which comes in a custom and dedicated format.
 
+Note (performance):
+GME Public Offer data is partitioned by date, offerType, status, and market. Requesting very narrow subsets (for example a single status or offerType in several separate requests) does not improve performance and can cause the same data to be fetched multiple times.
+For this reason, the examples below:
+• Use a large page size so that all data for a given (date, market, filters) is typically returned in a single page.
+• Loop over markets explicitly to cover all required markets without redundant fetches.
+
 ### Extract GME Public Offer
 
 ```Python
 from Artesian.GMEPublicOffers import GMEPublicOfferService, Market, Purpose, Status, Zone, Scope, UnitType, GenerationType, BAType
 
-qs = GMEPublicOfferService(cfg)
+/* your Artesian configuration */;
+
+var _cfg = new ArtesianServiceConfig(
+new Uri("https://arkive.artesian.cloud/{TENANT_NAME}/"),
+your_API_Key);
+
+qs = GMEPublicOfferService(_cfg)
+
+# Large page size: goal is to get all data for the given filters in one page
+PAGE_SIZE = 250000
+
+# List of markets to cover. Extend as needed for your use case.
+markets = [
+    Market.MGP,
+    # Market.MSD,
+    # Market.MIT,
+]
 
-data = qs.createQuery() \
+all_data = []
+
+for market in markets:
+
+data = (
+    qs.createQuery() \
     .forDate("2020-04-01") \
     .forMarket([Market.MGP]) \
     .forStatus(Status.ACC) \
     .forPurpose(Purpose.BID) \
     .forZone([Zone.NORD]) \
-    .withPagination(1,100) \
+    .withPagination(1,PAGE_SIZE) \
     .execute()
+    )
+    print(f"Fetched {len(data)} offers for market {market}")
+    all_data.extend(data)
 
-print(data)
+print(f"Total offers fetched: {len(all_data)}")
 ```
 
 To construct a GME Public Offer Extraction the following must be provided.
@@ -733,6 +763,76 @@ Extraction options for GME Public Offer queries.
  .withAggregationRule(AggregationRule.SumAndDivide)
 ```
 
+### DerivedTransform: Query Validation
+
+Use the `DerivedTransformQueryValidation` to validate and execute a derived transform query against a sample TimeSerieData.
+
+```Python
+from Artesian.MarketData import MarketDataService
+from Artesian.MarketData._Dto.DerivedTransformQueryValidation import DerivedTransformQueryValidation
+from Artesian.MarketData._Dto.TimeSerieData import TimeSerieData
+from Artesian import MarketDataType
+from datetime import datetime
+
+mds = MarketDataService(cfg)
+
+request = DerivedTransformQueryValidation(
+    data=TimeSerieData(
+        rows=[
+            (datetime(2018, 10, 1, 0, 0), 100),
+            (datetime(2018, 10, 1, 1, 0), 100)
+        ],
+        type=MarketDataType.ActualTimeSerie,
+    ),
+    transform="SELECT Time, (Value + 1) as Value FROM $table"
+)
+
+derivedTransformResponse = mds.derivedTransformQueryValidation(request)
+```
+
+#### Available Columns
+
+| Column | Type     |
+| ------ | -------- |
+| Time   | datetime |
+| Value  | double   |
+
+> `$table` is a virtual table exposed by the query engine.
+
+**Returned Object: DerivedTransformQueryValidationResponse**
+
+| Property | Description                                    |
+| -------- | ---------------------------------------------- |
+| Data     | Transformed time series generated by the query |
+| Valid    | Indicates whether the query is valid           |
+| Error    | Validation error details when `Valid = false`  |
+
+#### Query examples
+
+```sql
+SELECT Time + INTERVAL 1 DAY AS Time, Value FROM $table
+
+SELECT Time, CASE WHEN EXTRACT(HOUR FROM (Time AT TIME ZONE 'UTC') AT TIME ZONE 'Europe/Rome') < 10 THEN Value + 1 ELSE Value END AS Value FROM $table WHERE Time IS NOT NULL
+
+SELECT Time, Value FROM $table WHERE Version IS NOT NULL AND ((EXTRACT(hour FROM Version) < 10 AND Time >= date_trunc('day', Version + interval '1 day')) OR (EXTRACT(hour FROM Version) >= 10 AND Time >= date_trunc('day', Version + interval '2 day')))
+```
+
+#### SQL Dialect & Execution Model
+
+The query engine uses a DuckDB-compatible SQL dialect (PostgreSQL-like).
+
+Queries are executed against a virtual table named `$table`, which represents the provided sample TimeSerieData.
+
+The engine supports common SQL features including:
+
+- SELECT expressions
+- WHERE filtering
+- CASE WHEN expressions
+- Date/time arithmetic and functions
+- Timezone conversion functions
+
+Full DuckDB SQL reference: https://duckdb.org/docs/sql/introduction.html
+
 ## Write Data in Artesian
 
 Using the MarketDataService is possible to register MarketData and write curves into it using the UpsertData method.
@@ -786,9 +886,11 @@ mkservice.upsertData(data)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
@@ -796,12 +898,11 @@ deferDataGeneration (true/false) choose between syncronoys and asyncronous preco
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for ActualTimeSeries the two modes are equivalent. Leaving Null/None/Empty is equivalent to Merge.
 
-
-DerivedCfg can be of algorithm type: Coalesce, Sum, Muv.
+DerivedCfg can be of algorithm type: Coalesce, Sum, Muv, Transform.
 
 Updating the DerivedCfg can be performed with `updateDerivedConfiguration` on MarketDataService. A validation will be done on the existing DerivedCfg of the MarketData, that should be not null and with same type as the one used for the update.
 
-```csharp
+```Python
 derivedCfgUpdate = DerivedCfg(
     version=1,
     derivedAlgorithm=DerivedAlgorithm.Coalesce,
@@ -814,6 +915,31 @@ marketDataUpdated = mkdservice.updateDerivedConfiguration(
                         False)
 ```
 
+For DerivedCfg Transform, `orderedReferencedMarketDataIds` contains a single source series (the series where the transform is applied), and `transform` contains the query to apply.
+
+```Python
+derivedCfgUpdate = DerivedCfg(
+    version=1,
+    derivedAlgorithm=DerivedAlgorithm.Transform,
+    orderedReferencedMarketDataIds=[10001],
+    transform="SELECT Time, (Value + 1) as Value FROM $table"
+)
+
+marketDataUpdated = mkdservice.updateDerivedConfiguration(
+                        registeredDerived.marketDataId,
+                        derivedCfgUpdate,
+                        False)
+```
+
+#### Available Columns
+
+| Column | Type     |
+| ------ | -------- |
+| Time   | datetime |
+| Value  | double   |
+
+> `$table` is a virtual table exposed by the query engine.
+
 In case we want to write an hourly (or lower) time series the timezone for the upsert data must be UTC:
 
 ```Python
@@ -896,9 +1022,11 @@ mkservice.upsertData(data)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
@@ -906,25 +1034,24 @@ deferDataGeneration (true/false) choose between syncronoys and asyncronous preco
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for VersionedTimeSeries the merge writes in to the curve replacing existing data for an existing instant, replace writes the payload removing any previous data for the version. Leaving Null/None/Empty is equivalent to Merge.
 
-| DATETIME | EXISTING | PAYLOAD | MERGE | REPALACE |
-|---|---|---|---|---|
+| DATETIME     | EXISTING   | PAYLOAD    | MERGE      | REPLACE    |
+| ------------ | ---------- | ---------- | ---------- | ---------- |
 | VERSION NAME | 2025-01-01 | 2025-01-01 | 2025-01-01 | 2025-01-01 |
-| 2025-01-01 |        |        |        |        |
-| 2025-01-02 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-03 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-04 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-05 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-06 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-07 | 999.99 | 222.22 | 222.22 | 222.22 |
-| 2025-01-08 |        | 222.22 | 222.22 | 222.22 |
-| 2025-01-09 |        |        |        |        |
-| 2025-01-10 |        |        |        |        |
-| 2025-01-11 | 999.99 |        | 999.99 |        |
-| 2025-01-12 | 999.99 |        | 999.99 |        |
-| 2025-01-13 | 999.99 |        | 999.99 |        |
-| 2025-01-14 |        |        |        |        |
-| 2025-01-15 |        |        |        |        |
-
+| 2025-01-01   |            |            |            |            |
+| 2025-01-02   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-03   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-04   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-05   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-06   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-07   | 999.99     | 222.22     | 222.22     | 222.22     |
+| 2025-01-08   |            | 222.22     | 222.22     | 222.22     |
+| 2025-01-09   |            |            |            |            |
+| 2025-01-10   |            |            |            |            |
+| 2025-01-11   | 999.99     |            | 999.99     |            |
+| 2025-01-12   | 999.99     |            | 999.99     |            |
+| 2025-01-13   | 999.99     |            | 999.99     |            |
+| 2025-01-14   |            |            |            |            |
+| 2025-01-15   |            |            |            |            |
 
 ### Write Data in a Market Assessment Time Series
 
@@ -974,9 +1101,11 @@ mkservice.upsertData(marketAssessment)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
@@ -984,7 +1113,6 @@ deferDataGeneration (true/false) choose between syncronoys and asyncronous preco
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for MarketAssessment merge adds the new products to the existing and overwrites existing with the new ones while replace replaces all the existing products with the new ones. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ### Write Data in a Bid Ask Time Series
 
 ```Python
@@ -1032,9 +1160,11 @@ mkservice.upsertData(bidAsk)
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
@@ -1042,7 +1172,6 @@ deferDataGeneration (true/false) choose between syncronoys and asyncronous preco
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for BidAsk merge adds the new products to the existing and overwrites existing with the new ones while replace replaces all the existing products with the new ones. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ### Write Data in an Auction Time Series
 
 ```Python
@@ -1090,9 +1219,11 @@ auctionRows = MarketData.UpsertData(MarketData.MarketDataIdentifier('PROVIDER',
 ```
 
 Upsert has optional switches that can be applied
+
 ```
 mkservice.upsertData(data, deferCommandExecution, deferDataGeneration, keepNulls, upsertMode)
 ```
+
 The switch details are,
 
 deferCommandExecution (true/false) choose between syncronoys and asyncronous command execution, default is false.
@@ -1100,7 +1231,6 @@ deferDataGeneration (true/false) choose between syncronoys and asyncronous preco
 keepNulls (true/false) if true then nulls are written in the curve replacing any data present for the instant, default is false.
 upsertMode (Merge/Replace) for Auction merge and replace are equivalent. Leaving Null/None/Empty is equivalent to Merge.
 
-
 ## Delete Data in Artesian
 
 Using the MarketDataService is possible to delete MarketData and its curves.