learnml-sdk 0.1.0__tar.gz

learnml_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: learnml-sdk
+Version: 0.1.0
+Summary: Python SDK for the LearnML training data management platform
+Home-page: https://github.com/milindjain0/learnml
+Author: LearnML
+Author-email: milindjain0@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: grpcio>=1.50.0
+Requires-Dist: protobuf>=4.0.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# LearnML — ML Data Management System
+
+A unified platform for storing ML training data (traditional labeled data + LLM training data), model checkpoints, and training metrics.
+
+**Stack:** C++17 · gRPC · Protocol Buffers · PostgreSQL · Grafana
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+- Docker & Docker Compose
+- CMake 3.20+
+- C++17 compiler (GCC 9+ or Clang 10+)
+- System packages: `grpc`, `protobuf`, `libpqxx`, `openssl`
+
+### 1. Start PostgreSQL & Grafana
+
+```bash
+docker-compose up -d
+```
+
+This starts:
+- **PostgreSQL 16** on `localhost:5432` (auto-runs migrations)
+- **Grafana 10** on `http://localhost:3000` (admin/admin)
+
+### 2. Build the Server
+
+```bash
+mkdir build && cd build
+cmake .. -DCMAKE_BUILD_TYPE=Release
+make -j$(nproc)
+```
+
+### 3. Run the Server
+
+```bash
+./build/learnml_server ../config.json
+```
+
+Server starts on `0.0.0.0:50051` by default.
+
+### 4. Test with grpcurl
+
+```bash
+# Register a user
+grpcurl -plaintext -d '{"email":"alice@test.com","password":"secret123","display_name":"Alice"}' \
+  localhost:50051 learnml.AuthService/Register
+
+# Login
+grpcurl -plaintext -d '{"email":"alice@test.com","password":"secret123"}' \
+  localhost:50051 learnml.AuthService/Login
+# → Returns access_token and refresh_token
+
+# Create a universe (use the access_token from login)
+grpcurl -plaintext \
+  -H "Authorization: Bearer <access_token>" \
+  -d '{"name":"my-ml-project","description":"My first ML project"}' \
+  localhost:50051 learnml.UniverseService/CreateUniverse
+
+# Create a data point
+grpcurl -plaintext \
+  -H "Authorization: Bearer <token>" \
+  -H "x-universe-id: <universe_id>" \
+  -d '{"universe_id":"<id>","name":"sample_image","data_type":"IMAGE"}' \
+  localhost:50051 learnml.DataService/CreateDataPoint
+
+# Create a training run and log metrics
+grpcurl -plaintext \
+  -H "Authorization: Bearer <token>" \
+  -d '{"universe_id":"<id>","name":"gpt2-finetune","model_name":"gpt2"}' \
+  localhost:50051 learnml.MetricsService/CreateTrainingRun
+
+grpcurl -plaintext \
+  -H "Authorization: Bearer <token>" \
+  -d '{"universe_id":"<id>","run_id":"<run_id>","snapshot":{"step":100,"metrics":{"loss":0.5,"perplexity":12.3}}}' \
+  localhost:50051 learnml.MetricsService/LogMetrics
+```
+
+---
+
+## Architecture
+
+```
+┌──────────────┐  ┌──────────┐  ┌─────────┐
+│  CLI / SDK   │  │ React UI │  │ Grafana │
+└──────┬───────┘  └────┬─────┘  └────┬────┘
+       │               │             │ (PostgreSQL datasource)
+       └───────┬───────┘             │
+               ▼                     │
+    ┌──────────────────┐             │
+    │  Auth Interceptor│             │
+    │  (JWT validation)│             │
+    └────────┬─────────┘             │
+             ▼                       │
+    ┌──────────────────┐             │
+    │  gRPC Services   │             │
+    │  ┌─ Auth         │             │
+    │  ├─ Universe     │             │
+    │  ├─ Data         │             │
+    │  ├─ Collection   │             │
+    │  ├─ Checkpoint   │             │
+    │  └─ Metrics      │             │
+    └────────┬─────────┘             │
+             ▼                       ▼
+    ┌──────────────────┐  ┌──────────────────┐
+    │   PostgreSQL     │  │  File System     │
+    │   (metadata +    │  │  (blobs per      │
+    │    RLS policies) │  │   universe)      │
+    └──────────────────┘  └──────────────────┘
+```
+
+## Key Concepts
+
+### Universe
+The top-level isolation boundary. All data is scoped to a universe. Users are granted **READ**, **WRITE**, or **ADMIN** access.
+
+### Data Points
+Support traditional labeled data (image, video, PDF, text) and LLM training data (SFT, RL, pretraining).
+
+### Collections
+Logical groups of data points. A data point can belong to multiple collections (many-to-many).
+
+### Checkpoints
+Model checkpoint files with streaming upload/download APIs.
+
+### Training Metrics
+Logged per-step with batch or streaming APIs. Grafana queries PostgreSQL directly.
+
+---
+
+## Configuration
+
+Edit `config.json` or use environment variables:
+
+| JSON Key | Env Variable | Default | Description |
+|----------|-------------|---------|-------------|
+| `server_address` | `LEARNML_SERVER_ADDRESS` | `0.0.0.0:50051` | gRPC listen address |
+| `db_host` | `LEARNML_DB_HOST` | `localhost` | PostgreSQL host |
+| `db_port` | `LEARNML_DB_PORT` | `5432` | PostgreSQL port |
+| `db_name` | `LEARNML_DB_NAME` | `learnml` | Database name |
+| `jwt_secret` | `LEARNML_JWT_SECRET` | dev secret | JWT signing key |
+| `storage_root` | `LEARNML_STORAGE_ROOT` | `/tmp/learnml_storage` | File storage root |
+
+---
+
+## Project Structure
+
+```
+learnml/
+├── proto/learnml/          # Protobuf schemas (7 files)
+├── src/
+│   ├── auth/               # JWT, password hashing, gRPC interceptor
+│   ├── db/                 # Connection pool, repository, migrations
+│   ├── services/           # 6 gRPC service implementations
+│   ├── server/             # gRPC server + main entry point
+│   └── util/               # Config, file store
+├── grafana/                # Dashboard JSON + provisioning
+├── CMakeLists.txt
+├── docker-compose.yml
+├── config.json
+└── README.md
+```
+
+## License
+
+MIT

learnml_sdk-0.1.0/learnml/__init__.py

@@ -0,0 +1,4 @@
+from .client import LearnMLClient
+
+__version__ = "0.1.0"
+__all__ = ["LearnMLClient"]

learnml_sdk-0.1.0/learnml/auth_pb2.py

@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# NO CHECKED-IN PROTOBUF GENCODE
+# source: learnml/auth.proto
+# Protobuf Python Version: 6.31.1
+"""Generated protocol buffer code."""
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import runtime_version as _runtime_version
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf.internal import builder as _builder
+_runtime_version.ValidateProtobufRuntimeVersion(
+    _runtime_version.Domain.PUBLIC,
+    6,
+    31,
+    1,
+    '',
+    'learnml/auth.proto'
+)
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x12learnml/auth.proto\x12\x07learnml\x1a\x1fgoogle/protobuf/timestamp.proto\"g\n\x04User\x12\n\n\x02id\x18\x01 \x01(\t\x12\r\n\x05\x65mail\x18\x02 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x03 \x01(\t\x12.\n\ncreated_at\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\"H\n\x0fRegisterRequest\x12\r\n\x05\x65mail\x18\x01 \x01(\t\x12\x10\n\x08password\x18\x02 \x01(\t\x12\x14\n\x0c\x64isplay_name\x18\x03 \x01(\t\"/\n\x10RegisterResponse\x12\x1b\n\x04user\x18\x01 \x01(\x0b\x32\r.learnml.User\"/\n\x0cLoginRequest\x12\r\n\x05\x65mail\x18\x01 \x01(\t\x12\x10\n\x08password\x18\x02 \x01(\t\"Y\n\rLoginResponse\x12\x14\n\x0c\x61\x63\x63\x65ss_token\x18\x01 \x01(\t\x12\x15\n\rrefresh_token\x18\x02 \x01(\t\x12\x1b\n\x04user\x18\x03 \x01(\x0b\x32\r.learnml.User\",\n\x13RefreshTokenRequest\x12\x15\n\rrefresh_token\x18\x01 \x01(\t\"C\n\x14RefreshTokenResponse\x12\x14\n\x0c\x61\x63\x63\x65ss_token\x18\x01 \x01(\t\x12\x15\n\rrefresh_token\x18\x02 \x01(\t\"\x17\n\x15GetCurrentUserRequest\"5\n\x16GetCurrentUserResponse\x12\x1b\n\x04user\x18\x01 \x01(\x0b\x32\r.learnml.UserB\x03\xf8\x01\x01\x62\x06proto3')
+
+_globals = globals()
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'learnml.auth_pb2', _globals)
+if not _descriptor._USE_C_DESCRIPTORS:
+  _globals['DESCRIPTOR']._loaded_options = None
+  _globals['DESCRIPTOR']._serialized_options = b'\370\001\001'
+  _globals['_USER']._serialized_start=64
+  _globals['_USER']._serialized_end=167
+  _globals['_REGISTERREQUEST']._serialized_start=169
+  _globals['_REGISTERREQUEST']._serialized_end=241
+  _globals['_REGISTERRESPONSE']._serialized_start=243
+  _globals['_REGISTERRESPONSE']._serialized_end=290
+  _globals['_LOGINREQUEST']._serialized_start=292
+  _globals['_LOGINREQUEST']._serialized_end=339
+  _globals['_LOGINRESPONSE']._serialized_start=341
+  _globals['_LOGINRESPONSE']._serialized_end=430
+  _globals['_REFRESHTOKENREQUEST']._serialized_start=432
+  _globals['_REFRESHTOKENREQUEST']._serialized_end=476
+  _globals['_REFRESHTOKENRESPONSE']._serialized_start=478
+  _globals['_REFRESHTOKENRESPONSE']._serialized_end=545
+  _globals['_GETCURRENTUSERREQUEST']._serialized_start=547
+  _globals['_GETCURRENTUSERREQUEST']._serialized_end=570
+  _globals['_GETCURRENTUSERRESPONSE']._serialized_start=572
+  _globals['_GETCURRENTUSERRESPONSE']._serialized_end=625
+# @@protoc_insertion_point(module_scope)

learnml_sdk-0.1.0/learnml/auth_pb2_grpc.py

@@ -0,0 +1,24 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+"""Client and server classes corresponding to protobuf-defined services."""
+import grpc
+import warnings
+
+
+GRPC_GENERATED_VERSION = '1.78.0'
+GRPC_VERSION = grpc.__version__
+_version_not_supported = False
+
+try:
+    from grpc._utilities import first_version_is_lower
+    _version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
+except ImportError:
+    _version_not_supported = True
+
+if _version_not_supported:
+    raise RuntimeError(
+        f'The grpc package installed is at version {GRPC_VERSION},'
+        + ' but the generated code in learnml/auth_pb2_grpc.py depends on'
+        + f' grpcio>={GRPC_GENERATED_VERSION}.'
+        + f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+        + f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
+    )

learnml_sdk-0.1.0/learnml/checkpoint_pb2.py

@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# NO CHECKED-IN PROTOBUF GENCODE
+# source: learnml/checkpoint.proto
+# Protobuf Python Version: 6.31.1
+"""Generated protocol buffer code."""
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import runtime_version as _runtime_version
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf.internal import builder as _builder
+_runtime_version.ValidateProtobufRuntimeVersion(
+    _runtime_version.Domain.PUBLIC,
+    6,
+    31,
+    1,
+    '',
+    'learnml/checkpoint.proto'
+)
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from learnml import common_pb2 as learnml_dot_common__pb2
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x18learnml/checkpoint.proto\x12\x07learnml\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x14learnml/common.proto\"\xd1\x02\n\x0fModelCheckpoint\x12\n\n\x02id\x18\x01 \x01(\t\x12\x13\n\x0buniverse_id\x18\x02 \x01(\t\x12\x0e\n\x06run_id\x18\x03 \x01(\t\x12\x0c\n\x04step\x18\x04 \x01(\x03\x12\r\n\x05\x65poch\x18\x05 \x01(\x05\x12\x11\n\tfile_path\x18\x06 \x01(\t\x12\x17\n\x0f\x66ile_size_bytes\x18\x07 \x01(\x03\x12)\n\x06\x66ormat\x18\x08 \x01(\x0e\x32\x19.learnml.CheckpointFormat\x12\x38\n\x08metadata\x18\t \x03(\x0b\x32&.learnml.ModelCheckpoint.MetadataEntry\x12.\n\ncreated_at\x18\n \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"h\n\x15SaveCheckpointRequest\x12\x33\n\x08metadata\x18\x01 \x01(\x0b\x32\x1f.learnml.SaveCheckpointMetadataH\x00\x12\x0f\n\x05\x63hunk\x18\x02 \x01(\x0cH\x00\x42\t\n\x07payload\"\x89\x02\n\x16SaveCheckpointMetadata\x12\x13\n\x0buniverse_id\x18\x01 \x01(\t\x12\x0e\n\x06run_id\x18\x02 \x01(\t\x12\x0c\n\x04step\x18\x03 \x01(\x03\x12\r\n\x05\x65poch\x18\x04 \x01(\x05\x12)\n\x06\x66ormat\x18\x05 \x01(\x0e\x32\x19.learnml.CheckpointFormat\x12?\n\x08metadata\x18\x06 \x03(\x0b\x32-.learnml.SaveCheckpointMetadata.MetadataEntry\x12\x10\n\x08\x66ilename\x18\x07 \x01(\t\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"F\n\x16SaveCheckpointResponse\x12,\n\ncheckpoint\x18\x01 \x01(\x0b\x32\x18.learnml.ModelCheckpoint\"C\n\x15LoadCheckpointRequest\x12\x13\n\x0buniverse_id\x18\x01 \x01(\t\x12\x15\n\rcheckpoint_id\x18\x02 \x01(\t\"_\n\x13LoadCheckpointChunk\x12,\n\x08metadata\x18\x01 \x01(\x0b\x32\x18.learnml.ModelCheckpointH\x00\x12\x0f\n\x05\x63hunk\x18\x02 \x01(\x0cH\x00\x42\t\n\x07payload\"m\n\x16ListCheckpointsRequest\x12\x13\n\x0buniverse_id\x18\x01 \x01(\t\x12\x0e\n\x06run_id\x18\x02 \x01(\t\x12.\n\npagination\x18\x03 \x01(\x0b\x32\x1a.learnml.PaginationRequest\"y\n\x17ListCheckpointsResponse\x12-\n\x0b\x63heckpoints\x18\x01 \x03(\x0b\x32\x18.learnml.ModelCheckpoint\x12/\n\npagination\x18\x02 \x01(\x0b\x32\x1b.learnml.PaginationResponse\"A\n\x1aGetLatestCheckpointRequest\x12\x13\n\x0buniverse_id\x18\x01 \x01(\t\x12\x0e\n\x06run_id\x18\x02 \x01(\t\"K\n\x1bGetLatestCheckpointResponse\x12,\n\ncheckpoint\x18\x01 \x01(\x0b\x32\x18.learnml.ModelCheckpoint\"E\n\x17\x44\x65leteCheckpointRequest\x12\x13\n\x0buniverse_id\x18\x01 \x01(\t\x12\x15\n\rcheckpoint_id\x18\x02 \x01(\t\"\x1a\n\x18\x44\x65leteCheckpointResponse*\xd3\x01\n\x10\x43heckpointFormat\x12!\n\x1d\x43HECKPOINT_FORMAT_UNSPECIFIED\x10\x00\x12\x1d\n\x19\x43HECKPOINT_FORMAT_PYTORCH\x10\x01\x12 \n\x1c\x43HECKPOINT_FORMAT_TENSORFLOW\x10\x02\x12\x1a\n\x16\x43HECKPOINT_FORMAT_ONNX\x10\x03\x12!\n\x1d\x43HECKPOINT_FORMAT_SAFETENSORS\x10\x04\x12\x1c\n\x18\x43HECKPOINT_FORMAT_CUSTOM\x10\x05\x42\x03\xf8\x01\x01\x62\x06proto3')
+
+_globals = globals()
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'learnml.checkpoint_pb2', _globals)
+if not _descriptor._USE_C_DESCRIPTORS:
+  _globals['DESCRIPTOR']._loaded_options = None
+  _globals['DESCRIPTOR']._serialized_options = b'\370\001\001'
+  _globals['_MODELCHECKPOINT_METADATAENTRY']._loaded_options = None
+  _globals['_MODELCHECKPOINT_METADATAENTRY']._serialized_options = b'8\001'
+  _globals['_SAVECHECKPOINTMETADATA_METADATAENTRY']._loaded_options = None
+  _globals['_SAVECHECKPOINTMETADATA_METADATAENTRY']._serialized_options = b'8\001'
+  _globals['_CHECKPOINTFORMAT']._serialized_start=1522
+  _globals['_CHECKPOINTFORMAT']._serialized_end=1733
+  _globals['_MODELCHECKPOINT']._serialized_start=93
+  _globals['_MODELCHECKPOINT']._serialized_end=430
+  _globals['_MODELCHECKPOINT_METADATAENTRY']._serialized_start=383
+  _globals['_MODELCHECKPOINT_METADATAENTRY']._serialized_end=430
+  _globals['_SAVECHECKPOINTREQUEST']._serialized_start=432
+  _globals['_SAVECHECKPOINTREQUEST']._serialized_end=536
+  _globals['_SAVECHECKPOINTMETADATA']._serialized_start=539
+  _globals['_SAVECHECKPOINTMETADATA']._serialized_end=804
+  _globals['_SAVECHECKPOINTMETADATA_METADATAENTRY']._serialized_start=383
+  _globals['_SAVECHECKPOINTMETADATA_METADATAENTRY']._serialized_end=430
+  _globals['_SAVECHECKPOINTRESPONSE']._serialized_start=806
+  _globals['_SAVECHECKPOINTRESPONSE']._serialized_end=876
+  _globals['_LOADCHECKPOINTREQUEST']._serialized_start=878
+  _globals['_LOADCHECKPOINTREQUEST']._serialized_end=945
+  _globals['_LOADCHECKPOINTCHUNK']._serialized_start=947
+  _globals['_LOADCHECKPOINTCHUNK']._serialized_end=1042
+  _globals['_LISTCHECKPOINTSREQUEST']._serialized_start=1044
+  _globals['_LISTCHECKPOINTSREQUEST']._serialized_end=1153
+  _globals['_LISTCHECKPOINTSRESPONSE']._serialized_start=1155
+  _globals['_LISTCHECKPOINTSRESPONSE']._serialized_end=1276
+  _globals['_GETLATESTCHECKPOINTREQUEST']._serialized_start=1278
+  _globals['_GETLATESTCHECKPOINTREQUEST']._serialized_end=1343
+  _globals['_GETLATESTCHECKPOINTRESPONSE']._serialized_start=1345
+  _globals['_GETLATESTCHECKPOINTRESPONSE']._serialized_end=1420
+  _globals['_DELETECHECKPOINTREQUEST']._serialized_start=1422
+  _globals['_DELETECHECKPOINTREQUEST']._serialized_end=1491
+  _globals['_DELETECHECKPOINTRESPONSE']._serialized_start=1493
+  _globals['_DELETECHECKPOINTRESPONSE']._serialized_end=1519
+# @@protoc_insertion_point(module_scope)

learnml_sdk-0.1.0/learnml/checkpoint_pb2_grpc.py

@@ -0,0 +1,24 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+"""Client and server classes corresponding to protobuf-defined services."""
+import grpc
+import warnings
+
+
+GRPC_GENERATED_VERSION = '1.78.0'
+GRPC_VERSION = grpc.__version__
+_version_not_supported = False
+
+try:
+    from grpc._utilities import first_version_is_lower
+    _version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
+except ImportError:
+    _version_not_supported = True
+
+if _version_not_supported:
+    raise RuntimeError(
+        f'The grpc package installed is at version {GRPC_VERSION},'
+        + ' but the generated code in learnml/checkpoint_pb2_grpc.py depends on'
+        + f' grpcio>={GRPC_GENERATED_VERSION}.'
+        + f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+        + f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
+    )