datachain 0.37.5__tar.gz → 0.37.7__tar.gz

{datachain-0.37.5 → datachain-0.37.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.37.5
+Version: 0.37.7
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -55,9 +55,9 @@ Provides-Extra: docs
 Requires-Dist: mkdocs>=1.5.2; extra == "docs"
 Requires-Dist: mkdocs-gen-files>=0.5.0; extra == "docs"
 Requires-Dist: mkdocs-material==9.5.22; extra == "docs"
-Requires-Dist: mkdocs-section-index>=0.3.6; extra == "docs"
 Requires-Dist: mkdocstrings-python>=1.6.3; extra == "docs"
 Requires-Dist: mkdocs-literate-nav>=0.6.1; extra == "docs"
+Requires-Dist: mkdocs-section-index>=0.3.10; extra == "docs"
 Requires-Dist: eval-type-backport; extra == "docs"
 Provides-Extra: torch
 Requires-Dist: torch>=2.1.0; extra == "torch"

datachain-0.37.7/docs/studio/index.md

@@ -0,0 +1,62 @@
+# DataChain Studio
+
+DataChain Studio is a web application that enables Machine Learning and Data teams to seamlessly
+
+- [Run and track jobs](user-guide/jobs/index.md)
+- [Track experiments and manage models](user-guide/experiments/index.md) (via DVC integration)
+- [Collaborate on data projects](user-guide/team-collaboration.md)
+
+DataChain Studio supports multiple workflows:
+- **DataChain workflows**: For unstructured data processing and transformation
+- **DVC + Git workflows**: For ML experiment tracking and model registry, maintaining Git as the single-source-of-truth
+
+Sign in to DataChain Studio using your GitHub.com, GitLab.com, or Bitbucket.org account, or with your email address. Explore the demo projects and datasets, and [let us know](user-guide/troubleshooting.md#support) if you need any help getting started.
+
+## Why DataChain Studio?
+
+- Simplify data processing job tracking, visualization, and collaboration.
+- Support both modern DataChain workflows and traditional DVC experiment tracking.
+- Keep your code, data and processing connected at all times.
+- Apply your existing software engineering stack for data and ML teams.
+- Build a comprehensive data processing and ML platform for transparency and discovery across all your projects.
+- For DVC projects, maintain Git as the single-source-of-truth and use [GitOps](https://www.gitops.tech/) for deployment and automation.
+
+## Getting Started
+
+New to DataChain Studio? Start with these guides:
+
+- **[User Guide](user-guide/index.md)** - Learn how to use DataChain Studio features
+- **[API Reference](api/index.md)** - Integrate with Studio programmatically
+- **[Webhooks](webhooks.md)** - Set up event notifications
+- **[Self-hosting](self-hosting/index.md)** - Deploy your own Studio instance
+
+## Key Features
+
+### Dataset Management
+- Track and version your datasets
+- Visualize data processing pipelines
+- Share datasets across teams
+
+### Job Processing
+- Run data processing jobs in the cloud
+- Monitor job progress and logs
+- Schedule recurring data processing tasks
+
+### ML Experiment Tracking (DVC Integration)
+- Track and compare ML experiments
+- Manage model lifecycle and registry
+- Visualize metrics and plots
+- Git-based experiment versioning
+
+### Team Collaboration
+- Share projects with team members
+- Control access with role-based permissions
+- Integrate with development workflows
+
+### API Integration
+- RESTful API for programmatic access
+- Webhook notifications for automation
+- Command-line tools for developers
+
+
+Visit [studio.datachain.ai](https://studio.datachain.ai) to get started, or learn about [self-hosting](self-hosting/index.md) for enterprise deployments.

datachain-0.37.7/docs/studio/self-hosting/configuration/ca-certificates.md

@@ -0,0 +1,410 @@
+# CA Certificates
+
+This guide covers how to configure custom Certificate Authority (CA) certificates for your self-hosted DataChain Studio instance. This is necessary when your organization uses internal CAs or when connecting to services with custom certificates.
+
+## Overview
+
+DataChain Studio may need to trust custom CA certificates in several scenarios:
+
+- **Internal Git Servers**: Self-hosted GitLab, GitHub Enterprise with custom certificates
+- **Storage Services**: S3-compatible storage with custom certificates
+- **Corporate Proxies**: HTTPS proxies with internal certificates
+- **Database Connections**: PostgreSQL/Redis with SSL using custom CAs
+
+## Configuration Methods
+
+### Kubernetes/Helm Deployment
+
+#### Method 1: ConfigMap with Certificate Files
+
+1. Create a ConfigMap with your CA certificates:
+
+```bash
+kubectl create configmap custom-ca-certs \
+  --namespace datachain-studio \
+  --from-file=ca1.crt=/path/to/your/ca1.crt \
+  --from-file=ca2.crt=/path/to/your/ca2.crt
+```
+
+2. Configure Helm values to mount the certificates:
+
+```yaml
+# values.yaml
+global:
+  customCaCerts:
+    enabled: true
+    configMapName: custom-ca-certs
+
+# Alternative: Inline certificates
+global:
+  customCaCerts:
+    certificates:
+      - name: "internal-ca"
+        certificate: |
+          -----BEGIN CERTIFICATE-----
+          MIIDXTCCAkWgAwIBAgIJAKoK/heBjcOuMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV
+          BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
+          ... (certificate content) ...
+          -----END CERTIFICATE-----
+      - name: "corporate-ca"
+        certificate: |
+          -----BEGIN CERTIFICATE-----
+          ... (another certificate) ...
+          -----END CERTIFICATE-----
+```
+
+3. Apply the configuration:
+
+```bash
+helm upgrade datachain-studio datachain/studio \
+  --namespace datachain-studio \
+  --values values.yaml
+```
+
+#### Method 2: Direct Certificate Configuration
+
+Add certificates directly to your Helm values:
+
+```yaml
+# values.yaml
+global:
+  customCaCerts:
+    - |-
+      -----BEGIN CERTIFICATE-----
+      MIIDXTCCAkWgAwIBAgIJAKoK/heBjcOuMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV
+      BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
+      ... (rest of certificate) ...
+      -----END CERTIFICATE-----
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (another certificate) ...
+      -----END CERTIFICATE-----
+```
+
+### AWS AMI Deployment
+
+For AMI deployments, configure CA certificates directly on the instance:
+
+#### 1. Upload CA Certificates
+
+```bash
+# Copy certificate files to the instance
+scp -i your-key.pem ca-certificates.crt ubuntu@your-instance:/tmp/
+
+# SSH to the instance
+ssh -i your-key.pem ubuntu@your-instance
+
+# Install CA certificates
+sudo cp /tmp/ca-certificates.crt /usr/local/share/ca-certificates/
+sudo update-ca-certificates
+```
+
+#### 2. Configure DataChain Studio
+
+Update the configuration file:
+
+```yaml
+# /opt/datachain-studio/config.yml
+global:
+  ssl:
+    caCertificates:
+      - /usr/local/share/ca-certificates/ca-certificates.crt
+
+    # Alternatively, inline certificates
+    customCaCerts:
+      - |
+        -----BEGIN CERTIFICATE-----
+        ... certificate content ...
+        -----END CERTIFICATE-----
+```
+
+#### 3. Restart Services
+
+```bash
+sudo systemctl restart datachain-studio
+```
+
+## Use Cases and Examples
+
+### Internal GitLab Server
+
+Configure CA certificates for connecting to an internal GitLab server:
+
+```yaml
+global:
+  customCaCerts:
+    - |-
+      -----BEGIN CERTIFICATE-----
+      MIIDXTCCAkWgAwIBAgIJAKoK/heBjcOuMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV
+      ... (your GitLab CA certificate) ...
+      -----END CERTIFICATE-----
+
+  git:
+    gitlab:
+      enabled: true
+      url: "https://gitlab.internal.company.com"
+      clientId: "your-gitlab-client-id"
+      clientSecret: "your-gitlab-client-secret"
+
+      # SSL verification settings
+      ssl:
+        verify: true
+        caCertificate: true  # Use custom CA
+```
+
+### S3-Compatible Storage with Custom CA
+
+Configure certificates for custom S3-compatible storage:
+
+```yaml
+global:
+  customCaCerts:
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (your storage provider's CA certificate) ...
+      -----END CERTIFICATE-----
+
+storage:
+  type: s3
+  s3:
+    endpoint: "https://s3.internal.company.com"
+    bucket: "datachain-studio-storage"
+    region: "us-east-1"
+
+    # SSL settings
+    ssl:
+      enabled: true
+      verify: true
+      caCertificate: true
+```
+
+### Corporate Proxy with Custom CA
+
+Configure certificates for accessing external services through a corporate proxy:
+
+```yaml
+global:
+  # Proxy configuration
+  proxy:
+    enabled: true
+    http: "http://proxy.company.com:8080"
+    https: "https://proxy.company.com:8080"
+
+  # CA certificates for proxy SSL
+  customCaCerts:
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (your proxy's CA certificate) ...
+      -----END CERTIFICATE-----
+```
+
+### Multiple CA Certificates
+
+Configure multiple CA certificates for different services:
+
+```yaml
+global:
+  customCaCerts:
+    # Internal root CA
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (internal root CA) ...
+      -----END CERTIFICATE-----
+
+    # GitLab intermediate CA
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (GitLab intermediate CA) ...
+      -----END CERTIFICATE-----
+
+    # Storage service CA
+    - |-
+      -----BEGIN CERTIFICATE-----
+      ... (storage service CA) ...
+      -----END CERTIFICATE-----
+```
+
+## Certificate Chain Validation
+
+### Obtaining CA Certificates
+
+Get CA certificates from various sources:
+
+#### From a Website
+```bash
+# Extract certificate chain from a website
+openssl s_client -showcerts -connect gitlab.company.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab-ca.crt
+```
+
+#### From a Certificate File
+```bash
+# Extract CA from a certificate bundle
+openssl x509 -in certificate-bundle.crt -out ca-certificate.crt
+```
+
+#### From System Trust Store
+```bash
+# Export system CA certificates
+cat /etc/ssl/certs/ca-certificates.crt > system-cas.crt
+```
+
+### Validating Certificate Chains
+
+Test certificate validation:
+
+```bash
+# Verify certificate against CA
+openssl verify -CAfile ca-certificate.crt server-certificate.crt
+
+# Test SSL connection with custom CA
+openssl s_client -connect gitlab.company.com:443 -CAfile ca-certificate.crt
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**SSL verification errors:**
+```
+ERROR: SSL certificate verification failed
+```
+
+**Solution:**
+1. Verify CA certificate is correct and complete
+2. Check certificate format (PEM format required)
+3. Ensure certificate chain is complete
+
+**Certificate format issues:**
+```
+ERROR: Invalid certificate format
+```
+
+**Solution:**
+1. Ensure certificates are in PEM format
+2. Check for proper BEGIN/END markers
+3. Validate certificate using OpenSSL
+
+### Debugging CA Certificate Issues
+
+#### 1. Check Certificate Details
+
+```bash
+# View certificate information
+openssl x509 -in ca-certificate.crt -text -noout
+
+# Check certificate validity
+openssl x509 -in ca-certificate.crt -noout -dates
+```
+
+#### 2. Test Certificate Trust
+
+```bash
+# Test connection with custom CA
+curl --cacert ca-certificate.crt https://gitlab.company.com
+
+# Test with verbose output
+curl -v --cacert ca-certificate.crt https://gitlab.company.com
+```
+
+#### 3. Validate Certificate Chain
+
+```bash
+# Check certificate chain
+openssl verify -verbose -CAfile ca-certificate.crt intermediate.crt
+
+# Show certificate chain
+openssl s_client -connect gitlab.company.com:443 -showcerts
+```
+
+### Container-Level Debugging
+
+For Kubernetes deployments, debug CA certificate issues:
+
+```bash
+# Check if certificates are mounted correctly
+kubectl exec -it deployment/datachain-studio-backend -n datachain-studio -- ls -la /etc/ssl/certs/
+
+# Test certificate from within container
+kubectl exec -it deployment/datachain-studio-backend -n datachain-studio -- openssl s_client -connect gitlab.company.com:443 -CApath /etc/ssl/certs
+
+# Check certificate trust store
+kubectl exec -it deployment/datachain-studio-backend -n datachain-studio -- cat /etc/ssl/certs/ca-certificates.crt | grep -A 20 "Your CA Name"
+```
+
+## Security Best Practices
+
+### Certificate Management
+
+1. **Regular Updates**: Keep CA certificates updated
+2. **Secure Storage**: Store CA certificates securely
+3. **Access Control**: Limit access to CA certificate files
+4. **Validation**: Regularly validate certificate chains
+
+### Certificate Rotation
+
+```yaml
+# Automated certificate rotation
+global:
+  certificates:
+    rotation:
+      enabled: true
+      schedule: "0 2 * * 0"  # Weekly on Sunday at 2 AM
+      backup: true
+
+    validation:
+      enabled: true
+      checkExpiry: true
+      daysBeforeExpiry: 30
+```
+
+### Monitoring and Alerting
+
+```yaml
+# Certificate monitoring
+monitoring:
+  certificates:
+    enabled: true
+    alerts:
+      - name: "Certificate Expiry Warning"
+        condition: "certificate_days_until_expiry < 30"
+        severity: "warning"
+
+      - name: "Certificate Expiry Critical"
+        condition: "certificate_days_until_expiry < 7"
+        severity: "critical"
+```
+
+## Validation and Testing
+
+### Post-Configuration Validation
+
+```bash
+# Test DataChain Studio connectivity
+curl -I https://studio.company.com
+
+# Verify certificate trust
+openssl s_client -connect studio.company.com:443 -verify_return_error
+
+# Check service logs for SSL errors
+kubectl logs -f deployment/datachain-studio-backend -n datachain-studio | grep -i ssl
+```
+
+### Integration Testing
+
+```bash
+# Test Git integration
+curl -k https://studio.company.com/api/git/test-connection
+
+# Test storage connectivity
+curl -k https://studio.company.com/api/storage/health
+
+# Test webhook delivery
+curl -k https://studio.company.com/api/webhooks/test
+```
+
+## Next Steps
+
+- Configure [SSL/TLS certificates](ssl-tls.md) for secure communications
+- Set up [Git forge integrations](git-forges/index.md) with custom certificates
+- Review [troubleshooting guides](../troubleshooting/index.md) for SSL/TLS issues
+- Learn about [upgrading procedures](../upgrading/index.md) with certificate considerations