azure-blob 0.4.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 20f2d1c40dbc979782d017b025e5369f21e8ec9c1de02e50ef030648b8fd557b
4
- data.tar.gz: d818c311fb5a55db068d1b6d7e8d112d0fc44aa5f862144ec789ce3e3cd97a99
3
+ metadata.gz: 94edcaaaa7717925c1fb8238b9e7420949b26c98c90344cbf54930d5e0ff3b19
4
+ data.tar.gz: aae153da753212659590ed4a3ae029c86cf7ed66b4f924dac1a5f3a5ff3b2e05
5
5
  SHA512:
6
- metadata.gz: 9f58536a8295aa300c0be73a4654e2a888a51db9c7acc0bb793297c3878c39947aef7bf25c4231ab30e2ee764bd65a3f5c5863d0a2b5b555bf9c7bd976abeb13
7
- data.tar.gz: ae0f1e93eaff5c7f196fa78b6cf230f6be2a3c05cb4ba9635eb19828539cf381c8c23415453678ea19e790d860b65180260e9175ad1c825cb904047b09893047
6
+ metadata.gz: 102c3d5fd08761199413e96bd8e4bae8a84ac5478d7bb1202e232fbff346703f7cb3b8f344553983b607dbacec56942452ce18aa0afc184a4c994d31dc94118c
7
+ data.tar.gz: 646636874dd381757595f82999dd0b7b5647a740512c8e2baf86bdc2045e579cdf78354a4690c746452280efb7ff92cbf9c49c95775ccbcf6a690eb0ae05131f
@@ -0,0 +1,22 @@
1
+ # This file is maintained automatically by "terraform init".
2
+ # Manual edits may be lost in future updates.
3
+
4
+ provider "registry.terraform.io/hashicorp/azurerm" {
5
+ version = "3.113.0"
6
+ constraints = "~> 3.0"
7
+ hashes = [
8
+ "h1:eEUtt0lrLdpVaF6FiDq8BGQPgEcykmhj0aNIL7hTOGw=",
9
+ "zh:12479f5664288943400447b55e50df675c28ae82ad8d373cc2e5682f3a3411f0",
10
+ "zh:1b42a14e80e568429d3b55fed753ca3ef0df9dcdfa107890d7264599c020940f",
11
+ "zh:381be6ca617f848de3baa3985a6e1788e91a803afe04a3c5c727453528b6310d",
12
+ "zh:3e70e2e07b6db1c363de3e5d0ca47f27fc956473df03329c7d2e54d3ac29176b",
13
+ "zh:87c7633aeaa828098c6055da9e67d4acaf4b46748b6b3f0267e105e55f05de25",
14
+ "zh:8d0d98226901f874770dd5220d4701a12ae8bd586994615aa7dcba12b9736bec",
15
+ "zh:9fd913acd42a60c3a90a18ce803567ef861db8779a59aacced91f2cbd86de9d9",
16
+ "zh:b6f3f7ae0a055437fb36c139af9bb3135e7f4dad172157ae1eb0177dc74d703f",
17
+ "zh:b927027ba2bf40d34e03d742fd2b6c5299023b5ab8e6f05e50aac76a46ad1094",
18
+ "zh:ceb5187b9d2a439f4e48944f3ffeeeaf47a03dbe6f3325ea1775bf659ce0aa88",
19
+ "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
20
+ "zh:fb9d78dfeca7489bffca9b1a1f3abee7f16dbbcba31388aea1102062c1d6dce8",
21
+ ]
22
+ }
data/CHANGELOG.md CHANGED
@@ -1,5 +1,18 @@
1
1
  ## [Unreleased]
2
2
 
3
- ## [0.1.0] - 2024-04-05
3
+ ## [0.5.0] 2024-09-09
4
4
 
5
- - Initial release
5
+ - Added support for Managed Identities (Entra ID)
6
+
7
+ ## [0.4.2] 2024-06-06
8
+
9
+ - Documentation
10
+ - Fix an issue with integrity check on multi block upload
11
+
12
+
13
+ ## [0.4.1] 2024-05-27
14
+
15
+ First working release.
16
+
17
+ - Re-implemented the required parts of the azure-storage-blob API to make Active Storage work.
18
+ - Extracted the AzureStorage adapter from Rails.
data/README.md CHANGED
@@ -2,41 +2,113 @@
2
2
 
3
3
  This gem was built to replace azure-storage-blob (deprecated) in Active Storage, but was written to be Rails agnostic.
4
4
 
5
+ ## Active Storage
6
+
7
+ ### Migration
8
+ To migrate from azure-storage-blob to azure-blob:
9
+
10
+ 1. Replace `azure-storage-blob` in your Gemfile with `azure-blob`
11
+ 2. Run `bundle install`
12
+ 3. Change the `AzureStorage` service to `AzureBlob` in your Active Storage config (`config/storage.yml`)
13
+ 4. Restart or deploy the app.
14
+
15
+ ### Managed Identity (Entra ID)
16
+
17
+ AzureBlob supports managed identities on :
18
+ - Azure VM
19
+ - App Service
20
+ - Azure Functions (Untested but should work)
21
+ - Azure Containers (Untested but should work)
22
+
23
+ AKS support will likely require more work. Contributions are welcome.
24
+
25
+ To authenticate through managed identities instead of a shared key, omit `storage_access_key` from your `storage.yml` file.
26
+
27
+ It is recommended to add the identity's `principal_id` to the config.
28
+
29
+ ActiveStorage config example:
30
+
31
+ ```
32
+ prod:
33
+ service: AzureBlob
34
+ container: container_name
35
+ storage_account_name: account_name
36
+ principal_id: 71b34410-4c50-451d-b456-95ead1b18cce
37
+ ```
38
+
39
+ ## Standalone
40
+
41
+ Instantiate a client with your account name, an access key and the container name:
42
+
43
+ ```ruby
44
+ client = AzureBlob::Client.new(
45
+ account_name: @account_name,
46
+ access_key: @access_key,
47
+ container: @container,
48
+ )
49
+
50
+ path = "some/new/file"
51
+
52
+ # Upload
53
+ client.create_block_blob(path, "Hello world!")
54
+
55
+ # Download
56
+ client.get_blob(path) #=> "Hello world!"
57
+
58
+ # Delete
59
+ client.delete_blob(path)
60
+ ```
61
+
62
+ For the full list of methods: https://www.rubydoc.info/gems/azure-blob/AzureBlob/Client
63
+
5
64
  ## Contributing
6
65
 
7
- ### dev environment
66
+ ### Dev environment
8
67
 
9
- Ensure your version of Ruby fit the minimum version in `azure-blob.gemspec`
68
+ A dev environment is supplied through Nix with [devenv](https://devenv.sh/).
10
69
 
11
- and setup those Env variables:
70
+ 1. Install [devenv](https://devenv.sh/).
71
+ 2. Enter the dev environment by cd into the repo and running `devenv shell` (or `direnv allow` if you are a direnv user).
72
+ 3. Log into azure CLI with `az login`
73
+ 4. `terraform init`
74
+ 5. `terraform apply` This will generate the necessary infrastructure on azure.
75
+ 6. Generate devenv.local.nix with your private key and container information: `generate-env-file`
76
+ 7. If you are using direnv, the environment will reload automatically. If not, exit the shell and reopen it by hitting <C-d> and running `devenv shell` again.
12
77
 
13
- - `AZURE_ACCOUNT_NAME`
14
- - `AZURE_ACCESS_KEY`
15
- - `AZURE_PRIVATE_CONTAINER`
16
- - `AZURE_PUBLIC_CONTAINER`
78
+ #### Entra ID
17
79
 
80
+ To test with Entra ID, the `AZURE_ACCESS_KEY` environment variable must be unset and the code must be ran or proxied through a VPS with the proper roles.
18
81
 
19
- A dev environment setup is also supplied through Nix with [devenv](https://devenv.sh/).
82
+ For cost saving, the terraform variable `create_vm` is false by default.
83
+ To create the VPS, Create a var file `var.tfvars` containing:
20
84
 
21
- To use the Nix environment:
22
- 1- install [devenv](https://devenv.sh/)
23
- 2- Copy `devenv.local.nix.example` to `devenv.local.nix`
24
- 3- Insert your azure credentials into `devenv.local.nix`
25
- 4- Start the shell with `devenv shell` or with [direnv](https://direnv.net/).
85
+ ```
86
+ create_vm = true
87
+ ```
88
+ and re-apply terraform: `terraform apply -var-file=var.tfvars`.
26
89
 
27
- ### Tests
90
+ This will create the VPS and required managed identities.
28
91
 
29
- `bin/rake test`.
92
+ `bin/rake test_azure_vm` and `bin/rake test_app_service` will establish a VPN connection to the VM or App service container and run the test suite. You might be prompted for a sudo password when the VPN starts (sshuttle).
30
93
 
31
- # Active Storage
94
+ After you are done, run terraform again without the var file (`terraform apply`) to destroy the VPS and App service application.
32
95
 
33
- ## Migration
34
- To migrate from azure-storage-blob to azure-blob:
96
+ #### Cleanup
97
+
98
+ Some tests copied over from Rails don't clean after themselves. A rake task is provided to empty your containers and keep cost low: `bin/rake flush_test_container`
99
+
100
+ #### Run without devenv/nix
35
101
 
36
- 1- Replace `azure-storage-blob` in your Gemfile with `azure-blob`
37
- 2- Run `bundle install`
38
- 3- change the `AzureStorage` service to `AzureBlob` in your Active Storage config (`config/storage.yml`)
39
- 4- Restart or deploy the app.
102
+ If you prefer not using devenv/nix:
103
+
104
+ Ensure your version of Ruby fit the minimum version in `azure-blob.gemspec`
105
+
106
+ and setup those Env variables:
107
+
108
+ - `AZURE_ACCOUNT_NAME`
109
+ - `AZURE_ACCESS_KEY`
110
+ - `AZURE_PRIVATE_CONTAINER`
111
+ - `AZURE_PUBLIC_CONTAINER`
40
112
 
41
113
  ## License
42
114
 
data/Rakefile CHANGED
@@ -2,7 +2,57 @@
2
2
 
3
3
  require "bundler/gem_tasks"
4
4
  require "minitest/test_task"
5
+ require "azure_blob"
6
+ require_relative "test/support/app_service_vpn"
7
+ require_relative "test/support/azure_vm_vpn"
5
8
 
6
- Minitest::TestTask.create
9
+ Minitest::TestTask.create(:test_rails) do
10
+ self.test_globs = [ "test/rails/**/test_*.rb",
11
+ "test/rails/**/*_test.rb", ]
12
+ end
13
+
14
+ Minitest::TestTask.create(:test_client) do
15
+ self.test_globs = [ "test/client/**/test_*.rb",
16
+ "test/client/**/*_test.rb", ]
17
+ end
7
18
 
8
19
  task default: %i[test]
20
+
21
+ task :test do
22
+ Rake::Task["test_client"].execute
23
+ Rake::Task["test_rails"].execute
24
+ end
25
+
26
+ task :test_app_service do |t|
27
+ vpn = AppServiceVpn.new
28
+ ENV["IDENTITY_ENDPOINT"] = vpn.endpoint
29
+ ENV["IDENTITY_HEADER"] = vpn.header
30
+ Rake::Task["test_entra_id"].execute
31
+ ensure
32
+ vpn.kill
33
+ end
34
+
35
+ task :test_azure_vm do |t|
36
+ vpn = AzureVmVpn.new
37
+ Rake::Task["test_entra_id"].execute
38
+ ensure
39
+ vpn.kill
40
+ end
41
+
42
+ task :test_entra_id do |t|
43
+ ENV["AZURE_ACCESS_KEY"] = nil
44
+ Rake::Task["test"].execute
45
+ end
46
+
47
+ task :flush_test_container do |t|
48
+ AzureBlob::Client.new(
49
+ account_name: ENV["AZURE_ACCOUNT_NAME"],
50
+ access_key: ENV["AZURE_ACCESS_KEY"],
51
+ container: ENV["AZURE_PRIVATE_CONTAINER"],
52
+ ).delete_prefix ""
53
+ AzureBlob::Client.new(
54
+ account_name: ENV["AZURE_ACCOUNT_NAME"],
55
+ access_key: ENV["AZURE_ACCESS_KEY"],
56
+ container: ENV["AZURE_PUBLIC_CONTAINER"],
57
+ ).delete_prefix ""
58
+ end
data/azure-blob.gemspec CHANGED
@@ -9,13 +9,14 @@ Gem::Specification.new do |spec|
9
9
  spec.email = [ "joe@dupuis.io" ]
10
10
 
11
11
  spec.summary = "Azure blob client"
12
- spec.homepage = "https://github.com/JoeDupuis/azure-blob"
12
+ spec.homepage = "https://github.com/testdouble/azure-blob"
13
13
  spec.license = "MIT"
14
14
  spec.required_ruby_version = ">= 3.1"
15
15
 
16
+ spec.metadata["rubygems_mfa_required"] = "true"
16
17
  spec.metadata["homepage_uri"] = spec.homepage
17
18
  spec.metadata["source_code_uri"] = spec.homepage
18
- spec.metadata["changelog_uri"] = "https://github.com/JoeDupuis/azure-blob/blob/main/CHANGELOG.md"
19
+ spec.metadata["changelog_uri"] = "https://github.com/testdouble/azure-blob/blob/main/CHANGELOG.md"
19
20
 
20
21
  spec.add_dependency "rexml"
21
22
 
data/devenv.lock CHANGED
@@ -108,16 +108,16 @@
108
108
  },
109
109
  "nixpkgs": {
110
110
  "locked": {
111
- "lastModified": 1715542476,
111
+ "lastModified": 1725001927,
112
112
  "owner": "NixOS",
113
113
  "repo": "nixpkgs",
114
- "rev": "44072e24566c5bcc0b7aa9178a0104f4cfffab19",
115
- "treeHash": "3f9021e4c33de6fe59b88ac8c3019fc49136dc2a",
114
+ "rev": "6e99f2a27d600612004fbd2c3282d614bfee6421",
115
+ "treeHash": "1e85443cc9f0ba302df2cf61cacb8014943e2d19",
116
116
  "type": "github"
117
117
  },
118
118
  "original": {
119
119
  "owner": "NixOS",
120
- "ref": "nixos-23.11",
120
+ "ref": "nixos-24.05",
121
121
  "repo": "nixpkgs",
122
122
  "type": "github"
123
123
  }
@@ -129,11 +129,11 @@
129
129
  "nixpkgs": "nixpkgs_2"
130
130
  },
131
131
  "locked": {
132
- "lastModified": 1713939467,
132
+ "lastModified": 1724737223,
133
133
  "owner": "bobvanderlinden",
134
134
  "repo": "nixpkgs-ruby",
135
- "rev": "c1ba161adf31119cfdbb24489766a7bcd4dbe881",
136
- "treeHash": "0d32620317b29f94d6718684f030dd2fc2f30cb2",
135
+ "rev": "175b5867babcbc471b94be9fd5576f2973bbdb6d",
136
+ "treeHash": "2fe3404ac0eeb7bcb7ac7b5f5f8b9b6a7e460147",
137
137
  "type": "github"
138
138
  },
139
139
  "original": {
@@ -160,16 +160,16 @@
160
160
  },
161
161
  "nixpkgs_2": {
162
162
  "locked": {
163
- "lastModified": 1715542476,
163
+ "lastModified": 1725001927,
164
164
  "owner": "NixOS",
165
165
  "repo": "nixpkgs",
166
- "rev": "44072e24566c5bcc0b7aa9178a0104f4cfffab19",
167
- "treeHash": "3f9021e4c33de6fe59b88ac8c3019fc49136dc2a",
166
+ "rev": "6e99f2a27d600612004fbd2c3282d614bfee6421",
167
+ "treeHash": "1e85443cc9f0ba302df2cf61cacb8014943e2d19",
168
168
  "type": "github"
169
169
  },
170
170
  "original": {
171
171
  "owner": "NixOS",
172
- "ref": "nixos-23.11",
172
+ "ref": "nixos-24.05",
173
173
  "repo": "nixpkgs",
174
174
  "type": "github"
175
175
  }
data/devenv.nix CHANGED
@@ -1,12 +1,36 @@
1
- { pkgs, ... }:
1
+ { pkgs, config, ... }:
2
2
 
3
3
  {
4
+ env = {
5
+ LD_LIBRARY_PATH = "${config.devenv.profile}/lib";
6
+ };
7
+
4
8
  packages = with pkgs; [
5
9
  git
6
10
  libyaml
11
+ terraform
12
+ azure-cli
13
+ glib
14
+ vips
15
+ sshuttle
16
+ sshpass
17
+ rsync
7
18
  ];
8
19
 
9
-
10
20
  languages.ruby.enable = true;
11
- languages.ruby.version = "3.1.5";
21
+ languages.ruby.version = "3.1.6";
22
+
23
+ scripts.generate-env-file.exec = ''
24
+ terraform output -raw devenv_local_nix > devenv.local.nix
25
+ '';
26
+
27
+ scripts.proxy-vps.exec = ''
28
+ exec sshuttle -e "ssh -o CheckHostIP=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" -r "$(terraform output --raw vm_username)@$(terraform output --raw vm_ip)" 0/0
29
+ '';
30
+
31
+ scripts.start-app-service-ssh.exec = ''
32
+ resource_group=$(terraform output --raw "resource_group")
33
+ app_name=$(terraform output --raw "app_service_app_name")
34
+ exec az webapp create-remote-connection --resource-group $resource_group --name $app_name
35
+ '';
12
36
  }
data/devenv.yaml CHANGED
@@ -1,6 +1,6 @@
1
1
  allowUnfree: true
2
2
  inputs:
3
3
  nixpkgs:
4
- url: github:NixOS/nixpkgs/nixos-23.11
4
+ url: github:NixOS/nixpkgs/nixos-24.05
5
5
  nixpkgs-ruby:
6
6
  url: github:bobvanderlinden/nixpkgs-ruby
data/input.tf ADDED
@@ -0,0 +1,44 @@
1
+ variable "location" {
2
+ type = string
3
+ default = "westus2"
4
+ }
5
+
6
+ variable "prefix" {
7
+ type = string
8
+ default = "azure-blob"
9
+ }
10
+
11
+ variable "storage_account_name" {
12
+ type = string
13
+ default = "azureblobrubygemdev"
14
+ }
15
+
16
+ variable "create_vm" {
17
+ type = bool
18
+ default = false
19
+ }
20
+
21
+ variable "vm_size" {
22
+ type = string
23
+ default = "Standard_B2s"
24
+ }
25
+
26
+ variable "vm_username" {
27
+ type = string
28
+ default = "azureblob"
29
+ }
30
+
31
+ variable "vm_password" {
32
+ type = string
33
+ default = "qwe123QWE!@#"
34
+ }
35
+
36
+ variable "create_app_service" {
37
+ type = bool
38
+ default = false
39
+ }
40
+
41
+ variable "ssh_key" {
42
+ type = string
43
+ default = ""
44
+ }
@@ -25,17 +25,17 @@
25
25
  require "active_support/core_ext/numeric/bytes"
26
26
  require "active_storage/service"
27
27
 
28
- require 'azure_blob'
28
+ require "azure_blob"
29
29
 
30
30
  module ActiveStorage
31
- # = Active Storage \Azure Storage \Service
31
+ # = Active Storage \Azure Blob \Service
32
32
  #
33
33
  # Wraps the Microsoft Azure Storage Blob Service as an Active Storage service.
34
- # See ActiveStorage::Service for the generic API documentation that applies to all services.
34
+ # See {ActiveStorage::Service}[https://api.rubyonrails.org/classes/ActiveStorage/Service.html] for more details.
35
35
  class Service::AzureBlobService < Service
36
36
  attr_reader :client, :container, :signer
37
37
 
38
- def initialize(storage_account_name:, storage_access_key:, container:, public: false, **options)
38
+ def initialize(storage_account_name:, storage_access_key: nil, container:, public: false, **options)
39
39
  @container = container
40
40
  @public = public
41
41
  @client = AzureBlob::Client.new(
@@ -1,7 +1,13 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module AzureBlob
4
+ # AzureBlob::Blob holds the metada for a given Blob.
4
5
  class Blob
6
+ # You should not instanciate this object directly,
7
+ # but obtain one when calling relevant methods of AzureBlob::Client.
8
+ #
9
+ # Expects a Net::HTTPResponse object from a
10
+ # HEAD or GET request to a blob uri.
5
11
  def initialize(response)
6
12
  @response = response
7
13
  end
@@ -26,6 +32,7 @@ module AzureBlob
26
32
  response.code == "200"
27
33
  end
28
34
 
35
+ # Returns the custom Azure metada tagged on the blob.
29
36
  def metadata
30
37
  @metadata || response
31
38
  .to_hash
@@ -3,10 +3,28 @@
3
3
  require "rexml"
4
4
 
5
5
  module AzureBlob
6
+ # Enumerator class to lazily iterate over a list of Blob keys.
6
7
  class BlobList
7
8
  include REXML
8
9
  include Enumerable
9
10
 
11
+ # You should not instanciate this object directly,
12
+ # but obtain one when calling relevant methods of AzureBlob::Client.
13
+ #
14
+ # Expects a callable object that takes an Azure API page marker as an
15
+ # argument and returns the raw body response of a call to the list blob endpoint.
16
+ #
17
+ # Example:
18
+ #
19
+ # fetcher = ->(marker) do
20
+ # uri.query = URI.encode_www_form(
21
+ # marker: marker,
22
+ # ...
23
+ # )
24
+ # response = Http.new(uri, signer:).get
25
+ # end
26
+ # AzureBlob::BlobList.new(fetcher)
27
+ #
10
28
  def initialize(fetcher)
11
29
  @fetcher = fetcher
12
30
  end
@@ -3,7 +3,9 @@
3
3
  require "rexml"
4
4
 
5
5
  module AzureBlob
6
- class BlockList
6
+ class BlockList # :nodoc:
7
+ # Internal
8
+ # BlockList builds the XML list of blocks to commit to a blob
7
9
  include REXML
8
10
  def initialize(blocks)
9
11
  @blocks = blocks
@@ -1,5 +1,5 @@
1
1
  module AzureBlob
2
- class CanonicalizedHeaders
2
+ class CanonicalizedHeaders # :nodoc:
3
3
  STANDARD_HEADERS = [
4
4
  :"x-ms-version",
5
5
  ]
@@ -1,7 +1,7 @@
1
1
  require "cgi"
2
2
 
3
3
  module AzureBlob
4
- class CanonicalizedResource
4
+ class CanonicalizedResource # :nodoc:
5
5
  def initialize(uri, account_name, service_name: nil, url_safe: true)
6
6
  # This next line is needed because CanonicalizedResource
7
7
  # need to be escaped for auhthorization headers, but not SAS tokens