data_gov_au.rb 0.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: cc8124fb0b1af53acff6ba03881b9ee09d90378693f99b1d1951bb8f59b43073
4
+ data.tar.gz: 8b0599da2d95fa93c122bec731d38222f279dfee101b9617f32bd3aeda9097e3
5
+ SHA512:
6
+ metadata.gz: 9f6ab6c06cb9fbff1dd8d39a7672e1c4839c7a9795bf565b670e3ccfb0d88dcc0f8e89519889f53ed65e1d773df241108e65b6a172329bd47183d972add457cd
7
+ data.tar.gz: 35d9daee433aa62cf27c16a0c0129ac8ff58724cc21ac2f47de83aa1dd93730909aac0e3103901af8b68884c7299d715da83ed47208076f0d377dcf130dc6eb9
data/CHANGELOG ADDED
@@ -0,0 +1,13 @@
1
+ # data_gov_au/CHANGELOG
2
+
3
+ ## 20261103
4
+
5
+ 0.0.0: A client for data.gov.au's CKAN API: a dataset found by its name or id, its resources described, and a resource downloaded once.
6
+
7
+ 1. + lib/data_gov_au.rb, lib/DataGovAu/VERSION.rb.
8
+ 2. + data_gov_au.gemspec, data_gov_au.rb.gemspec: The same gem, published under both names.
9
+ 3. + lib/DataGovAu/Catalogue.rb: + dataset(), through package_show. The user agent and the API's URL are arguments with defaults rather than settings read from a file, a library having no machine of its own to configure. Either can also be set once it is made, or in a block.
10
+ 4. + lib/DataGovAu/Dataset.rb: + name, title, organisation, licence, modified_at, resources, to_h. CKAN gives its times without a zone, and they are read as UTC.
11
+ 5. + lib/DataGovAu/Resource.rb: + download(), written to a .partial file and renamed into place once whole, so that an interrupted download leaves nothing which would pass for the file, and stopping upon a body other than the length the catalogue reports. + first_bytes(), which asks by range and stops upon anything but 206, holding the whole body being what asking by range is meant to avoid. + held_at?(). + filename, taken from the URL, the resource's own name being prose.
12
+ 6. + lib/DataGovAu/Error.rb: + DataGovAu::Error, CatalogueUnavailable, DownloadFailed.
13
+ 7. + test/DataGovAu/Catalogue_test.rb, Dataset_test.rb, Resource_test.rb: 32 runs.
data/Gemfile ADDED
@@ -0,0 +1,5 @@
1
+ # Gemfile
2
+
3
+ source 'https://rubygems.org'
4
+
5
+ gemspec name: 'data_gov_au'
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 thoran
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,74 @@
1
+ # data_gov_au
2
+
3
+ A client for data.gov.au's CKAN API.
4
+
5
+
6
+ ## Installation
7
+
8
+ Needs Ruby 3.2 or later.
9
+
10
+ ```shell
11
+ $ gem install data_gov_au
12
+ ```
13
+
14
+ Or from a checkout:
15
+
16
+ ```shell
17
+ $ rake install
18
+ ```
19
+
20
+
21
+ ## Usage
22
+
23
+ ```ruby
24
+ require 'data_gov_au'
25
+
26
+ dataset = DataGovAu::Catalogue.new.dataset('abn-bulk-extract')
27
+
28
+ dataset.title # => "ABN Bulk Extract"
29
+ dataset.organisation # => "Australian Business Register"
30
+ dataset.licence # => "Creative Commons Attribution 3.0 Australia"
31
+ dataset.modified_at # => 2026-10-27 21:41:00.366366 UTC
32
+
33
+ resource = dataset.resources.find{|resource| resource.format == 'ZIP'}
34
+ resource.filename # => "public_split_1_10.zip"
35
+ resource.size # => 500487995
36
+ resource.download(File.join('releases', resource.filename))
37
+ ```
38
+
39
+ A dataset is asked for by its name or its id, CKAN taking either. It is asked for afresh each time rather than held, a resource's URL being the catalogue's to change.
40
+
41
+ A dataset gives its `name`, `title`, `organisation`, `licence`, `modified_at` and `resources`, and a resource its `name`, `format`, `size`, `url`, `last_modified` and `filename`. Each one's `to_h` is the entry exactly as the catalogue gave it.
42
+
43
+ ### Downloads
44
+
45
+ `download` writes to a `.partial` file beside the path given and renames it into place once whole, so that a download cut short leaves nothing which would pass for the file. A body other than the length the catalogue reports stops with `DataGovAu::DownloadFailed`, and leaves nothing behind.
46
+
47
+ `held_at?(path)` says whether the file at a path is the one the catalogue describes: present, and at the length it reports. Where the catalogue gives no length, presence alone stands for it.
48
+
49
+ `first_bytes(length)` asks for the start of a file by range, which is enough to read a zip's first local file header, say, for a few hundred bytes rather than the whole file. A server which ignores the range and answers with the whole body stops it, holding that body being what asking by range is meant to avoid.
50
+
51
+ ### Times
52
+
53
+ CKAN gives its times without a zone. They are UTC, and `modified_at` and `last_modified` are read as such.
54
+
55
+ ### Settings
56
+
57
+ ```ruby
58
+ DataGovAu::Catalogue.new(user_agent: 'mine/1.0', api_url: 'https://data.gov.au/data/api/3/action')
59
+ ```
60
+
61
+ Both have defaults: the user agent names this library and its version, and the API is data.gov.au's. Any CKAN portal's action API answers the same calls. Either can also be set once the catalogue is made, or in a block:
62
+
63
+ ```ruby
64
+ DataGovAu::Catalogue.new{|c| c.user_agent = 'mine/1.0'}
65
+ ```
66
+
67
+ ### Errors
68
+
69
+ Everything raised is a `DataGovAu::Error`: `CatalogueUnavailable` where the catalogue could not be read or reported no result, as for a name it does not know, and `DownloadFailed` where a resource could not be retrieved or arrived at the wrong length.
70
+
71
+
72
+ ## Licence
73
+
74
+ MIT. See LICENSE.
data/Rakefile ADDED
@@ -0,0 +1,22 @@
1
+ # Rakefile
2
+
3
+ require 'rake/testtask'
4
+
5
+ Rake::TestTask.new(:test) do |t|
6
+ t.libs << 'lib'
7
+ t.libs << 'test'
8
+ t.test_files = FileList['test/**/*_test.rb']
9
+ t.verbose = true
10
+ t.warning = false
11
+ end
12
+
13
+ task default: :test
14
+
15
+ desc "Run tests"
16
+ task :spec => :test
17
+
18
+ desc "Show version"
19
+ task :version do
20
+ require_relative './lib/DataGovAu/VERSION'
21
+ puts "data_gov_au #{DataGovAu::VERSION}"
22
+ end
data/TODO ADDED
@@ -0,0 +1,7 @@
1
+ # TODO
2
+
3
+ 1. DataGovAu::ABR, the Australian Business Register's weekly ABN bulk extract: its current release found through the catalogue and fetched once, kept in a directory of its own beside the catalogue entry it was found through.
4
+ 2. DataGovAu::ABR: the parser, reading every record of a release out of the zips as they lie, so that neither a zip nor an XML file is ever held whole.
5
+ 3. DataGovAu::ABR: storage, every record and every name stored with the dates over which it held, nothing filtered and nothing derived at ingest.
6
+ 4. DataGovAu::ABR: querying, every criterion a WHERE clause, none of them applied at ingest.
7
+ 5. DataGovAu::ABR: output, CSV through dsv, carrying the ABR's attribution.
@@ -0,0 +1,51 @@
1
+ # data_gov_au.rb.gemspec
2
+
3
+ require_relative './lib/DataGovAu/VERSION'
4
+
5
+ class Gem::Specification
6
+ def development_dependencies=(gems)
7
+ gems.each{|gem| add_development_dependency(*gem)}
8
+ end
9
+
10
+ def runtime_dependencies=(gems)
11
+ gems.each{|gem| add_runtime_dependency(*gem)}
12
+ end
13
+ end
14
+
15
+ Gem::Specification.new do |spec|
16
+ spec.name = 'data_gov_au.rb'
17
+ spec.version = DataGovAu::VERSION
18
+
19
+ spec.summary = "A client for data.gov.au's CKAN API."
20
+ spec.description = "Find a dataset through data.gov.au's CKAN catalogue by its name or id, read what the catalogue says of it and of each of its resources, and download a resource once, checked against the length the catalogue reports."
21
+
22
+ spec.author = 'thoran'
23
+ spec.email = 'code@thoran.com'
24
+ spec.homepage = "https://github.com/thoran/data_gov_au"
25
+ spec.license = 'MIT'
26
+
27
+ spec.required_ruby_version = '>= 3.2'
28
+ spec.require_paths = ['lib']
29
+
30
+ spec.files = [
31
+ 'data_gov_au.rb.gemspec',
32
+ Dir['lib/**/*.rb'],
33
+ Dir['test/**/*.rb'],
34
+ 'CHANGELOG',
35
+ 'Gemfile',
36
+ 'LICENSE',
37
+ 'Rakefile',
38
+ 'README.md',
39
+ 'TODO',
40
+ ].flatten
41
+
42
+ spec.runtime_dependencies = [
43
+ 'http.rb',
44
+ ]
45
+
46
+ spec.development_dependencies = [
47
+ ['minitest', '~> 6.0'],
48
+ 'rake',
49
+ 'webmock',
50
+ ]
51
+ end
@@ -0,0 +1,43 @@
1
+ # DataGovAu/Catalogue.rb
2
+ # DataGovAu::Catalogue
3
+
4
+ require 'json'
5
+
6
+ require 'http.rb'
7
+
8
+ require_relative './Dataset'
9
+ require_relative './Error'
10
+ require_relative './VERSION'
11
+
12
+ module DataGovAu
13
+ # data.gov.au's CKAN action API, through which a dataset and its resources are found.
14
+ class Catalogue
15
+ API_URL = 'https://data.gov.au/data/api/3/action'
16
+ USER_AGENT = "data_gov_au/#{VERSION}"
17
+
18
+ attr_accessor :api_url, :user_agent
19
+
20
+ # A dataset by its name or its id, CKAN taking either. Asked for each time rather than held, its resources being the catalogue's to change.
21
+ def dataset(name_or_id)
22
+ Dataset.new(self, action('package_show', 'id' => name_or_id))
23
+ end
24
+
25
+ private
26
+
27
+ def initialize(api_url: API_URL, user_agent: USER_AGENT)
28
+ @api_url = api_url
29
+ @user_agent = user_agent
30
+ yield(self) if block_given?
31
+ end
32
+
33
+ # One call to the action API, giving up its result where the answer reports success.
34
+ def action(name, parameters = {})
35
+ url = "#{@api_url}/#{name}"
36
+ response = HTTP.get(url, parameters, {'User-Agent' => @user_agent})
37
+ raise CatalogueUnavailable, "#{url} answered #{response.code}" unless response.success?
38
+ body = JSON.parse(response.body)
39
+ raise CatalogueUnavailable, "#{url} reported no result" unless body['success'] && body['result']
40
+ body['result']
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,49 @@
1
+ # DataGovAu/Dataset.rb
2
+ # DataGovAu::Dataset
3
+
4
+ require 'time'
5
+
6
+ require_relative './Resource'
7
+
8
+ module DataGovAu
9
+ # A dataset's entry in the catalogue, as package_show gave it.
10
+ class Dataset
11
+ def name
12
+ @package['name']
13
+ end
14
+
15
+ def title
16
+ @package['title']
17
+ end
18
+
19
+ def organisation
20
+ @package.dig('organization', 'title')
21
+ end
22
+
23
+ def licence
24
+ @package['license_title']
25
+ end
26
+
27
+ # CKAN gives its times without a zone, and they are UTC.
28
+ def modified_at
29
+ @modified_at ||= @package['metadata_modified'] && Time.parse("#{@package['metadata_modified']}Z")
30
+ end
31
+
32
+ # In the catalogue's order.
33
+ def resources
34
+ @resources ||= (@package['resources'] || []).collect{|resource| Resource.new(@catalogue, resource)}
35
+ end
36
+
37
+ # The entry exactly as the catalogue gave it, so that what was found can be kept beside what was downloaded.
38
+ def to_h
39
+ @package
40
+ end
41
+
42
+ private
43
+
44
+ def initialize(catalogue, package)
45
+ @catalogue = catalogue
46
+ @package = package
47
+ end
48
+ end
49
+ end
@@ -0,0 +1,12 @@
1
+ # DataGovAu/Error.rb
2
+ # DataGovAu::Error
3
+
4
+ module DataGovAu
5
+ class Error < StandardError; end
6
+
7
+ # The catalogue could not be read, so what a dataset holds now remains unknown.
8
+ class CatalogueUnavailable < Error; end
9
+
10
+ # A resource could not be retrieved, or arrived at a length other than the one the catalogue reported.
11
+ class DownloadFailed < Error; end
12
+ end
@@ -0,0 +1,95 @@
1
+ # DataGovAu/Resource.rb
2
+ # DataGovAu::Resource
3
+
4
+ require 'fileutils'
5
+ require 'time'
6
+ require 'uri'
7
+
8
+ require 'http.rb'
9
+
10
+ require_relative './Error'
11
+
12
+ module DataGovAu
13
+ # One file of a dataset, as the catalogue describes it.
14
+ class Resource
15
+ PARTIAL_CONTENT = 206
16
+ PARTIAL_SUFFIX = '.partial'
17
+
18
+ def name
19
+ @resource['name']
20
+ end
21
+
22
+ def format
23
+ @resource['format']
24
+ end
25
+
26
+ # In bytes, or nil where the catalogue leaves it out.
27
+ def size
28
+ @resource['size']
29
+ end
30
+
31
+ def url
32
+ @resource['url']
33
+ end
34
+
35
+ # CKAN gives its times without a zone, and they are UTC.
36
+ def last_modified
37
+ @last_modified ||= @resource['last_modified'] && Time.parse("#{@resource['last_modified']}Z")
38
+ end
39
+
40
+ # The name data.gov.au serves the file under, which is the last segment of its URL; the resource's own name is prose, and holds a trailing space in at least one case.
41
+ def filename
42
+ @filename ||= File.basename(URI.parse(url).path)
43
+ end
44
+
45
+ # The first bytes, asked for by range, so that what a file begins with costs a few hundred bytes rather than the whole of it. A server which ignored the range would answer 200 with the whole body, which is worth stopping for: holding it is the one thing asking by range is meant to avoid.
46
+ def first_bytes(length)
47
+ response = get('Range' => "bytes=0-#{length - 1}")
48
+ raise DownloadFailed, "#{url} answered #{response.code} where a range was asked for" unless response.code.to_i == PARTIAL_CONTENT
49
+ response.body
50
+ end
51
+
52
+ # Written to a partial file and renamed into place once whole, so that a download cut short leaves nothing which would pass for the file. A body other than the length the catalogue reports stops here rather than at whatever reads it.
53
+ def download(path)
54
+ partial_path = path + PARTIAL_SUFFIX
55
+ FileUtils.mkdir_p(File.dirname(path))
56
+ File.binwrite(partial_path, body)
57
+ verify(partial_path)
58
+ File.rename(partial_path, path)
59
+ path
60
+ end
61
+
62
+ # Present at the length the catalogue reports. A length the catalogue leaves out cannot be checked, so presence alone stands for it.
63
+ def held_at?(path)
64
+ File.exist?(path) && (size.nil? || File.size(path) == size)
65
+ end
66
+
67
+ def to_h
68
+ @resource
69
+ end
70
+
71
+ private
72
+
73
+ def initialize(catalogue, resource)
74
+ @catalogue = catalogue
75
+ @resource = resource
76
+ end
77
+
78
+ def get(headers = {})
79
+ HTTP.get(url, {}, {'User-Agent' => @catalogue.user_agent}.merge(headers))
80
+ end
81
+
82
+ def body
83
+ response = get
84
+ raise DownloadFailed, "#{url} answered #{response.code}" unless response.success?
85
+ response.body
86
+ end
87
+
88
+ def verify(path)
89
+ return if size.nil? || File.size(path) == size
90
+ found = File.size(path)
91
+ File.delete(path)
92
+ raise DownloadFailed, "#{url} gave #{found} bytes where the catalogue reports #{size}"
93
+ end
94
+ end
95
+ end
@@ -0,0 +1,6 @@
1
+ # DataGovAu/VERSION.rb
2
+ # DataGovAu::VERSION
3
+
4
+ module DataGovAu
5
+ VERSION = '0.0.0'
6
+ end
@@ -0,0 +1,8 @@
1
+ # data_gov_au.rb
2
+ # DataGovAu
3
+
4
+ require_relative './DataGovAu/Catalogue'
5
+ require_relative './DataGovAu/Dataset'
6
+ require_relative './DataGovAu/Error'
7
+ require_relative './DataGovAu/Resource'
8
+ require_relative './DataGovAu/VERSION'
@@ -0,0 +1,71 @@
1
+ # test/DataGovAu/Catalogue_test.rb
2
+
3
+ require_relative '../helper'
4
+
5
+ describe DataGovAu::Catalogue do
6
+ describe "the dataset it finds" do
7
+ it "is the one package_show gives for the name asked" do
8
+ stub_package
9
+ _(DataGovAu::Catalogue.new.dataset('abn-bulk-extract').title).must_equal 'ABN Bulk Extract'
10
+ end
11
+
12
+ it "is asked for by id as readily as by name" do
13
+ id = '5bd7fcab-e315-42cb-8daf-50b7efc2027e'
14
+ stub_package(id)
15
+ _(DataGovAu::Catalogue.new.dataset(id).name).must_equal 'abn-bulk-extract'
16
+ end
17
+ end
18
+
19
+ describe "the request it makes" do
20
+ it "identifies itself" do
21
+ stub = stub_request(:get, "#{PACKAGE_SHOW_URL}?id=abn-bulk-extract").
22
+ with(headers: {'User-Agent' => "data_gov_au/#{DataGovAu::VERSION}"}).
23
+ to_return(status: 200, body: JSON.generate({'success' => true, 'result' => package}))
24
+ DataGovAu::Catalogue.new.dataset('abn-bulk-extract')
25
+ assert_requested(stub)
26
+ end
27
+
28
+ it "identifies itself as it is told to" do
29
+ stub = stub_request(:get, "#{PACKAGE_SHOW_URL}?id=abn-bulk-extract").
30
+ with(headers: {'User-Agent' => 'mine/1.0'}).
31
+ to_return(status: 200, body: JSON.generate({'success' => true, 'result' => package}))
32
+ DataGovAu::Catalogue.new(user_agent: 'mine/1.0').dataset('abn-bulk-extract')
33
+ assert_requested(stub)
34
+ end
35
+
36
+ it "goes to the API it is told to" do
37
+ stub = stub_package(url: 'https://ckan.example/api/3/action/package_show')
38
+ DataGovAu::Catalogue.new(api_url: 'https://ckan.example/api/3/action').dataset('abn-bulk-extract')
39
+ assert_requested(stub)
40
+ end
41
+
42
+ it "goes to the API it is set to after it is made" do
43
+ stub = stub_package(url: 'https://ckan.example/api/3/action/package_show')
44
+ catalogue = DataGovAu::Catalogue.new
45
+ catalogue.api_url = 'https://ckan.example/api/3/action'
46
+ catalogue.dataset('abn-bulk-extract')
47
+ assert_requested(stub)
48
+ end
49
+
50
+ it "identifies itself as a block tells it to" do
51
+ stub = stub_request(:get, "#{PACKAGE_SHOW_URL}?id=abn-bulk-extract").
52
+ with(headers: {'User-Agent' => 'mine/1.0'}).
53
+ to_return(status: 200, body: JSON.generate({'success' => true, 'result' => package}))
54
+ DataGovAu::Catalogue.new{|c| c.user_agent = 'mine/1.0'}.dataset('abn-bulk-extract')
55
+ assert_requested(stub)
56
+ end
57
+ end
58
+
59
+ describe "when the catalogue cannot be read" do
60
+ it "stops upon a failed response" do
61
+ stub_package('abn-bulk-extract', {}, status: 503)
62
+ _(->{DataGovAu::Catalogue.new.dataset('abn-bulk-extract')}).must_raise DataGovAu::CatalogueUnavailable
63
+ end
64
+
65
+ # CKAN answers a name it does not know with success false.
66
+ it "stops where the answer reports no success" do
67
+ stub_package('abn-bulk-extract', {'success' => false})
68
+ _(->{DataGovAu::Catalogue.new.dataset('abn-bulk-extract')}).must_raise DataGovAu::CatalogueUnavailable
69
+ end
70
+ end
71
+ end
@@ -0,0 +1,30 @@
1
+ # test/DataGovAu/Dataset_test.rb
2
+
3
+ require_relative '../helper'
4
+
5
+ describe DataGovAu::Dataset do
6
+ it "reports what the catalogue says of it" do
7
+ _(dataset.name).must_equal 'abn-bulk-extract'
8
+ _(dataset.title).must_equal 'ABN Bulk Extract'
9
+ _(dataset.organisation).must_equal 'Australian Business Register'
10
+ _(dataset.licence).must_equal 'Creative Commons Attribution 3.0 Australia'
11
+ end
12
+
13
+ # CKAN's times carry no zone and are UTC, so read as local they would be wrong by the offset.
14
+ it "reads its modified time as UTC" do
15
+ _(dataset.modified_at).must_equal Time.utc(2026, 10, 27, 21, 41, 0, 366366)
16
+ end
17
+
18
+ it "gives its resources in the catalogue's order" do
19
+ _(dataset.resources.collect(&:filename)).must_equal ['bulkextract.xsd', 'public_split_1_10.zip']
20
+ end
21
+
22
+ it "gives no resources where the catalogue lists none" do
23
+ stub_package('abn-bulk-extract', {'success' => true, 'result' => package.merge('resources' => nil)})
24
+ _(DataGovAu::Catalogue.new.dataset('abn-bulk-extract').resources).must_equal []
25
+ end
26
+
27
+ it "gives up its entry as the catalogue gave it" do
28
+ _(dataset.to_h).must_equal package
29
+ end
30
+ end
@@ -0,0 +1,161 @@
1
+ # test/DataGovAu/Resource_test.rb
2
+
3
+ require_relative '../helper'
4
+
5
+ describe DataGovAu::Resource do
6
+ let(:resource){dataset.resources.last}
7
+ let(:url){resource.url}
8
+
9
+ describe "what it reports" do
10
+ it "reports what the catalogue says of it" do
11
+ _(resource.name).must_equal 'ABN Bulk Extract Part 1'
12
+ _(resource.format).must_equal 'ZIP'
13
+ _(resource.size).must_equal 40
14
+ end
15
+
16
+ it "reads its last modified time as UTC" do
17
+ _(resource.last_modified).must_equal Time.utc(2026, 10, 27, 21, 39, 23, 580175)
18
+ end
19
+
20
+ it "has no last modified time where the catalogue gives none" do
21
+ sizeless = DataGovAu::Resource.new(DataGovAu::Catalogue.new, resource.to_h.merge('last_modified' => nil))
22
+ _(sizeless.last_modified).must_be_nil
23
+ end
24
+
25
+ # The resource's own name is prose, and in at least one case carries a trailing space.
26
+ it "takes its file's name from its URL rather than from its own name" do
27
+ _(resource.filename).must_equal 'public_split_1_10.zip'
28
+ end
29
+ end
30
+
31
+ describe "a download which succeeds" do
32
+ it "writes the body to the path given" do
33
+ Dir.mktmpdir do |directory|
34
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 40)
35
+ path = File.join(directory, 'public_split_1_10.zip')
36
+ _(resource.download(path)).must_equal path
37
+ _(File.size(path)).must_equal 40
38
+ end
39
+ end
40
+
41
+ it "makes the directory it writes into" do
42
+ Dir.mktmpdir do |directory|
43
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 40)
44
+ path = File.join(directory, 'releases', '20261027', 'public_split_1_10.zip')
45
+ resource.download(path)
46
+ _(File.exist?(path)).must_equal true
47
+ end
48
+ end
49
+
50
+ # The partial file is the means, not an artefact: a directory holding one would be read as a cut-short download.
51
+ it "leaves no partial file behind" do
52
+ Dir.mktmpdir do |directory|
53
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 40)
54
+ path = File.join(directory, 'public_split_1_10.zip')
55
+ resource.download(path)
56
+ _(File.exist?(path + DataGovAu::Resource::PARTIAL_SUFFIX)).must_equal false
57
+ end
58
+ end
59
+
60
+ it "identifies itself as its catalogue does" do
61
+ Dir.mktmpdir do |directory|
62
+ resource = dataset(DataGovAu::Catalogue.new(user_agent: 'mine/1.0')).resources.last
63
+ stub = stub_request(:get, url).with(headers: {'User-Agent' => 'mine/1.0'}).to_return(status: 200, body: 'x' * 40)
64
+ resource.download(File.join(directory, 'f.zip'))
65
+ assert_requested(stub)
66
+ end
67
+ end
68
+
69
+ it "takes any length where the catalogue reports none" do
70
+ Dir.mktmpdir do |directory|
71
+ sizeless = DataGovAu::Resource.new(DataGovAu::Catalogue.new, resource.to_h.merge('size' => nil))
72
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 7)
73
+ path = File.join(directory, 'f.zip')
74
+ sizeless.download(path)
75
+ _(File.size(path)).must_equal 7
76
+ end
77
+ end
78
+ end
79
+
80
+ describe "a download which fails" do
81
+ it "stops upon a failed response" do
82
+ Dir.mktmpdir do |directory|
83
+ stub_request(:get, url).to_return(status: 404, body: '')
84
+ path = File.join(directory, 'public_split_1_10.zip')
85
+ _(->{resource.download(path)}).must_raise DataGovAu::DownloadFailed
86
+ _(File.exist?(path)).must_equal false
87
+ end
88
+ end
89
+
90
+ # A body short of what the catalogue reported is a truncated download, and is worth stopping for here rather than wherever it is read.
91
+ it "stops where the body is not the length the catalogue reported" do
92
+ Dir.mktmpdir do |directory|
93
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 39)
94
+ _(->{resource.download(File.join(directory, 'public_split_1_10.zip'))}).must_raise DataGovAu::DownloadFailed
95
+ end
96
+ end
97
+
98
+ it "leaves nothing in place where the length is wrong" do
99
+ Dir.mktmpdir do |directory|
100
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 39)
101
+ path = File.join(directory, 'public_split_1_10.zip')
102
+ begin
103
+ resource.download(path)
104
+ rescue DataGovAu::DownloadFailed
105
+ end
106
+ _(File.exist?(path)).must_equal false
107
+ _(File.exist?(path + DataGovAu::Resource::PARTIAL_SUFFIX)).must_equal false
108
+ end
109
+ end
110
+ end
111
+
112
+ describe "its first bytes" do
113
+ it "asks by range and returns what came back" do
114
+ stub = stub_request(:get, url).
115
+ with(headers: {'User-Agent' => "data_gov_au/#{DataGovAu::VERSION}", 'Range' => 'bytes=0-255'}).
116
+ to_return(status: 206, body: 'PK')
117
+ _(resource.first_bytes(256)).must_equal 'PK'
118
+ assert_requested(stub)
119
+ end
120
+
121
+ # Were the range ignored, the whole body would already be in hand, which is the one thing asking by range is meant to avoid; stopping says so rather than letting it pass unremarked.
122
+ it "stops where the range was ignored" do
123
+ stub_request(:get, url).to_return(status: 200, body: 'x' * 40)
124
+ _(->{resource.first_bytes(256)}).must_raise DataGovAu::DownloadFailed
125
+ end
126
+ end
127
+
128
+ describe "whether it is held" do
129
+ it "is not held where the file is absent" do
130
+ Dir.mktmpdir do |directory|
131
+ _(resource.held_at?(File.join(directory, 'public_split_1_10.zip'))).must_equal false
132
+ end
133
+ end
134
+
135
+ it "is held where the file is present at the length the catalogue reports" do
136
+ Dir.mktmpdir do |directory|
137
+ path = File.join(directory, 'public_split_1_10.zip')
138
+ File.binwrite(path, 'x' * 40)
139
+ _(resource.held_at?(path)).must_equal true
140
+ end
141
+ end
142
+
143
+ # A download cut short leaves a short file, which must not pass for a held one.
144
+ it "is not held where the file is present at another length" do
145
+ Dir.mktmpdir do |directory|
146
+ path = File.join(directory, 'public_split_1_10.zip')
147
+ File.binwrite(path, 'x' * 39)
148
+ _(resource.held_at?(path)).must_equal false
149
+ end
150
+ end
151
+
152
+ it "is held on presence alone where the catalogue reports no length" do
153
+ Dir.mktmpdir do |directory|
154
+ sizeless = DataGovAu::Resource.new(DataGovAu::Catalogue.new, resource.to_h.merge('size' => nil))
155
+ path = File.join(directory, 'public_split_1_10.zip')
156
+ File.binwrite(path, 'x')
157
+ _(sizeless.held_at?(path)).must_equal true
158
+ end
159
+ end
160
+ end
161
+ end
data/test/helper.rb ADDED
@@ -0,0 +1,66 @@
1
+ # test/helper.rb
2
+
3
+ lib_dir = File.expand_path(File.join(__FILE__, '..', '..', 'lib'))
4
+ $LOAD_PATH.unshift(lib_dir) unless $LOAD_PATH.include?(lib_dir)
5
+
6
+ # webmock's http_rb adapter's require 'http' finds thoran's http.rb gem, takes
7
+ # HTTP::VERSION for the http gem's, and crashes trying to monkey-patch an
8
+ # HTTP::Client which isn't there. The same workaround as http.rb's own suite.
9
+ def skip_webmocks_http_rb_adapter
10
+ gem 'webmock'
11
+
12
+ $LOADED_FEATURES << File.join(
13
+ Gem.loaded_specs['webmock'].full_gem_path,
14
+ 'lib/webmock/http_lib_adapters/http_rb_adapter.rb'
15
+ )
16
+ end
17
+
18
+ skip_webmocks_http_rb_adapter
19
+
20
+ require 'json'
21
+ require 'minitest/autorun'
22
+ require 'minitest/spec'
23
+ require 'tmpdir'
24
+ require 'webmock/minitest'
25
+
26
+ require 'data_gov_au'
27
+
28
+ PACKAGE_SHOW_URL = "#{DataGovAu::Catalogue::API_URL}/package_show"
29
+
30
+ def resource_entry(name, filename, format, last_modified, size)
31
+ {
32
+ 'name' => name,
33
+ 'format' => format,
34
+ 'size' => size,
35
+ 'last_modified' => last_modified,
36
+ 'url' => "https://data.gov.au/data/dataset/abc/resource/def/download/#{filename}",
37
+ }
38
+ end
39
+
40
+ # A package_show result shaped as data.gov.au's.
41
+ def package(resources: nil)
42
+ {
43
+ 'name' => 'abn-bulk-extract',
44
+ 'title' => 'ABN Bulk Extract',
45
+ 'organization' => {'name' => 'australian-business-register', 'title' => 'Australian Business Register'},
46
+ 'license_title' => 'Creative Commons Attribution 3.0 Australia',
47
+ 'metadata_modified' => '2026-10-27T21:41:00.366366',
48
+ 'resources' => resources || default_resources,
49
+ }
50
+ end
51
+
52
+ def default_resources
53
+ [
54
+ resource_entry('Bulk Extract Schema', 'bulkextract.xsd', 'XML', '2023-01-19T00:00:00', 12866),
55
+ resource_entry('ABN Bulk Extract Part 1', 'public_split_1_10.zip', 'ZIP', '2026-10-27T21:39:23.580175', 40),
56
+ ]
57
+ end
58
+
59
+ def stub_package(name = 'abn-bulk-extract', body = {'success' => true, 'result' => package}, status: 200, url: PACKAGE_SHOW_URL)
60
+ stub_request(:get, "#{url}?id=#{name}").to_return(status: status, body: JSON.generate(body))
61
+ end
62
+
63
+ def dataset(catalogue = DataGovAu::Catalogue.new)
64
+ stub_package
65
+ catalogue.dataset('abn-bulk-extract')
66
+ end
metadata ADDED
@@ -0,0 +1,114 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: data_gov_au.rb
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.0
5
+ platform: ruby
6
+ authors:
7
+ - thoran
8
+ bindir: bin
9
+ cert_chain: []
10
+ date: 1980-01-02 00:00:00.000000000 Z
11
+ dependencies:
12
+ - !ruby/object:Gem::Dependency
13
+ name: http.rb
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - ">="
17
+ - !ruby/object:Gem::Version
18
+ version: '0'
19
+ type: :runtime
20
+ prerelease: false
21
+ version_requirements: !ruby/object:Gem::Requirement
22
+ requirements:
23
+ - - ">="
24
+ - !ruby/object:Gem::Version
25
+ version: '0'
26
+ - !ruby/object:Gem::Dependency
27
+ name: minitest
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - "~>"
31
+ - !ruby/object:Gem::Version
32
+ version: '6.0'
33
+ type: :development
34
+ prerelease: false
35
+ version_requirements: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: '6.0'
40
+ - !ruby/object:Gem::Dependency
41
+ name: rake
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - ">="
45
+ - !ruby/object:Gem::Version
46
+ version: '0'
47
+ type: :development
48
+ prerelease: false
49
+ version_requirements: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - ">="
52
+ - !ruby/object:Gem::Version
53
+ version: '0'
54
+ - !ruby/object:Gem::Dependency
55
+ name: webmock
56
+ requirement: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - ">="
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ type: :development
62
+ prerelease: false
63
+ version_requirements: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - ">="
66
+ - !ruby/object:Gem::Version
67
+ version: '0'
68
+ description: Find a dataset through data.gov.au's CKAN catalogue by its name or id,
69
+ read what the catalogue says of it and of each of its resources, and download a
70
+ resource once, checked against the length the catalogue reports.
71
+ email: code@thoran.com
72
+ executables: []
73
+ extensions: []
74
+ extra_rdoc_files: []
75
+ files:
76
+ - CHANGELOG
77
+ - Gemfile
78
+ - LICENSE
79
+ - README.md
80
+ - Rakefile
81
+ - TODO
82
+ - data_gov_au.rb.gemspec
83
+ - lib/DataGovAu/Catalogue.rb
84
+ - lib/DataGovAu/Dataset.rb
85
+ - lib/DataGovAu/Error.rb
86
+ - lib/DataGovAu/Resource.rb
87
+ - lib/DataGovAu/VERSION.rb
88
+ - lib/data_gov_au.rb
89
+ - test/DataGovAu/Catalogue_test.rb
90
+ - test/DataGovAu/Dataset_test.rb
91
+ - test/DataGovAu/Resource_test.rb
92
+ - test/helper.rb
93
+ homepage: https://github.com/thoran/data_gov_au
94
+ licenses:
95
+ - MIT
96
+ metadata: {}
97
+ rdoc_options: []
98
+ require_paths:
99
+ - lib
100
+ required_ruby_version: !ruby/object:Gem::Requirement
101
+ requirements:
102
+ - - ">="
103
+ - !ruby/object:Gem::Version
104
+ version: '3.2'
105
+ required_rubygems_version: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - ">="
108
+ - !ruby/object:Gem::Version
109
+ version: '0'
110
+ requirements: []
111
+ rubygems_version: 4.0.23
112
+ specification_version: 4
113
+ summary: A client for data.gov.au's CKAN API.
114
+ test_files: []