argos-ruby 1.0.5 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,15 +1,15 @@
1
1
  ---
2
2
  !binary "U0hBMQ==":
3
3
  metadata.gz: !binary |-
4
- MDNkNTlmYzhjMjk2YzhlNGMzNjVjNDY3NDM5MjZjNzE4YzYwMzMxZA==
4
+ NWZlNzc5MzczNTY4MTkyMDljZmMzYTQwYTBlODIwYThkMjg5MDZhZQ==
5
5
  data.tar.gz: !binary |-
6
- MzVlNThiZTM1NzUxMTQ1ODQwMDZiNjU2NGFjYzIwMDY4MDRjZTIyMA==
6
+ MzQ0MzQ0NjVhM2Y0NDMxMmZjYjEzODRlNDg4ZDQxNTc4MDM1YTgxMQ==
7
7
  SHA512:
8
8
  metadata.gz: !binary |-
9
- Zjc1OWM3ZjE1MmI3ZDZiMzIyYThkMzYwODZhZTMxNjM3MWRiMTQ0MTY4YTFh
10
- YzU1YTY0NzZjNmE1YmQxYTgwOGZlMWM5YTgwNTczZjJmNGIzODkxOGI0NmNl
11
- ODk3ZTAwY2FhNGZkMTA1YjFkMDdlMGM4ZmY5MjY5ZjYyMmJlZjI=
9
+ N2U3OWFmOWU4NWEwNWEwZGI1YmE2OTNjODMyYjMyMTIwZDdmMTkwZGQyZGZl
10
+ ZWQ1MjNiN2VlMDQ3ZWRhZjU0YTE1ZmFjYzQyYTY2ZjViNDkyODNhMzMxMzEy
11
+ MWZkY2VjNDUxZjVhMzU0OWU5MTFhYTk1Nzc2MmIzZGIxZGIxZWM=
12
12
  data.tar.gz: !binary |-
13
- NmY0NDdmM2NmYzc2ODdjNjQwZmQ1NzQ4N2M3YzBhYTg2YTMwZWVmMzg3MDNm
14
- ODNlNDJlNTRiNTY3YzM2ZmI0YmM5NTFkODEzODJjMjk1MjRmN2ZhYWRlMDg3
15
- OGE5M2M3MGE1MDFiYjY0MDM5NjYyYjg0ZTcxNTNiM2E2Zjk5NDE=
13
+ N2Q0ZTg0NWUwOWRkNWYwY2VlZDBjNzc3NWQ3ZmIxMGVhYzg2YjYyZDVlZjQ4
14
+ YmYzMGEwZjRjNTM2Y2ZiMmJiMjRmMmIwNThjYmNiYjZjZTlmNGVjNDA0MjBh
15
+ OTNmZTRjMTZhM2U2ZDk0Y2M5OTEzOWEzZmI2N2RlMGMxZjdmYmM=
data/.gitignore CHANGED
@@ -18,3 +18,4 @@ todo.txt
18
18
  .yardoc
19
19
  _yardoc
20
20
  doc/
21
+ *.pdf
data/Gemfile CHANGED
@@ -4,7 +4,7 @@ gem "yajl-ruby", git: "git://github.com/brianmario/yajl-ruby.git"
4
4
  gem "uuidtools"
5
5
  gem "hashie", git: "git://github.com/intridea/hashie.git"
6
6
  #gem "json-schema", git: "git://github.com/hoxworth/json-schema.git"
7
-
7
+ gem "savon", github: 'savonrb/savon'
8
8
 
9
9
  group :development, :test do
10
10
  gem "thin"
data/LICENSE CHANGED
@@ -652,7 +652,7 @@ Also add information on how to contact you by electronic and paper mail.
652
652
  If the program does terminal interaction, make it output a short
653
653
  notice like this when it starts in an interactive mode:
654
654
 
655
- argos-ruby Copyright (C) 2013 Norwegian Polar Data
655
+ argos-ruby Copyright (C) 2013 Norwegian Polar Institute
656
656
  This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657
657
  This is free software, and you are welcome to redistribute it
658
658
  under certain conditions; type `show c' for details.
data/README.md CHANGED
@@ -1,92 +1,57 @@
1
1
  # argos-ruby
2
2
 
3
- A Ruby library and command-line tool for parsing Argos tracking data files.
3
+ A Ruby library and command-line tool accessing data from the [Argos](http://www.argos-system.org) tracking system.
4
4
 
5
- ## Usage
6
- ### DS/DIAG to JSON
5
+ ## argos-soap (webservice client)
7
6
 
8
- ```argos-ruby``` converts Argos files to JSON. DS header data is repeated in each message
9
- ```sh
10
- curl "https://raw.github.com/npolar/argos-ruby/master/spec/argos/_ds/fragment.ds" > /tmp/fragment.ds
11
- argos-ruby /tmp/fragment.ds
12
- ```
7
+ $ argos-soap --operations
13
8
  ```json
14
- [
15
- {
16
- "program": 9660,
17
- "platform": 10783,
18
- "lines": 3,
19
- "sensors": 3,
20
- "satellite": "K",
21
- "lc": "3",
22
- "positioned": "1999-12-30T17:58:26Z",
23
- "latitude": 79.824,
24
- "longitude": 22.363,
25
- "altitude": 0.0,
26
- "headers": 12,
27
- "measured": "1999-12-30T17:56:30Z",
28
- "identical": 2,
29
- "sensor_data": [
30
- "78",
31
- "00",
32
- "00"
33
- ],
34
- "technology": "argos",
35
- "type": "ds",
36
- "location": "file:///tmp/fragment.ds",
37
- "source": "a9c02cf81978a9fafecac582309c7c8161e5a76c",
38
- "parser": "argos-ruby-1.0.4",
39
- "id": "2d833010d13714dfb771f73470417405b887e8f4",
40
- "bundle": "bc30f5e5733087f730202bea53fa730092a192e3"
41
- },
42
- {
43
- "program": 9660,
44
- "platform": 10783,
45
- "lines": 3,
46
- "sensors": 3,
47
- "satellite": "K",
48
- "lc": "3",
49
- "positioned": "1999-12-30T17:58:26Z",
50
- "latitude": 79.824,
51
- "longitude": 22.363,
52
- "altitude": 0.0,
53
- "headers": 12,
54
- "measured": "1999-12-30T18:01:20Z",
55
- "identical": 5,
56
- "sensor_data": [
57
- "78",
58
- "00",
59
- "00"
60
- ],
61
- "technology": "argos",
62
- "type": "ds",
63
- "location": "file:///tmp/fragment.ds",
64
- "source": "a9c02cf81978a9fafecac582309c7c8161e5a76c",
65
- "parser": "argos-ruby-1.0.4",
66
- "id": "26af9092d01fbd49ff8cb9041e63df352886dba6",
67
- "bundle": "bc30f5e5733087f730202bea53fa730092a192e3"
68
- }
69
- ]
70
-
9
+ ["getCsv","getStreamXml","getKml","getXml","getXsd","getPlatformList","getObsCsv","getObsXml"]
71
10
  ```
11
+ **All programs**
12
+ Data from all programs and all platforms are fetched by default
72
13
 
73
- ## About
14
+ $ argos-soap --operation=getXml > [getXml.xml](https://github.com/npolar/argos-ruby/blob/master/spec/argos/_soap/getXml.xml)
15
+
16
+ $ argos-soap --operation=getXml --format=json > [getXml.json](https://github.com/npolar/argos-ruby/blob/master/spec/argos/_soap/getXml.json)
17
+
18
+ $ argos-soap --operation=getKml > [getKml.xml](https://github.com/npolar/argos-ruby/blob/master/spec/argos/_soap/getKml.xml)
19
+
20
+ $ argos-soap --operation=getCsv --format=text > [getCsv.csv](https://github.com/npolar/argos-ruby/blob/master/spec/argos/_soap/getCsv.csv)
21
+
22
+ **Select program(s)**
23
+ argos-soap --operation getXml --programNumber 9660
74
24
 
75
- [argos-ruby](https://github.com/npolar/argos-ruby) has been developed to parse [Argos](http://www.argos-system.org)
76
- satellite tracking data files collected by the [Norwegian Polar Institute]
77
- (http://npolar.no/en) since 1989.
25
+ **Select platformId(s)**
26
+ $ argos-soap --operation getXml --platformId 81308
78
27
 
79
- Be warned, the Argos file formats have changed over time. No promises are
28
+
29
+ **Authentication**
30
+ Either set your credentials on the command line
31
+ $ argos-soap --operation=getPlatformList --username=myusername --password=mypassword
32
+ Or set environmental variables ARGOS_SOAP_USERNAME and ARGOS_SOAP_PASSWORD
33
+
34
+ ## Argos file parsing
35
+ ### DS/DIAG to JSON
36
+ $ #
37
+
38
+ ## About
39
+
40
+ [argos-ruby](https://github.com/npolar/argos-ruby) was developed to parse [Argos](http://www.argos-system.org)
41
+ satellite tracking data text files collected by the [Norwegian Polar Institute]
42
+ (http://npolar.no/en) since 1989. Be warned, the Argos text file formats have changed over time. No promises are
80
43
  made that the library will work outside of Norway :).
81
44
 
82
- Currently, the library parses Argos DS/DIAG files dating from August 1990
83
- and onwards.
45
+ Currently, the library parses Argos DS/DIAG files dating from August 1990 and onwards.
84
46
 
85
47
  ## Install
86
48
  $ gem install argos-ruby
87
49
 
88
50
  ## Links
89
- * [http://api.npolar.no/tracking/?q=&filter-technology=argos](http://api.npolar.no/tracking/?q=&filter-technology=argos)
90
- * [Argos users manual v1.5](http://www.argos-system.org/files/pmedia/public/r363_9_argos_users_manual-v1.5.pdf) (PDF)
51
+
52
+ * https://github.com/npolar/api.npolar.no/wiki/Tracking-API-JSON
53
+ * [Argos User's Manual](http://www.argos-system.org/manual/)
54
+ * [Argos Web Service Interface Specification v1.5](http://www.argos-system.org/manual/argos_webservices-1_4.pdf)
91
55
  * http://alaska.usgs.gov/science/biology/spatial/
92
- * http://gis-lab.info/programs/argos/argos-manual-eng.html
56
+ * http://gis-lab.info/programs/argos/argos-manual-eng.html
57
+
data/argos-ruby.gemspec CHANGED
@@ -9,8 +9,8 @@ Gem::Specification.new do |s|
9
9
  s.authors = ["Espen Egeland", "Conrad Helgeland"]
10
10
  s.email = ["data*npolar.no"]
11
11
  s.homepage = "http://github.com/npolar/argos-ruby"
12
- s.summary = %q{Argos satellite tracking data parsers}
13
- s.description = %q{Parses Argos (http://www.argos-system.org/) DS/DAT and DIAG/DIA files.}
12
+ s.summary = %q{Argos satellite tracking library and command-line tools}
13
+ s.description = %q{Argos (http://www.argos-system.org/) webservice client and Argos legacy file (DS/DAT and DIAG/DIA) parser.}
14
14
  s.license = "GPL-3.0"
15
15
  s.add_development_dependency "rspec"
16
16
  s.files = `git ls-files`.split("\n")
@@ -3,9 +3,10 @@
3
3
 
4
4
  # argos-ruby /path/argos/ --dest=/path/dest --level=info 2>> /path/argos-json.log
5
5
  require_relative "../lib/argos"
6
+
6
7
  require "optparse"
7
- require "yaml"
8
8
  require "fileutils"
9
+ require "csv"
9
10
 
10
11
  param = { format: "json",
11
12
  action: "parse",
@@ -66,7 +67,7 @@ log = Logger.new(STDERR)
66
67
  log.level = param[:level]
67
68
 
68
69
 
69
- if not Argos.argos? glob
70
+ if not Argos::Ascii.argos? glob
70
71
  glob = glob.gsub(/\/$/, "")
71
72
  if glob != /\*/
72
73
  glob += "/**/*"
@@ -80,14 +81,14 @@ argosfiles = 0
80
81
 
81
82
  Dir[glob].reject {|f| File.directory? f}.select {|f|
82
83
  argosfiles = argosfiles + 1
83
- if not Argos.argos? f
84
+ if not Argos::Ascii.argos? f
84
85
  log.warn "Not Argos: #{f}"
85
86
  end
86
87
 
87
- Argos.argos? f
88
+ Argos::Ascii.argos? f
88
89
  }.map {|filename|
89
90
  log.debug "*"*80
90
- argos = Argos.factory(Argos.type(filename))
91
+ argos = Argos::Ascii.factory(Argos::Ascii.type(filename))
91
92
  argos.filename = filename
92
93
  argos.filter = param[:filter]
93
94
  argos.bundle = bundle
@@ -95,7 +96,7 @@ Dir[glob].reject {|f| File.directory? f}.select {|f|
95
96
 
96
97
  case param[:action]
97
98
  when "source","metadata"
98
- result << Argos.source(argos).merge(glob: glob, bundle: bundle)
99
+ result << Argos::Ascii.source(argos).merge(glob: glob, bundle: bundle)
99
100
 
100
101
  when "parse"
101
102
 
@@ -150,6 +151,14 @@ if param[:dest].nil?
150
151
  puts result.to_yaml
151
152
  when /(ruby|rb)/
152
153
  puts result
154
+ when "csv"
155
+
156
+ data = result.map {|r| r.values }
157
+ csv = CSV.generate() do |csv|
158
+ csv << result[0].keys
159
+ data.each { |row| csv << row }
160
+ end
161
+ puts csv
153
162
  end
154
163
  end
155
164
 
data/bin/argos-soap ADDED
@@ -0,0 +1,13 @@
1
+ #!/usr/bin/env ruby
2
+ # encoding: utf-8
3
+
4
+ # Ruby-based command line client for Argos' Soap webservice
5
+ #
6
+ # For more information:
7
+ # $ ./bin/argos-soap --help
8
+ # https://github.com/npolar/argos-ruby/blob/master/README.md
9
+
10
+ require_relative "../lib/argos"
11
+ require_relative "../lib/argos/soap_command"
12
+
13
+ Argos::SoapCommand.run
data/lib/argos.rb CHANGED
@@ -1,155 +1,33 @@
1
+ require "bundler/setup"
1
2
  require "bigdecimal"
2
3
  require "date"
3
4
  require "time" # for iso8601
4
5
  require "digest/sha1"
5
6
  require "json"
6
7
  require "logger"
8
+ require "cgi"
7
9
 
8
10
  require_relative "argos/exception"
11
+ require_relative "argos/ascii"
9
12
  require_relative "argos/ds"
10
13
  require_relative "argos/diag"
11
-
12
- # Argos module
13
- # Contains parsers for Argos DS/DAT and DIAG files.
14
- #
15
- # Code written by staff at the Norwegian Polar Data Centre
16
- # http://data.npolar.no - a service run by the [Norwegian Polar Institute](http://npolar.no)
14
+ require_relative "argos/soap"
15
+ require_relative "argos/download"
16
+
17
+ # Argos library containg
18
+ # * Parsers for Argos legacy ASCII files (DS/DAT and DIAG files)
19
+ # * Soap web service client and Argos XML download tool
20
+ #
21
+ # https://github.com/npolar/argos-ruby
22
+ #
23
+ # Code written by staff at the [Norwegian Polar Data Centre](http://data.npolar.no),
24
+ # [Norwegian Polar Institute](http://npolar.no)
17
25
  #
18
26
  # For information about Argos, see: http://www.argos-system.org
19
27
  module Argos
20
- VERSION = "1.0.5"
21
- # Detect Argos type ("ds" or "diag" or nil)
22
- #
23
- # @param filename [String] Argos (DS or DIAG) file
24
- # @return [String]
25
- #"ds"|"diag"
26
- def self.type filename
27
-
28
- if File.file? filename
29
- # Avoid invalid byte sequence in UTF-8 (ArgumentError)
30
- firstline = File.open(filename, :encoding => "iso-8859-1") {|f| f.readline}
31
- else
32
- firstline = filename
33
- end
34
-
35
- case firstline
36
- when Argos::Ds::START_REGEX, Argos::Ds::START_REGEX_LEGACY
37
- "ds"
38
- when Argos::Diag::START_REGEX
39
- "diag"
40
- when "", nil
41
- raise ArgumentError, "Not a file or empty string: #{filename}"
42
- else nil
43
- end
44
- end
45
-
46
- # Argos file?
47
- #
48
- # @param filename [String] Argos (DS or DIAG) file
49
- # @return [Boolean]
50
- def self.argos?(filename)
51
- case type(filename)
52
- when "ds", "diag"
53
- true
54
- else
55
- false
56
- end
57
- end
58
-
59
- # Factory for Argos::Ds / Argos::Diag
60
- #
61
- # @param type [String]: Argos (DS or DIAG) file type (or filename)
62
- # @return [Argos::Ds Argos::Diag]
63
- # @throws ArgumentError
64
- def self.factory(type)
65
-
66
- # Auto-detect file format if not "ds" or "diag"
67
- if not ["ds","diag"].include? type
68
- if argos? type
69
- type = self.type(type)
70
- end
71
- end
72
-
73
- case type
74
- when "ds"
75
- Ds.new
76
- when "diag"
77
- Diag.new
78
- else
79
- raise ArgumentError, "Unknown Argos type: #{type}"
80
- end
81
- end
82
-
83
-
28
+ VERSION = "1.1.0"
29
+
84
30
  def self.library_version
85
31
  "argos-ruby-#{VERSION}"
86
32
  end
87
-
88
- # Source fingerprint of Argos file (sha1 hash, segment and document counts, etc.)
89
- #
90
- # @param  [Argos::Ds Argos::Diag] argos
91
- # @return [Hash] Source hash
92
- def self.source(argos)
93
-
94
- argos.parse(argos.filename)
95
-
96
- latitude_mean = longitude_mean = nil
97
- if argos.latitudes.any?
98
- latitude_mean = (argos.latitudes.inject{ |sum, latitude| sum + latitude } / argos.latitudes.size).round(3)
99
- end
100
- if argos.longitudes.any?
101
- longitude_mean = (argos.longitudes.inject{ |sum, longitude| sum + longitude } / argos.latitudes.size).round(3)
102
- end
103
-
104
-
105
- source = {
106
- id: argos.source,
107
- technology: "argos",
108
- collection: "tracking",
109
- type: argos.type,
110
- programs: argos.programs,
111
- platforms: argos.platforms,
112
- start: argos.start,
113
- stop: argos.stop,
114
- north: argos.latitudes.max,
115
- east: argos.longitudes.max,
116
- south: argos.latitudes.min,
117
- west: argos.longitudes.min,
118
- latitude_mean: latitude_mean,
119
- longitude_mean: longitude_mean,
120
- location: "file://"+argos.filename,
121
- bytes: argos.filesize,
122
- modified: argos.updated.utc.iso8601,
123
- messages: argos.messages.size,
124
- filter: argos.filtername.nil? ? argos.filter : argos.filtername,
125
- size: argos.size,
126
- parser: library_version
127
- }
128
- if argos.multiplicates.any?
129
- source[:multiplicates] = argos.multiplicates.map {|a| a[:id]}
130
- end
131
- if argos.errors.any?
132
- source[:errors] = argos.errors
133
- end
134
- source
135
-
136
- end
137
-
138
-
139
- def latitudes
140
- select {|a| a.key? :latitude and a[:latitude].is_a? Float }.map {|a| a[:latitude]}.sort
141
- end
142
-
143
- def longitudes
144
- select {|a| a.key? :longitude and a[:longitude].is_a? Float }.map {|a| a[:longitude]}.sort
145
- end
146
-
147
- def platforms
148
- map {|a| a[:platform]}.uniq.sort
149
- end
150
-
151
- def programs
152
- map {|a| a[:program]}.uniq.sort
153
- end
154
-
155
33
  end
@@ -0,0 +1,134 @@
1
+ module Argos
2
+ module Ascii
3
+
4
+ # Detect Argos ASCII filetype ("ds" or "diag" or nil)
5
+ #
6
+ # @param filename [String] Argos (DS or DIAG) file
7
+ # @return [String]
8
+ #"ds"|"diag"
9
+ def self.type filename
10
+
11
+ if File.file? filename
12
+ # Avoid invalid byte sequence in UTF-8 (ArgumentError)
13
+ firstline = File.open(filename, :encoding => "iso-8859-1") {|f| f.readline}
14
+ else
15
+ firstline = filename
16
+ end
17
+
18
+ case firstline
19
+ when Argos::Ds::START_REGEX, Argos::Ds::START_REGEX_LEGACY
20
+ "ds"
21
+ when Argos::Diag::START_REGEX
22
+ "diag"
23
+ when "", nil
24
+ raise ArgumentError, "Not a file or empty string: #{filename}"
25
+ else nil
26
+ end
27
+ end
28
+
29
+ # Argos ASCII file?
30
+ #
31
+ # @param filename [String] Argos (DS or DIAG) file
32
+ # @return [Boolean]
33
+ def self.argos?(filename)
34
+ case type(filename)
35
+ when "ds", "diag"
36
+ true
37
+ else
38
+ false
39
+ end
40
+ end
41
+
42
+ # Factory for Argos::Ds / Argos::Diag
43
+ #
44
+ # @param type [String]: Argos (DS or DIAG) file type (or filename)
45
+ # @return [Argos::Ds Argos::Diag]
46
+ # @throws ArgumentError
47
+ def self.factory(type)
48
+
49
+ # Auto-detect file format if not "ds" or "diag"
50
+ if not ["ds","diag"].include? type
51
+ if argos? type
52
+ type = self.type(type)
53
+ end
54
+ end
55
+
56
+ case type
57
+ when "ds"
58
+ Ds.new
59
+ when "diag"
60
+ Diag.new
61
+ else
62
+ raise ArgumentError, "Unknown Argos type: #{type}"
63
+ end
64
+ end
65
+
66
+ # Source fingerprint of Argos file (sha1 hash, segment and document counts, etc.)
67
+ #
68
+ # @param  [Argos::Ds Argos::Diag] argos
69
+ # @return [Hash] Source hash
70
+ def self.source(argos)
71
+
72
+ argos.parse(argos.filename)
73
+
74
+ latitude_mean = longitude_mean = nil
75
+ if argos.latitudes.any?
76
+ latitude_mean = (argos.latitudes.inject{ |sum, latitude| sum + latitude } / argos.latitudes.size).round(3)
77
+ end
78
+ if argos.longitudes.any?
79
+ longitude_mean = (argos.longitudes.inject{ |sum, longitude| sum + longitude } / argos.latitudes.size).round(3)
80
+ end
81
+
82
+
83
+ source = {
84
+ id: argos.source,
85
+ technology: "argos",
86
+ collection: "tracking",
87
+ type: argos.type,
88
+ programs: argos.programs,
89
+ platforms: argos.platforms,
90
+ start: argos.start,
91
+ stop: argos.stop,
92
+ north: argos.latitudes.max,
93
+ east: argos.longitudes.max,
94
+ south: argos.latitudes.min,
95
+ west: argos.longitudes.min,
96
+ latitude_mean: latitude_mean,
97
+ longitude_mean: longitude_mean,
98
+ file: "file://"+argos.filename,
99
+ bytes: argos.filesize,
100
+ modified: argos.updated.utc.iso8601,
101
+ messages: argos.messages.size,
102
+ filter: argos.filtername.nil? ? argos.filter : argos.filtername,
103
+ size: argos.size,
104
+ parser: Argos.library_version
105
+ }
106
+ if argos.multiplicates.any?
107
+ source[:multiplicates] = argos.multiplicates.map {|a| a[:id]}
108
+ end
109
+ if argos.errors.any?
110
+ source[:errors] = argos.errors
111
+ end
112
+ source
113
+
114
+ end
115
+
116
+
117
+ def latitudes
118
+ select {|a| a.key? :latitude and a[:latitude].is_a? Float }.map {|a| a[:latitude]}.sort
119
+ end
120
+
121
+ def longitudes
122
+ select {|a| a.key? :longitude and a[:longitude].is_a? Float }.map {|a| a[:longitude]}.sort
123
+ end
124
+
125
+ def platforms
126
+ map {|a| a[:platform]}.uniq.sort
127
+ end
128
+
129
+ def programs
130
+ map {|a| a[:program]}.uniq.sort
131
+ end
132
+
133
+ end
134
+ end