aipp 0.2.4 → 0.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70da2e53874e0c550cbbacc863ff421cbd38dfc2cb1b7e4d2c77e2821839fdea
-  data.tar.gz: 62e9d5c5413756490d03f057e8851aba54c7d8802f4012a2f9626f78eb241d6f
+  metadata.gz: cddca26d9635de7aa2ded8bde0def36adf97d548e9c133358443f7812c778ef6
+  data.tar.gz: cadd71e6709c70f03a6f2ceda4af1ef38b049d70f3aaad2cd45ef3fc792df192
 SHA512:
-  metadata.gz: 6508c2a1565fdd92ed4fe2593b883278940e10fa10e68a0ffc75391b7e6d8ba8c66ca19c7e91495da39f60c4257e7882611a235ad9c8a5314b928140625a6e90
-  data.tar.gz: ade8abd0e5f521faf63b684d719c1e71a85278ec57f56edf4c83349c1ca4fc68422c9948c74f059c064cbb6ec21d66c91216f6418d3749bd53dd6eb1ecca24cc
+  metadata.gz: 5d511918c12c5c877c64cb4d5f28e5d968cdd98e08f9dfc72b8f188579da214aa63e953b36a5f17dbdd8b2d5d0281209f29d2693a458c4b7613d0d012c4111e7
+  data.tar.gz: b3b4a1dc343049c3aec1c58290e52d09fb43adb476a6c085271d31f0d20c016db19cb9ed0f4355e8eb78f199fa3e6b9580a2f2a30d2e6ff8bcc6548f818d8e08

data/.github/workflows/test.yml

@@ -0,0 +1,26 @@
+name: Test
+
+on:
+  push:
+    branches:
+    - master
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        ruby:
+        - 2.6.x
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Ruby ${{ matrix.ruby }}
+      uses: actions/setup-ruby@v1
+      with:
+        ruby-version: ${{ matrix.ruby }}
+    - name: Build and test
+      run: |
+        gem install bundler
+        bundle install --jobs 4 --retry 3
+        bundle exec rake

data/.ruby-version

@@ -1 +1 @@
-ruby-2.6.3
+ruby-2.6.5

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 0.2.5
+
+#### Additions
+* LF/AD-2>2.19 (AD navigational aids relevant to VFR)
+* Write build and manifest to `~/.aipp/<region>/builds`
+
+#### Changes
+* Renamed `~/.aipp/<region>/archive` to `~/.aipp/<region>/sources`
+
 ## 0.2.4
 
 #### Additions

data/README.md

@@ -1,20 +1,31 @@
 [![Version](https://img.shields.io/gem/v/aipp.svg?style=flat)](https://rubygems.org/gems/aipp)
-[![Continuous Integration](https://img.shields.io/travis/svoop/aipp/master.svg?style=flat)](https://travis-ci.org/svoop/aipp)
+[![Tests](https://github.com/svoop/aipp/workflows/Test/badge.svg)](https://github.com/svoop/aipp/actions?workflow=Test)
 [![Donorbox](https://img.shields.io/badge/donate-on_donorbox-yellow.svg)](https://donorbox.org/bitcetera)
 
 # AIPP
 
 Parser for Aeronautical Information Publication (AIP) available online.
 
-This gem incluces two executables to download and parse aeronautical data as HTML or PDF, then export is as [AIXM](https://github.com/svoop/aixm) or [OFMX](https://github.com/openflightmaps/ofmx/wiki).
+This gem incluces two executables to download and parse aeronautical data as HTML or PDF, then build and export is as [AIXM](https://github.com/svoop/aixm) or [OFMX](https://github.com/openflightmaps/ofmx/wiki).
 
 * [Homepage](https://github.com/svoop/aipp)
 * [Rubydoc](https://www.rubydoc.info/gems/aipp/AIPP)
 * Author: [Sven Schwyn - Bitcetera](http://www.bitcetera.com)
 
+## Table of Contents
+
+[Install](#install)<br>
+[Usage](#usage)<br>
+[Storage](#storage)<br>
+[Regions](#regions)<br>
+[AIP Parsers](#aip-parsers)<br>
+[References](#references)<br>
+[AIRAC Date Calculations](#airac-date-calculations)<br>
+[Development](#development)
+
 ## Install
 
-Add this to your <tt>Gemfile</tt>:
+Add this to your <tt>Gemfile</tt> or <tt>gems.rb</tt>:
 
 ```ruby
 gem aipp
@@ -22,15 +33,56 @@ gem aipp
 
 ## Usage
 
+See the built-in help for all options:
+
 ```
 aip2aixm --help
 aip2ofmx --help
 ```
 
+Say, you with to build the complete OFMX file for the current AIRAC cycle of the region LF:
+
+```
+aip2ofmx -r LF
+```
+
+You'll find the OFMX file in the current directory if the binary exits successfully.
+
 ## Storage
 
 AIPP uses a storage directory for configuration, caching and in order to keep the results of previous runs. The default location is `~/.aipp`, however, you can pass a different directory with the `--storage` argument.
 
+You'll find a directory for each region which contains the following items:
+
+* `sources/`<br>This directory contains one ZIP archive per AIRAC cycle which incrementially caches all source files used to build the AIXM/OFMX file. Therefore, to make sure it contains all source files for a region, you have to build at least one complete AIXM/OFMX file for that region.
+* `builds/`<br>This directory contains one ZIP archive per AIRAC cycle which is overwritten on every run. Therefore, to make sure it contains the complete build for a region, you have to make sure that your last run builds the complete AIXM/OFMX for that region. This archive contains:
+  * the built AIXM/OFMX file
+  * `build.yaml` – context of the build process
+  * `manifest.csv` – diffable manifest (see below)
+* `config.yml`<br>This file contains configuration which will be read on subsequent runs, most notably the namespace UUID used to identify the creator of OFMX files.
+
+The manifest is a CSV which lists every feature on a separate line along with its hashes and comment. You can `diff` or `git diff` two manifests:
+
+```diff
+$ git diff -U0 2019-09-12/manifest.csv 2019-10-10/manifest.csv
+
+--- a/2019-09-12/manifest.csv
++++ b/2019-10-10/manifest.csv
+@@ -89,0 +90 @@ Aha,449f4791,d6d4bff8,Address: PHONE for LFF958FC61
++Aha,44b9a044,cf55a3cf,Address: PHONE for LF49929DD6
+@@ -134 +135 @@ Aha,6acfbcc5,c4f57e3f,Address: FAX for LF897E1A6F
+-Aha,6b381b32,fb947716,Address: RADIO for LFPO
++Aha,6b381b32,b9723b7e,Address: RADIO for LFPO
+@@ -253,0 +255 @@ Aha,d01c4282,7cc8db7f,Address: PHONE for LF0C413AE2
++Aha,d05cfbff,74fb76bc,Address: FAX for LF49929DD6
+@@ -327 +329 @@ Ahp,068f4f01,1dc2fb5c,Airport: LFEA BELLE ILE
+-Ahp,06d04910,e18af958,Airport: LFFDE9077D TONNERRE CENTRE HOSPITALIER
++Ahp,06d04910,cc522711,Airport: LFFDE9077D TONNERRE CENTRE HOSPITALIER
+(...)
+```
+
+The advantage of `git diff` is it's ability to hightlight exactly which part of a line has changed. [Check out this post to learn how](https://www.viget.com/articles/dress-up-your-git-diffs-with-word-level-highlights/).
+
 ## Regions
 
 The reference implementation is region "LF" (France).
@@ -83,7 +135,7 @@ Inside the `parse` method, you have access to the following methods:
 
 * [`read`](https://www.rubydoc.info/gems/aipp/AIPP/AIP#read-instance_method) – download and read an AIP file
 * [`add`](https://www.rubydoc.info/gems/aipp/AIPP/AIP#add-instance_method) – add a [`AIXM::Feature`](https://www.rubydoc.info/gems/aixm/AIXM/Feature)
-* [`select`](https://www.rubydoc.info/gems/aipp/AIPP/AIP#find-instance_method – search previously written [`AIXM::Feature`s](https://www.rubydoc.info/gems/aixm/AIXM/Feature)
+* [`select`](https://www.rubydoc.info/gems/aipp/AIPP/AIP#find-instance_method) – search previously written [`AIXM::Feature`s](https://www.rubydoc.info/gems/aixm/AIXM/Feature)
 * some core extensions from ActiveSupport – [`Object#blank`](https://www.rubydoc.info/gems/activesupport/Object#blank%3F-instance_method) and [`String`](https://www.rubydoc.info/gems/activesupport/String)
 * core extensions from this gem – [`Object`](https://www.rubydoc.info/gems/aipp/Object), [`NilClass`](https://www.rubydoc.info/gems/aipp/NilClass), [`Integer`](https://www.rubydoc.info/gems/aipp/Integer), [`String`](https://www.rubydoc.info/gems/aipp/String), [`Hash`](https://www.rubydoc.info/gems/aipp/Hash) and [`Enumerable`](https://www.rubydoc.info/gems/aipp/Enumerable)
 
@@ -129,7 +181,7 @@ See [`AIPP::Border`](https://www.rubydoc.info/gems/aipp/AIPP/Border) for more on
 
 ### Helpers
 
-Helpers are modules defined in the <tt>lib/aipp/regions/{REGION}/helpers/</tt> directory. All helper modules are required automatically.
+Helpers are modules defined in the <tt>lib/aipp/regions/{REGION}/helpers/</tt> directory. All helper modules are required automatically in alphabetic order.
 
 There is one mandatory helper called `URL.rb` which must define the following method to build URLs from which to download AIPs:
 
@@ -155,7 +207,7 @@ Feel free to add more helpers to DRY code which is shared by multiple AIP parser
 module AIPP
   module LF
     module Helpers
-      module Common
+      module Base
 
         def source(position:, aip_file: nil)
           (...)
@@ -174,7 +226,7 @@ module AIPP
   module LF
     class AD2 < AIP
 
-      include AIPP::LF::Helpers::Common
+      include AIPP::LF::Helpers::Base
 
     end
   end

data/aipp.gemspec

@@ -25,18 +25,19 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'minitest'
   spec.add_development_dependency 'minitest-reporters'
   spec.add_development_dependency 'minitest-sound'
+  spec.add_development_dependency 'minitest-focus'
   spec.add_development_dependency 'minitest-matchers'
   spec.add_development_dependency 'spy'
   spec.add_development_dependency 'guard'
   spec.add_development_dependency 'guard-minitest'
 
-  spec.add_runtime_dependency 'aixm', '>= 0.3.6'
+  spec.add_runtime_dependency 'aixm', '>= 0.3.8'
   spec.add_runtime_dependency 'activesupport', '~> 5'
   spec.add_runtime_dependency 'nokogiri', '~> 1'
   spec.add_runtime_dependency 'nokogumbo', '~> 2'
   spec.add_runtime_dependency 'pdf-reader', '~> 2'
   spec.add_runtime_dependency 'json', '~> 2'
-  spec.add_runtime_dependency 'rubyzip', '~> 1'
+  spec.add_runtime_dependency 'rubyzip', '~> 2'
   spec.add_runtime_dependency 'colorize', '~> 0'
   spec.add_runtime_dependency 'pry', '~> 0'
   spec.add_runtime_dependency 'pry-rescue', '~> 1'

data/lib/aipp.rb

@@ -4,7 +4,9 @@ require 'pry'
 require 'pry-rescue'
 require 'optparse'
 require 'yaml'
+require 'csv'
 require 'pathname'
+require 'tmpdir'
 require 'open-uri'
 require 'securerandom'
 require 'tsort'

data/lib/aipp/aip.rb

@@ -35,9 +35,8 @@ module AIPP
 
     # Read an AIP source file
     #
-    # Depending on whether a local copy of the file exists, either:
-    # * download from URL to local storage and read from local archive
-    # * read from local archive
+    # Read the cached source file if it exists in the source archive. Otherwise,
+    # download it from URL and cache it.
     #
     # An URL builder method +url_for(aip_file)+ must be defined either in
     # +helper.rb+ or in the AIP parser definition (e.g. +ENR-2.1.rb+).
@@ -69,15 +68,8 @@ module AIPP
     #   e.g. :airport or :vor as listed in AIXM::CLASSES are recognized as well
     # @param attributes [Hash] filter by these attributes and their values
     # @return [Array<AIXM::Feature>]
-    def select(klass, attributes={})
-      klass = AIXM::CLASSES.fetch(klass) if klass.is_a? Symbol
-      aixm.features.select do |feature|
-        if feature.is_a? klass
-          attributes.reduce(true) do |memo, (attribute, value)|
-            memo && feature.send(attribute) == value
-          end
-        end
-      end
+    def select(*args)
+      aixm.select_features(*args)
     end
   end
 

data/lib/aipp/downloader.rb

@@ -3,15 +3,15 @@ module AIPP
   # AIP downloader infrastructure
   #
   # The downloader operates in the +storage+ directory where it creates two
-  # subdirectories "archive" and "work". The initializer looks for +archive+
-  # in "archives" and (if found) unzips its contents into "work". When reading
-  # a +document+, the downloader looks for the +document+ in "work" and
+  # subdirectories "sources" and "work". The initializer looks for the +source+
+  # archive in "sources" and (if found) unzips its contents into "work". When
+  # reading a +document+, the downloader looks for the +document+ in "work" and
   # (unless found) downloads it from +url+. HTML documents are parsed to
   # +Nokogiri::HTML5::Document+, PDF documents are parsed to +AIPP::PDF+.
-  # Finally, the contents of "work" are written back to +archive+.
+  # Finally, the contents of "work" are written back to the +source+ archive.
   #
   # @example
-  #   AIPP::Downloader.new(storage: options[:storage], archive: "2018-11-08") do |downloader|
+  #   AIPP::Downloader.new(storage: options[:storage], source: "2018-11-08") do |downloader|
   #     html = downloader.read(
   #       document: 'ENR-5.1',
   #       url: 'https://www.sia.aviation-civile.gouv.fr/dvd/eAIP_08_NOV_2018/FRANCE/AIRAC-2018-11-08/html/eAIP/FR-ENR-5.1-fr-FR.html'
@@ -26,20 +26,20 @@ module AIPP
     # @return [Pathname] directory to operate within
     attr_reader :storage
 
-    # @return [String] name of the archive (without extension ".zip")
-    attr_reader :archive
+    # @return [String] name of the source archive (without extension ".zip")
+    attr_reader :source
 
-    # @return [Pathname] full path to the archive
-    attr_reader :archive_file
+    # @return [Pathname] full path to the source archive
+    attr_reader :source_file
 
     # @param storage [Pathname] directory to operate within
-    # @param archive [String] name of the archive (without extension ".zip")
-    def initialize(storage:, archive:)
-      @storage, @archive = storage, archive
+    # @param source [String] name of the source archive (without extension ".zip")
+    def initialize(storage:, source:)
+      @storage, @source = storage, source
       fail(ArgumentError, 'bad storage directory') unless Dir.exist? storage
-      @archive_file = archives_path.join("#{@archive}.zip")
+      @source_file = sources_path.join("#{@source}.zip")
       prepare
-      unzip if @archive_file.exist?
+      unzip if @source_file.exist?
       yield self
       zip
     ensure
@@ -65,8 +65,8 @@ module AIPP
 
     private
 
-    def archives_path
-      @storage.join('archives')
+    def sources_path
+      @storage.join('sources')
     end
 
     def work_path
@@ -75,7 +75,7 @@ module AIPP
 
     def prepare
       teardown
-      archives_path.mkpath
+      sources_path.mkpath
       work_path.mkpath
     end
 
@@ -87,15 +87,15 @@ module AIPP
     end
 
     def unzip
-      Zip::File.open(archive_file).each do |entry|
+      Zip::File.open(source_file).each do |entry|
         entry.extract(work_path.join(entry.name))
       end
     end
 
     def zip
-      backup_file = archive_file.sub(/$/, '.old') if archive_file.exist?
-      archive_file.rename(backup_file) if backup_file
-      Zip::File.open(archive_file, Zip::File::CREATE) do |zip|
+      backup_file = source_file.sub(/$/, '.old') if source_file.exist?
+      source_file.rename(backup_file) if backup_file
+      Zip::File.open(source_file, Zip::File::CREATE) do |zip|
         work_path.children.each do |entry|
           zip.add(entry.basename.to_s, entry) unless entry.basename.to_s[0] == '.'
         end

data/lib/aipp/executable.rb

@@ -8,7 +8,8 @@ module AIPP
       @options = options
       @options[:airac] = AIPP::AIRAC.new
       @options[:storage] = Pathname(Dir.home).join('.aipp')
-      @options[:force] = $VERBOSE_INFO = $PRY_ON_WARN = $PRY_ON_ERROR = false
+      @options[:force] = @options[:mid] = false
+      $VERBOSE_INFO = $PRY_ON_WARN = $PRY_ON_ERROR = false
       OptionParser.new do |o|
         o.banner = <<~END
           Download online AIP and convert it to #{options[:schema].upcase}.
@@ -17,6 +18,7 @@ module AIPP
         o.on('-d', '--airac DATE', String, %Q[AIRAC date (default: "#{@options[:airac].date.xmlschema}")]) { |v| @options[:airac] = AIPP::AIRAC.new(v) }
         o.on('-r', '--region STRING', String, 'region (e.g. "LF")') { |v| @options[:region] = v.upcase }
         o.on('-a', '--aip STRING', String, 'process this AIP only (e.g. "ENR-5.1")') { |v| @options[:aip] = v.upcase }
+        o.on('-m', '--[no-]mid', 'insert mid attributes into all Uid elements (default: false)') { |v| @options[:mid] = v } 
         o.on('-s', '--storage DIR', String, 'storage directory (default: "~/.aipp")') { |v| @options[:storage] = Pathname(v) }
         o.on('-f', '--[no-]force', 'ignore XML schema validation (default: false)') { |v| @options[:force] = v }
         o.on('-v', '--[no-]verbose', 'verbose output (default: false)') { |v| $VERBOSE_INFO = v }
@@ -41,6 +43,7 @@ module AIPP
           parser.parse_aip
           parser.validate_aixm
           parser.write_aixm
+          parser.write_build
           parser.write_config
         end
       rescue => error

data/lib/aipp/parser.rb

@@ -3,6 +3,8 @@ module AIPP
   # AIP parser infrastructure
   class Parser
 
+    using AIXM::Refinements
+
     # @return [Hash] passed command line arguments
     attr_reader :options
 
@@ -24,7 +26,7 @@ module AIPP
     def initialize(options:)
       @options = options
       @options[:storage] = options[:storage].join(options[:region])
-      @options[:storage].mkpath unless @options[:storage].exist?
+      @options[:storage].mkpath
       @config = {}
       @aixm = AIXM.document(region: @options[:region], effective_at: @options[:airac].date)
       @dependencies = THash.new
@@ -76,7 +78,7 @@ module AIPP
     # Parse AIP by invoking the parser classes for the current region.
     def parse_aip
       info("AIRAC #{options[:airac].id} effective #{options[:airac].date}", color: :green)
-      AIPP::Downloader.new(storage: options[:storage], archive: options[:airac].date.xmlschema) do |downloader|
+      AIPP::Downloader.new(storage: options[:storage], source: options[:airac].date.xmlschema) do |downloader|
         @dependencies.tsort(options[:aip]).each do |aip|
           info("Parsing #{aip}")
           ("AIPP::%s::%s" % [options[:region], aip.remove(/\W/).classify]).constantize.new(
@@ -102,9 +104,52 @@ module AIPP
 
     # Write the AIXM document.
     def write_aixm
-      file = "#{options[:region]}_#{options[:airac].date.xmlschema}.#{options[:schema]}"
-      info("Writing #{file}")
-      File.write(file, aixm.to_xml)
+      info("Writing #{aixm_file}")
+      AIXM.config.mid_region = options[:region] if options[:mid]
+      File.write(aixm_file, aixm.to_xml)
+    end
+
+    # Write the AIXM document and context information.
+    def write_build
+      info("Writing build")
+      builds_path.mkpath
+      build_file = builds_path.join("#{@options[:airac].date.xmlschema}.zip")
+      build_file.delete if build_file.exist?
+      Dir.mktmpdir do |tmp_dir|
+        tmp_dir = Pathname(tmp_dir)
+        AIXM.config.mid_region = options[:region]
+        # AIXM/OFMX file
+        File.write(tmp_dir.join(aixm_file), aixm.to_xml)
+        # Build details
+        File.write(
+          tmp_dir.join('build.yaml'), {
+            version: AIPP::VERSION,
+            config: @config,
+            options: @options
+          }.to_yaml
+        )
+        # Manifest
+        manifest, buffer, feature, uid, comment = [], '', '', '', ''
+        File.open(tmp_dir.join(aixm_file)).each do |line|
+          buffer << line
+          case line
+          when /^ {2}<(\w{3}) / then buffer, feature = line, $1
+          when /^ {4}<#{feature}Uid mid="(.*?)"/ then uid = $1
+          when /^ {2}<!-- (.*) -->/ then comment = $1
+          when /^ {2}<\/#{feature}>/
+            manifest << [feature, uid[0,8], buffer.payload_hash(region: options[:region])[0,8], comment].to_csv
+            feature, uid = '', ''
+          end
+        end
+        manifest = manifest.sort.prepend "Feature,Short Uid Hash,Short Feature Hash,Comment\n"
+        File.write(tmp_dir.join('manifest.csv'), manifest.join)
+        # Zip it
+        Zip::File.open(build_file, Zip::File::CREATE) do |zip|
+          tmp_dir.children.each do |entry|
+            zip.add(entry.basename.to_s, entry) unless entry.basename.to_s[0] == '.'
+          end
+        end
+      end
     end
 
     # Write the configuration to config.yml.
@@ -115,6 +160,14 @@ module AIPP
 
     private
 
+    def aixm_file
+      "#{options[:region]}_#{options[:airac].date.xmlschema}.#{options[:schema]}"
+    end
+
+    def builds_path
+      options[:storage].join('builds')
+    end
+
     def config_file
       options[:storage].join('config.yml')
     end