github-to-canvas 0.1.0 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cae13e5551446c7306ce3d8d90a12fa47a3caedbcebdc749b1852456b558c695
-  data.tar.gz: 2501db731b91ab03fafae8a42eeaed104228cab7f5dcffaf21723558e9ac03f3
+  metadata.gz: b18946099c8b4fd23e1c2230f71fac458f932079fd22ada5297732358fec4d0e
+  data.tar.gz: '092e1b7854f2b4c4026f0713599d423d8eac6431ae16e4f9372c1a00ebb9775c'
 SHA512:
-  metadata.gz: daf5758513ab73925cb7b7f402112ead3e9c0b3bcb4a59d6f227d5eb04ab710c7c9ecf06d3edf9326172d5758a8625a158e508f6eb68e740f8c5efd8cf31bc31
-  data.tar.gz: 38cf70d8025eed5b8128e7885bb21ac9c8a25b94c89a0bd2fb00df7421ecc80df02615c26e335cb86bdce16d1d5653898617332a31249ce0478050b5a89f3818
+  metadata.gz: d11be9f366fc3a457a671669a57802d6a1a8cc8ff3b0a7e9fa3ee10b4b8b91334cbaa336d39c1e3057c41e787619716d76df3e8523d918393cbb1b6fa9737549
+  data.tar.gz: 85d91f81000c82d55cb45e02f31c9aa3b2e241c785cb58366f5fbef99b72c7ce76e7e9d70f101cedb7f54c5082b2061926e0453cd8a162227cdba78f96c54e71

data/README.md

@@ -4,19 +4,15 @@
 
 The `github-to-canvas` gem is designed to aid in integrating GitHub and the
 Canvas LMS. This gem takes a GitHub repository's `README.md` file, converts it
-to HTML, and pushes it to Canvas using the Canvas API. The gem also has the ability
-to updates a repository to include a `.canvas` file containing Canvas specific
-information.
-
-With  the `.canvas` file in place, this gem can be used to continuously align
-content between GitHub and Canvas using the GitHub repository as the single
-source of truth.
+to HTML, and pushes it to Canvas using the Canvas API. This gem can also update
+existing Canvas lessons, allowing continuous alignment of content between GitHub
+and Canvas.
 
 This gem is built for use internally at [Flatiron School][], so some features may be
 specific to Flatiron School branding and needs. Access to the
 [Canvas LMS API][] and the ability to add pages and assignments to a Canvas
 course are required. Write access to the GitHub repository being converted is
-also required for committing `.canvas` files.
+also required for some features.
 
 ## Installation
 
@@ -47,7 +43,7 @@ echo "$(export 'CANVAS_API_KEY=your-new-API-key-here' | cat - ~/.bash_profile)"
 
 ### Add Canvas API Base Path
 
-The exact Canvas API path is specific to where you Canvas LMS is located. For example,
+The exact Canvas API path is specific to where your Canvas LMS is located. For example,
 Flatiron School's base path is `https://learning.flatironschool.com/api/v1`. Add this path
 as an `ENV` variable like the API key. **Do not add a `/` at the end after `/api/v1`.**
 
@@ -65,67 +61,366 @@ After both the API key and path are added to `~/.zshrc`, run `source ~/.zshrc` (
 to make them available in your current terminal. You can verify these variables
 are present by running `ENV` and finding them in the output list.
 
-## Usage
+## Common Uses
+
+The GitHub to Canvas gem can be used for the following:
+
+- [Create a Canvas Lesson from a Local Repository](#create)
+- [Create a Canvas Lesson from a Remote Repository](#createremote)
+- [Read a Remote Repository as HTML](#read)
+- [Update a Canvas Lesson from a Local Repository](#update)
+- [Update a Canvas Lesson from a Remote Repository](#updateremote)
+- [Retrieve Canvas Course Information as YAML Markdown](#query)
+- [Map GitHub Repositories to a Canvas Course YAML file](#map)
+- [Create New Canvas Course from a YAML file](#buildcourse)
+- [Update Lessons in an Existing Course from a YAML file](#updatecourse)
+
+### Creating and Updating Canvas Lessons
+
+#### Create a Canvas Lesson from a Local Repository <a name="create"></a>
+
+Navigate into a repository folder cloned down to your local machine and run:
+
+```sh
+github-to-canvas -c <CANVAS_COURSE_ID> -lr --forkable
+```
+
+The command above will create a Canvas lesson in the course provided. It will
+also remove the repositories top-level header and remove any old Flatiron
+branded footers. It will also add an HTML header for Canvas that includes links
+back to the repository.
+
+If the lesson type is an assignment, a Fork button will also be added to the
+HTML header. Because the command didn't specify, the type of lesson is determined
+based on the local repo structure - if it has sub-folders, the lesson will become
+an assignment; if there are no sub-folders, the lesson will become a page. If the
+lesson type is a page, the `--forkable` option will be ignored.
+
+Creating a lesson this way will also produce a `.canvas` file. This file
+contains info about the Canvas lesson that was created.
+
+#### Create a Canvas Lesson from a Remote Repository <a name="createremote"></a>
+
+To create from a remote repo, run the following command:
+
+```sh
+github-to-canvas --create-from-github <URL> --course <CANVAS_COURSE_ID> --type <TYPE> -lr --forkable
+```
+
+This command will read a GitHub markdown based on the provided URL and create a
+Canvas lesson based on it. We need to provide the Course ID and the type of
+lesson (`page` or `assignment`) as type can't be determined automatically. We'll
+also continue to use `-lr --forkable` like the previous command to set up the
+lesson the same way as before.
+
+The repository must be public in order to read the markdown file.
 
-### Create a Canvas Lesson
+#### Read a Remote Repository as HTML <a name="read"></a>
 
-To create a Canvas lesson from a GitHub `README.md` file, at minimum, you will
-need to know the Canvas course id you are going to add to. This id can be found
-in the URL of the course.
+To read the contents of a remote repo:
 
-Once you have the course id, you will need to do the following:
+```sh
+github-to-canvas --read-from-github <URL>
+```
+
+This will produce an HTML conversion of the repository's markdwon. This HTML can
+be directly pasted into Canvas' HTML editor if a manual process is needed.
 
-1. Clone down the repository you'd like to push to Canvas.
-2. Change directory into the new local repository
-3. Run `github-to-canvas --create <your-course-id>` from inside the local repo
+#### Update a Canvas Lesson from a Local Repository <a name="update"></a>
 
-If everything is set up properly, `github-to-canvas` will create a Canvas lesson
-using `master` branch `README.md` and the name of the current folder. By
-default, if the repository contains folders, an **assignment** will be created.
-Otherwise, a **page** will be created.
+If you previously created a Canvas lesson from a local repository, you should
+have a `.canvas` file present in the repo. If that file is present, you can run
+the following command to update the listed Canvas lesson automatically:
+
+```sh
+github-to-canvas -a -lr --forkable
+```
 
-### Saving `.canvas` to GitHub
+If you don't have or want to use the `.canvas` file, you can also specify the
+course ID and lesson ID:
 
-Using `--save-to-github` will create a `.canvas` YAML file in the local repo and attempt to commit
-and push it to the remote repository. This file contains the course id, the newly
-created page id, and the Canvas URL to the lesson for future reference. 
+```sh
+github-to-canvas -a --course <CANVAS_COURSE_ID> --id <CANVAS_LESSON_ID> -lr --forkable
+```
 
-If you create multiple Canvas lessons from the same repository, each lesson's
-Canvas data will be stored in the `.canvas` file.
+Canvas course and lesson IDs can be found in the URL.
 
-### Update an Existing Canvas Lesson
+#### Update a Canvas Lesson from a Remote Repository <a name="updateremote"></a>
 
-To update an existing Canvas lesson using a local repository, **a `.canvas` file
-must be present in the repo**, as it contains the lesson information for the
-Canvas API.
+You can update an existing Canvas course using a remote GitHub repository like so:
 
-1. Clone down and/or change directory into the repository you'd like to update
-2. Run `github-to-canvas --align` from inside the local repo
+```sh
+github-to-canvas --align-from-github <URL> --course <COURSE_ID> --id <LESSON_ID> --type <TYPE> -lr --forkable
+```
 
-`github-to-canvas` will get the course id and page/assignment id from the
-`.canvas` file and update the associated Canvas lesson. If there are multiple
-Canvas lessons included in the `.canvas` file, each lesson will be updated (i.e.
-you have the same lesson in two courses created from one repository).
+This will read remote markdown using the provided URL and update a Canvas course
+using the info provided. Type must match the existing lesson type.
 
-### Options
+### Course Creation
 
-This gem provides to the following options:
+This gem can create Canvas courses from scratch. These features
+are still in development and may not work for all course designs. Quiz and
+Discussion Topic lesson creation is still under development and will not work.
 
-* `-c, --create-lesson COURSE` - Creates a new Canvas lesson, converting the local repository's README.md to HTML. Adds .canvas file to remote repository
-* `-a, --align` - Updates a canvas lesson based on the local repository's README.md
-* `-n, --name NAME` - Sets the name of the new Canvas lesson to be created. If no name is given, repository folder name is used
-* `-t, --type TYPE` - Sets the type of Canvas lesson to be created (page, assignment or discussion). If no type, type decided based on repository structure
-* `-f, --file FILE` - Looks for and uses a markdown file in the currentt folder as source for conversion. Default file is README.md
-* `-b, --branch BRANCH` - Sets the repository branch used for lesson creation
-* `-s, --save-to-github` - Creates a local .canvas file and attempts to commit and push it to the GitHub repository
-* `-l, --fis-links` - Adds additional Flatiron School HTML after markdown conversion           
-* `-r, --remove-header-and-footer` - Removes top lesson header and any Learn.co specific footer links before converting to HTML             
-* `-o, --only-content` - For align functionality only - updates the HTML content of a lesson without changing the name
-* `-h, --help` - Outputs examples commands and all options
-                   
+By providing a YAML file of the desired course structure, this gem can create a
+course, add in modules, and populate those modules with pages and assignments.
+The required YAML file must follow a specific structure. Using the steps below,
+this gem can create the necessary YAML markup from existing Canvas courses.
+
+#### Retrieve Canvas Course Information as YAML Markdown <a name="query"></a>
+
+To create YAML markup of an existing Canvas course, use the following:
+
+```sh
+github-to-canvas --query <CANVAS_COURSE_ID>
+```
+
+This command will use the Canvas API to read a course's information, including
+its module and lesson structure and lesson URLs. YAML markup will be produced as a result. This
+output can be directly saved to a file, which will make the next step a little
+easier:
+
+```sh
+github-to-canvas --query <CANVAS_COURSE_ID> > query_results.yml
+```
+
+The output will look similar to this:
+
+```yml
+---
+:name: The Course's Title
+:id: 111
+:modules:
+- :id: 2020
+  :name: First Module's Name
+  :lessons:
+  - id: slugified-page-name
+    title: This is the Title Of The First Lesson, a Page
+    indent: 0
+    type: Page
+    html_url: https://learning.flatironschool.com/courses/111/modules/items/27001
+    page_url: slugified-page-name
+    url: https://learning.flatironschool.com/api/v1/courses/111/pages/slugified-page-name
+    published: false
+    repository: ''
+  - id: 333
+    title: This is the Title Of The Second Lesson, an Assignment
+    indent: 1
+    type: Assignment
+    html_url: https://learning.flatironschool.com/courses/111/modules/items/27002
+    page_url: ''
+    url: https://learning.flatironschool.com/api/v1/courses/111/assignments/333
+    published: false
+    repository: ''
+- :id: 2021
+  :name: Second Module's Name
+  :lessons:
+```
+
+The output YAML will not include associated GitHub repository information.
+
+#### Map GitHub Repositories to a Canvas Course YAML file <a name="map"></a>
+
+To associate repositories to an existing course YAML, the following command can be used:
+
+```sh
+github-to-canvas --map <YAML_FILE>
+```
+
+For this feature to work, all Canvas lessons must have the GitHub HTML header
+that is added using `-l`. This header contains the necessary repo information.
+
+Save the output YAML to another file if you want to use it for later course
+creation. For example:
+
+```sh
+github-to-canvas --map query_results.yml > course_structure.yml
+```
+
+The resulting YAML file will contain course information, the module and lesson
+structure, and each lesson's associated GitHub repository.
+
+#### Create New Canvas Course from a YAML file <a name="buildcourse"></a>
+
+To create a Canvas course with this gem, you will need a correctly structured
+YAML file with the course info, modules, lessons and associated lesson
+repositories. In addition, all lessons must have a type (`Page` or
+`Assignment`), a repository, and a title. Other lesson options are currently
+ignored. 
+
+With the necessary data configured, use the `--build-course` option and pass in
+the course's YAML file:
+
+```sh
+github-to-canvas --build-course course_structure.yml -lr --forkable
+```
+
+This command will cause the following to happen:
+
+- Create a Canvas course using the top level `:name:` in the YAML data
+- Create the first module
+- Create the first lesson using the repository, title and type, as well as additional command options
+- Add the newly created lesson to the current module
+- Create the second lesson and add it to the module...
+- Repeate process until all modules and lessons are created
+
+#### Update Lessons in an Existing Course from a YAML file <a name="updatecourse"></a>
+
+The GitHub to Canvas gem can be used to update all lessons in a course
+with a single command. To do this, you will need an up-to-date course YAML file with repositories
+mapped to each lesson.
+
+```sh
+github-to-canvas --query <COURSE_ID> > query_results.yml
+github-to-canvas --map query_results.yml > your_new_course.yml
+```
+
+Use the resulting file (in this example, `your_new_course.yml`) to update all lessons in
+a course based on their GitHub repo:
+
+```sh
+github-to-canvas --update-course YAML_FILE -lr --forkable
+```
+
+The gem will iterate over the data and update every lesson according to the
+associated repository.
+
+## Known Issues
+
+### HTML Code Snippets Do Not Render
+
+The Canvas API renders all HTML it receives. If your repository's markdown
+includes HTML that is not meant to be rendered, the content will be rendered as
+part of the page's HTML, resulting in unusual display errors in Canvas. Examples of
+this would be lessons on HTML or JavaScript that include HTML code snippets.
+
+To prevent HTML from being rendered, include the `--contains-html` option when
+running the GitHub to Canvas gem. This replaces `<` and `>` characters with HTML
+charset values wrapped in `span` elements. This will stop Canvas from rendering
+the HTML.
+
+If your markdown contains a mix of HTML that should and should not be rendered,
+you will need to either replace HTML that you want to be rendered with markdown
+syntax equivalents. For example, HTML you want to display as code and an `<img>`
+element you want to render as the image itself, replace the `<img>` tag with
+markdown syntax (`![alt text](url)`).
+
+The one exception is the `<iframe>` element. There is no way to easily embed
+videos in GitHub markdown without HTML, so this tag will always be allowed to
+render in Canvas, whether or not you use `--contains-html`.
+
+If you have HTML related rendering issues in Canvas that can't be fixed with
+`--contains-html`:
+
+- Go to the Canvas WYSIWYG editor for the afflicted lesson.
+- Click the HTML editor option (`</>` button in the lower right) to switch to
+  HTML.
+- Read the GitHub repo as HTML:
+
+  ```sh
+  github-to-canvas --read-from-github URL
+  ```
+
+- Copy the output HTML and paste it in to the Canvas editor. This should clear up
+some larger page rendering issues, but may not fix all code snippets issues. 
+- Switch back to the regular Canvas WYSIWYG editor
+- Open a second tab to the GitHub repo you're converting from.
+- Copy any HTML code snippets from GitHub and paste them into the Canvas editor
+  where they should be displayed.
+
+The Canvas editor will treat the pasted HTML content as code and will
+automatically replace some characters, escaping the code from the
+normal rendering process.
+
+Note that realigning after fixing this content may overwrite fixes.
+
+### Multi-Line Code Snippets Render as a Single Line
+
+The Github to Canvas gem uses an existing gem, Redcarpet, to handle the Markdown to HTML
+conversion. This gem does not currently support multi-line codeblocks. This
+functionality has been added in the Github to Canvas gem through monkeypatching.
+This error should be resolved, but if you encounter code snippet rendering
+issues, please open a new issue with a markdown example to replicate the error.
+
+### Markdown Formatting Issues Cause Errors in Canvas
+
+An empty line should separate individual markdown headers, paragraphs and code snippets
+in the markdown. Without these empty lines, the contents will be interpretted as one
+continuous paragraph and ignore formatting.
+
+### New Repos That Use a `main` Branch
+
+If you are using a new GitHub repository that uses a `main` branch, you may not be able to
+create or update from a remote repository. You can still create and update from a local
+repository by using the `--branch` (`-b`) option and specifying `main` as the branch.
+
+A fix is planned for this issue, but not implemented.
+
+## Overview of GitHub to Canvas workflows
+
+Using this gem, you can maintain Canvas courses in multiple ways - either by
+creating and updating individual Canvas lessons or through the YAML file process.
+
+At Flatiron School, we use the Canvas blueprint course feature. We use the
+github-to-canvas gem to update the blueprint copy of lessons. These updates will appear
+in future copies of the course or when synced down associated courses in Canvas.
+
+![github-to-canvas workflow chart](./images/github-to-canvas_workflow.png)
+
+This can either be done with individual lessons or using a YAML files to keep
+entire courses updated.
+
+If you are using github-to-canvas in this way, changes should always be made on
+the repository, not on Canvas. Any changes made only on Canvas will get
+overwritten if the lesson is updated using the gem.
+
+## Common Options
+
+- `--create-lesson`, `-c`: Requires a Canvas course ID. Creates a new Canvas
+  lesson, converting the local repository's README.md to HTML. Adds `.canvas`
+  file to remote repository
+- `--align`, `-a`: Updates a canvas lesson based on the local repository's
+  README.md. If no other options are used, `--align` will look for a `.canvas`
+  file to know what to lesson to update
+- `--course`: Provide a specific course ID. When used with `--id`, this can
+  override the default behavior for `--align`, allowing you to update any
+  existing lesson and ignore the `.canvas` file if present.
+- `--id`: Provide a specific lesson ID. This can be found in the URL of the
+  specific lesson. For Pages, used the slugified lesson title.
+- `--type`: Sets the type of Canvas lesson to be created (page, assignment or
+  discussion). If no type, type decided based on repository structure.
+- `--name`: Can be used to override default naming behavior. By default, Canvas
+  lesson names are determined by the first top-level (`#`) header in the
+  repository's markdown file.
+- `--fis-links`, `-l`: Adds additional Flatiron School HTML header after
+  markdown conversion, including links back to the GitHub repo and it's issue
+  form.
+- `--forkable`: Adds a **Fork** button to the Flatiron School HTML header. For
+  use with custom Canvas JS to enable Canvas assignment forking workflow for
+  Flatiron School students.
+- `--remove-header-and-footer`, `-r`: Removes top lesson header and any Flatiron
+  School specific footer links before converting to HTML. Removing top lesson
+  header prevents duplicate titles while viewing in Canvas.
+- `--create-from-github`: Requires a GitHub repository URL. Also requires
+  `--course` and `--type`. Creates a Canvas lesson, reading from the remote repo
+  instead of a local repository. Repository must be public.
+- `--align-from-github`: Requires a GitHub repo URL, `--course`, `--id`, and
+  `--type`. Updates a Canvas lesson from a remote repository.
+- `--read-from-github`: Requires a GitHub repo URL. Reads a remote repository
+  and converts its contents to HTML, but does not push to Canvas.
+- `--branch`, `-b`: Can be used when creating or aligning with a local repo to
+  specify which branch to use. Use this if you have a new repository that uses a
+  `main` branch instead of `master`.
+- `--save-to-github`, `-s`: If you are are creating or aligning content locally,
+  the `.canvas` file is not automatically committed to the repo. This option will
+  attempt to commit and push the `.canvas` file to the remote repository.
+
+Run `github-to-canvas --help` for additional options not listed in this Readme.
 
 ## Examples of Valid Images This Gem Can Convert
 
+This gem can convert both standard inline markdown and HTML images. 
+
 Inline Markdown:
 
 ![example in-line image](https://curriculum-content.s3.amazonaws.com/fewpjs/fewpjs-fetch-lab/Image_25_AsynchronousJavaScript.png)

data/bin/github-to-canvas

@@ -1,4 +1,7 @@
 #!/usr/bin/env ruby
+
+require 'byebug'
+
 require 'optparse'
 require 'github-to-canvas'
 
@@ -75,12 +78,15 @@ OptionParser.new do |opts|
           end
   opts.on("-tTYPE", "--type TYPE", 
           "Sets the type Canvas lesson to be created (page or assignment). If no type, type decided based on repository structure") do |type| 
-            if type == 'page' || type == 'assignment' || type == 'discussion'
-              options[:type] = type
-            else
-              puts "Invalid type. Defaulting to page"
-              options[:type] = "page"
-            end
+            # byebug
+            options[:type] = type.downcase
+            abort if type == 'quiz' || type == 'discussion'
+            # if type == 'page' || type == 'assignment' || type == 'discussion' || type == 'quiz' || type == 'Page' || type == 'Assignment' || type == 'Discussion' || type == 'Quiz'
+              
+            # else
+            #   puts "Invalid type. Defaulting to page"
+            #   options[:type] = "page"
+            # end
           end
   opts.on("-fFILE", "--file FILE", 
           "Looks for and uses a markdown file in the currentt folder as source for conversion. Default file is README.md. Skips writing .canvas to repository") do |file| 
@@ -102,6 +108,10 @@ OptionParser.new do |opts|
           "Adds additional Flatiron School HTML after markdown conversion") do |f|
             options[:fis] = true
           end
+  opts.on("--aaq", 
+          "Adds AAQ flag to HTML header appended with --fis-links") do |aaq|
+            options[:aaq] = aaq
+          end
   opts.on("--forkable",
           "Used with --fis-links, adds fork button to HTML header injected into Canvas lesson") do |remote|
             options[:forkable] = true
@@ -130,6 +140,14 @@ OptionParser.new do |opts|
           "REQUIRES -f or --file Associates canvas lessons with repositories. Use query to create required YAML file") do |file|
             options[:map] = file
           end
+  opts.on("--urls-only", 
+          "Use with --map. Outputs repo URLs instead of YAML") do |urls|
+            options[:urls_only] = urls
+          end
+#  opts.on("--csv COURSE", 
+#          "Returns a course's lesson struction as CSV") do |course|
+#            options[:csv] = course
+#          end
   opts.on("--read-from-canvas CANVAS_URL",
           "Retrieves an existing Canvas lesson using the provided URL") do |url|
             options[:read_from_canvas] = url
@@ -142,10 +160,10 @@ OptionParser.new do |opts|
           "Creates a new Canvas lesson from a remote GitHub Readme. --course and --type options required") do |url|
             options[:create_from_github] = url
           end
-  opts.on("--create-quiz-from-github GITHUB_URL",
-          "Creates a new Canvas quiz from a remote GitHub YAML file. --course option required") do |url|
-            options[:quiz_from_github] = url
-          end
+  # opts.on("--create-quiz-from-github GITHUB_URL",
+  #         "Creates a new Canvas quiz from a remote GitHub YAML file. --course option required") do |url|
+  #           options[:quiz_from_github] = url
+  #         end
   opts.on("--align-from-github GITHUB_URL",
           "Aligns an existing Canvas lesson using a remote GitHub Readme. --course, --id, and --type options required") do |url|
             options[:align_from_github] = url
@@ -162,6 +180,14 @@ OptionParser.new do |opts|
           "Updates all lessons in a course using remote repos in provided YAML file") do |file|
             options[:update_course_lessons] = file
           end
+  opts.on("--clone-from-yaml YAML_FILE", 
+          "Iterates over provided course YAML file and clones repos locally") do |file|
+            options[:clone_from_yaml] = file
+          end
+  opts.on("--contains-html", 
+          "Escapes all HTML included in source markdown by replacing '<' and '>' with HTML charset values") do |html|
+            options[:contains_html] = html
+          end
   
 end.parse!
 
@@ -171,12 +197,19 @@ if options[:version]
 end
 
 if options[:read_from_canvas]
-  GithubToCanvas.new(mode: 'canvas_read', filepath: options[:read_from_canvas])
+  GithubToCanvas.new(mode: 'canvas_read', 
+                    filepath: options[:read_from_canvas])
   abort
 end
 
 if options[:read_from_github]
-  GithubToCanvas.new(mode: 'github_read', filepath: options[:read_from_github])
+  GithubToCanvas.new(mode: 'github_read', 
+                    filepath: options[:read_from_github],
+                    remove_header_and_footer: !!options[:remove_header_and_footer],
+                    forkable: !!options[:forkable],
+                    fis_links: !!options[:fis],
+                    aaq: !!options[:aaq],
+                    contains_html: options[:contains_html])
   abort
 end
 
@@ -186,9 +219,12 @@ if options[:create_from_github]
                       filepath: options[:create_from_github], 
                       course_id: options[:course_id], 
                       type: options[:type],
+                      name: options[:name],
                       remove_header_and_footer: !!options[:remove_header_and_footer],
                       forkable: !!options[:forkable],
-                      fis_links: !!options[:fis])
+                      fis_links: !!options[:fis],
+                      aaq: !!options[:aaq],
+                      contains_html: options[:contains_html])
   else
     puts 'Canvas course ID and lesson type required. Example: github-to-canvas --create-from-github URL --course ID --type TYPE'
   end
@@ -202,9 +238,12 @@ if options[:align_from_github]
                       course_id: options[:course_id], 
                       type: options[:type], 
                       id: options[:id],
+                      name: options[:name],
                       remove_header_and_footer: !!options[:remove_header_and_footer],
                       forkable: !!options[:forkable],
-                      fis_links: !!options[:fis])
+                      fis_links: !!options[:fis],
+                      aaq: !!options[:aaq],
+                      contains_html: options[:contains_html])
   else
     puts 'Canvas course ID, lesson ID, and type required. Example: github-to-canvas --create-from-github URL --course COURSE_ID --id LESSON_ID --type TYPE'
   end
@@ -217,7 +256,14 @@ if options[:query]
 end
 
 if options[:map]
-  GithubToCanvas.new(mode: 'map', file_to_convert: options[:map])
+  GithubToCanvas.new(mode: 'map', 
+                      file_to_convert: options[:map],
+                      urls_only: !!options[:urls_only])
+  abort
+end
+
+if options[:csv]
+  GithubToCanvas.new(mode: 'csv', file_to_convert: options[:csv])
   abort
 end
 
@@ -226,6 +272,7 @@ if options[:build_course]
                     file_to_convert: options[:build_course],
                     fis_links: !!options[:fis], 
                     remove_header_and_footer: !!options[:remove_header_and_footer],
+                    aaq: !!options[:aaq],
                     forkable: !!options[:forkable])
   abort
 end
@@ -237,7 +284,9 @@ if options[:add_to_course]
                       file_to_convert: options[:add_to_course],
                       fis_links: !!options[:fis], 
                       remove_header_and_footer: !!options[:remove_header_and_footer],
-                      forkable: !!options[:forkable])
+                      forkable: !!options[:forkable],
+                      aaq: !!options[:aaq],
+                      contains_html: options[:contains_html])
   else
     puts '--course required'
   end
@@ -249,10 +298,17 @@ if options[:update_course_lessons]
                     file_to_convert: options[:update_course_lessons],
                     fis_links: !!options[:fis], 
                     remove_header_and_footer: !!options[:remove_header_and_footer],
-                    forkable: !!options[:forkable])
+                    forkable: !!options[:forkable],
+                    aaq: !!options[:aaq],
+                    contains_html: options[:contains_html])
   abort
 end
 
+if options[:clone_from_yaml]
+  GithubToCanvas.new(mode: 'clone_course', 
+                    file_to_convert: options[:clone_from_yaml])
+  abort
+end
 
 if !options[:type]
   if Dir.glob("**/*/").empty?
@@ -306,7 +362,9 @@ if options[:create_lesson]
                     fis_links: !!options[:fis], 
                     remove_header_and_footer: !!options[:remove_header_and_footer],
                     only_update_content: !!options[:only_content],
-                    forkable: !!options[:forkable])
+                    forkable: !!options[:forkable],
+                    aaq: !!options[:aaq],
+                    contains_html: options[:contains_html])
 end
 
 if options[:align]
@@ -322,5 +380,7 @@ if options[:align]
                     fis_links: !!options[:fis], 
                     remove_header_and_footer: !!options[:remove_header_and_footer],
                     only_update_content: !!options[:only_content],
-                    forkable: !!options[:forkable])
+                    forkable: !!options[:forkable],
+                    aaq: !!options[:aaq],
+                    contains_html: options[:contains_html])
 end

data/lib/github-to-canvas.rb

@@ -21,12 +21,14 @@ class GithubToCanvas
     when 'query'
       CanvasInterface.get_course_info(options[:course_id], options[:id])
     when 'map'
-      CanvasInterface.map_course_info(options[:file_to_convert])
+      CanvasInterface.map_course_info(options)
+    when 'csv'
+      CanvasInterface.csv(options[:file_to_convert])
     when 'canvas_read'
       puts CanvasInterface.read_lesson(options[:filepath])
     when 'github_read'
-      markdown = GithubInterface.read_remote(options[:filepath])
-      puts RepositoryConverter.convert_to_html(markdown)
+      html = RepositoryConverter.remote_file_conversion(options)
+      puts RepositoryConverter.adjust_converted_html(options, html)
     when 'create' # used with a local repo
       html = RepositoryConverter.local_file_conversion(options)
       name = RepositoryInterface.get_name(options[:filepath], html)
@@ -36,13 +38,13 @@ class GithubToCanvas
       puts "Canvas lesson created. Lesson available at #{response['html_url']}"
     when 'align' # used with a local repo
       html = RepositoryConverter.local_file_conversion(options)
-      name = RepositoryInterface.get_name(options[:filepath], html)
+      name = options[:name] ? options[:name] : RepositoryInterface.get_name(options[:filepath], html)
       html = RepositoryConverter.adjust_converted_html(options, html)
       CanvasInterface.update_all_related_lessons(options, name, html)
       
     when 'github_create'
       html = RepositoryConverter.remote_file_conversion(options)
-      name = RepositoryInterface.get_name(options[:filepath], html)
+      name = options[:name] ? options[:name] : RepositoryInterface.get_name(options[:filepath], html)
       html = RepositoryConverter.adjust_converted_html(options, html)
       
       response = CanvasInterface.create_lesson(options, name, html)
@@ -50,7 +52,7 @@ class GithubToCanvas
       puts "Canvas lesson created. Lesson available at #{response['html_url']}"
     when 'github_align'
       html = RepositoryConverter.remote_file_conversion(options)
-      name = RepositoryInterface.get_name(options[:filepath], html)
+      name = options[:name] ? options[:name] : RepositoryInterface.get_name(options[:filepath], html)
       html = RepositoryConverter.adjust_converted_html(options, html)
       response = CanvasInterface.update_existing_lesson(options, name, html)
       puts "Canvas lesson updated. Lesson available at #{response['html_url']}"
@@ -70,10 +72,8 @@ class GithubToCanvas
           options[:course_id] = created_course_info["id"]
           options[:filepath] = lesson["repository"]
           
-          
           html = RepositoryConverter.remote_file_conversion(options)
           # Add each lesson to it's module
-          
           html = RepositoryConverter.adjust_converted_html(options, html)
           created_lesson_info = CanvasInterface.create_lesson(options, lesson["title"], html)
           lesson = lesson.merge(created_lesson_info)
@@ -134,6 +134,24 @@ class GithubToCanvas
           sleep(1)
         }
       }
+    when 'clone_course'
+      course_yaml = YAML.load(File.read(options[:file_to_convert]))
+      new_dir = "#{course_yaml[:name].downcase.gsub(' ','-')}"
+      cmd = "mkdir #{new_dir}"
+      `#{cmd}`
+      course_yaml[:modules].each { |module_info|
+      puts "Cloning #{module_info[:name]}"
+        module_info[:lessons].each { |lesson|
+        if lesson["repository"] == ""
+          puts "No repository found for #{lesson['title']}"
+          next
+        else
+          cmd = "git clone #{lesson['repository']}"
+          puts cmd
+          GithubInterface.cd_into_and(new_dir, cmd)
+        end
+        }
+      }
     else
       puts VERSION
     end

data/lib/github-to-canvas/canvas_interface.rb

@@ -47,10 +47,6 @@ class CanvasInterface
     JSON.parse(response.body)
   end
 
-  def self.create_quiz(options, quiz_data)
-    
-  end
-
   def self.add_to_module(course_id, module_info, lesson_info)
     # POST /api/v1/courses/:course_id/modules/:module_id/items
     url = "#{ENV['CANVAS_API_PATH']}/courses/#{course_id}/modules/#{module_info["id"]}/items"
@@ -60,6 +56,7 @@ class CanvasInterface
       'module_item[title]' => lesson_info["title"],
       'module_item[type]' => lesson_info["type"],
       'module_item[indent]' => 0,
+      'module_item[page_url]' => lesson_info["id"],
       'module_item[completion_requirement][type]' => 'must_view'
     }
     elsif lesson_info["type"] == "Quiz"
@@ -70,19 +67,17 @@ class CanvasInterface
         'module_item[title]' => lesson_info["title"],
         'module_item[type]' => lesson_info["type"],
         'module_item[indent]' => 1,
+        'module_item[content_id]' =>  lesson_info["id"],
         'module_item[completion_requirement][type]' => 'must_submit'
       }
     end
     begin
-      byebug
       response = RestClient.post(url, payload, self.headers)
     rescue
-      puts "Something went wrong while add lesson #{lesson_info["id"]} to module #{module_info["id"]} in course #{course_id}" if lesson_info["type"] == "Assignment"
-      puts "Something went wrong while add lesson #{lesson_info["page_url"]} to module #{module_info["id"]} in course #{course_id}" if lesson_info["type"] == "Page"
+      puts "Something went wrong while adding lesson #{lesson_info["id"]} to module #{module_info["id"]} in course #{course_id}" if lesson_info["type"] == "Assignment"
+      puts "Something went wrong while adding lesson #{lesson_info["page_url"]} to module #{module_info["id"]} in course #{course_id}" if lesson_info["type"] == "Page"
       abort
     end
-
-    
     response
     
   end
@@ -97,7 +92,7 @@ class CanvasInterface
     
     begin
       headers = self.headers
-      if options[:type] == 'page'
+      if options[:type] == 'page' || options[:type] == 'Page'
         response = RestClient.get(url, headers)
         lesson_info = JSON.parse(response)
         url = url.sub(/[^\/]+$/, lesson_info["page_id"].to_s)
@@ -172,6 +167,7 @@ class CanvasInterface
     [info, type]
   end
 
+
   def self.get_course_info(course, id)
     if id
       lesson_data = self.get_lesson_info(course, id)
@@ -197,6 +193,7 @@ class CanvasInterface
         while !!index
           url = "#{ENV['CANVAS_API_PATH']}/courses/#{course}/modules?page=#{index}&per_page=20"
           index += 1
+          
           response = RestClient.get(url, self.headers)
           modules = JSON.parse(response.body)
           
@@ -204,7 +201,7 @@ class CanvasInterface
             course_info[:modules] = course_info[:modules] + modules
           else
             index = nil
-          end 
+          end
         end
         
         course_info[:modules] = course_info[:modules].map do |mod|
@@ -217,11 +214,12 @@ class CanvasInterface
           while !!index
             url = "#{ENV['CANVAS_API_PATH']}/courses/#{course}/modules/#{mod['id']}/items?page=#{index}&per_page=20"
             index += 1
-            response = RestClient.get(url, headers={
-              "Authorization" => "Bearer #{ENV['CANVAS_API_KEY']}"
-            })
+            response = RestClient.get(url, self.headers)
             lessons = JSON.parse(response.body)
             lessons = lessons.map do |lesson|
+              if lesson["type"] == "ExternalUrl"
+                next
+              end
               lesson = lesson.slice("id","title","name","indent","type","html_url","page_url","url","completion_requirement", "published")
               lesson["repository"] = ""
               lesson['id'] = lesson['url'].gsub(/^(.*[\\\/])/,'')
@@ -236,17 +234,18 @@ class CanvasInterface
           end
           new_mod
         end
-
+        
         puts course_info.to_yaml
       
     rescue
+      byebug
       puts "Something went wrong while getting info about course #{course}"
       abort
     end
   end
 
-  def self.map_course_info(file_to_convert)
-    course_info = YAML.load(File.read("#{Dir.pwd}/#{file_to_convert}"))
+  def self.map_course_info(options)
+    course_info = YAML.load(File.read("#{Dir.pwd}/#{options[:file_to_convert]}"))
     course_info[:modules] = course_info[:modules].map do |mod|
       mod[:lessons] = mod[:lessons].map do |lesson|
 
@@ -282,6 +281,7 @@ class CanvasInterface
             lesson["repository"] = repo
           else
             lesson["repository"] = "https://github.com/learn-co-curriculum/" + repo
+            puts lesson["repository"] if options[:urls_only]
           end
         end
         sleep(1)
@@ -289,11 +289,55 @@ class CanvasInterface
       end
       mod
     end
-    puts course_info.to_yaml
+    puts course_info.to_yaml if !options[:urls_only]
   end
 
-  def self.submit_to_canvas(course_id, type, name, readme)
-    
+  def self.csv(file_to_convert)
+    course_info = YAML.load(File.read("#{Dir.pwd}/#{file_to_convert}"))
+    course_info[:modules] = course_info[:modules].map do |mod|
+      mod[:lessons] = mod[:lessons].map do |lesson|
+
+        url = lesson["url"]
+        response = RestClient.get(url, headers={
+          "Authorization" => "Bearer #{ENV['CANVAS_API_KEY']}"
+        })
+        begin
+          lesson_data = JSON.parse(response)
+          contents = lesson_data["body"] if lesson["type"] == "Page"
+          contents = lesson_data["message"] if lesson["type"] == "Discussion"
+          contents = lesson_data["description"] if lesson["type"] == "Assignment" || lesson["type"] == "Quiz"
+          if contents.nil?
+            repo = ""
+          else
+            if contents[/data-repo=\"(.*?)"/]
+              repo = contents[/data-repo=\"(.*?)"/]
+              repo = repo.slice(11..-2)
+            elsif contents[/class=\"fis-git-link\" href=\"(.*?)"/]
+              repo = contents[/class=\"fis-git-link\" href=\"(.*?)"/]
+              repo = repo.slice(27..-2)
+            else
+              repo = ""
+            end
+          end
+        rescue
+          puts 'Error while mapping course info.'
+          abort
+        end
+        
+        if repo != nil && repo != ""
+          if repo.include?('https://github.com/learn-co-curriculum/')
+            lesson["repository"] = repo
+          else
+            lesson["repository"] = "https://github.com/learn-co-curriculum/" + repo
+          end
+        end
+        sleep(1)
+        lesson
+      end
+      mod
+    end
+    byebug
+    puts course_info.to_yaml
   end
 
   

data/lib/github-to-canvas/github_interface.rb

@@ -1,4 +1,3 @@
-require 'byebug'
 require 'json'
 require 'rest-client'
 class GithubInterface
@@ -62,12 +61,13 @@ class GithubInterface
     rescue
       begin
         response = RestClient.get(url_fallback)
+        return response.body
       rescue
         puts 'Error reading ' + url
       end
     end
     
-    response.body
+    
   end
 
   def self.save_to_github(filepath, branch)

data/lib/github-to-canvas/repository_converter.rb

@@ -1,5 +1,4 @@
 require 'redcarpet'
-require 'byebug'
 
 class CustomRender < Redcarpet::Render::HTML
   def block_code(code, lang)
@@ -18,7 +17,7 @@ class RepositoryConverter
     GithubInterface.get_updated_repo(options[:filepath], options[:branch])
     markdown = RepositoryInterface.read_local_file(options[:filepath], options[:file_to_convert])
     raw_remote_url = self.set_raw_image_remote_url(options[:filepath])
-    
+    markdown = self.escape_existing_html(markdown) if options[:contains_html]
     markdown = self.fix_local_images(options, markdown, raw_remote_url)
     html = self.convert_to_html(markdown)
     # self.fix_local_html_links(options, html, options[:filepath])
@@ -27,6 +26,11 @@ class RepositoryConverter
   def self.remote_file_conversion(options)
     markdown = GithubInterface.read_remote(options[:filepath])
     raw_remote_url = self.set_raw_image_remote_url(options[:filepath])
+    if options[:contains_html]
+      puts "Contains HTML"
+      puts options[:contains_html]
+      markdown = self.escape_existing_html(markdown)
+    end
     markdown = self.fix_local_images(options, markdown, raw_remote_url)
     html = self.convert_to_html(markdown)
     # self.fix_local_html_links(options, html, options[:filepath])
@@ -47,9 +51,26 @@ class RepositoryConverter
     if options[:fis_links]
       html = self.add_fis_links(options, html)
     end
+
+    if options[:contains_html]
+      html = self.fix_escaped_inline_html_code(html)
+    end
+
     html
   end
 
+  def self.fix_escaped_inline_html_code(html)
+    html = html.gsub("<code>&amp;lt;", "<code>&lt;")
+    html = html.gsub("&amp;gt;</code>", "&gt;</code>")
+  end
+
+
+  def self.escape_existing_html(markdown)
+    markdown = markdown.gsub(/<\/(?!iframe)/, "&lt;/")
+    markdown = markdown.gsub(/<(?!iframe)/, "&lt;")
+    markdown = markdown.gsub(/(?<!iframe)>/, "&gt;")
+  end
+
   def self.remove_header_and_footer(html)
     new_html = self.remove_html_header(html)
     new_html = self.remove_footer(new_html)
@@ -57,7 +78,7 @@ class RepositoryConverter
   end
 
   def self.remove_header(readme)
-    readme.gsub!(/^# .+?\n\n/,"")
+    readme = readme.gsub(/^# .+?\n\n/,"")
     readme.gsub(/^# .+?\n/,"")
   end
 
@@ -115,7 +136,7 @@ class RepositoryConverter
   end
 
   def self.adjust_local_markdown_images(readme, raw_remote_url, branch)
-    readme.gsub!(/\!\[.+\]\(.+\)/) {|image_markdown|
+    readme.gsub(/\!\[.+\]\(.+\)/) {|image_markdown|
       if !image_markdown.match?('amazonaws.com') && !image_markdown.match?('https://') && !image_markdown.match?('http://') && !image_markdown.match?('youtube')
         image_markdown.gsub!(/\(.+\)/) { |path|
           path.delete_prefix!("(")
@@ -128,11 +149,12 @@ class RepositoryConverter
   end
 
   def self.adjust_local_html_images(readme, raw_remote_url, branch)
-    readme.gsub!(/src=(\'|\")[\s\S]*?(\'|\")/) { |image_source|
-      if !image_source.match?('amazonaws.com') && !image_source.match?('https://') && !image_source.match?('http://') && !image_source.match?('youtube')
-        image_source.gsub!(/(\'|\")/, "")
-        image_source.gsub!(/src=/, '')
-        image_source.strip!
+    readme.gsub(/src=(\'|\")[\s\S]*?(\'|\")/) { |image_source|
+      
+      if !image_source.match?('amazonaws.com') && !image_source.match?('https://') && !image_source.match?('http://') && !image_source.match?('youtube') && !image_source.match(/src=(\'|\")(?=<%)/)
+        image_source = image_source.gsub(/(\'|\")/, "")
+        image_source = image_source.gsub(/src=/, '')
+        image_source = image_source.strip
         'src="' + raw_remote_url + '/' + branch + '/' + image_source + '"'
       else
         image_source
@@ -164,7 +186,7 @@ class RepositoryConverter
     repo_info = self.get_repo_info(options[:filepath])
     html = html.sub(/<div id="git-data-element.*<header class="fis-header.*<\/header>/,'') # remove existing fis header
     header = self.create_github_link_header(repo_info[:repo_path], options[:forkable])
-    data_element = self.create_data_element(repo_info[:repo_org], repo_info[:repo_name])
+    data_element = self.create_data_element(repo_info[:repo_org], repo_info[:repo_name], options[:aaq])
     data_element + header + html
   end
 
@@ -184,8 +206,12 @@ class RepositoryConverter
     end
   end
 
-  def self.create_data_element(repo_org, repo_name)
-    "<div id='git-data-element' data-org='#{repo_org}' data-repo='#{repo_name}'></div>"
+  def self.create_data_element(repo_org, repo_name, aaq)
+    if (aaq)
+      "<div id='git-data-element' data-aaq='enabled' data-org='#{repo_org}' data-repo='#{repo_name}'></div>"
+    else
+      "<div id='git-data-element' data-org='#{repo_org}' data-repo='#{repo_name}'></div>"
+    end
   end
 
   

data/lib/github-to-canvas/version.rb

@@ -1,3 +1,3 @@
 class GithubToCanvas
-  VERSION = "0.1.0"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: github-to-canvas
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.5
 platform: ruby
 authors:
 - maxwellbenton
@@ -108,7 +108,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.4
+rubygems_version: 3.0.8
 signing_key:
 specification_version: 4
 summary: github-to-canvas is a tool for migrating and aligning GitHub content with