tractive 1.0.0

data/README.adoc

@@ -0,0 +1,742 @@
+= Tractive: migrating from Trac to GitHub
+
+== Purpose
+
+Tractive is a tool that helps you migrate Trac instances to GitHub.
+
+Specifically, it converts http://trac.edgewall.org[Trac] tickets into
+https://github.com/[GitHub] Issues by accessing the Trac database's issues and
+posts the change history of each ticket as issue comments.
+
+Tractive is written using re-tooled code based on the excellent
+https://github.com/mavam/trac-hub[trac-hub] migration tool.
+
+NOTE: This tool was created in order to perform Trac-to-GitHub migration
+for the https://www.ietf.org/[IETF].
+
+
+== Installation
+
+=== Prerequisites
+
+. Ruby
+. https://git-scm.com/book/en/v2/Getting-Started-Installing-Git[Git]
+. http://www.catb.org/~esr/reposurgeon/[`reposurgeon`], which works on Linux
+  (manual compile recommended for the latest versions) and macOS (use Homebrew)
+. `trac-admin` must be installed in order to extract Trac attachments
+
+=== Steps
+
+Install it via:
+
+[source,sh]
+----
+$ gem install tractive
+----
+
+
+== Usage
+
+=== Prerequisites
+
+. A https://github.com/[GitHub] Account as the importer
+
+
+=== Steps to migrate from Trac to GitHub
+
+. Migrate your Trac-managed SVN repositories to GitHub. (<<migrate-svn>>)
+
+. Build the SVN Revision to Git Commit RevMap. (<<build-map>>)
+
+. Bootstrap your Tractive config file. (<<bootstrap-config>>)
+
+. Configure mapping in your Tractive config file. (<<create-config>>)
+
+. Run Tractive (<<running-tractive>>)
++
+[source,sh]
+----
+$ tractive <options>
+----
+
+
+[[migrate-svn]]
+=== Migrating SVN to GitHub using `reposurgeon`
+
+`reposurgeon` is the leading tool today (and under active development)
+that allows conversion among multiple version control systems.
+
+We strongly recommend you to use the excellent `reposurgeon` for this task.
+
+In the case of Trac to GitHub, we are mostly looking at SVN to Git.
+
+NOTE: For further details, please see the
+http://www.catb.org/~esr/reposurgeon/repository-editing.html[reposurgeon guide].
+
+
+Typically these are the steps to take:
+
+. In an empty directory, run `repotool --initialize {name-of-repo}`.
+  Enter that the source VCS is `svn`, the destination is `git`.
+  The directory will then be populated with a number of files, including:
+
+** `Makefile` includes multiple useful tasks, requires user configuration.
+
+** `{name-of-repo}.lift`, a file to store `reposurgeon` conversion configuration.
+
+** `{name-of-repo}.map`, a `reposurgeon` specific map of SVN users to GitHub users/emails)
+
+** `{name-of-repo}.opts`, to enter pre-read `reposurgeon` options.
+
+. Customize these lines of the `Makefile` with the SVN URL and reposurgeon "read
+  options":
++
+[source,make]
+----
+REMOTE_URL = https://svn.ietf.org/svn/tools/mailarch
+READ_OPTIONS = --branchify=trunk:branch/*:personal/*/*:tags/*:*
+----
+
+. Update the `{name-of-repo}.lift` with additional reposurgeon commands (see the
+  reference guide) if necessary. e.g.
++
+[source]
+----
+branch rename refs/heads/master refs/heads/main
+
+msgin --create <<EOF
+Tag-Name: git-conversion
+Tagger: IETF Bot <noreply@ietf.org> America/Los_Angeles
+
+Marks the spot where this repository was migrated from Subversion to git.
+EOF
+----
+
+. Download the SVN repository in mirror mode to the `{name-of-repo}-mirror/`
+  directory. This is necessary for faster processing and so that reposurgeon
+  won't attempt to download from the `REMOTE_URL`.
+
+. Generate the user map from the SVN repository with `make stubmap`.
+  Edit the `{name-of-repo}.map` to correct/fix it if necessary.
+
+. Run `make` to generate `{name-of-repo}.svn` and `{name-of-repo}.fo`.
+  The resulting Git repository will be at `{name-of-repo}-git/`.
+
+. The `{name-of-repo}.fo` file is a SVN Revision to timestamp mapping.
+  Tractive uses this file to build the SVN Revision to Git Commit SHA hash
+  mapping.
+
+
+[[build-map]]
+=== Building the Trac Revision to Git Commit RevMap
+
+RevMap is a mapping file that contains a one-to-many mapping
+of an SVN revision number to its corresponding Git commit SHA hash(es).
+
+NOTE: An SVN revision may map to multiple, identical Git commits; see
+`reposurgeon` documentation for more details.
+
+With the `{name-of-repo}.fo` file generated by `reposurgeon` (or any other tool
+for that matter, as long as the format is identical), we can build the SVN
+revision to Git commit SHA "RevMap".
+
+A typical `{name-of-repo}.fo` file looks like this:
+
+[source]
+----
+SVN:9	2012-10-10T23:50:08Z!rcross@amsl.com
+SVN:10	2012-10-10T23:50:56Z!rcross@amsl.com
+SVN:12	2012-10-10T23:52:56Z!rcross@amsl.com
+SVN:13	2012-10-10T23:52:56Z!rcross@amsl.com
+...
+SVN:33	2013-02-07T04:01:56Z!rcross@amsl.com
+SVN:34	2013-02-12T17:46:48Z!rcross@amsl.com
+SVN:35.2	2013-02-12T20:15:49Z!henrik@levkowetz.com
+...
+----
+
+The left column is the SVN revision, while the right column provides a timestamp
+for which you can find with Git. If you enter `{name-of-repo}-git/`, you can use
+the `git scommit` and `git scommits` commands (defined in the `reposurgeon`
+guide) to view the corresponding Git commit(s).
+
+Notice that:
+
+* typically there is a single timestamp per SVN Revision. In this case each
+  timestamp can be identified with one Git commit via `git scommit {timestamp}`.
+
+* sometimes two SVN revisions have identical timestamps. In this case you will
+  need to use `git scommits {timestamp}` to see all these commits.
+
+* sometimes one SVN revision maps to multiple Git commits. In this case you will
+  need to use `git scommits {timestamp}` to see all these commits.
+
+With this information you can now build the RevMap with:
+
+[source,sh]
+----
+tractive generate-revmap \
+  --svn-url <url of the SVN repository> \
+  --rev-timestamp-file <reposurgeon timestamp map, e.g. {name-of-repo}.fo> \
+  --git-local-repo-path <path to converted Git repository, e.g. {name-of-repo}-git> \
+  --revmap-output-file <output path of RevMap.txt>
+----
+
+The generated RevMap will be then used in a Tractive run so that the migrated
+issues and commits will replace references to SVN revisions with the
+corresponding Git commit SHA, enabling GitHub to expose those linkages in the
+user interface.
+
+.Options for `tractive generate-revmap` command
+[cols="3a,5a,2a",options="header"]
+|===
+| Option | Description | Type
+
+| `--svn-url`
+|
+(required) SVN repository URL that should be used in RevMap generation.
+The URL must start with the `http://` or `https://` prefix (not the `svn://`
+prefix).
+| String
+
+| `--svn-local-path`
+| SVN local repository path that should be used in RevMap generation. You can clone
+the svn repo locally and provide its local path if the svn repository is offline. (`--svn-url` is not
+required if this option is given).
+| String
+
+| `--rev-timestamp-file`
+|
+(required) Specify the input file that contains SVN revision and timestamps that
+should be used in RevMap generation. The format of the file must follow the
+`reposurgeon` `.fo` format.
+
+| String
+
+| `--git-local-repo-path`
+|
+(required) Local Git repository path that should be used in RevMap generation.
+The path must be a Git repository with a `.git` folder within.
+| String
+
+| `--revmap-output-file`
+| (required) Output file path to save the generated RevMap.
+| String
+
+|===
+
+
+[[bootstrap-config]]
+=== Boostrapping the Tractive configuration file
+
+==== General
+
+Tractive uses a YAML configuration file that contains configuration to export
+data from Trac and then import to GitHub.
+
+==== Setting up the configuration file
+
+Copy the link:config.example.yaml[example YAML configuration] and adapt it
+as necessary:
+
+[source,sh]
+----
+cp config.example.yaml config.yaml
+vi config.yaml
+----
+
+Then, at a minimum, you need to setup the following sections for
+bootstrapping (the next step) to work:
+
+. Trac configuration (<<config-trac>>)
+
+. GitHub configuration (<<config-github>>)
+
+
+[[gather-info]]
+==== Bootstrapping the configuration file with information
+
+The Tractive configuration file is used to perform the actual migration actions,
+and therefore depends heavily on the contents of the content to be migrated.
+
+Tractive can read certain information from the Trac database to allow for
+easier mapping of users, labels and milestones.
+
+The following command provides that information.
+
+[source,sh]
+----
+tractive -i
+----
+
+
+[[create-config]]
+=== Working with the Tractive configuration file, and mapping
+
+The configuration file contains the following sections.
+
+[[config-trac]]
+==== Trac configuration
+
+`trac:`:: (mandatory) Trac (data source) configuration options.
+
+`database:`::: Database access URL. The database URL follows the
+http://sequel.jeremyevans.net/rdoc/classes/Sequel.html#method-c-connect[Sequel scheme].
+SQLite, MySQL, Postgres endpoints supported.
+
+`ticketbaseurl:`::: URL of the Trac "tickets" interface.
+
+EXAMPLE:
+
+[source,yaml]
+----
+trac:
+  # For MySQL: mysql2://user:password@host:port/database
+  database: sqlite://db/trac.db
+  ticketbaseurl: https://example.org/trac/foobar/ticket
+----
+
+[[config-github]]
+==== GitHub configuration
+
+`github:`:: (mandatory) GitHub (migration target) configuration options.
+
+`repo:`::: Target GitHub organization and repo name as `{github-org}/{repo-name}`.
+
+`token:`::: Personal Access Token of the GitHub user used to import.
+This token can be generated under GitHub
+https://github.com/settings/tokens[Settings > Personal Access Tokens].
+e.g. 'ghp_fpsc4de1f0c46e01576810740c9242097cba4619486'.
+
+`local_repo_path:`::: Local path to the migrated Git repository.
+e.g. '/Users/user/repo-git'.
+
+`revmap_path:`::: Local path to the RevMap file generated via the
+`tractive generate-revmap` command.
+
+
+EXAMPLE:
+
+[source,yaml]
+----
+github:
+  repo: 'example-org/target-repository'
+  token: 'ghp_fpsc4de1f0c46e01576810740c9242097cba4619486'
+  local_repo_path: '/Users/user/repo-git'
+  revmap_path: ./example-revmap.txt
+----
+
+
+==== Attribute/Label mapping
+
+Since GitHub's issue tracker does not have a first-class notion of ticket
+priority, type, and version information, Tractive supports expressing these in
+the form of labels.
+
+The pattern of a mapping is like:
++
+----
+{trac-ticket-type}:
+  {trac-ticket-type-value}: {github-label-value}
+----
+
+`labels:`:: provides custom label mappings.
+
+`type:`::: Type of the Trac ticket.  e.g.
++
+[source,yaml]
+----
+  defect:      defect
+  task:        task
+  enhancement: feature request
+  cleanup:     cleanup
+----
+
+`component:`::: Component of the Trac ticket. e.g.
++
+[source,yaml]
+----
+  configuration:      conf
+  documentation:      doc
+----
+
+`resolution:`::: Resolution of the Trac ticket. e.g.
++
+[source,yaml]
+----
+  fixed:      fixed
+  invalid:    invalid
+  wontfix:    wontfix
+  duplicate:  duplicated
+  worksforme: worksforme
+  obe:        obe
+----
+
+`platform:`::: Platform related to the Trac ticket. e.g.
++
+[source,yaml]
+----
+  Linux:   Linux
+  Windows: Windows
+----
+
+`severity:`::: Severity of the Trac ticket. (also called Priority in Trac) e.g.
++
+[source,yaml]
+----
+  blocker:  '#blocker'
+  critical: '#critical'
+  major:    '#major'
+  medium:   '#medium'
+  minor:    '#minor'
+  trivial:  '#easy'
+  waiting:  '#pending'
+  n/a:
+----
+
+`priority:`::: Priority of the Trac ticket.
++
+[source,yaml]
+----
+  Low:  '@low'
+  High: '@high'
+----
+
+
+==== User mapping
+
+`users:`:: a one-to-one mapping between Trac usernames or email addresses to
+GitHub usernames for users, in the following pattern:
++
+[source,yaml]
+----
+users:
+  {Trac email or username}: {username on GitHub}
+  ...
+----
+
+EXAMPLE:
+
+[source,yaml]
+----
+users:
+  matthew@example.org: example-matt
+  valencia: example-vale
+----
+
+==== Milestone mapping
+
+`milestones:`:: mapping of milestones.
+
+`{milestone-name}:`::: ID of the milestone.
+`name:`:::: Name of the milestone.
+`due:`:::: Due date in POSIX msec.
+`completed:`:::: Date of completion in POSIX msec.
+`description:`:::: Description of the milestone
+
+EXAMPLE:
+
+[source,yaml]
+----
+milestones:
+  '2021_02':
+    name: '2021_02'
+    due: 1392595200000000
+    completed: 1415959156000000
+    description: ''
+----
+
+
+==== Attachments migration configuration
+
+`attachments:`:: specifies method of obtaining attachments from Trac.
+
+`url:`::: URL to obtain Trac attachments from
+
+`export_folder:`::: folder where the attachments will be downloaded to from Trac.
+
+`export_script:`::: output of a script that utilizes `trac-admin` to download
+all attachments from Trac.
+
+[source,yaml]
+----
+attachments:
+  url: https://abc.com/raw-attachment/ticket
+  export_folder: ./attachments
+  export_script: attachments.sh
+----
+
+By using the <<gather-info,`-i` option>>, you can easily produce a YAML file
+with labels, users, milestones etc. You can copy the output into the
+`config.yaml` file and adapt it as required.
+
+Once the command is completed you should run `attachments.sh` to export the
+attachments.
+
+
+[[running-tractive]]
+=== Running Tractive
+
+Thereafter just invoke `tractive` to read from the default `config.yaml` path:
+
+[source,sh]
+----
+tractive
+----
+
+By default, `tractive` assumes the file `config.yaml` to be in the same
+directory as the command was run.
+
+You can also specify the configuration file on the command line:
+[source,sh]
+----
+tractive -c foo.yaml
+----
+
+
+=== Filtering Trac tickets by status
+
+Add the `-o` flag to only import the tickets that are not in a `closed`
+status:
+
+[source,sh]
+----
+tractive -o
+----
+
+=== Single post mode
+
+If you want all Trac comments and changes within one Trac ticket to be compiled
+into a single issue at GitHub:
+
+[source,sh]
+----
+tractive -S
+----
+
+
+=== Resuming Trac ticket import
+
+To resume the migration at a given Trac ticket ID, use `-s`:
+
+[source,sh]
+----
+tractive -s 42
+----
+
+
+=== Matching Trac ticket IDs and GitHub issues IDs
+
+You may want to ensure that the imported Trac ticket IDs are identical to the
+GitHub Issues ID. This section applies when migrating to an existing or empty
+GitHub repository,
+
+Tractive can help you create dummy tickets (and close them) for IDs missing in
+Trac (because they were deleted). This works even if you run it multiple times.
+
+[source,sh]
+----
+tractive -M
+----
+
+NOTE: When converting your Trac setup to GitHub, it is prudent to first try the
+migration into a test repository which you can delete afterwards. If the run was
+smooth and delivers the expected results, you can re-run the migration for the
+real target repository.
+
+As the process can be interrupted, you can always specify the first ID number
+to migrate. In this case you need to provide the `-s` argument for the first ID
+not available in Github.
+
+[source,sh]
+----
+tractive -M -s 601
+----
+
+
+=== Fast issue import
+
+By default, Tractive will verify that the created issue numbers match the ticket
+IDs of the corresponding Trac ticket and error-exit if the numbers do not match.
+
+The *fast import* option allows you to disable this safe-checking behavior.
+
+In order to utilize this feature, you should also disable user interactions by
+setting **Limit to repository collaborators** under your repository
+settings. Alternatively, when migrating issues to a new repository,
+import the issues on a test-repository and rename the repository to the
+final name when the import went satisfactory.
+
+You can disable this check by using the *fast* option:
+
+[source,sh]
+----
+tractive -F
+----
+
+In effect your import will be much faster since there is no ID synchronization
+(but after the script has finished, it can still take some time until the issues
+are created on github).
+
+If you know that the ticket IDs will not match, e.g. there are existing GitHub issues
+that are not created by import, using the fast import option is obligatory.
+In this case, you must specify the ID of the first Trac ticket to be migrated
+(even if it is 1):
+
+[source,sh]
+----
+tractive -F -s 1
+----
+
+
+
+== References
+
+=== Command line options
+
+[cols="3a,5a,2a",options="header"]
+|===
+| Option | Description | Type
+
+| `-A`, `--attachment-exporter`
+| Generate an attachment exporter script according to `config.yaml`.
+| String
+
+| `-a`, `--attachment-url`
+| Add URL to Cloud Host where attachment files are available.
+| String
+
+| `-c`
+| Set the configuration file. Default value: `tractive.config.yaml`.
+| String
+
+| `-d`, `--dry-run`
+| Write exported data to a local file instead of pushing it to Github.
+| Boolean
+
+| `-e`, `--export-attachments`
+| Export attachments from the database according to `config.yaml`.
+| String
+
+| `-F`, `--fast-import`
+| Import without safety-checking GitHub issue numbers.
+| Boolean
+
+| `-f`, `--filter`
+|
+Filter Trac tickets that you want to import.
+
+The following options are allowed (at least one necessary):
+
+* `column-name`
+* `operator`
+* `column-value`
+
+| Boolean
+
+| `--column-name`
+| Name of the column to filter.
+| String
+
+| `--operator`
+| Operator for filter. Example of operators include `LIKE` and `=`.
+| String
+
+| `--column-value`
+| Value of the column to filter.
+| String
+
+| `-h`, `help`
+| Display the Tractive help message, or you can provide a command to know more
+  about a single command via `tractive help {command}`.
+| N/A
+
+| `-I <path>`, `--import-from-file=<path>`
+| Import issues from a JSON file specified at `<path>`.
+| String
+
+| `-i`, `--info`
+| Reports existing labels and users in the database.
+| Boolean
+
+| `-L`, `--log-file`
+| Name of the log file to output logs.
+| String
+
+| `-M`, `--mockup`
+| Create mocked closed issues on Github for deleted tickets on Trac.
+| Boolean
+
+| `-o`, `--opened-only`
+| Skips the import of closed tickets.
+| Boolean
+
+| `-r`, `--rev-map-file`
+| Specify path of the RevMap file for migration.
+| String
+
+| `-S`, `--single-post`
+| Put all Trac ticket comments in the first GitHub issue comment.
+| Boolean
+
+| `-s <ID>`, `--start-at <ID>`
+| Start migration from ticket with number <ID>.
+| String
+
+| `-v`, `--verbose`
+| Enable verbose mode.
+| Boolean
+
+|===
+
+
+== Implementation details
+
+=== Usage of GitHub Issue Import API
+
+Tractive uses the GitHub
+https://gist.github.com/jonmagic/5282384165e0f86ef105[Issue Import API]
+to create Issues.
+
+While this API is not available via GitHub's official API bindings (e.g.
+Octokit), it offers several advantages over the normal Issue creation API:
+
+* will not trigger abuse detection warnings and will not get blocked
+* does not send out email notifications on issue changes
+* does not increase your contribution count (especially if you attempt multiple import tries)
+* faster than with the https://developer.github.com/v3/issues[GitHub v3 Issues API]
+* allows setting the correct creation/closed date
+* creates atomic changes without allowing users to interfere in the creation of
+  a single issue and its comments.
+
+The caveat is that there is still no way of migrating an issue or comment
+attributing to a third-party user account without using that user's GitHub
+account, but this is likely a security concern that will not be addressed by
+GitHub.
+
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/ietf-ribose/tractive. This project is intended to be a safe,
+welcoming space for collaboration, and contributors are expected to adhere to
+the
+https://github.com/ietf-ribose/tractive/blob/main/CODE_OF_CONDUCT.md[code of conduct].
+
+
+== Code of conduct
+
+Everyone interacting in the Tractive project's codebases, issue trackers, chat
+rooms and mailing lists is expected to follow the
+https://github.com/ietf-ribose/tractive/blob/main/CODE_OF_CONDUCT.md[code of conduct].
+
+
+== License
+
+Tractive and its supporting code are licensed under a
+link:LICENSE.md[BSD-style licence].
+
+Code inherited from `trac-hub` is under Matthias Vallentin's
+link:LICENSE.md[BSD 3-Clause License].
+
+Tractive is funded and developed by https://github.com/riboseinc[Ribose Inc.]