ki-repo 0.1.0 → 0.1.1

data/README.md

@@ -1,18 +1,32 @@
 # ki-repo
 
-Repository for storing versions and metadata.
+Repository for storing packages and metadata.
 
-## Contributing to ki-repo
- 
-* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
-* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
-* Fork the project.
-* Start a feature/bugfix branch.
-* Commit and push until you are happy with your contribution.
-* Make sure to add tests for it.
-* Please make your commit tidy.
+note: Currently Ki is not ready for any kind of use.
 
-## Copyright
+# Links
 
-Copyright (c) 2012 Mikko Apo. See LICENSE.txt for further details.
+* Rubygem: https://rubygems.org/gems/ki-repo
+* Documentation for latest released version is available at http://rubydoc.info/gems/ki-repo
+
+# Documentation
+
+* {file:docs/repository_basics.md Repository basics} includes a simple tutorial and explains basic concepts.
+* Ki command line utilies are documented in {file:docs/ki_commands.md}.
+* {file:docs/development.md Development} provides additional development related information.
+
+# Plan
+
+1. Local repository features: create package, import, export, list, test, dependencies
+2. Metadata support: statuses, file tagging
+3. Repository cleanup and removal
+4. Use script files in repository to add additional commands to ki
+5. Documentation
+
+Once these features are implemented Ki is ready for use on local server. In the future, the goal is to provide tools
+to manage distributed repositories: downloads and replication.
+
+# Copyright
+
+Copyright (c) 2012 Mikko Apo. See {file:LICENSE.txt} for further details.
 

data/VERSION

@@ -1 +1 @@
-0.1.0
+0.1.1

data/bin/ki

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+# encoding: UTF-8
+
+# Copyright 2012 Mikko Apo
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require_relative '../lib/ki_repo_all'
+
+Ki::KiCommand.new.execute(ARGV)
+

data/docs/backlog.md

@@ -0,0 +1,35 @@
+# @title Ki: Backlog
+
+# Backlog for 0.2
+* fix defined? checks
+* fix double repository entries in finder.all_repositories "should show imported version"
+* cleanup gem
+* create release notes
+
+# 0.3
+* cleanup and removal operations
+* version() returns sometime nil, sometimes Version
+* version!()
+* fix Version.exists? - there should be a better way to check if version exists
+* VersionTester should support test_version(metadata, source)
+* Download & replication
+* website command
+* daemon: long running processes, web site monitoring
+* multiple local repositories: command line tools and helpers
+* support for separate binaries directory
+* pref: define repository lookup order
+
+# Future releases
+* Digital signing
+* Encrypted/packed packages
+* Support for using files from other repositories
+* when building version create file operations based on user's changes
+
+# Maybe at some point...
+* replace popen4.spawn with Kernel.spawn
+* alias
+* package dep operations: dep-rm, dep-mv, dep-cp, dep-switch
+* named version lists for component my/component#released:Smoke=green
+
+# Future backwards compatability issues
+* how to store version directories so that per directory limits can be bypassed

data/docs/development.md

@@ -0,0 +1,45 @@
+Warning! Ki is currently not ready for public use. APIs, classes and functionality will change.
+
+# Setting up development environment
+
+Follow the instructions here: {file:docs/development_setup.md}
+
+# Writing extensions
+
+{file:docs/writing_extensions.md} describes how to extend Ki by writing extension scripts.
+
+# Important Ruby classes
+
+## Command line utilities
+
+* {Ki::KiCommand} - Main ki entry point
+* {Ki::KiCommandHelp}, {Ki::KiInfoCommand} - help, ki-info
+* {Ki::UserPrefCommand} - Stored preferences
+* {Ki::BuildVersionMetadataFile}, {Ki::TestVersion}, {Ki::ImportVersion}, {Ki::ExportVersion}, {Ki::VersionStatus}, {Ki::ShowVersion}, {Ki::VersionSearch} - repository management
+
+## Data storage - files and directories
+
+* {Ki::KiHome}, {Ki::Repository::Repository}, {Ki::Repository::Component}, {Ki::Repository::Version} - Repository directory objects
+* {Ki::VersionMetadataFile}, {Ki::Dependency}, {Ki::VersionStatusFile} - Version metadata and statuses
+* {Ki::DirectoryBaseModule}, {Ki::DirectoryBase} - Base classes for file and directory management
+* {Ki::KiJSONFile}, {Ki::KiJSONListFile}, {Ki::KiJSONHashFile}, {Ki::KiJSONHashFile::CachedMapDataAccessor} - Base classes for JSON files
+* {Ki::DirectoryWithChildrenInListFile} - Helper to generate list file class and related methods for a repository object
+* {Ki::RepositoryMethods}, {Ki::RepositoryMethods::RepositoryListFile} - Repository helper methods
+* {Ki::UserPrefFile} - Stored preferences
+
+## Data access - finders, helpers, iterators
+
+* {Ki::VersionTester}, {Ki::VersionImporter}, {Ki::VersionExporter} - Utility classes for managing version
+* {Ki::RepositoryFinder} - Loads components from all available repositories and finds components and versions
+* {Ki::VersionIterator} - Provides method to iterate through all component's matching versions
+* {Ki::FileFinder} - Finds matchins files from a Version
+* {Ki::Component}, {Ki::Version} - Classes that manage combined information for versions and components from all repositories
+* {Ki::VersionFileOperations} - Helper class to process Version's file operations
+
+## Utilities
+
+* {Ki::Tester} - Helper class for tests, makes it easy to clear resources
+* {AttrChain} - Chained accessor methods
+* {ExceptionCatcher} - Execute multiple blocks and process exceptions in the end
+* {Ki::ServiceRegistry} - Class for storing all Ki extensions
+* {Ki::SimpleOptionParser} - Simplified OptionParser

data/docs/development_setup.md

@@ -0,0 +1,49 @@
+# @title Ki: Setting up development environment
+# Setting up development environment
+
+## Install Ruby 1.9
+
+Rvm makes it easy to use multiple ruby installations. By default it installs rubies to users home directory.
+Rvm is available at http://beginrescueend.com/
+
+Follow the rvm installation instructions, once the rvm command line tool works execute:
+
+    rvm install 1.9.2
+    rvm use 1.9.2
+    rvm gemset create ki
+    rvm use 1.9.2@ki
+
+Now ruby 1.9.2 should be available and all gems will be installed for Ki use only.
+
+## Check out the source
+
+    git clone git@github.com:mikko-apo/ki-repo.git
+    cd ki-repo
+
+## Install gems
+
+Bundler is used to download and install the required gems to the ruby environment in use. if you are using rvm, gems are installed to rvm's directories.
+
+    gem install bundler
+    bundle install
+
+## Run tests
+
+Tests are run with
+
+    rake spec
+
+All tests should pass. Code coverage report is generated to coverage/index.html
+
+## Generate documentation
+
+Documentation is generated to doc/index.html with
+
+    rake yard
+
+You can also start up the yard in server mode, which makes it easier to update the documentation. When server starts
+up, the documentation will be available at http://localhost:8808/
+
+    yard server --reload
+
+

data/docs/ki_commands.md

@@ -0,0 +1,202 @@
+# @title Ki: Command line utilities
+# Command line utilities for Ki Repository v0.1.0
+"ki" is the main command line tool that starts all other Ki processes. Whenever ki command line tools
+are executed, ki goes through the following startup process
+
+1. Common command line parameters are parsed. These can used to set execution parameters for this invocation.
+2. Extension scripts are loaded from repository. Version configuration is from either -u or user preferences
+3. Find command by name
+4. Execute the command and pass rest of the command line parameters
+
+Examples
+
+    ki build-version *.txt
+    ki -u my/tools compile
+    ki -u my/tools:scripts,tools compile
+
+note: By default only files with tag "ki-cmd" are used. Use the 'my/tools:scripts,tools' to define additional tags.
+
+Common parameters:
+
+    -h, --home                       Path to Ki root directory
+    -u, --use                        Use defined scripts
+
+## help: Displays help for given Ki command
+
+"ki help" shows information Ki and its commands.
+
+### Examples
+
+    ki help
+    ki help version-build
+
+
+## ki-info: Show information about Ki
+
+"ki ki-info" shows information about Ki.
+
+### Examples
+
+    ki ki-info -c
+    ki ki-info -r
+
+### Parameters
+    -c, --commands                   List commands
+    -r, --registered                 List all registered extensions
+
+## version-build: Create version metadata file
+
+"ki version-build" can be used to generate version metadata files. Version metadata files
+contain information about files (size, permission bits, hash checksums), version origins
+and dependencies.
+
+After version metadata file is ready, it can be imported to repository using version-import.
+
+### Usage
+
+    ki version-build <parameters> file_pattern1*.* file_pattern2*.*
+
+### Examples
+
+    ki version-build test.sh
+    ki version-build readme* -t doc
+    ki version-build -d my/component/1,name=comp,path=doc,internal -O "mv doc/test.sh helloworld.sh"
+    ki version-import
+
+### Parameters
+
+    -f, --file FILE                  Version file target
+    -i, --input-directory INPUT-DIR  Input directory
+    -v, --version-id VERSION-ID      Version's id
+        --source-url URL             Build source parameter url
+        --source-tag-url TAG-URL     Build source parameter tag-url
+        --source-author AUTHOR       Build source parameter author
+        --source-repotype REPOTYPE   Build source parameter repotype
+    -t, --tags TAGS                  Tag files with keywords
+        --hashes HASHES              Calculate checksums using defined hash algos. Default: sha1. Available: sha1, sha2, md5
+    -d, --dependency DEPENDENCY      Dependency definition my/component/123[,name=AA][,path=aa][,internal]
+    -o, --operation OP               Add operation to previous dependency
+    -O, --version-operation OP       Add operation to version
+
+
+## version-test: Tests versions and their dependencies
+
+"ki version-test" tests versions, their files and their dependencies. Can also test version that has not been imported yet.
+
+### Examples
+
+    ki version-test -r my/product other/product
+    ki version-test -f ki-version.json -i file-directory
+
+### Parameters
+
+    -f, --file FILE                  Version source file. By default uses file's directory as source for binary files.'
+    -i, --input-directory INPUT-DIR  Binary file input directory
+    -r, --recursive                  Tests version's dependencies also.'
+
+
+## version-import: Imports version metadata and files to repository
+
+"ki version-import" imports version and its files to repository.
+
+Version name can be defined either during "version-build",
+or generated automatically for component at import (with -c my/component) or defined to be a specific version (-v).
+Can also move files (-m), test dependencies before import (-t).
+
+### Examples
+
+    ki version-import -m -t -c my/product
+    ki version-import -f ki-version.json -i file-directory
+
+### Parameters
+
+    -f, --file FILE                  Version source file. By default uses file's directory as source for binary files.'
+    -i, --input-directory INPUT-DIR  Input directory
+    -t, --test-recursive             Tests version's dependencies before importing.'
+    -m, --move                       Moves files to repository'
+    -c COMPONENT,                    Creates new version number for defined component'
+        --create-new-version
+    -v, --version-id VERSION         Imports version with defined version id'
+
+
+## version-export: Export version to a directory
+
+"ki version-export" exports version and its dependencies to target directory.
+
+### Usage
+
+    ki version-export <parameters> <file_export_pattern*.*>
+
+### Examples
+
+    ki version-export -o export-dir --tags -c bin my/product
+    ki version-export -o scripts -c -t my/admin-tools '*.sh'
+
+### Parameters
+
+    -o, --output-directory INPUT-DIR Input directory
+        --tags TAGS                  Select files with matching tag
+    -t, --test                       Test version before export
+    -c, --copy                       Exported files are copied instead of linked
+
+
+## version-status: Add status values to version
+
+"ki version-status" sets status values to versions and sets status value order to component.
+
+Status value order is used to determine which statuses match version queries:
+
+    my/component:maturity>alpha
+
+### Examples
+
+    ki version-status add my/component/1.2.3 Smoke=Green action=path/123
+    ki version-status order my/component maturity alpha,beta,gamma
+
+## version-show: Prints information about version or versions
+
+"ki version-show" prints information about version or versions and their dependencies
+
+### Examples
+
+    ki version-show -r -d my/component/23 my/product/127
+    ki version-show -f ki-version.json -i binary-dir
+
+## version-search: Searches for versions and components
+
+"ki version-search" searches for versions and components.
+
+### Examples
+
+    ki version-search my/component
+    ki version-search my/*
+
+## pref: Sets user preferences
+
+Sets user preferences
+Syntax: ki pref prefix|use parameters...
+
+### Examples for command prefixes:
+    ki pref prefix
+    - shows command prefixes, when a "ki command" is executed ki looks for the command with all prefix combinations
+    ki pref prefix version package
+    - sets two command prefixes, looks for "command", "version-command" and "package-command"
+    ki pref prefix + foo
+    - adds one command prefix to existing ones, looks for "command", "version-command", "package-command", "foo-command"
+    ki pref prefix - package foo
+    - removes two command prefixes from list
+    ki pref prefix -c
+    - clears command prefix list
+
+### Examples for automatic script loading:
+    ki pref use
+    - shows list of automatically loading scripts. when ki starts up, it looks for all defined versions and loads all files tagged with ki-cmd
+    ki pref use ki/http ki/ftp/123:ki-extra
+    - scripts are loaded from two different version. ki/http uses latest available version and files tagged with "ki-cmd", ki/ftp uses specific version and files tagged with "ki-extra"
+    ki pref use + ki/scp
+    - adds one more script package version
+    ki pref use - ki/scp ki/ftp/123:ki-extra
+    - removes two configurations
+    ki pref use -c
+    - clear use list
+

data/docs/repository_basics.md

@@ -0,0 +1,171 @@
+# @title Ki: Repository basics
+
+# Getting started
+
+1. Create a version and import it to repository
+
+        echo 'Hello World!' > test.sh
+        echo "Simple demo" > "readme.txt"
+        chmod u+x test.sh
+        ki version-build test.sh
+        ki version-build readme* -t doc
+        ki version-import -m -c my/component
+        ki version-show my/component
+
+    * `version-import -c` creates a new version under `"my/component`` and `-m` moves the files to version's repository directory
+    * `version-show` looks for the latest available version and shows information about that
+
+2. Create another version, with dependency to `my/component/1` and import that too
+
+        ki pref prefix version
+        ki build -d my/component/1,name=comp,path=doc,internal -O "mv doc/test.sh helloworld.sh"
+        ki import -m -c my/product
+        ki show -r my/product
+        ki export my/product -o export
+        find export
+
+    * `ki pref prefix version` configures a shortcut to call version commands with shorter syntax
+    * `version-build` generates a dependency to `"my/component/1"` and puts the files from `"my/component/1"` to doc directory
+    * `version-build -O` operations are executed when the product version is exported, `"doc/test.sh"` is moved to helloworld.sh
+    * the contents of "export" directory should be:
+
+            export
+            export/doc
+            export/doc/readme.txt
+            export/helloworld.sh
+
+# Repository structure: versions, components
+
+Ki-Repo is a repository for storing file packages and metadata about those packages.
+
+An example repository has following structure:
+
+* Repository has three components: `"my/componentA"`, `"my/componentB"`, `"my/product"`.
+* Each component maintains a chronological list of versions. For example, `"my/component"` has versions `"3"`, `"2"` and `"1"`.
+* Component's name should be a unique identifier and it can include any number of identifiers. Valid component names include `"ki/repo"`, `"ki-repo"` or `"my/test/builds/ki/repo/"`.
+* Version contains a set of files. Version can also define dependencies and other metadata.
+
+The repository and its directories and files would look like this:
+
+    repository/
+      my/
+        component/
+          23/
+            jar/lib.jar
+            readme.txt
+            start.sh
+          22/
+            jar/lib.jar
+            readme.txt
+            start.sh
+          21/
+            jar/lib.jar
+            readme.txt
+            start.sh
+        componentB/
+          build-211/
+            lib/utils.rb
+          build-210/
+            ...
+          build-209/
+          ...
+        product/
+          98/
+          97/
+          ...
+
+## Version
+
+Each version identifies a unique combination of files and dependencies. After version has been built and imported to
+repository it does not change. This ensures that when ever the version is used, the contents stay the same.
+
+In addition to having files, version can define metadata about its files (size, permission bits, hash checksums), origins and dependencies.
+Files can also be tagged with identifiers to make it easier to identify files of different types. Versions can also have status
+information (for example "IntegrationTest=green"), which makes it easier to search for versions.
+
+Version's full name is the name of the component and the version name: `"my/component/23"`
+
+### File metadata
+
+    { "path": "test.sh", "size": 2, "executable": true, "tags": [ "test-start" ], "sha1": "9a900f538965a426994e1e90600920aff0b4e8d2" }
+
+* path identifies the actual file (stored in the version)
+* size and sha1 are calculated when version is built
+* executable is stored when version is built and ki ensures that when the file is in repository or exported is has its executable flag set on
+* file tags can be used to tag files with different identifiers
+
+### Source information
+
+    { "url": "http://test.repo/component@21331", "tag-url": "http://test.repo/component/tags/23", "repotype": "git", "author": "john" }
+
+* source information is used to store reference to the original source of the package
+
+### Dependencies
+
+Version can have dependencies, that include additional versions in to the main version. Dependencies refer to other
+versions with full version name.
+
+        { "version_id": "my/component/23", "name": "comp", "path": "comp", "internal": true,
+          "operations": [
+            [
+              "cp", "comp/test.sh", "test.bat"
+            ]
+          ]
+        }
+
+* version_id defines the full name of the required version
+* path defines a subdirectory where files from this dependency are placed
+* dependency can have a name, which makes it possible to compare version's dependencies and also navigate version hierarchies
+    * navigation syntax: my/product/1>comp -> my/component/1
+    * my/product/1>comp = my/component/23, my/product/2>comp -> my/component/24
+* internal dependencies are visible in the version hierarchy only for top level version. comp is visible from my/product,
+but if other version has a dependency on product, files from comp are not visible
+
+### File Operations
+
+Version can define file operations to modify the exported file structure. Operations can be defined at version level, so
+that they affect version's all files (the ones defined in the version and brought by the dependencies). File operations
+can also be defined per dependency.
+
+Available file operations:
+
+* cp - copy pattern1 pattern2 ... dest
+* mv - move pattern1 pattern2 ... dest
+* rm - remove.
+
+Examples, command line format - json storage format
+
+    "rm *.txt" - ["rm", "*.txt"]
+    "cp *.txt sub-directory" - ["cp", "*.txt", "sub-directory"]
+    "mv *.sh scripts" - ["mv", "*.sh", "scripts"]
+
+File operations are defined for "version-build" with -o and -O parameters
+
+    ki version-build -d my/tests/a/123,name=tests -o "rm *.txt"
+    ki version-build -O "cp scripts/start.sh start.sh"
+
+## Component
+
+Component has only a two responsibilities:
+
+1. Keep a chronological list of versions
+2. Store status value order for different status fields
+
+Status value order is used to determine which statuses match version queries:
+
+    my/component:maturity>alpha
+
+## Repository
+
+Repository directory keeps a list of all components that have data in the repository.
+
+## KiHome
+
+KiHome is the root directory for Ki. It contains repositories.
+
+# Multiple repositories (TODO)
+
+Ki supports storing information to multiple repositories. It will be possible to use one repository for shared
+artifacts like binaries and store statuses to a different repository. This way it's easy to support for example
+team specific status values for artifacts produced by other team.
+

data/docs/writing_extensions.md

@@ -0,0 +1,50 @@
+# @title Ki: Writing extensions
+
+# Writing extensions
+
+Ki can be extended by packaging extension code in to version packages and using the `ki -u version/id` command line
+parameter to load scripts in to the Ruby VM. Extension versions can also be configured to be automatically loaded
+with the `ki pref prefix version/id` command.
+
+"ki" is the main command line tool that starts all other Ki processes. Whenever ki command line tools
+are executed, ki goes through the following startup process
+
+1. Common command line parameters are parsed. These can used to set execution parameters for this invocation.
+2. Extension scripts are loaded from repository. Version configuration is from either -u or user preferences
+3. Find command by name
+4. Execute the command and pass rest of the command line parameters
+
+Examples
+
+    ki build-version *.txt
+    ki -u my/tools compile
+    ki -u my/tools:scripts,tools compile
+
+note: By default only files with tag "ki-cmd" are used. Use the 'my/tools:scripts,tools' to define additional tags.
+
+Ki's extension mechanism makes it easy to manage different scenarios:
+* write and distribute command line utilities
+* use different versions of those utilities at the same time on the same machine (backwards compatability)
+* add new features to existing utilities: hashing algorithms, integrations to different tools like git, mercurial and svn
+
+# Command line utility
+
+Command classes are registered with KiCommand.register_cmd
+
+    KiCommand.register_cmd("version-build", BuildVersionMetadataFile)
+
+They should implement following methods:
+* execute(ctx, args)
+* help, summary
+* attr_chain :shell_command, :require is optional but help method can use it to generate more useful help texts
+
+For more information, see {Ki::ImportVersion}
+
+# Extension points
+
+{Ki::KiCommand} stores all registered extensions to its class variable {Ki::KiCommand::KiExtensions} ({Ki::ServiceRegistry}).
+
+Currently used extension points are:
+
+* /commands/
+* /hashing/