hydra 10.0.0 → 10.0.1

data/doc/Lesson---Start-FCRepo-and-Solr.md

@@ -0,0 +1,76 @@
+# Goals
+* Install FCRepo and Solr
+* Learn to start and stop FCRepo and Solr
+
+# Explanation
+In order to use blacklight and hydra-head you need an installation of Solr.  In addition, hydra-head requires a copy of Fedora.  Fedora and Solr are both Java web applications.
+
+For developer environments, we have created a `solr_wrapper` and `fcrepo_wrapper` -- commands to assist you in starting Solr and Fedora. Whenever you need Fedora and Solr running in your development environment, just start or stop that copy of `solr_wrapper` or `fcrepo_wrapper`
+
+>
+**TIP** *DO NOT* use `solr_wrapper` or `fcrepo_wrapper` for production installations.
+>
+
+>
+**WORKSHOPS & HYDRA CAMP** if you're using a VM provided for the workshop, there's probably a copy of solr and fedora already on your VM, follow the instructions your facilitator provides to copy the files instead of having to download them again.  Probably something like
+```
+cp ~/downloads/solr-5.4.1.zip /tmp
+cp ~/downloads/fcrepo-webapp-4.5.0-jetty-console.jar /tmp
+```
+>
+
+### Step 1: Start Solr 
+Open a new terminal window and type:
+```
+cd <hydra-demo app path>
+solr_wrapper -d solr/config/ --collection_name hydra-development
+```
+
+You can check to see if Solr is started by going to [[http://localhost:8983/]]
+
+>
+**TIP** Running solr_wrapper in a tmux panel can silently fail. Use a separate new terminal window, or install https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard for Mac OSX.
+>
+
+
+
+### Step 2: Start FCRepo
+Open a new terminal window and type:
+
+```
+fcrepo_wrapper -p 8984
+```
+
+You can check to see if FCRepo is started by going to [[http://localhost:8984/]]
+
+### Step 3: Make git ignore the `fcrepo4-data` directory
+
+We want git to ignore the `fcrepo4-data` directory for the same reasons that we don't check our development databases into git -- because it's big and bulky and you don't actually need other developers to have exact copies of your data as long as they have all the other code. 
+
+We do that by editing `.gitignore` and adding the something like this:
+
+```text
+# Ignore Fedora data files
+/fcrepo4-data
+
+```
+
+Now commit this change
+
+```text
+git add .gitignore
+git commit -m "Adds /fcrepo4-data to .gitignore"
+```
+
+#### FYI: Stopping FCRepo and Solr
+In order to stop either service, open the window they are running in and type `<Control>-C`. Wait a few moments for the process to terminate.
+
+
+>
+**Tip:** Sometimes people are confused about whether they need to restart Fedora or Solr when they restart their Rails application.  In most cases it is fine to leave Fedora and Solr running when you start, stop, and restart the Rails application.  The only exception is when you make changes to Solr's configuration or Fedora's configuration -- these would be changes to files inside of your copy of the solr configuration (i.e. solr/config), not changes to files in your Rails application's Ruby code.  In those cases, where you have made changes to Solr or Fedora's configuration, you need to restart the processes in order for those changes to take effect.  The most common change that requires restarting Solr is when you modify the solrconfig.xml or schema.xml in your solr config directory. Normally, changes to your data steams or models do not require restarts to Solr and Fedora because these changes are indexed dynamically by Hydra.
+> 
+
+
+
+# Next Step
+Go on to [[Lesson - Start the Application & Search for Results]] or return to the [[Dive into Hydra]] page.

data/doc/{Lesson:-add-the-Hydra-dependencies.md → Lesson---add-the-Hydra-dependencies.md}

@@ -15,10 +15,10 @@ Hydra builds on and extends the features provided by Blacklight, the hydra gener
 Open up `Gemfile` in your editor.   We're going to add the following lines after the `source` line:
 
 ```ruby
-gem 'hydra'
+gem 'hydra', '9.1.0'
 ```
 
-This includes the hydra-gem in our application.  Bundler will then ensure that the hydra-head, blacklight, active-fedora and other gems required by hydra get included (required) correctly. This includes a dependency for the jettywrapper gem (installed automatically). The jettywrapper gem is used to install and configure a preconfigured instance of jetty that loads and runs local development instances of Fedora and Solr for you to run and test your application against.
+This includes the hydra gem in our application.  Bundler will then ensure that the hydra-head, blacklight, active-fedora and other gems required by hydra get included (required) correctly.
 
 Now save the change and install the dependencies by running bundler:
 ```text
@@ -67,4 +67,4 @@ git commit -m "Ran hydra generator"
 ```
 
 # Next Step
-Go on to [[Lesson: Install hydra-jetty]] or return to the [[Dive into Hydra]] page.
+Go on to [[Lesson - Start FCRepo and Solr]] or return to the [[Dive into Hydra]] page.

data/doc/{Lesson:-generate-a-rails-application.md → Lesson---generate-a-rails-application.md}

@@ -1,91 +1,78 @@
-# Goals
-* Create your new Ruby on Rails Application
-* Initialize the local git repository for your project
-
-# Explanation
-
-**Note:** This lesson is roughly covers the [Getting Started](http://curriculum.railsbridge.org/curriculum/getting_started) and [Create A New Git Repo](http://curriculum.railsbridge.org/curriculum/create_a_new_git_repo) steps in the RailsBridge Curriculum.
-
-This lesson assumes you are using a **3.2, 4.0, or 4.1 version of rails**.  To avoid confusion, it's better to have a clean gemset with only one version of rails installed.  Most people use either [RVM](http://rvm.io) or [rbenv](https://github.com/sstephenson/rbenv) to handle gemsets and ruby versions.
-
-The first step to creating a Hydra Head, or any other type of Rails Application, is to generate the basic skeleton of the application code.
-
-We'll also initialize our Git repository in this lesson so we can track incremental changes to our code. In order to track the changes you make to your code, to share your changes with others, and to pull other people's changes into your code, you need some form of Version Control.  The Hydra community uses Git for version control and to share work on Github.
-
-# Steps
-
-### Step 1: Create a new rails application
-
-Once you have installed a suitable rails gem (any 3.2, 4.0, or 4.1 release), begin by using it to generate a new rails application.  You can choose any name for your application.  In this tutorial we are calling it hydra-demo 
-
-```text
-rails new hydra-demo
-```
-
-This generates the file structure for an empty rails application. And it runs 'bundler' which loads in all of the external dependencies for rails.
-
-Enter the directory for the new rails app:
-
-```text
-cd hydra-demo
-```
-
-When you type `ls` at the command prompt, you should see a file structure like this:
-
->
-```text
-Gemfile		Rakefile	config.ru	lib		script		vendor
-Gemfile.lock	app		db		log		test
-README.rdoc	config		doc		public		tmp
-```
->
-
-### Step 1a: *(Linux only)* Enable javascript runtime
-
-Find the line in your Gemfile that has ```# gem 'therubyracer', :platforms => :ruby``` and uncomment that line.  This allows your system to identify the appropriate javascript runtime.
-
-Now save the Gemfile and run ```bundle install```. This tells bundler to update your dependencies to reflect the change in your Gemfile.
-
-### Step 1b: *(Rails 3 only)* Remove the default rails 3 homepage
-
-Rails 3 generates a placeholder homepage at `public/index.html`.  You will need to remove this file in order for the blacklight generated homepage to be used:
-
-```text
-rm public/index.html
-```
-
-### Step 2: Set your Ruby version for this project
-
-A file called ```.ruby-version``` in your project directory (hydra-demo) identifies which version of ruby your application is built with. Put your Ruby's version inside that file. For example, if you're using ruby-2.1.1, you can create the file by entering this at the command prompt:
-
-```text
-echo 'ruby-2.1.1' > .ruby-version
-```
-
-If you're using a Ruby environment manager such as RVM or rbenv, this will cause your system to automatically load the correct version of Ruby when you enter this directory.
-
-### Step 3: Initialize your git repository
-
-Now, let's turn the application directory into a git repository.  Type the following:
-
-```text
-git init .
-```
-
-Then you should see something like this:
-
->
-```text
-Initialized empty Git repository in /Users/camper/hydra-demo/.git/
-```
->
-
-Next, we'll add all the files rails created into the repository.  This way we can jump back to this state later if the need arises.
-
-```text
-git add .
-git commit -m "Initial rails application"
-```
-
-# Next Step
-Go on to [[Lesson: Add the Hydra Dependencies]] or return to the [[Dive into Hydra]] page.
+# Goals
+* Create your new Ruby on Rails Application
+* Initialize the local git repository for your project
+
+# Explanation
+
+**Note:** This lesson is roughly covers the [Getting Started](http://curriculum.railsbridge.org/curriculum/getting_started) and [Create A New Git Repo](http://curriculum.railsbridge.org/curriculum/create_a_new_git_repo) steps in the RailsBridge Curriculum.
+
+This lesson assumes you are using a **4.2.x version of rails**.  To avoid confusion, it's better to have a clean gemset with only one version of rails installed.  Most people use either [RVM](http://rvm.io) or [rbenv](https://github.com/sstephenson/rbenv) to handle gemsets and ruby versions.
+
+The first step to creating a Hydra Head, or any other type of Rails Application, is to generate the basic skeleton of the application code.
+
+We'll also initialize our Git repository in this lesson so we can track incremental changes to our code. In order to track the changes you make to your code, to share your changes with others, and to pull other people's changes into your code, you need some form of Version Control.  The Hydra community uses Git for version control and to share work on Github.
+
+# Steps
+
+### Step 1: Create a new rails application
+
+Once you have installed a suitable rails gem (4.2.x releases work best for this tutorial), begin by using it to generate a new rails application.  You can choose any name for your application.  In this tutorial we are calling it hydra-demo 
+
+```text
+rails new hydra-demo
+```
+
+This generates the file structure for an empty rails application. And it runs 'bundler' which loads in all of the external dependencies for rails.
+
+Enter the directory for the new rails app:
+
+```text
+cd hydra-demo
+```
+
+When you type `ls` at the command prompt, you should see a file structure like this:
+
+>
+```text
+Gemfile		Rakefile	config.ru	lib		script		vendor
+Gemfile.lock	app		db		log		test
+README.rdoc	config		doc		public		tmp
+```
+>
+
+**Windows Only**: if you're running this tutorial directly on a Windows system, you'll need to use `dir` instead of `ls` anywhere in these instructions where you're asked to list files.
+
+#### Step 1a: (**Linux only**) Enable javascript runtime
+
+Find the line in your Gemfile that has ```# gem 'therubyracer', platforms: :ruby``` and uncomment that line.  This allows your system to identify the appropriate javascript runtime.
+
+Now save the Gemfile and run ```bundle install```. This tells bundler to update your dependencies to reflect the change in your Gemfile.
+
+Alternatively, a Node.js runtime will be resolved without adding `therubyracer`. Joyent maintains simple instructions for [installing Node.js with various Linux package managers](https://github.com/joyent/node/wiki/installing-node.js-via-package-manager).
+
+
+### Step 2: Initialize your git repository
+
+Now, let's turn the application directory into a git repository.  Type the following:
+
+```text
+git init .
+```
+
+Then you should see something like this:
+
+>
+```text
+Initialized empty Git repository in /Users/camper/hydra-demo/.git/
+```
+>
+
+Next, we'll add all the files rails created into the repository.  This way we can jump back to this state later if the need arises.
+
+```text
+git add .
+git commit -m "Initial rails application"
+```
+
+# Next Step
+Go on to [[Lesson - Add the Hydra Dependencies]] or return to the [[Dive into Hydra]] page.

data/doc/Lesson---make-blacklight-return-search-results.md

@@ -0,0 +1,71 @@
+# Goals
+* *(for now)* Turn off access controls for Blacklight-based searches and "show" views 
+* Tell Blacklight which fields to use in searches
+* Run a search in Blacklight and see results rendered
+
+# Explanation
+
+In [[Lesson - Build a book model with RDF]] and/or [[Lesson - Build a codex model with XML]] you defined an object model and stored a new digital object in your repository. We saved the objects and saw that their metadata was indexed in Solr.  Now we will make your repository objects appear in your Blacklight searches.
+
+One of the main features that Hydra adds to Blacklight is the ability to control who has access to which information in search results.  That topic gets a little bit complicated.  For the purpose of this Tutorial we want to stay focused on showing you how to set up an app, define models and create objects based on those models, so in order to keep things simple we will make this Hydra Head behave like an open access repository where everyone can see everything.  Once you've completed this tutorial, you can check out [Access Controls with Hydra](https://github.com/projecthydra/hydra-head/wiki/Access-Controls-with-Hydra) to learn how to assert access controls on objects and enforce those access controls in a Hydra Head.
+
+Once you've turned off access controls in step #1, we show you how to tell blacklight which fields you want to use for default searches in the remaining steps.
+
+# Steps
+
+### Step 1: Comment out and change the lines that enforce access controls in Blacklight's CatalogController
+
+If you open ```app/controllers/catalog_controller.rb``` and look at the code near lines 8-12 you should see this:
+```ruby
+  # These before_filters apply the hydra access controls
+  before_filter :enforce_show_permissions, :only=>:show
+  # This applies appropriate access controls to all solr queries
+  Hydra::SearchBuilder.default_processor_chain += [:add_access_controls_to_solr_params]
+```
+
+This code tells blacklight to enforce access controls on the search and result view pages.  For the time being we will turn this off by commenting out the before_filter and changing ``` += ``` to ``` -= ``` in order to remove ``` :add_access_controls_to_solr_params ``` from the default_processor_chain, the code should look like this:
+
+```ruby
+  # These before_filters apply the hydra access controls
+  #before_filter :enforce_show_permissions, :only=>:show
+  # This applies appropriate access controls to all solr queries
+  Hydra::SearchBuilder.default_processor_chain -= [:add_access_controls_to_solr_params]
+```
+Then, save the file.
+
+### Step 2: Run a search in the CatalogController
+
+Visit or reload the page at [[http://localhost:3000/]].  You should see the Blacklight search interface with a search box.  If you just hit enter on the blank search box (effectively asking blacklight to return all objects) you should get a response with the object(s) you've created using the rails console.  **IN ADDITION** If you search for your title or author by typing part of it into your search box and then hitting enter, you'll see any matching results.  This happens because Blacklight configures 'title', 'author', and 'subject' as default search fields out of the box.  
+
+### Step 3: Tell Blacklight which fields to use in Queries
+
+Sometimes, we want to tell Blacklight explicitly which fields to search in.  Let's fix that by setting the default 'qf' solr parameter.  Open `app/controllers/catalog_controller.rb` and set the `default_solr_params` section (around line 18) to this:
+
+```ruby
+    config.default_solr_params = { 
+      qf: 'title_tesim author_tesim',
+      qt: 'search',
+      rows: 10 
+    }
+```
+
+> NOTE: Because we used the same names, 'author' and 'title' for both our book and codex models, blacklight will return results that match either digital object type.  We'll talk about filtering object types in one of the bonus lessons.
+
+### Step 4: Re-run your search
+
+Save the changes to your `catalog_controller.rb` file, and restart your web server from the terminal (Ctrl-C to stop the server and `rails server` to restart it.)  Now, in your browser, search for one of the words in your author or title fields and you should see a list of matching results.
+
+~~**Tip:** When you make changes like this, you *don't* need to restart the Rails server.  This is because in development mode (which is the default environment for the Rails server), the Rails server reloads any files in app/models, app/controllers, app/views, etc. for every request it receives from a browser.  This makes the server slower, but it makes life much smoother when you're actively developing and making changes.~~
+
+### Step 5: Commit your changes
+
+Now that we've updated our search functionality, it's a great time to commit to git:
+
+```text
+git add .
+git commit -m "Disabled access controls and set default search fields"
+```
+
+# Next Step
+Go on to **BONUS** [[Lesson - Define Relationships Between Objects]] or 
+explore other [Dive into Hydra](Dive into Hydra#bonus) tutorial bonus lessons.

data/doc/Lesson---start-the-application-&-search-for-results.md

@@ -0,0 +1,55 @@
+# Goals
+* Configure the application
+* Start and Stop a local "development" copy of the application
+* Remove the default Welcome page provided by Rails
+* Run a Search in Blacklight through your browser
+
+# Steps
+
+### Step 1: Configure the application
+
+There are a couple configurations that need to be set, or verified, before starting the application. 
+
+#### config/solr.yml
+The urls and collection name must be set to match what is running by solr_wrapper.
+``` 
+development:
+  url: http://127.0.0.1:<%= ENV['SOLR_TEST_PORT'] || 8983 %>/solr/hydra-development
+```
+
+#### config/fedora.yml
+The urls to the Fedora server must be properly set for production and what is running by fcrepo_wrapper.
+```
+development:
+  url: http://127.0.0.1:<%= ENV['FCREPO_DEVELOPMENT_PORT'] || 8984 %>/rest
+  ...
+```
+
+### Step 2: Start the Rails server
+
+Now, let's see our model on the web.  You can start 'webrick', a development server that comes with rails by typing:
+
+```bash
+rails server -b 0.0.0.0
+```
+
+>
+*NOTE:* -b 0.0.0.0 isn't strictly required, but lets you access your server from your host browser if you're running it in a VM
+>
+
+Leave this terminal window open through the rest of the tutorial.  It will print info whenever the server does anything.  You will return to this window when you want to stop or restart the server.
+
+### Step 3: Look at the application in your Browser and customize the home page
+
+Now you can visit your local copy of the application with a web browser when you go to [[http://localhost:3000/]].
+
+The default home page gives instructions for customizing the home page: essentially, create a ```app/views/catalog/_home_text.html.erb``` with the content you want. After saving that file and reloading in the browser you should see your changes.
+ 
+### Step 4: Run a Search
+
+If you enter a search query you don't see any results because we haven't created any objects yet -- your Solr index is empty.
+
+In the next lessons we will create a objects in your Fedora repository, then we will show you how to make them appear in your search results.
+
+# Next Step
+If you're interested in how RDF object modeling works, go on to [[Lesson - Build a book model with RDF]]; otherwise go on to [[Lesson - Build a codex model with XML]] or return to the [[Dive into Hydra]] page.

data/doc/Recipe:-Add-full-text-indexing-to-your-app.md

@@ -0,0 +1,27 @@
+There are number of areas in the Hydra stack that need to be touched to do full-text indexing.  Sufia supports full-text indexing using Apache Tika (which is provided in Apache Solr), and here's how it's implemented. (Note: if you're using Sufia, this is already done for you!)
+
+# Solr
+
+The [Solr schema](https://github.com/projecthydra/sufia/blob/eb714cb91f4ecd49dec67b0e9996dc3d4e918f1a/solr_conf/conf/schema.xml#L174-L175) contains a field called `all_text_timv`.
+
+The [Solr config](https://github.com/projecthydra/sufia/blob/7cedb837b2d6388d81ba7d9a815edc0e4b4c1251/solr_conf/conf/solrconfig.xml#L16-L17) pulls in a bunch of extraction libraries and [adds the `all_text_timv` field to the default qf and pf](https://github.com/projecthydra/sufia/blob/7cedb837b2d6388d81ba7d9a815edc0e4b4c1251/solr_conf/conf/solrconfig.xml#L42-L47). The [ExtractingRequestHandler](https://github.com/projecthydra/sufia/blob/7cedb837b2d6388d81ba7d9a815edc0e4b4c1251/solr_conf/conf/solrconfig.xml#L110-L121) must be enabled as well.
+
+# Extraction libraries
+
+Sufia uses [a rake task](https://github.com/projecthydra/sufia/blob/eb714cb91f4ecd49dec67b0e9996dc3d4e918f1a/sufia-models/lib/tasks/sufia-models_tasks.rake#L27-L96) to download extraction libraries and store them where Solr looks for them.
+
+# Blacklight catalog
+
+The `all_text_timv` field is added to the `all_fields` search qf in the [Catalog controller](https://github.com/projecthydra/sufia/blob/a064489d3cf4d228abf978d7126644e2c653aa5e/lib/generators/sufia/templates/catalog_controller.rb#L125)
+
+# Modeling
+
+Sufia's `GenericFile` model mixes in [a module that knows how to talk to Solr's ExtractingRequestHandler](https://github.com/projecthydra/sufia/blob/a064489d3cf4d228abf978d7126644e2c653aa5e/sufia-models/app/models/concerns/sufia/generic_file/full_text_indexing.rb). (The `#extract_content` method is where that happens.)
+
+# Indexing
+
+Sufia has [an indexing service](https://github.com/projecthydra/sufia/blob/7644ff5b04d66cfe1c4d7bee75a6840eb09eb664/sufia-models/app/services/sufia/generic_file_indexing_service.rb#L10) that takes the output of Apache Tika and indexes it in Solr. (This is the equivalent of overriding `#to_solr` on an ActiveFedora model.)
+
+# Workflow
+
+When a file is uploaded, Sufia spawns [a background job that characterizes the file](https://github.com/projecthydra/sufia/blob/a064489d3cf4d228abf978d7126644e2c653aa5e/sufia-models/app/jobs/characterize_job.rb#L7). The `#characterize` method calls [#append_metadata](https://github.com/projecthydra/sufia/blob/a064489d3cf4d228abf978d7126644e2c653aa5e/sufia-models/app/models/concerns/sufia/generic_file/characterization.rb#L75). That method in turn calls the `#extract_content` method which hits Apache Tika via the Solr API.

data/doc/Using-rdf:resource-within-your-models.md

@@ -0,0 +1,43 @@
+Some data models may call for adding properties to objects that consist of URI references using the `rdf:resource` syntax. For example, you may want to define an object's language by reference to a standard list available online, such as that supplied by Library of Congress. In this example, we will add a language property to the Book example from Dive Into Hydra which consists of a URI reference.
+
+First, you add the required property to the model:
+
+```ruby
+property :language, predicate: ::RDF::DC.language, multiple: false
+```
+
+Then you add a getter and setter method:
+
+```ruby
+def language_uri=(uri)
+  self.language = ::RDF::URI.new(uri) if uri.present?
+end
+
+def language_uri
+  self.language.to_term.value unless language.nil?
+end
+```
+
+In your rails console try assigning a language to an object, for example:
+
+```
+b = Book.new
+b.language_uri = "http://id.loc.gov/vocabulary/languages/abk"
+b.language_uri
+ => "http://id.loc.gov/vocabulary/languages/abk" 
+b.language
+ => #<ActiveTriples::Resource:0x42c2258(default)> 
+b.resource.dump(:rdf)
+ => "<?xml version='1.0' encoding='utf-8' ?>
+<rdf:RDF xmlns:ns0='info:fedora/fedora-system:def/model#' xmlns:ns1='http://purl.org/dc/terms/' xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'>
+<rdf:Description rdf:about=''>
+<ns1:language rdf:resource='http://id.loc.gov/vocabulary/languages/abk' />
+<ns0:hasModel>Book</ns0:hasModel>
+</rdf:Description>
+</rdf:RDF>" 
+```
+
+As you can see, the link has been added to the property as an `rdf:resource` attribute rather than as a value. If you are updating objects via a web form, you will use the language_uri attribute in the form rather than the language attribute. You will also need to add the `language_uri` field to the whitelisted parameters in the controller. 
+
+
+