ae_easy-login 0.0.2 → 0.0.3

data/doc/index.html

@@ -73,16 +73,438 @@ src="http://img.shields.io/badge/license-MIT-yellowgreen.svg"></a></p>
 handle login and session recovery, quite useful when scraping websites with
 login features and expiring sessions.</p>
 
-<p>Install gem: <code>gem install &#39;ae_easy-login&#39;</code></p>
+<p>Install gem: <code>ruby gem install &#39;ae_easy-login&#39; </code></p>
 
-<p>Require gem: <code>require &#39;ae_easy/login&#39;</code></p>
+<p>Require gem: <code>ruby  require &#39;ae_easy/login&#39; </code></p>
 
-<p>Documentation can be found <a
+<p>Code documentation can be found <a
 href="http://rubydoc.org/gems/ae_easy-login/frames">here</a>.</p>
+
+<h2 id="label-How+to+implement">How to implement</h2>
+
+<h3 id="label-Before+you+start">Before you start</h3>
+
+<p>It is true that most user cases for <code>ae_easy-login</code> gem applies
+to websites with login pages and create sessions, so we will cover this
+scenario on our example.</p>
+
+<p>Therefore, <code>ae_easy-login</code> gem is designed to handle
+<strong>ANY</strong> kind of session recovery, even those that doesn&#39;t
+requires a login form <code>POST</code> by just changing the flow from:</p>
+
+<pre class="code ruby"><code class="ruby">login -&gt; login_post -&gt; restore
+</code></pre>
+
+<p>To whatever you need like for example:</p>
+
+<pre class="code ruby"><code class="ruby">home -&gt; search_page -&gt; restore
+</code></pre>
+
+<p>Here are some user case examples that can be fixed by
+<code>ae_easy-login</code> gem:</p>
+<ul><li>
+<p>Websites that invalidate requests with fast expiring cookies created on
+first request.</p>
+</li><li>
+<p>Websites that generates tokens on every search (either on cookies or
+query_params) that are required to fetch a detail page.</p>
+</li><li>
+<p>Websites that expires session due inactivity.</p>
+</li><li>
+<p>Websites that uses complex login flows.</p>
+</li><li>
+<p>etc.</p>
+</li></ul>
+
+<p>Feel confident to expirement with it until it fit all your needs.</p>
+
+<h3 id="label-Adding+ae_easy-login+to+your+project">Adding ae_easy-login to your project</h3>
+
+<p>Let&#39;s assume a simple project implementing <code>ae_easy</code> like
+the one described on <a
+href="https://github.com/answersengine/ae_easy/blob/master/README.md">ae_easy
+README.md</a> that scrapers your website.</p>
+
+<p>Now lets assume your website has a login page
+<code>https://example.com/login</code> with a session that expires before
+our sample project scrape job finish, causing all remaining webpages to
+respond <code>403</code> HTTP response code and fail… quite the problem
+isn&#39;t it? Well, not anymore, <code>ae_easy-login</code> gem to the
+rescue!</p>
+
+<p>First, let&#39;s create our base module that will contain our session
+validation and recovery logic, for this example, we will call it
+<code>LoginEnable</code> :</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./lib/login_enable.rb
+</span>
+<span class='kw'>module</span> <span class='const'>LoginEnable</span>
+  <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login.html" title="AeEasy::Login (module)">Login</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Plugin.html" title="AeEasy::Login::Plugin (module)">Plugin</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Plugin/EnabledBehavior.html" title="AeEasy::Login::Plugin::EnabledBehavior (module)">EnabledBehavior</a></span></span>
+  
+  <span class='comment'># Hook to initialize login_flow configuration.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_initialize_hook_login_plugin_enabled_behavior'>initialize_hook_login_plugin_enabled_behavior</span> <span class='id identifier rubyid_opts'>opts</span> <span class='op'>=</span> <span class='lbrace'>{</span><span class='rbrace'>}</span>
+    <span class='id identifier rubyid_opts'>opts</span> <span class='op'>=</span> <span class='lbrace'>{</span><span class='label'>app_config:</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Config</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lparen'>(</span><span class='id identifier rubyid_opts'>opts</span><span class='rparen'>)</span><span class='rbrace'>}</span><span class='period'>.</span><span class='id identifier rubyid_merge'>merge</span> <span class='id identifier rubyid_opts'>opts</span>
+    <span class='ivar'>@login_flow</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login.html" title="AeEasy::Login (module)">Login</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Flow.html" title="AeEasy::Login::Flow (class)">Flow</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span> <span class='id identifier rubyid_opts'>opts</span>
+    <span class='ivar'>@cookie</span> <span class='op'>=</span> <span class='kw'>nil</span>
+  <span class='kw'>end</span>
+
+  <span class='comment'># Get cookie after applying response cookie.
+</span>  <span class='comment'># @return [String] Cookie string.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_cookie'>cookie</span>
+      <span class='kw'>return</span> <span class='ivar'>@cookie</span> <span class='kw'>if</span> <span class='ivar'>@cookie</span><span class='period'>.</span><span class='id identifier rubyid_nil?'>nil?</span>
+      
+      <span class='id identifier rubyid_raw_cookie'>raw_cookie</span> <span class='op'>=</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span> <span class='op'>||</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_headers</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Set-Cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span>
+      <span class='ivar'>@cookie</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Helper</span><span class='op'>::</span><span class='const'>Cookie</span><span class='period'>.</span><span class='id identifier rubyid_update'>update</span><span class='lparen'>(</span><span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='comma'>,</span> <span class='id identifier rubyid_raw_cookie'>raw_cookie</span><span class='rparen'>)</span>
+      <span class='ivar'>@cookie</span>
+    <span class='kw'>end</span>
+  
+  <span class='comment'># Validates session.
+</span>  <span class='comment'># @return [Boolean] `true` when session is valid, else `false`.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_valid_session?'>valid_session?</span>
+    <span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>200</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>404</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_include?'>include?</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_status_code</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_to_s'>to_s</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+  <span class='kw'>end</span>
+
+  <span class='comment'># Fix page session when session is invalid.
+</span>  <span class='comment'># @return [Boolean] `true` when session is valid, else `false`.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+    <span class='kw'>return</span> <span class='kw'>true</span> <span class='kw'>if</span> <span class='id identifier rubyid_valid_session?'>valid_session?</span>
+    
+    <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_session'>fix_session</span> <span class='kw'>do</span>
+      <span class='id identifier rubyid_save_pages'>save_pages</span> <span class='lbracket'>[</span><span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>9</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>freshness</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='const'>Time</span><span class='period'>.</span><span class='id identifier rubyid_now'>now</span><span class='period'>.</span><span class='id identifier rubyid_iso8601'>iso8601</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>stl=</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_salt'>salt</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span>
+          <span class='comment'># Add any extra header you need here
+</span>          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>stl=</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_salt'>salt</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span>
+        <span class='rbrace'>}</span>
+      <span class='rbrace'>}</span><span class='rbracket'>]</span>
+    <span class='kw'>end</span>
+    
+    <span class='kw'>false</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Notice that our example <code>valid_session</code> method uses
+<code>200</code> and <code>404</code> HTTP response codes to validate that
+our session hasn&#39;t expired yet, therefore, <strong><em>this might not
+be the case for your website</em></strong>, so make sure to modify this
+method to fit your needs.</p>
+
+<p>Our <code>fix_session</code> method will store any page with a failed
+session by creating an output so it can be restored later once we have the
+new active session cookie.</p>
+
+<p><code>fix_session</code> method will also mark the current session cookie
+as expired and <strong><em>enqueue a new <code>login</code> page with HIGH
+priority as long as another parser hasn&#39;t already did it to avoid
+duplicates</em></strong>.</p>
+
+<p><code>cookie</code> method will merge the request cookies with the response
+cookies, so we can be sure that the cookies are always updated when needed.</p>
+
+<p>Next step is to create a simple parser that enqueue the <code>POST</code>
+of our login page:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/login.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Login</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>http://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login_post</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>10</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>method</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>POST</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_cookie'>cookie</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span>
+          <span class='comment'># Add any extra header you need here
+</span>          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_cookie'>cookie</span>
+        <span class='rbrace'>}</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Now let&#39;s handle the login response, seed and restore any page with an
+expired session:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/login_post.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>LoginPost</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed!'>seed!</span>
+      <span class='kw'>return</span> <span class='kw'>if</span> <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_seeded?'>seeded?</span>
+
+      <span class='const'>Seeders</span><span class='op'>::</span><span class='const'>Seeder</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lparen'>(</span><span class='label'>context:</span> <span class='id identifier rubyid_context'>context</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_seed'>seed</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_new_page'>new_page</span><span class='op'>|</span>
+        <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_page!'>fix_page!</span> <span class='id identifier rubyid_new_page'>new_page</span>
+      <span class='kw'>end</span>
+      
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_seeded!'>seeded!</span>
+    <span class='kw'>end</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_update_config'>update_config</span><span class='lparen'>(</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_get_cookie'>get_cookie</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>expired</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='kw'>false</span>
+      <span class='rparen'>)</span>
+
+      <span class='comment'># Wait for any pending fetch to be hold
+</span>      <span class='id identifier rubyid_sleep'>sleep</span> <span class='int'>10</span>
+
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_restore_held_pages'>restore_held_pages</span>
+      <span class='id identifier rubyid_seed!'>seed!</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Notice something interesting? that&#39;s right, the seeding happens
+<strong>AFTER</strong> we got our new active session cookie, so the pages
+we seed includes the session cookie. We use
+<code>login_flow.fix_page!</code> method to add our latest active session
+cookie along some internal <code>page['vars']</code> (used to handle page
+recovery) to our seeded pages.</p>
+
+<p><strong>IMPORTANT:</strong> This example assumes that
+<code>login_post</code> pages will never fails, but you might need to add
+some extra validations to make sure the login attempt was successful before
+restoring your pages.</p>
+
+<p><strong><em>Note:</em></strong> This example assumes that all pages to be
+seeded requires an active session, so we will add it to all pages we seed,
+but this will likely not apply to all pages to be seeded in a real life
+scenario, so make sure to add it only to those pages that requires an
+active session.</p>
+
+<p>So next step is to modify our seeder so it allow the cookie inclusion by
+adding a <code>block</code> param that will be used by our
+<code>Parsers::LoginPost#seed!</code> method:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./seeder/seeder.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Seeder</span>
+  <span class='kw'>class</span> <span class='const'>Seeder</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Seeder</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed'>seed</span> <span class='op'>&amp;</span><span class='id identifier rubyid_block'>block</span>
+      <span class='id identifier rubyid_new_page'>new_page</span> <span class='op'>=</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login.rb?query=food</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>search</span><span class='tstring_end'>&#39;</span></span>
+      <span class='rbrace'>}</span>
+      <span class='id identifier rubyid_block'>block</span><span class='period'>.</span><span class='id identifier rubyid_call'>call</span><span class='lparen'>(</span><span class='id identifier rubyid_page'>page</span><span class='rparen'>)</span> <span class='kw'>unless</span> <span class='id identifier rubyid_block'>block</span><span class='period'>.</span><span class='id identifier rubyid_nil?'>nil?</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_new_page'>new_page</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Now we will need to create a new seeder to seed login page:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./seeder/login.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Seeder</span>
+  <span class='kw'>class</span> <span class='const'>Login</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Seeder</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed'>seed</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>9</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Now let&#39;s modify our <code>./config.yaml</code> to add our new page
+types on it, as well as let us parse failed fetched pages since our example
+assumes that website will return <code>403</code> HTTP response code when
+session has expired:</p>
+
+<pre class="code ruby"><code class="ruby"># ./config.yaml
+
+parse_failed_pages: true
+
+seeder:
+  file: ./router/seeder.rb
+  disabled: false
+
+parsers:
+  - page_type: search
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: product
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: login
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: login_post
+    file: ./router/parser.rb
+    disabled: false
+</code></pre>
+
+<p>And don&#39;t forget to modify <code>./ae_easy.yaml</code> to add our new
+routes and change our seeder so login page can be seed first instead of our
+old seeder:</p>
+
+<pre class="code ruby"><code class="ruby"># ./ae_easy.yaml
+
+router:
+  parser:
+    routes:
+      - page_type: search
+        class: Parsers::Search
+      - page_type: product
+        class: Parsers::Product
+      - page_type: login
+        class: Parsers::Login
+      - page_type: login_post
+        class: Parsers::LoginPost
+
+  seeder:
+    routes:
+      - class: Seeder::Login
+</code></pre>
+
+<p>Now, let&#39;s will need to modify our routers as well since we modified
+our <code>ae_easy.yaml</code> routes and added new classes:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./router/seeder.rb
+</span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/router</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./seeder/login</span><span class='tstring_end'>&#39;</span></span>
+
+<span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Router</span><span class='op'>::</span><span class='const'>Seeder</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='period'>.</span><span class='id identifier rubyid_route'>route</span> <span class='label'>context:</span> <span class='kw'>self</span>
+</code></pre>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./router/parser.rb
+</span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cgi</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/router</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/login</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./lib/login_enable</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./seeder/seeder</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/search</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/product</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/login</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/login_post</span><span class='tstring_end'>&#39;</span></span>
+
+<span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Router</span><span class='op'>::</span><span class='const'>Parser</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='period'>.</span><span class='id identifier rubyid_route'>route</span> <span class='label'>context:</span> <span class='kw'>self</span>
+</code></pre>
+
+<p>Next, we need to include our <code>LoginEnable</code> module on every
+parser that requires session validation to fix any expired session request.
+To do this, we will be using our <code>LoginEnable#fix_session</code>
+function as the first thing to do on each parser&#39;s <code>parse</code>
+method:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/search.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Search</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+      
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.name</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_element'>element</span><span class='op'>|</span>
+        <span class='id identifier rubyid_name'>name</span> <span class='op'>=</span> <span class='id identifier rubyid_element'>element</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+        <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>https://example.com/product/</span><span class='embexpr_beg'>#{</span><span class='const'>CGI</span><span class='op'>::</span><span class='id identifier rubyid_escape'>escape</span> <span class='id identifier rubyid_name'>name</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_name'>name</span><span class='rbrace'>}</span>
+        <span class='rbrace'>}</span>
+      <span class='kw'>end</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/product.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Product</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_description'>description</span> <span class='op'>=</span> <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.description</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+      <span class='id identifier rubyid_outputs'>outputs</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>_collection</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>description</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_description'>description</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p><strong><em>Note:</em></strong> This example asumes that all pages requires
+an active session, so we will add it to all parsers, but this will likely
+not apply to all parsers in a real life scenario since not all web pages
+will require session, so make sure to add it to only the parsers that needs
+it.</p>
+
+<p>Finally, we need to make sure that every page that requires an active
+session is enqueued within our latest active session cookie, so we need to
+use <code>login_flow.fix_page!</code> method on all pages to be enqueued
+that applies.</p>
+
+<p>As for this example, we already add it to our search pages enqueued by our
+seeder, so the only place left to modify is
+<code>./parsers/search.rb</code> parser since it enqueues
+<code>product</code> pages:</p>
+
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/search.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Search</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+      
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.name</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_element'>element</span><span class='op'>|</span>
+        <span class='id identifier rubyid_name'>name</span> <span class='op'>=</span> <span class='id identifier rubyid_element'>element</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+        <span class='id identifier rubyid_new_page'>new_page</span> <span class='op'>=</span> <span class='lbrace'>{</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>https://example.com/product/</span><span class='embexpr_beg'>#{</span><span class='const'>CGI</span><span class='op'>::</span><span class='id identifier rubyid_escape'>escape</span> <span class='id identifier rubyid_name'>name</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_name'>name</span><span class='rbrace'>}</span>
+        <span class='rbrace'>}</span>
+        <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_page!'>fix_page!</span> <span class='id identifier rubyid_new_page'>new_page</span>
+        <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_new_page'>new_page</span>
+      <span class='kw'>end</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+
+<p>Hurray! Now you have implemented a fully functional login flow with auto
+recovery capabilities on your project.</p>
 </div></div>
 
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:55 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>

data/doc/top-level-namespace.html

@@ -100,7 +100,7 @@
 </div>
 
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:55 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>

data/lib/ae_easy/login/flow.rb

@@ -230,11 +230,13 @@ module AeEasy
       # @param [Hash] held_page Held page to fix.
       def fix_page! held_page
         clean_page_response! held_page
-        held_page['cookie'] = merge_cookie held_page['cookie']
-        held_page['headers'] = {} unless held_page.has_key? 'headers'
-        if held_page['headers'].has_key? 'Cookie'
-          held_page['headers']['Cookie'] = merge_cookie held_page['headers']['Cookie']
-        end
+        cookie_key = held_page.has_key?(:cookie) ? :cookie : 'cookie'
+        headers_key = held_page.has_key?(:headers) ? :headers : 'headers'
+        held_page[cookie_key] = merge_cookie held_page[cookie_key]
+        held_page[headers_key] = {} unless held_page.has_key? headers_key
+        header_cookie_key = held_page[headers_key].has_key?('cookie') ? 'cookie' : 'Cookie'
+        held_page[headers_key][header_cookie_key] = '' unless  held_page[headers_key].has_key? header_cookie_key
+        held_page[headers_key][header_cookie_key] = merge_cookie held_page[headers_key][header_cookie_key]
         add_vars! held_page
         custom_fix! held_page
       end