RubyGems - ae_easy-login - Versions diffs - 0.0.2 → 0.0.3 - Mend

ae_easy-login 0.0.2 → 0.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (14) hide show

checksums.yaml +4 -4
data/README.md +374 -3
data/doc/AeEasy.html +1 -1
data/doc/AeEasy/Login.html +2 -2
data/doc/AeEasy/Login/Flow.html +19 -15
data/doc/AeEasy/Login/Plugin.html +1 -1
data/doc/AeEasy/Login/Plugin/EnabledBehavior.html +1 -1
data/doc/_index.html +1 -1
data/doc/file.README.html +426 -4
data/doc/index.html +426 -4
data/doc/top-level-namespace.html +1 -1
data/lib/ae_easy/login/flow.rb +7 -5
data/lib/ae_easy/login/version.rb +1 -1
metadata +2 -2

data/doc/AeEasy/Login/Plugin.html CHANGED

@@ -107,7 +107,7 @@
 </div>
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:56 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>

data/doc/AeEasy/Login/Plugin/EnabledBehavior.html CHANGED

@@ -580,7 +580,7 @@ AeEasy::Login::Flow#initialize_hook_login_flow).</p>
 </div>
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:56 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>

data/doc/_index.html CHANGED

@@ -157,7 +157,7 @@
 </div>
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:55 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>

data/doc/file.README.html CHANGED

@@ -73,16 +73,438 @@ src="http://img.shields.io/badge/license-MIT-yellowgreen.svg"></a></p>
 handle login and session recovery, quite useful when scraping websites with
 login features and expiring sessions.</p>
-<p>Install gem: <code>gem install &#39;ae_easy-login&#39;</code></p>
+<p>Install gem: <code>ruby gem install &#39;ae_easy-login&#39; </code></p>
-<p>Require gem: <code>require &#39;ae_easy/login&#39;</code></p>
+<p>Require gem: <code>ruby  require &#39;ae_easy/login&#39; </code></p>
-<p>Documentation can be found <a
+<p>Code documentation can be found <a
 href="http://rubydoc.org/gems/ae_easy-login/frames">here</a>.</p>
+<h2 id="label-How+to+implement">How to implement</h2>
+<h3 id="label-Before+you+start">Before you start</h3>
+<p>It is true that most user cases for <code>ae_easy-login</code> gem applies
+to websites with login pages and create sessions, so we will cover this
+scenario on our example.</p>
+<p>Therefore, <code>ae_easy-login</code> gem is designed to handle
+<strong>ANY</strong> kind of session recovery, even those that doesn&#39;t
+requires a login form <code>POST</code> by just changing the flow from:</p>
+<pre class="code ruby"><code class="ruby">login -&gt; login_post -&gt; restore
+</code></pre>
+<p>To whatever you need like for example:</p>
+<pre class="code ruby"><code class="ruby">home -&gt; search_page -&gt; restore
+</code></pre>
+<p>Here are some user case examples that can be fixed by
+<code>ae_easy-login</code> gem:</p>
+<ul><li>
+<p>Websites that invalidate requests with fast expiring cookies created on
+first request.</p>
+</li><li>
+<p>Websites that generates tokens on every search (either on cookies or
+query_params) that are required to fetch a detail page.</p>
+</li><li>
+<p>Websites that expires session due inactivity.</p>
+</li><li>
+<p>Websites that uses complex login flows.</p>
+</li><li>
+<p>etc.</p>
+</li></ul>
+<p>Feel confident to expirement with it until it fit all your needs.</p>
+<h3 id="label-Adding+ae_easy-login+to+your+project">Adding ae_easy-login to your project</h3>
+<p>Let&#39;s assume a simple project implementing <code>ae_easy</code> like
+the one described on <a
+href="https://github.com/answersengine/ae_easy/blob/master/README.md">ae_easy
+README.md</a> that scrapers your website.</p>
+<p>Now lets assume your website has a login page
+<code>https://example.com/login</code> with a session that expires before
+our sample project scrape job finish, causing all remaining webpages to
+respond <code>403</code> HTTP response code and fail… quite the problem
+isn&#39;t it? Well, not anymore, <code>ae_easy-login</code> gem to the
+rescue!</p>
+<p>First, let&#39;s create our base module that will contain our session
+validation and recovery logic, for this example, we will call it
+<code>LoginEnable</code> :</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./lib/login_enable.rb
+</span>
+<span class='kw'>module</span> <span class='const'>LoginEnable</span>
+  <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login.html" title="AeEasy::Login (module)">Login</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Plugin.html" title="AeEasy::Login::Plugin (module)">Plugin</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Plugin/EnabledBehavior.html" title="AeEasy::Login::Plugin::EnabledBehavior (module)">EnabledBehavior</a></span></span>
+  <span class='comment'># Hook to initialize login_flow configuration.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_initialize_hook_login_plugin_enabled_behavior'>initialize_hook_login_plugin_enabled_behavior</span> <span class='id identifier rubyid_opts'>opts</span> <span class='op'>=</span> <span class='lbrace'>{</span><span class='rbrace'>}</span>
+    <span class='id identifier rubyid_opts'>opts</span> <span class='op'>=</span> <span class='lbrace'>{</span><span class='label'>app_config:</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Config</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lparen'>(</span><span class='id identifier rubyid_opts'>opts</span><span class='rparen'>)</span><span class='rbrace'>}</span><span class='period'>.</span><span class='id identifier rubyid_merge'>merge</span> <span class='id identifier rubyid_opts'>opts</span>
+    <span class='ivar'>@login_flow</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login.html" title="AeEasy::Login (module)">Login</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="AeEasy/Login/Flow.html" title="AeEasy::Login::Flow (class)">Flow</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span> <span class='id identifier rubyid_opts'>opts</span>
+    <span class='ivar'>@cookie</span> <span class='op'>=</span> <span class='kw'>nil</span>
+  <span class='kw'>end</span>
+  <span class='comment'># Get cookie after applying response cookie.
+</span>  <span class='comment'># @return [String] Cookie string.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_cookie'>cookie</span>
+      <span class='kw'>return</span> <span class='ivar'>@cookie</span> <span class='kw'>if</span> <span class='ivar'>@cookie</span><span class='period'>.</span><span class='id identifier rubyid_nil?'>nil?</span>
+      <span class='id identifier rubyid_raw_cookie'>raw_cookie</span> <span class='op'>=</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span> <span class='op'>||</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_headers</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Set-Cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span>
+      <span class='ivar'>@cookie</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Helper</span><span class='op'>::</span><span class='const'>Cookie</span><span class='period'>.</span><span class='id identifier rubyid_update'>update</span><span class='lparen'>(</span><span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='comma'>,</span> <span class='id identifier rubyid_raw_cookie'>raw_cookie</span><span class='rparen'>)</span>
+      <span class='ivar'>@cookie</span>
+    <span class='kw'>end</span>
+  <span class='comment'># Validates session.
+</span>  <span class='comment'># @return [Boolean] `true` when session is valid, else `false`.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_valid_session?'>valid_session?</span>
+    <span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>200</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>404</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_include?'>include?</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>response_status_code</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_to_s'>to_s</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+  <span class='kw'>end</span>
+  <span class='comment'># Fix page session when session is invalid.
+</span>  <span class='comment'># @return [Boolean] `true` when session is valid, else `false`.
+</span>  <span class='kw'>def</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+    <span class='kw'>return</span> <span class='kw'>true</span> <span class='kw'>if</span> <span class='id identifier rubyid_valid_session?'>valid_session?</span>
+    <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_session'>fix_session</span> <span class='kw'>do</span>
+      <span class='id identifier rubyid_save_pages'>save_pages</span> <span class='lbracket'>[</span><span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>9</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>freshness</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='const'>Time</span><span class='period'>.</span><span class='id identifier rubyid_now'>now</span><span class='period'>.</span><span class='id identifier rubyid_iso8601'>iso8601</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>stl=</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_salt'>salt</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span>
+          <span class='comment'># Add any extra header you need here
+</span>          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>stl=</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_salt'>salt</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span>
+        <span class='rbrace'>}</span>
+      <span class='rbrace'>}</span><span class='rbracket'>]</span>
+    <span class='kw'>end</span>
+    <span class='kw'>false</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Notice that our example <code>valid_session</code> method uses
+<code>200</code> and <code>404</code> HTTP response codes to validate that
+our session hasn&#39;t expired yet, therefore, <strong><em>this might not
+be the case for your website</em></strong>, so make sure to modify this
+method to fit your needs.</p>
+<p>Our <code>fix_session</code> method will store any page with a failed
+session by creating an output so it can be restored later once we have the
+new active session cookie.</p>
+<p><code>fix_session</code> method will also mark the current session cookie
+as expired and <strong><em>enqueue a new <code>login</code> page with HIGH
+priority as long as another parser hasn&#39;t already did it to avoid
+duplicates</em></strong>.</p>
+<p><code>cookie</code> method will merge the request cookies with the response
+cookies, so we can be sure that the cookies are always updated when needed.</p>
+<p>Next step is to create a simple parser that enqueue the <code>POST</code>
+of our login page:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/login.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Login</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>http://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login_post</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>10</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>method</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>POST</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_cookie'>cookie</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>headers</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span>
+          <span class='comment'># Add any extra header you need here
+</span>          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>Cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_cookie'>cookie</span>
+        <span class='rbrace'>}</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Now let&#39;s handle the login response, seed and restore any page with an
+expired session:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/login_post.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>LoginPost</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed!'>seed!</span>
+      <span class='kw'>return</span> <span class='kw'>if</span> <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_seeded?'>seeded?</span>
+      <span class='const'>Seeders</span><span class='op'>::</span><span class='const'>Seeder</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lparen'>(</span><span class='label'>context:</span> <span class='id identifier rubyid_context'>context</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_seed'>seed</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_new_page'>new_page</span><span class='op'>|</span>
+        <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_page!'>fix_page!</span> <span class='id identifier rubyid_new_page'>new_page</span>
+      <span class='kw'>end</span>
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_seeded!'>seeded!</span>
+    <span class='kw'>end</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_update_config'>update_config</span><span class='lparen'>(</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cookie</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_get_cookie'>get_cookie</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>expired</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='kw'>false</span>
+      <span class='rparen'>)</span>
+      <span class='comment'># Wait for any pending fetch to be hold
+</span>      <span class='id identifier rubyid_sleep'>sleep</span> <span class='int'>10</span>
+      <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_restore_held_pages'>restore_held_pages</span>
+      <span class='id identifier rubyid_seed!'>seed!</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Notice something interesting? that&#39;s right, the seeding happens
+<strong>AFTER</strong> we got our new active session cookie, so the pages
+we seed includes the session cookie. We use
+<code>login_flow.fix_page!</code> method to add our latest active session
+cookie along some internal <code>page['vars']</code> (used to handle page
+recovery) to our seeded pages.</p>
+<p><strong>IMPORTANT:</strong> This example assumes that
+<code>login_post</code> pages will never fails, but you might need to add
+some extra validations to make sure the login attempt was successful before
+restoring your pages.</p>
+<p><strong><em>Note:</em></strong> This example assumes that all pages to be
+seeded requires an active session, so we will add it to all pages we seed,
+but this will likely not apply to all pages to be seeded in a real life
+scenario, so make sure to add it only to those pages that requires an
+active session.</p>
+<p>So next step is to modify our seeder so it allow the cookie inclusion by
+adding a <code>block</code> param that will be used by our
+<code>Parsers::LoginPost#seed!</code> method:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./seeder/seeder.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Seeder</span>
+  <span class='kw'>class</span> <span class='const'>Seeder</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Seeder</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed'>seed</span> <span class='op'>&amp;</span><span class='id identifier rubyid_block'>block</span>
+      <span class='id identifier rubyid_new_page'>new_page</span> <span class='op'>=</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login.rb?query=food</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>search</span><span class='tstring_end'>&#39;</span></span>
+      <span class='rbrace'>}</span>
+      <span class='id identifier rubyid_block'>block</span><span class='period'>.</span><span class='id identifier rubyid_call'>call</span><span class='lparen'>(</span><span class='id identifier rubyid_page'>page</span><span class='rparen'>)</span> <span class='kw'>unless</span> <span class='id identifier rubyid_block'>block</span><span class='period'>.</span><span class='id identifier rubyid_nil?'>nil?</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_new_page'>new_page</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Now we will need to create a new seeder to seed login page:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./seeder/login.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Seeder</span>
+  <span class='kw'>class</span> <span class='const'>Login</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Seeder</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_seed'>seed</span>
+      <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>https://example.com/login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>login</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>priority</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='int'>9</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Now let&#39;s modify our <code>./config.yaml</code> to add our new page
+types on it, as well as let us parse failed fetched pages since our example
+assumes that website will return <code>403</code> HTTP response code when
+session has expired:</p>
+<pre class="code ruby"><code class="ruby"># ./config.yaml
+parse_failed_pages: true
+seeder:
+  file: ./router/seeder.rb
+  disabled: false
+parsers:
+  - page_type: search
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: product
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: login
+    file: ./router/parser.rb
+    disabled: false
+  - page_type: login_post
+    file: ./router/parser.rb
+    disabled: false
+</code></pre>
+<p>And don&#39;t forget to modify <code>./ae_easy.yaml</code> to add our new
+routes and change our seeder so login page can be seed first instead of our
+old seeder:</p>
+<pre class="code ruby"><code class="ruby"># ./ae_easy.yaml
+router:
+  parser:
+    routes:
+      - page_type: search
+        class: Parsers::Search
+      - page_type: product
+        class: Parsers::Product
+      - page_type: login
+        class: Parsers::Login
+      - page_type: login_post
+        class: Parsers::LoginPost
+  seeder:
+    routes:
+      - class: Seeder::Login
+</code></pre>
+<p>Now, let&#39;s will need to modify our routers as well since we modified
+our <code>ae_easy.yaml</code> routes and added new classes:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./router/seeder.rb
+</span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/router</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./seeder/login</span><span class='tstring_end'>&#39;</span></span>
+<span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Router</span><span class='op'>::</span><span class='const'>Seeder</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='period'>.</span><span class='id identifier rubyid_route'>route</span> <span class='label'>context:</span> <span class='kw'>self</span>
+</code></pre>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./router/parser.rb
+</span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>cgi</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/router</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>ae_easy/login</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./lib/login_enable</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./seeder/seeder</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/search</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/product</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/login</span><span class='tstring_end'>&#39;</span></span>
+<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>./parsers/login_post</span><span class='tstring_end'>&#39;</span></span>
+<span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Router</span><span class='op'>::</span><span class='const'>Parser</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='period'>.</span><span class='id identifier rubyid_route'>route</span> <span class='label'>context:</span> <span class='kw'>self</span>
+</code></pre>
+<p>Next, we need to include our <code>LoginEnable</code> module on every
+parser that requires session validation to fix any expired session request.
+To do this, we will be using our <code>LoginEnable#fix_session</code>
+function as the first thing to do on each parser&#39;s <code>parse</code>
+method:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/search.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Search</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.name</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_element'>element</span><span class='op'>|</span>
+        <span class='id identifier rubyid_name'>name</span> <span class='op'>=</span> <span class='id identifier rubyid_element'>element</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+        <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>https://example.com/product/</span><span class='embexpr_beg'>#{</span><span class='const'>CGI</span><span class='op'>::</span><span class='id identifier rubyid_escape'>escape</span> <span class='id identifier rubyid_name'>name</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_name'>name</span><span class='rbrace'>}</span>
+        <span class='rbrace'>}</span>
+      <span class='kw'>end</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/product.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Product</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_description'>description</span> <span class='op'>=</span> <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.description</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+      <span class='id identifier rubyid_outputs'>outputs</span> <span class='op'>&lt;&lt;</span> <span class='lbrace'>{</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>_collection</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_page'>page</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span><span class='rbracket'>]</span><span class='comma'>,</span>
+        <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>description</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_description'>description</span>
+      <span class='rbrace'>}</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p><strong><em>Note:</em></strong> This example asumes that all pages requires
+an active session, so we will add it to all parsers, but this will likely
+not apply to all parsers in a real life scenario since not all web pages
+will require session, so make sure to add it to only the parsers that needs
+it.</p>
+<p>Finally, we need to make sure that every page that requires an active
+session is enqueued within our latest active session cookie, so we need to
+use <code>login_flow.fix_page!</code> method on all pages to be enqueued
+that applies.</p>
+<p>As for this example, we already add it to our search pages enqueued by our
+seeder, so the only place left to modify is
+<code>./parsers/search.rb</code> parser since it enqueues
+<code>product</code> pages:</p>
+<pre class="code ruby"><code class="ruby"><span class='comment'># ./parsers/search.rb
+</span>
+<span class='kw'>module</span> <span class='const'>Parsers</span>
+  <span class='kw'>class</span> <span class='const'>Search</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'><span class='object_link'><a href="AeEasy.html" title="AeEasy (module)">AeEasy</a></span></span><span class='op'>::</span><span class='const'>Core</span><span class='op'>::</span><span class='const'>Plugin</span><span class='op'>::</span><span class='const'>Parser</span>
+    <span class='id identifier rubyid_include'>include</span> <span class='const'>LoginEnable</span>
+    <span class='kw'>def</span> <span class='id identifier rubyid_parse'>parse</span>
+      <span class='kw'>return</span> <span class='kw'>unless</span> <span class='id identifier rubyid_fix_session'>fix_session</span>
+      <span class='id identifier rubyid_html'>html</span> <span class='op'>=</span> <span class='const'>Nokogiri</span><span class='period'>.</span><span class='const'>HTML</span> <span class='id identifier rubyid_content'>content</span>
+      <span class='id identifier rubyid_html'>html</span><span class='period'>.</span><span class='id identifier rubyid_css'>css</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>.name</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_element'>element</span><span class='op'>|</span>
+        <span class='id identifier rubyid_name'>name</span> <span class='op'>=</span> <span class='id identifier rubyid_element'>element</span><span class='period'>.</span><span class='id identifier rubyid_text'>text</span><span class='period'>.</span><span class='id identifier rubyid_strip'>strip</span>
+        <span class='id identifier rubyid_new_page'>new_page</span> <span class='op'>=</span> <span class='lbrace'>{</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>url</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>https://example.com/product/</span><span class='embexpr_beg'>#{</span><span class='const'>CGI</span><span class='op'>::</span><span class='id identifier rubyid_escape'>escape</span> <span class='id identifier rubyid_name'>name</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page_type</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>product</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span>
+          <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>vars</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='lbrace'>{</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>name</span><span class='tstring_end'>&#39;</span></span> <span class='op'>=&gt;</span> <span class='id identifier rubyid_name'>name</span><span class='rbrace'>}</span>
+        <span class='rbrace'>}</span>
+        <span class='id identifier rubyid_login_flow'>login_flow</span><span class='period'>.</span><span class='id identifier rubyid_fix_page!'>fix_page!</span> <span class='id identifier rubyid_new_page'>new_page</span>
+        <span class='id identifier rubyid_pages'>pages</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_new_page'>new_page</span>
+      <span class='kw'>end</span>
+    <span class='kw'>end</span>
+  <span class='kw'>end</span>
+<span class='kw'>end</span>
+</code></pre>
+<p>Hurray! Now you have implemented a fully functional login flow with auto
+recovery capabilities on your project.</p>
 </div></div>
       <div id="footer">
-  Generated on Fri Sep 27 22:03:08 2019 by
+  Generated on Wed Oct 23 21:36:55 2019 by
   <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
   0.9.20 (ruby-2.5.3).
 </div>