gcloud 0.0.4 → 0.0.6

data/packages/gsutil/boto/docs/source/cloudsearch_tut.rst

@@ -0,0 +1,411 @@
+.. cloudsearch_tut:
+
+===============================================
+An Introduction to boto's Cloudsearch interface
+===============================================
+
+This tutorial focuses on the boto interface to AWS' Cloudsearch_. This tutorial
+assumes that you have boto already downloaded and installed.
+
+.. _Cloudsearch: http://aws.amazon.com/cloudsearch/
+
+Creating a Connection
+---------------------
+The first step in accessing CloudSearch is to create a connection to the service.
+
+The recommended method of doing this is as follows::
+
+    >>> import boto.cloudsearch
+    >>> conn = boto.cloudsearch.connect_to_region("us-east-1", aws_access_key_id= '<aws access key'>, aws_secret_access_key='<aws secret key>')
+
+At this point, the variable conn will point to a CloudSearch connection object
+in the us-east-1 region. Currently, this is the only region which has the
+CloudSearch service. In this example, the AWS access key and AWS secret key are
+passed in to the method explicitly. Alternatively, you can set the environment
+variables:
+
+* `AWS_ACCESS_KEY_ID` - Your AWS Access Key ID
+* `AWS_SECRET_ACCESS_KEY` - Your AWS Secret Access Key
+
+and then simply call::
+
+   >>> import boto.cloudsearch
+   >>> conn = boto.cloudsearch.connect_to_region("us-east-1")
+
+In either case, conn will point to the Connection object which we will use
+throughout the remainder of this tutorial.
+
+Creating a Domain
+-----------------
+
+Once you have a connection established with the CloudSearch service, you will
+want to create a domain. A domain encapsulates the data that you wish to index,
+as well as indexes and metadata relating to it.
+
+    >>> from boto.cloudsearch.domain import Domain
+    >>> domain = Domain(conn, conn.create_domain('demo'))
+
+This domain can be used to control access policies, indexes, and the actual
+document service, which you will use to index and search.
+
+Setting access policies
+-----------------------
+
+Before you can connect to a document service, you need to set the correct access properties.
+For example, if you were connecting from 192.168.1.0, you could give yourself access as follows:
+
+    >>> our_ip = '192.168.1.0'
+
+    >>> # Allow our IP address to access the document and search services
+    >>> policy = domain.get_access_policies()
+    >>> policy.allow_search_ip(our_ip)
+    >>> policy.allow_doc_ip(our_ip)
+
+You can use the allow_search_ip() and allow_doc_ip() methods to give different
+CIDR blocks access to searching and the document service respectively.
+
+Creating index fields
+---------------------
+
+Each domain can have up to twenty index fields which are indexed by the
+CloudSearch service. For each index field, you will need to specify whether
+it's a text or integer field, as well as optionaly a default value.
+
+    >>> # Create an 'text' index field called 'username'
+    >>> uname_field = domain.create_index_field('username', 'text')
+
+    >>> # Epoch time of when the user last did something
+    >>> time_field = domain.create_index_field('last_activity', 'uint', default=0)
+
+It is also possible to mark an index field as a facet. Doing so allows a search
+query to return categories into which results can be grouped, or to create
+drill-down categories
+   
+    >>> # But it would be neat to drill down into different countries    
+    >>> loc_field = domain.create_index_field('location', 'text', facet=True)
+
+Finally, you can also mark a snippet of text as being able to be returned
+directly in your search query by using the results option.
+
+    >>> # Directly insert user snippets in our results
+    >>> snippet_field = domain.create_index_field('snippet', 'text', result=True)
+
+You can add up to 20 index fields in this manner:
+
+    >>> follower_field = domain.create_index_field('follower_count', 'uint', default=0)
+
+Adding Documents to the Index
+-----------------------------
+
+Now, we can add some documents to our new search domain. First, you will need a
+document service object through which queries are sent:
+
+    >>> doc_service = domain.get_document_service()
+
+For this example, we will use a pre-populated list of sample content for our
+import. You would normally pull such data from your database or another
+document store.
+
+    >>> users = [
+        {
+            'id': 1,
+            'username': 'dan',
+            'last_activity': 1334252740,
+            'follower_count': 20,
+            'location': 'USA',
+            'snippet': 'Dan likes watching sunsets and rock climbing',
+        },
+        {
+            'id': 2,
+            'username': 'dankosaur',
+            'last_activity': 1334252904,
+            'follower_count': 1,
+            'location': 'UK',
+            'snippet': 'Likes to dress up as a dinosaur.',
+        },
+        {
+            'id': 3,
+            'username': 'danielle',
+            'last_activity': 1334252969,
+            'follower_count': 100,
+            'location': 'DE',
+            'snippet': 'Just moved to Germany!'
+        },
+        {
+            'id': 4,
+            'username': 'daniella',
+            'last_activity': 1334253279,
+            'follower_count': 7,
+            'location': 'USA',
+            'snippet': 'Just like Dan, I like to watch a good sunset, but heights scare me.',
+        }
+    ]
+
+When adding documents to our document service, we will batch them together. You
+can schedule a document to be added by using the add() method. Whenever you are
+adding a document, you must provide a unique ID, a version ID, and the actual
+document to be indexed. In this case, we are using the user ID as our unique
+ID. The version ID is used to determine which is the latest version of an
+object to be indexed. If you wish to update a document, you must use a higher
+version ID. In this case, we are using the time of the user's last activity as
+a version number.
+
+    >>> for user in users:
+    >>>     doc_service.add(user['id'], user['last_activity'], user)
+
+When you are ready to send the batched request to the document service, you can
+do with the commit() method. Note that cloudsearch will charge per 1000 batch
+uploads. Each batch upload must be under 5MB.
+
+    >>> result = doc_service.commit() 
+
+The result is an instance of `cloudsearch.CommitResponse` which will
+make the plain dictionary response a nice object (ie result.adds,
+result.deletes) and raise an exception for us if all of our documents
+weren't actually committed.
+
+After you have successfully committed some documents to cloudsearch, you must
+use :py:meth:`clear_sdf
+<boto.cloudsearch.document.DocumentServiceConnection.clear_sdf>`, if you wish
+to use the same document service connection again so that its internal cache is
+cleared.
+
+Searching Documents
+-------------------
+
+Now, let's try performing a search. First, we will need a SearchServiceConnection:
+
+    >>> search_service = domain.get_search_service()
+
+A standard search will return documents which contain the exact words being
+searched for.
+
+    >>> results = search_service.search(q="dan")
+    >>> results.hits
+    2
+    >>> map(lambda x: x['id'], results)
+    [u'1', u'4']
+
+The standard search does not look at word order:
+
+    >>> results = search_service.search(q="dinosaur dress")
+    >>> results.hits
+    1
+    >>> map(lambda x: x['id'], results)
+    [u'2']
+
+It's also possible to do more complex queries using the bq argument (Boolean
+Query). When you are using bq, your search terms must be enclosed in single
+quotes.
+
+    >>> results = search_service.search(bq="'dan'")
+    >>> results.hits
+    2
+    >>> map(lambda x: x['id'], results)
+    [u'1', u'4']
+
+When you are using boolean queries, it's also possible to use wildcards to
+extend your search to all words which start with your search terms:
+
+    >>> results = search_service.search(bq="'dan*'")
+    >>> results.hits
+    4
+    >>> map(lambda x: x['id'], results)
+    [u'1', u'2', u'3', u'4']
+
+The boolean query also allows you to create more complex queries. You can OR
+term together using "|", AND terms together using "+" or a space, and you can
+remove words from the query using the "-" operator.
+
+    >>> results = search_service.search(bq="'watched|moved'")
+    >>> results.hits
+    2
+    >>> map(lambda x: x['id'], results)
+    [u'3', u'4']
+
+By default, the search will return 10 terms but it is possible to adjust this
+by using the size argument as follows:
+
+    >>> results = search_service.search(bq="'dan*'", size=2)
+    >>> results.hits
+    4
+    >>> map(lambda x: x['id'], results)
+    [u'1', u'2']
+
+It is also possible to offset the start of the search by using the start argument as follows:
+
+    >>> results = search_service.search(bq="'dan*'", start=2)
+    >>> results.hits
+    4
+    >>> map(lambda x: x['id'], results)
+    [u'3', u'4']
+
+
+Ordering search results and rank expressions
+--------------------------------------------
+
+If your search query is going to return many results, it is good to be able to sort them 
+You can order your search results by using the rank argument. You are able to
+sort on any fields which have the results option turned on.
+
+    >>> results = search_service.search(bq=query, rank=['-follower_count'])
+
+You can also create your own rank expressions to sort your results according to
+other criteria:
+
+    >>> domain.create_rank_expression('recently_active', 'last_activity')  # We'll want to be able to just show the most recently active users
+    
+    >>> domain.create_rank_expression('activish', 'text_relevance + ((follower_count/(time() - last_activity))*1000)')  # Let's get trickier and combine text relevance with a really dynamic expression
+
+    >>> results = search_service.search(bq=query, rank=['-recently_active'])
+
+Viewing and Adjusting Stemming for a Domain
+-------------------------------------------
+
+A stemming dictionary maps related words to a common stem. A stem is
+typically the root or base word from which variants are derived. For
+example, run is the stem of running and ran. During indexing, Amazon
+CloudSearch uses the stemming dictionary when it performs
+text-processing on text fields. At search time, the stemming
+dictionary is used to perform text-processing on the search
+request. This enables matching on variants of a word. For example, if
+you map the term running to the stem run and then search for running,
+the request matches documents that contain run as well as running.
+
+To get the current stemming dictionary defined for a domain, use the
+``get_stemming`` method of the Domain object.
+
+    >>> stems = domain.get_stemming()
+    >>> stems
+    {u'stems': {}}
+    >>>
+
+This returns a dictionary object that can be manipulated directly to
+add additional stems for your search domain by adding pairs of term:stem
+to the stems dictionary.
+
+    >>> stems['stems']['running'] = 'run'
+    >>> stems['stems']['ran'] = 'run'
+    >>> stems
+    {u'stems': {u'ran': u'run', u'running': u'run'}}
+    >>>
+
+This has changed the value locally.  To update the information in
+Amazon CloudSearch, you need to save the data.
+
+    >>> stems.save()
+
+You can also access certain CloudSearch-specific attributes related to
+the stemming dictionary defined for your domain.
+
+    >>> stems.status
+    u'RequiresIndexDocuments'
+    >>> stems.creation_date
+    u'2012-05-01T12:12:32Z'
+    >>> stems.update_date
+    u'2012-05-01T12:12:32Z'
+    >>> stems.update_version
+    19
+    >>>
+
+The status indicates that, because you have changed the stems associated
+with the domain, you will need to re-index the documents in the domain
+before the new stems are used.
+
+Viewing and Adjusting Stopwords for a Domain
+--------------------------------------------
+
+Stopwords are words that should typically be ignored both during
+indexing and at search time because they are either insignificant or
+so common that including them would result in a massive number of
+matches.
+
+To view the stopwords currently defined for your domain, use the
+``get_stopwords`` method of the Domain object.
+
+    >>> stopwords = domain.get_stopwords()
+    >>> stopwords
+    {u'stopwords': [u'a',
+     u'an',
+     u'and',
+     u'are',
+     u'as',
+     u'at',
+     u'be',
+     u'but',
+     u'by',
+     u'for',
+     u'in',
+     u'is',
+     u'it',
+     u'of',
+     u'on',
+     u'or',
+     u'the',
+     u'to',
+     u'was']}
+     >>>
+
+You can add additional stopwords by simply appending the values to the
+list.
+
+    >>> stopwords['stopwords'].append('foo')
+    >>> stopwords['stopwords'].append('bar')
+    >>> stopwords
+
+Similarly, you could remove currently defined stopwords from the list.
+To save the changes, use the ``save`` method.
+
+    >>> stopwords.save()
+
+The stopwords object has similar attributes defined above for stemming
+that provide additional information about the stopwords in your domain.
+
+
+Viewing and Adjusting Stopwords for a Domain
+--------------------------------------------
+
+You can configure synonyms for terms that appear in the data you are
+searching. That way, if a user searches for the synonym rather than
+the indexed term, the results will include documents that contain the
+indexed term.
+
+If you want two terms to match the same documents, you must define
+them as synonyms of each other. For example:
+
+    cat, feline
+    feline, cat
+
+To view the synonyms currently defined for your domain, use the
+``get_synonyms`` method of the Domain object.
+
+    >>> synonyms = domain.get_synonyms()
+    >>> synonyms
+    {u'synonyms': {}}
+    >>>
+
+You can define new synonyms by adding new term:synonyms entries to the
+synonyms dictionary object.
+
+    >>> synonyms['synonyms']['cat'] = ['feline', 'kitten']
+    >>> synonyms['synonyms']['dog'] = ['canine', 'puppy']
+
+To save the changes, use the ``save`` method.
+
+    >>> synonyms.save()
+
+The synonyms object has similar attributes defined above for stemming
+that provide additional information about the stopwords in your domain.
+
+Deleting Documents
+------------------
+
+    >>> import time
+    >>> from datetime import datetime
+
+    >>> doc_service = domain.get_document_service()
+
+    >>> # Again we'll cheat and use the current epoch time as our version number
+     
+    >>> doc_service.delete(4, int(time.mktime(datetime.utcnow().timetuple())))
+    >>> service.commit()

data/packages/gsutil/boto/docs/source/cloudwatch_tut.rst

@@ -0,0 +1,116 @@
+.. cloudwatch_tut:
+
+==========
+CloudWatch
+==========
+
+First, make sure you have something to monitor.  You can either create a
+LoadBalancer or enable monitoring on an existing EC2 instance.  To enable
+monitoring, you can either call the monitor_instance method on the
+EC2Connection object or call the monitor method on the Instance object.
+
+It takes a while for the monitoring data to start accumulating but once
+it does, you can do this::
+
+    >>> import boto
+    >>> c = boto.connect_cloudwatch()
+    >>> metrics = c.list_metrics()
+    >>> metrics
+    [Metric:NetworkIn,
+     Metric:NetworkOut,
+     Metric:NetworkOut(InstanceType,m1.small),
+     Metric:NetworkIn(InstanceId,i-e573e68c),
+     Metric:CPUUtilization(InstanceId,i-e573e68c),
+     Metric:DiskWriteBytes(InstanceType,m1.small),
+     Metric:DiskWriteBytes(ImageId,ami-a1ffb63),
+     Metric:NetworkOut(ImageId,ami-a1ffb63),
+     Metric:DiskWriteOps(InstanceType,m1.small),
+     Metric:DiskReadBytes(InstanceType,m1.small),
+     Metric:DiskReadOps(ImageId,ami-a1ffb63),
+     Metric:CPUUtilization(InstanceType,m1.small),
+     Metric:NetworkIn(ImageId,ami-a1ffb63),
+     Metric:DiskReadOps(InstanceType,m1.small),
+     Metric:DiskReadBytes,
+     Metric:CPUUtilization,
+     Metric:DiskWriteBytes(InstanceId,i-e573e68c),
+     Metric:DiskWriteOps(InstanceId,i-e573e68c),
+     Metric:DiskWriteOps,
+     Metric:DiskReadOps,
+     Metric:CPUUtilization(ImageId,ami-a1ffb63),
+     Metric:DiskReadOps(InstanceId,i-e573e68c),
+     Metric:NetworkOut(InstanceId,i-e573e68c),
+     Metric:DiskReadBytes(ImageId,ami-a1ffb63),
+     Metric:DiskReadBytes(InstanceId,i-e573e68c),
+     Metric:DiskWriteBytes,
+     Metric:NetworkIn(InstanceType,m1.small),
+     Metric:DiskWriteOps(ImageId,ami-a1ffb63)]
+
+The list_metrics call will return a list of all of the available metrics
+that you can query against.  Each entry in the list is a Metric object.
+As you can see from the list above, some of the metrics are generic metrics
+and some have Dimensions associated with them (e.g. InstanceType=m1.small).
+The Dimension can be used to refine your query.  So, for example, I could
+query the metric Metric:CPUUtilization which would create the desired statistic
+by aggregating cpu utilization data across all sources of information available
+or I could refine that by querying the metric
+Metric:CPUUtilization(InstanceId,i-e573e68c) which would use only the data
+associated with the instance identified by the instance ID i-e573e68c.
+
+Because for this example, I'm only monitoring a single instance, the set
+of metrics available to me are fairly limited.  If I was monitoring many
+instances, using many different instance types and AMI's and also several
+load balancers, the list of available metrics would grow considerably.
+
+Once you have the list of available metrics, you can actually
+query the CloudWatch system for that metric.  Let's choose the CPU utilization
+metric for our instance.::
+
+    >>> m = metrics[5]
+    >>> m
+    Metric:CPUUtilization(InstanceId,i-e573e68c)
+
+The Metric object has a query method that lets us actually perform
+the query against the collected data in CloudWatch.  To call that,
+we need a start time and end time to control the time span of data
+that we are interested in.  For this example, let's say we want the
+data for the previous hour::
+
+    >>> import datetime
+    >>> end = datetime.datetime.now()
+    >>> start = end - datetime.timedelta(hours=1)
+
+We also need to supply the Statistic that we want reported and
+the Units to use for the results.  The Statistic can be one of these
+values::
+
+    ['Minimum', 'Maximum', 'Sum', 'Average', 'SampleCount']
+
+And Units must be one of the following::
+
+    ['Seconds', 'Percent', 'Bytes', 'Bits', 'Count',
+    'Bytes/Second', 'Bits/Second', 'Count/Second']
+
+The query method also takes an optional parameter, period.  This
+parameter controls the granularity (in seconds) of the data returned.
+The smallest period is 60 seconds and the value must be a multiple
+of 60 seconds.  So, let's ask for the average as a percent::
+
+    >>> datapoints = m.query(start, end, 'Average', 'Percent')
+    >>> len(datapoints)
+    60
+
+Our period was 60 seconds and our duration was one hour so
+we should get 60 data points back and we can see that we did.
+Each element in the datapoints list is a DataPoint object
+which is a simple subclass of a Python dict object.  Each
+Datapoint object contains all of the information available
+about that particular data point.::
+
+    >>> d = datapoints[0]
+    >>> d
+    {u'Average': 0.0,
+     u'SampleCount': 1.0,
+     u'Timestamp': u'2009-05-21T19:55:00Z',
+     u'Unit': u'Percent'}
+
+My server obviously isn't very busy right now!