gcloud 0.0.4 → 0.0.6

data/packages/gsutil/boto/docs/source/ec2_tut.rst

@@ -0,0 +1,86 @@
+.. _ec2_tut:
+
+=======================================
+An Introduction to boto's EC2 interface
+=======================================
+
+This tutorial focuses on the boto interface to the Elastic Compute Cloud
+from Amazon Web Services.  This tutorial assumes that you have already
+downloaded and installed boto.
+
+Creating a Connection
+---------------------
+
+The first step in accessing EC2 is to create a connection to the service.
+The recommended way of doing this in boto is::
+
+    >>> from boto.ec2
+    >>> conn = boto.ec2.connect_to_region("us-east-1",
+    ...    aws_access_key_id='<aws access key>',
+    ...    aws_secret_access_key='<aws secret key>')
+
+At this point the variable ``conn`` will point to an EC2Connection object.  In
+this example, the AWS access key and AWS secret key are passed in to the method
+explicitly.  Alternatively, you can set the boto config environment variables
+and then simply specify which region you want as follows::
+
+    >>> conn = boto.ec2.connect_to_region("us-east-1")
+
+In either case, conn will point to an EC2Connection object which we will
+use throughout the remainder of this tutorial.
+
+Launching Instances
+-------------------
+
+Possibly, the most important and common task you'll use EC2 for is to launch,
+stop and terminate instances. In its most primitive form, you can launch an
+instance as follows::
+
+    >>> conn.run_instances('<ami-image-id>')
+
+This will launch an instance in the specified region with the default parameters.
+You will not be able to SSH into this machine, as it doesn't have a security
+group set. See :doc:`security_groups` for details on creating one.
+
+Now, let's say that you already have a key pair, want a specific type of
+instance, and you have your :doc:`security group <security_groups>` all setup.
+In this case we can use the keyword arguments to accomplish that::
+
+    >>> conn.run_instances(
+            '<ami-image-id>',
+            key_name='myKey',
+            instance_type='c1.xlarge',
+            security_groups=['your-security-group-here'])
+
+The main caveat with the above call is that it is possible to request an
+instance type that is not compatible with the provided AMI (for example, the
+instance was created for a 64-bit instance and you choose a m1.small instance_type).
+For more details on the plethora of possible keyword parameters, be sure to
+check out boto's :doc:`EC2 API reference <ref/ec2>`.
+
+Stopping Instances
+------------------
+Once you have your instances up and running, you might wish to shut them down
+if they're not in use. Please note that this will only de-allocate virtual
+hardware resources (as well as instance store drives), but won't destroy your
+EBS volumes -- this means you'll pay nominal provisioned EBS storage fees
+even if your instance is stopped. To do this, you can do so as follows::
+
+    >>> conn.stop_instances(instance_ids=['instance-id-1','instance-id-2', ...])
+
+This will request a 'graceful' stop of each of the specified instances. If you
+wish to request the equivalent of unplugging your instance(s), simply add
+``force=True`` keyword argument to the call above. Please note that stop
+instance is not allowed with Spot instances.
+
+Terminating Instances
+---------------------
+Once you are completely done with your instance and wish to surrender both
+virtual hardware, root EBS volume and all other underlying components
+you can request instance termination. To do so you can use the call bellow::
+
+    >>> conn.terminate_instances(instance_ids=['instance-id-1','instance-id-2', ...])
+
+Please use with care since once you request termination for an instance there
+is no turning back.
+

data/packages/gsutil/boto/docs/source/elb_tut.rst

@@ -0,0 +1,257 @@
+.. _elb_tut:
+
+==========================================================
+An Introduction to boto's Elastic Load Balancing interface
+==========================================================
+
+This tutorial focuses on the boto interface for `Elastic Load Balancing`_
+from Amazon Web Services. This tutorial assumes that you have already
+downloaded and installed boto, and are familiar with the boto ec2 interface.
+
+.. _Elastic Load Balancing: http://aws.amazon.com/elasticloadbalancing/
+
+Elastic Load Balancing Concepts
+-------------------------------
+`Elastic Load Balancing`_ (ELB) is intimately connected with Amazon's `Elastic
+Compute Cloud`_ (EC2) service. Using the ELB service allows you to create a load
+balancer - a DNS endpoint and set of ports that distributes incoming requests
+to a set of EC2 instances. The advantages of using a load balancer is that it
+allows you to truly scale up or down a set of backend instances without
+disrupting service. Before the ELB service, you had to do this manually by
+launching an EC2 instance and installing load balancer software on it (nginx,
+haproxy, perlbal, etc.) to distribute traffic to other EC2 instances.
+
+Recall that the EC2 service is split into Regions, which are further
+divided into Availability Zones (AZ).
+For example, the US-East region is divided into us-east-1a, us-east-1b,
+us-east-1c, us-east-1d, and us-east-1e. You can think of AZs as data centers -
+each runs off a different set of ISP backbones and power providers.
+ELB load balancers can span multiple AZs but cannot span multiple regions.
+That means that if you'd like to create a set of instances spanning both the
+US and Europe Regions you'd have to create two load balancers and have some
+sort of other means of distributing requests between the two load balancers.
+An example of this could be using GeoIP techniques to choose the correct load
+balancer, or perhaps DNS round robin. Keep in mind also that traffic is
+distributed equally over all AZs the ELB balancer spans. This means you should
+have an equal number of instances in each AZ if you want to equally distribute
+load amongst all your instances.
+
+.. _Elastic Compute Cloud: http://aws.amazon.com/ec2/
+
+Creating a Connection
+---------------------
+
+The first step in accessing ELB is to create a connection to the service.
+
+>>> import boto
+>>> conn = boto.connect_elb(
+        aws_access_key_id='YOUR-KEY-ID-HERE',
+        aws_secret_access_key='YOUR-SECRET-HERE'
+    )
+
+
+A Note About Regions and Endpoints
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Like EC2, the ELB service has a different endpoint for each region. By default
+the US East endpoint is used. To choose a specific region, instantiate the
+ELBConnection object with that region's information.
+
+>>> from boto.regioninfo import RegionInfo
+>>> reg = RegionInfo(
+        name='eu-west-1',
+        endpoint='elasticloadbalancing.eu-west-1.amazonaws.com'
+    )
+>>> conn = boto.connect_elb(
+        aws_access_key_id='YOUR-KEY-ID-HERE',
+        aws_secret_access_key='YOUR-SECRET-HERE',
+        region=reg
+    )
+
+Another way to connect to an alternative region is like this:
+
+>>> import boto.ec2.elb
+>>> elb = boto.ec2.elb.connect_to_region('eu-west-1')
+
+Here's yet another way to discover what regions are available and then
+connect to one:
+
+>>> import boto.ec2.elb
+>>> regions = boto.ec2.elb.regions()
+>>> regions
+[RegionInfo:us-east-1,
+ RegionInfo:ap-northeast-1,
+ RegionInfo:us-west-1,
+ RegionInfo:ap-southeast-1,
+ RegionInfo:eu-west-1]
+>>> elb = regions[-1].connect()
+
+Alternatively, edit your boto.cfg with the default ELB endpoint to use::
+
+    [Boto]
+    elb_region_name = eu-west-1
+    elb_region_endpoint = elasticloadbalancing.eu-west-1.amazonaws.com
+
+Getting Existing Load Balancers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To retrieve any exiting load balancers:
+
+>>> conn.get_all_load_balancers()
+[LoadBalancer:load-balancer-prod, LoadBalancer:load-balancer-staging]
+
+You can also filter by name
+
+>>> conn.get_all_load_balancers(load_balancer_names=['load-balancer-prod'])
+[LoadBalancer:load-balancer-prod]
+
+:py:meth:`get_all_load_balancers <boto.ec2.elb.ELBConnection.get_all_load_balancers>`
+returns a :py:class:`boto.resultset.ResultSet` that contains instances
+of :class:`boto.ec2.elb.loadbalancer.LoadBalancer`, each of which abstracts
+access to a load balancer. :py:class:`ResultSet <boto.resultset.ResultSet>`
+works very much like a list.
+
+>>> balancers = conn.get_all_load_balancers()
+>>> balancers[0]
+[LoadBalancer:load-balancer-prod]
+
+Creating a Load Balancer
+------------------------
+To create a load balancer you need the following:
+ #. The specific **ports and protocols** you want to load balancer over, and what port
+    you want to connect to all instances.
+ #. A **health check** - the ELB concept of a *heart beat* or *ping*. ELB will use this health
+    check to see whether your instances are up or down. If they go down, the load balancer
+    will no longer send requests to them.
+ #. A **list of Availability Zones** you'd like to create your load balancer over.
+
+Ports and Protocols
+^^^^^^^^^^^^^^^^^^^
+An incoming connection to your load balancer will come on one or more ports -
+for example 80 (HTTP) and 443 (HTTPS). Each can be using a protocol -
+currently, the supported protocols are TCP and HTTP.  We also need to tell the
+load balancer which port to route connects *to* on each instance.  For example,
+to create a load balancer for a website that accepts connections on 80 and 443,
+and that routes connections to port 8080 and 8443 on each instance, you would
+specify that the load balancer ports and protocols are:
+
+ * 80, 8080, HTTP
+ * 443, 8443, TCP
+
+This says that the load balancer will listen on two ports - 80 and 443.
+Connections on 80 will use an HTTP load balancer to forward connections to port
+8080 on instances. Likewise, the load balancer will listen on 443 to forward
+connections to 8443 on each instance using the TCP balancer. We need to
+use TCP for the HTTPS port because it is encrypted at the application
+layer. Of course, we could specify the load balancer use TCP for port 80,
+however specifying HTTP allows you to let ELB handle some work for you -
+for example HTTP header parsing.
+
+.. _elb-configuring-a-health-check:
+
+Configuring a Health Check
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+A health check allows ELB to determine which instances are alive and able to
+respond to requests. A health check is essentially a tuple consisting of:
+
+ * *Target*: What to check on an instance. For a TCP check this is comprised of::
+
+        TCP:PORT_TO_CHECK
+
+   Which attempts to open a connection on PORT_TO_CHECK. If the connection opens
+   successfully, that specific instance is deemed healthy, otherwise it is marked
+   temporarily as unhealthy. For HTTP, the situation is slightly different::
+
+        HTTP:PORT_TO_CHECK/RESOURCE
+
+   This means that the health check will connect to the resource /RESOURCE on
+   PORT_TO_CHECK. If an HTTP 200 status is returned the instance is deemed healthy.
+ * *Interval*: How often the check is made. This is given in seconds and defaults
+   to 30. The valid range of intervals goes from 5 seconds to 600 seconds.
+ * *Timeout*: The number of seconds the load balancer will wait for a check to
+   return a result.
+ * *Unhealthy threshold*: The number of consecutive failed checks to deem the
+   instance as being dead. The default is 5, and the range of valid values lies
+   from 2 to 10.
+
+The following example creates a health check called *instance_health* that
+simply checks instances every 20 seconds on port 80 over HTTP at the
+resource /health for 200 successes.
+
+>>> from boto.ec2.elb import HealthCheck
+>>> hc = HealthCheck(
+        interval=20,
+        healthy_threshold=3,
+        unhealthy_threshold=5,
+        target='HTTP:8080/health'
+    )
+
+Putting It All Together
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, let's create a load balancer in the US region that listens on ports
+80 and 443 and distributes requests to instances on 8080 and 8443 over HTTP
+and TCP. We want the load balancer to span the availability zones
+*us-east-1a* and *us-east-1b*:
+
+>>> regions = ['us-east-1a', 'us-east-1b']
+>>> ports = [(80, 8080, 'http'), (443, 8443, 'tcp')]
+>>> lb = conn.create_load_balancer('my-lb', regions, ports)
+>>> # This is from the previous section.
+>>> lb.configure_health_check(hc)
+
+The load balancer has been created. To see where you can actually connect to
+it, do:
+
+>>> print lb.dns_name
+my_elb-123456789.us-east-1.elb.amazonaws.com
+
+You can then CNAME map a better name, i.e. www.MYWEBSITE.com to the
+above address.
+
+Adding Instances To a Load Balancer
+-----------------------------------
+
+Now that the load balancer has been created, there are two ways to add
+instances to it:
+
+ #. Manually, adding each instance in turn.
+ #. Mapping an autoscale group to the load balancer. Please see the
+    :doc:`Autoscale tutorial <autoscale_tut>` for information on how to do this.
+
+Manually Adding and Removing Instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Assuming you have a list of instance ids, you can add them to the load balancer
+
+>>> instance_ids = ['i-4f8cf126', 'i-0bb7ca62']
+>>> lb.register_instances(instance_ids)
+
+Keep in mind that these instances should be in Security Groups that match the
+internal ports of the load balancer you just created (for this example, they
+should allow incoming connections on 8080 and 8443).
+
+To remove instances:
+
+>>> lb.deregister_instances(instance_ids)
+
+Modifying Availability Zones for a Load Balancer
+------------------------------------------------
+
+If you wanted to disable one or more zones from an existing load balancer:
+
+>>> lb.disable_zones(['us-east-1a'])
+
+You can then terminate each instance in the disabled zone and then deregister then from your load
+balancer.
+
+To enable zones:
+
+>>> lb.enable_zones(['us-east-1c'])
+
+Deleting a Load Balancer
+------------------------
+
+>>> lb.delete()
+
+

data/packages/gsutil/boto/docs/source/emr_tut.rst

@@ -0,0 +1,108 @@
+.. _emr_tut:
+
+=====================================================
+An Introduction to boto's Elastic Mapreduce interface
+=====================================================
+
+This tutorial focuses on the boto interface to Elastic Mapreduce from
+Amazon Web Services.  This tutorial assumes that you have already
+downloaded and installed boto.
+
+Creating a Connection
+---------------------
+The first step in accessing Elastic Mapreduce is to create a connection
+to the service.  There are two ways to do this in boto.  The first is:
+
+>>> from boto.emr.connection import EmrConnection
+>>> conn = EmrConnection('<aws access key>', '<aws secret key>')
+
+At this point the variable conn will point to an EmrConnection object.
+In this example, the AWS access key and AWS secret key are passed in to
+the method explicitly.  Alternatively, you can set the environment variables:
+
+AWS_ACCESS_KEY_ID - Your AWS Access Key ID \
+AWS_SECRET_ACCESS_KEY - Your AWS Secret Access Key
+
+and then call the constructor without any arguments, like this:
+
+>>> conn = EmrConnection()
+
+There is also a shortcut function in the boto package called connect_emr
+that may provide a slightly easier means of creating a connection:
+
+>>> import boto
+>>> conn = boto.connect_emr()
+
+In either case, conn points to an EmrConnection object which we will use
+throughout the remainder of this tutorial.
+
+Creating Streaming JobFlow Steps
+--------------------------------
+Upon creating a connection to Elastic Mapreduce you will next 
+want to create one or more jobflow steps.  There are two types of steps, streaming
+and custom jar, both of which have a class in the boto Elastic Mapreduce implementation.
+
+Creating a streaming step that runs the AWS wordcount example, itself written in Python, can be accomplished by:
+
+>>> from boto.emr.step import StreamingStep
+>>> step = StreamingStep(name='My wordcount example',
+...                      mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py',
+...                      reducer='aggregate',
+...                      input='s3n://elasticmapreduce/samples/wordcount/input',
+...                      output='s3n://<my output bucket>/output/wordcount_output')
+
+where <my output bucket> is a bucket you have created in S3.
+
+Note that this statement does not run the step, that is accomplished later when we create a jobflow.  
+
+Additional arguments of note to the streaming jobflow step are cache_files, cache_archive and step_args.  The options cache_files and cache_archive enable you to use the Hadoops distributed cache to share files amongst the instances that run the step.  The argument step_args allows one to pass additional arguments to Hadoop streaming, for example modifications to the Hadoop job configuration.
+
+Creating Custom Jar Job Flow Steps
+----------------------------------
+
+The second type of jobflow step executes tasks written with a custom jar.  Creating a custom jar step for the AWS CloudBurst example can be accomplished by:
+
+>>> from boto.emr.step import JarStep
+>>> step = JarStep(name='Coudburst example',
+...                jar='s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar',
+...                step_args=['s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br',
+...                           's3n://elasticmapreduce/samples/cloudburst/input/100k.br',
+...                           's3n://<my output bucket>/output/cloudfront_output',
+...                            36, 3, 0, 1, 240, 48, 24, 24, 128, 16])
+
+Note that this statement does not actually run the step, that is accomplished later when we create a jobflow.  Also note that this JarStep does not include a main_class argument since the jar MANIFEST.MF has a Main-Class entry.
+
+Creating JobFlows
+-----------------
+Once you have created one or more jobflow steps, you will next want to create and run a jobflow.  Creating a jobflow that executes either of the steps we created above can be accomplished by:
+
+>>> import boto
+>>> conn = boto.connect_emr()
+>>> jobid = conn.run_jobflow(name='My jobflow', 
+...                          log_uri='s3://<my log uri>/jobflow_logs', 
+...                          steps=[step])
+
+The method will not block for the completion of the jobflow, but will immediately return.  The status of the jobflow can be determined by:
+
+>>> status = conn.describe_jobflow(jobid)
+>>> status.state
+u'STARTING'
+
+One can then use this state to block for a jobflow to complete.  Valid jobflow states currently defined in the AWS API are COMPLETED, FAILED, TERMINATED, RUNNING, SHUTTING_DOWN, STARTING and WAITING.
+
+In some cases you may not have built all of the steps prior to running the jobflow.  In these cases additional steps can be added to a jobflow by running:
+
+>>> conn.add_jobflow_steps(jobid, [second_step])
+
+If you wish to add additional steps to a running jobflow you may want to set the keep_alive parameter to True in run_jobflow so that the jobflow does not automatically terminate when the first step completes.
+
+The run_jobflow method has a number of important parameters that are worth investigating.  They include parameters to change the number and type of EC2 instances on which the jobflow is executed, set a SSH key for manual debugging and enable AWS console debugging.
+
+Terminating JobFlows
+--------------------
+By default when all the steps of a jobflow have finished or failed the jobflow terminates.  However, if you set the keep_alive parameter to True or just want to halt the execution of a jobflow early you can terminate a jobflow by:
+
+>>> import boto
+>>> conn = boto.connect_emr()
+>>> conn.terminate_jobflow('<jobflow id>') 
+