googlecloud 0.0.4 → 0.0.6

data/packages/gsutil/gslib/addlhelp/prod.py

@@ -0,0 +1,160 @@
+# Copyright 2012 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from gslib.help_provider import HELP_NAME
+from gslib.help_provider import HELP_NAME_ALIASES
+from gslib.help_provider import HELP_ONE_LINE_SUMMARY
+from gslib.help_provider import HelpProvider
+from gslib.help_provider import HELP_TEXT
+from gslib.help_provider import HelpType
+from gslib.help_provider import HELP_TYPE
+
+_detailed_help_text = ("""
+<B>OVERVIEW</B>
+  If you use gsutil in large production tasks (such as uploading or
+  downloading many GB of data each night), there are a number of things
+  you can do to help ensure success. Specifically, this section discusses
+  how to script large production tasks around gsutil's resumable transfer
+  mechanism.
+
+
+<B>BACKGROUND ON RESUMABLE TRANSFERS</B>
+  First, it's helpful to understand gsutil's resumable transfer mechanism,
+  and how your script needs to be implemented around this mechanism to work
+  reliably. gsutil uses the resumable transfer support in the boto library
+  when you attempt to upload or download a file larger than a configurable
+  threshold (by default, this threshold is 1MB). When a transfer fails
+  partway through (e.g., because of an intermittent network problem),
+  boto uses a randomized binary exponential backoff-and-retry strategy:
+  wait a random period between [0..1] seconds and retry; if that fails,
+  wait a random period between [0..2] seconds and retry; and if that
+  fails, wait a random period between [0..4] seconds, and so on, up to a
+  configurable number of times (the default is 6 times). Thus, the retry
+  actually spans a randomized period up to 1+2+4+8+16+32=63 seconds.
+
+  If the transfer fails each of these attempts with no intervening
+  progress, gsutil gives up on the transfer, but keeps a "tracker" file
+  for it in a configurable location (the default location is ~/.gsutil/,
+  in a file named by a combination of the SHA1 hash of the name of the
+  bucket and object being transferred and the last 16 characters of the
+  file name). When transfers fail in this fashion, you can rerun gsutil
+  at some later time (e.g., after the networking problem has been
+  resolved), and the resumable transfer picks up where it left off.
+
+
+<B>SCRIPTING DATA TRANSFER TASKS</B>
+  To script large production data transfer tasks around this mechanism,
+  you can implement a script that runs periodically, determines which file
+  transfers have not yet succeeded, and runs gsutil to copy them. Below,
+  we offer a number of suggestions about how this type of scripting should
+  be implemented:
+
+  1. When resumable transfers fail without any progress 6 times in a row
+  over the course of up to 63 seconds, it probably won't work to simply
+  retry the transfer immediately. A more successful strategy would be to
+  have a cron job that runs every 30 minutes, determines which transfers
+  need to be run, and runs them. If the network experiences intermittent
+  problems, the script picks up where it left off and will eventually
+  succeed (once the network problem has been resolved).
+
+  2. If your business depends on timely data transfer, you should consider
+  implementing some network monitoring. For example, you can implement
+  a task that attempts a small download every few minutes and raises an
+  alert if the attempt fails for several attempts in a row (or more or less
+  frequently depending on your requirements), so that your IT staff can
+  investigate problems promptly. As usual with monitoring implementations,
+  you should experiment with the alerting thresholds, to avoid false
+  positive alerts that cause your staff to begin ignoring the alerts.
+
+  3. There are a variety of ways you can determine what files remain to be
+  transferred. We recommend that you avoid attempting to get a complete
+  listing of a bucket containing many objects (e.g., tens of thousands
+  or more). One strategy is to structure your object names in a way that
+  represents your transfer process, and use gsutil prefix wildcards to
+  request partial bucket listings. For example, if your periodic process
+  involves downloading the current day's objects, you could name objects
+  using a year-month-day-object-ID format and then find today's objects by
+  using a command like gsutil ls gs://bucket/2011-09-27-*. Note that it
+  is more efficient to have a non-wildcard prefix like this than to use
+  something like gsutil ls gs://bucket/*-2011-09-27. The latter command
+  actually requests a complete bucket listing and then filters in gsutil,
+  while the former asks Google Storage to return the subset of objects
+  whose names start with everything up to the *.
+
+  For data uploads, another technique would be to move local files from a "to
+  be processed" area to a "done" area as your script successfully copies files
+  to the cloud. You can do this in parallel batches by using a command like:
+
+    gsutil -m cp -R to_upload/subdir_$i gs://bucket/subdir_$i
+
+  where i is a shell loop variable. Make sure to check the shell $status
+  variable is 0 after each gsutil cp command, to detect if some of the copies
+  failed, and rerun the affected copies.
+
+  With this strategy, the file system keeps track of all remaining work to
+  be done.
+
+  4. If you have really large numbers of objects in a single bucket
+  (say hundreds of thousands or more), you should consider tracking your
+  objects in a database instead of using bucket listings to enumerate
+  the objects. For example this database could track the state of your
+  downloads, so you can determine what objects need to be downloaded by
+  your periodic download script by querying the database locally instead
+  of performing a bucket listing.
+
+  5. Make sure you don't delete partially downloaded files after a transfer
+  fails: gsutil picks up where it left off (and performs an MD5 check of
+  the final downloaded content to ensure data integrity), so deleting
+  partially transferred files will cause you to lose progress and make
+  more wasteful use of your network. You should also make sure whatever
+  process is waiting to consume the downloaded data doesn't get pointed
+  at the partially downloaded files. One way to do this is to download
+  into a staging directory and then move successfully downloaded files to
+  a directory where consumer processes will read them.
+
+  6. If you have a fast network connection, you can speed up the transfer of
+  large numbers of files by using the gsutil -m (multi-threading /
+  multi-processing) option. Be aware, however, that gsutil doesn't attempt to
+  keep track of which files were downloaded successfully in cases where some
+  files failed to download. For example, if you use multi-threaded transfers
+  to download 100 files and 3 failed to download, it is up to your scripting
+  process to determine which transfers didn't succeed, and retry them. A
+  periodic check-and-run approach like outlined earlier would handle this case.
+
+  If you use parallel transfers (gsutil -m) you might want to experiment with
+  the number of threads being used (via the parallel_thread_count setting
+  in the .boto config file). By default, gsutil uses 24 threads. Depending
+  on your network speed, available memory, CPU load, and other conditions,
+  this may or may not be optimal. Try experimenting with higher or lower
+  numbers of threads, to find the best number of threads for your environment.
+""")
+
+
+class CommandOptions(HelpProvider):
+  """Additional help about using gsutil for production tasks."""
+
+  help_spec = {
+    # Name of command or auxiliary help info for which this help applies.
+    HELP_NAME : 'prod',
+    # List of help name aliases.
+    HELP_NAME_ALIASES : ['production', 'resumable', 'resumable upload',
+                         'resumable transfer', 'resumable download',
+                         'scripts', 'scripting'],
+    # Type of help:
+    HELP_TYPE : HelpType.ADDITIONAL_HELP,
+    # One line summary of this help.
+    HELP_ONE_LINE_SUMMARY : 'Scripting production data transfers with gsutil',
+    # The full help text.
+    HELP_TEXT : _detailed_help_text,
+  }

data/packages/gsutil/gslib/addlhelp/projects.py

@@ -0,0 +1,130 @@
+# Copyright 2012 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from gslib.help_provider import HELP_NAME
+from gslib.help_provider import HELP_NAME_ALIASES
+from gslib.help_provider import HELP_ONE_LINE_SUMMARY
+from gslib.help_provider import HelpProvider
+from gslib.help_provider import HELP_TEXT
+from gslib.help_provider import HelpType
+from gslib.help_provider import HELP_TYPE
+
+_detailed_help_text = ("""
+<B>OVERVIEW</B>
+  This section discusses how to work with projects in Google Cloud Storage.
+
+  For more information about using the Google APIs Console to administer
+  project memberships (which are automatically included in ACLs for buckets
+  you create) see https://code.google.com/apis/console#:storage:access.
+
+
+<B>PROJECT MEMBERS AND PERMISSIONS</B>
+  There are three groups of users associated with each project:
+
+    - Project Owners are allowed to list, create, and delete buckets,
+      and can also perform administrative tasks like adding and removing team
+      members and changing billing. The project owners group is the owner
+      of all buckets within a project, regardless of who may be the original
+      bucket creator.
+
+    - Project Editors are allowed to list, create, and delete buckets.
+
+    - All Project Team Members are allowed to list buckets within a project.
+
+  These projects make it easy to set up a bucket and start uploading objects
+  with access control appropriate for a project at your company, as the three
+  group memberships can be configured by your administrative staff. Control
+  over projects and their associated memberships is provided by the Google
+  APIs Console (https://code.google.com/apis/console).
+
+
+<B>HOW PROJECT MEMBERSHIP IS REFLECTED IN BUCKET ACLS</B>
+  When you create a bucket without specifying an ACL the bucket is given a
+  "project-private" ACL, which grants the permissions described in the previous
+  section. Here's an example of such an ACL:
+
+  <AccessControlList>
+    <Owner>
+      <ID>
+        00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7
+      </ID>
+    </Owner>
+    <Entries>
+      <Entry>
+        <Scope type="GroupById">
+          <ID>
+            00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7
+          </ID>
+        </Scope>
+        <Permission>
+          FULL_CONTROL
+        </Permission>
+      </Entry>
+      <Entry>
+        <Scope type="GroupById">
+          <ID>
+            00b4903a977fd817e9da167bc81306489181a110456bb635f466d71cf90a0d51
+          </ID>
+        </Scope>
+        <Permission>
+          FULL_CONTROL
+        </Permission>
+      </Entry>
+      <Entry>
+        <Scope type="GroupById">
+          <ID>
+            00b4903a974898cc8fc309f2f2835308ba3d3df1b889d3fc7e33e187d52d8e71
+          </ID>
+        </Scope>
+        <Permission>
+          READ
+        </Permission>
+      </Entry>
+    </Entries>
+  </AccessControlList>
+
+  The three "GroupById" scopes are the canonical IDs for the Project Owners,
+  Project Editors, and All Project Team Members groups.
+
+  You can edit the bucket ACL if you want to (see "gsutil help setacl"),
+  but for many cases you'll never need to, and instead can change group
+  membership via the APIs console.
+
+<B>IDENTIFYING PROJECTS WHEN CREATING AND LISTING BUCKETS</B>
+  When you create a bucket or list your buckets, you need to provide the
+  project ID that want to create or list (using the gsutil mb -p option or
+  the gsutil ls -p option, respectively). The project's name shown in the
+  Google APIs Console is a user-friendly name that you can choose; this is
+  not the project ID required by the gsutil mb and ls commands. To find the
+  project ID, go to the Storage Access pane in the Google APIs Console. Your
+  project ID is listed under Identifying your project.
+""")
+
+
+class CommandOptions(HelpProvider):
+  """Additional help about Access Control Lists."""
+
+  help_spec = {
+    # Name of command or auxiliary help info for which this help applies.
+    HELP_NAME : 'projects',
+    # List of help name aliases.
+    HELP_NAME_ALIASES : ['apis console', 'console', 'dev console', 'project',
+                         'proj', 'project-id'],
+    # Type of help:
+    HELP_TYPE : HelpType.ADDITIONAL_HELP,
+    # One line summary of this help.
+    HELP_ONE_LINE_SUMMARY : 'Working with projects',
+    # The full help text.
+    HELP_TEXT : _detailed_help_text,
+  }

data/packages/gsutil/gslib/addlhelp/subdirs.py

@@ -0,0 +1,110 @@
+# Copyright 2012 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from gslib.help_provider import HELP_NAME
+from gslib.help_provider import HELP_NAME_ALIASES
+from gslib.help_provider import HELP_ONE_LINE_SUMMARY
+from gslib.help_provider import HelpProvider
+from gslib.help_provider import HELP_TEXT
+from gslib.help_provider import HelpType
+from gslib.help_provider import HELP_TYPE
+
+_detailed_help_text = ("""
+<B>OVERVIEW</B>
+  This section provides details about how subdirectories work in gsutil.
+  Most users probably don't need to know these details, and can simply use
+  the commands (like cp -R) that work with subdirectories. We provide this
+  additional documentation to help users understand how gsutil handles
+  subdirectories differently than most GUI / web-based tools (e.g., why
+  those other tools create "dir_$folder$" objects), and also to explain cost and
+  performance implications of the gsutil approach, for those interested in such
+  details.
+
+  gsutil provides the illusion of a hierarchical file tree atop the "flat"
+  name space supported by the Google Cloud Storage service. To the service,
+  the object gs://bucket/abc/def/ghi.txt is just an object that happens to have
+  "/" characters in its name. There are no "abc" or "abc/def" directories;
+  just a single object with the given name.
+
+  gsutil achieves the hierarchical file tree illusion by applying a variety of
+  rules, to try to make naming work the way users would expect. For example, in
+  order to determine whether to treat a destination URI as an object name or the
+  root of a directory under which objects should be copied gsutil uses these
+  rules:
+
+  1. If the destination object ends with a "/" gsutil treats it as a directory.
+     For example, if you run the command:
+
+       gsutil cp file gs://bucket/abc/
+
+     gsutil will create the object gs://bucket/abc/file.
+
+  2. If you attempt to copy multiple source files to a destination URI, gsutil
+     treats the destination URI as a directory. For example, if you run
+     the command:
+
+       gsutil cp -R dir gs://bucket/abc
+
+     gsutil will create objects like gs://bucket/abc/dir/file1, etc. (assuming
+     file1 is a file under the source dir).
+
+  3. If neither of the above rules applies, gsutil performs a bucket listing to
+     determine if the target of the operation is a prefix match to the
+     specified string. For example, if you run the command:
+
+       gsutil cp file gs://bucket/abc
+
+     gsutil will make a bucket listing request for the named bucket, using
+     delimiter="/" and prefix="abc". It will then examine the bucket listing
+     results and determine whether there are objects in the bucket whose path
+     starts with gs://bucket/abc/, to determine whether to treat the target as
+     an object name or a directory name. In turn this impacts the name of the
+     object you create: If the above check indicates there is an "abc" directory
+     you will end up with the object gs://bucket/abc/file; otherwise you will
+     end up with the object gs://bucket/abc. (See "HOW NAMES ARE CONSTRUCTED"
+     under "gsutil help cp" for more details.)
+
+  This rule-based approach stands in contrast to the way many tools work, which
+  create objects to mark the existence of folders (such as "dir_$folder$").
+  gsutil understands several conventions used by such tools but does not
+  require such marker objects to implement naming behavior consistent with
+  UNIX commands.
+
+  A downside of the gsutil approach is it requires an extra bucket listing
+  before performing the needed cp or mv command. However those listings are
+  relatively inexpensive, because they use delimiter and prefix parameters to
+  limit result data. Moreover, gsutil makes only one bucket listing request
+  per cp/mv command, and thus amortizes the bucket listing cost across all
+  transferred objects (e.g., when performing a recursive copy of a directory
+  to the cloud).
+""")
+
+
+class CommandOptions(HelpProvider):
+  """Additional help about subdirectory handling in gsutil."""
+
+  help_spec = {
+    # Name of command or auxiliary help info for which this help applies.
+    HELP_NAME : 'subdirs',
+    # List of help name aliases.
+    HELP_NAME_ALIASES : ['dirs', 'directory', 'directories', 'folder',
+                         'folders', 'hierarchy', 'subdir', 'subdirectory',
+                         'subdirectories'],
+    # Type of help:
+    HELP_TYPE : HelpType.ADDITIONAL_HELP,
+    # One line summary of this help.
+    HELP_ONE_LINE_SUMMARY : 'How subdirectories work in gsutil',
+    # The full help text.
+    HELP_TEXT : _detailed_help_text,
+  }

data/packages/gsutil/gslib/addlhelp/support.py

@@ -0,0 +1,86 @@
+# Copyright 2012 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from gslib.help_provider import HELP_NAME
+from gslib.help_provider import HELP_NAME_ALIASES
+from gslib.help_provider import HELP_ONE_LINE_SUMMARY
+from gslib.help_provider import HelpProvider
+from gslib.help_provider import HELP_TEXT
+from gslib.help_provider import HelpType
+from gslib.help_provider import HELP_TYPE
+
+_detailed_help_text = ("""
+<B>TECHNICAL SUPPORT</B>
+  If you have any questions or encounter any problems with Google Cloud Storage,
+  please first read the FAQ at https://developers.google.com/storage/docs/faq.
+  If you still need help, please post your question to the gs-discussion forum
+  (https://developers.google.com/storage/forum) or to Stack Overflow with the
+  Google Cloud Storage tag
+  (http://stackoverflow.com/questions/tagged/google-cloud-storage). Our support
+  team actively monitors these forums and we'll do our best to respond. To help
+  us diagnose any issues you encounter, please provide these details in addition
+  to the description of your problem:
+
+  - The resource you are attempting to access (bucket name, object name)
+  - The operation you attempted (GET, PUT, etc.)
+  - The time and date (including timezone) at which you encountered the problem
+  - The tool or library you use to interact with Google Cloud Storage
+  - If you can use gsutil to reproduce your issue, specify the -D option to
+    display your request's HTTP details. Provide these details with your post
+    to the forum as they can help us further troubleshoot your issue.
+
+  Warning: The gsutil -D, -d, and -DD options will also print the authentication
+  header with authentication credentials for your Google Cloud Storage account.
+  Make sure to remove any "Authorization:" headers before you post HTTP details
+  to the forum.
+
+  If you make any local modifications to gsutil, please make sure to use
+  a released copy of gsutil (instead of your locally modified copy) when
+  providing the gsutil -D output noted above. We cannot support versions
+  of gsutil that include local modifications. (However, we're open to user
+  contributions; see "gsutil help dev".)
+
+  As an alternative to posting to the gs-discussion forum, we also
+  actively monitor http://stackoverflow.com for questions tagged with
+  "google-cloud-storage".
+
+
+<B>BILLING AND ACCOUNT QUESTIONS</B>
+  For questions about billing or account issues, please visit
+  http://code.google.com/apis/console-help/#billing.  If you want to cancel
+  billing, you can do so on the Billing pane of the Google APIs Console. For
+  more information, see
+  http://code.google.com/apis/console-help/#BillingCancelled. Caution: When you
+  disable billing, you also disable the Google Cloud Storage service. Make sure
+  you want to disable the Google Cloud Storage service before you disable
+  billing.
+""")
+
+
+class CommandOptions(HelpProvider):
+  """Additional help about tech and billing support."""
+
+  help_spec = {
+    # Name of command or auxiliary help info for which this help applies.
+    HELP_NAME : 'support',
+    # List of help name aliases.
+    HELP_NAME_ALIASES : ['techsupport', 'tech support', 'technical support',
+                         'billing', 'faq', 'questions'],
+    # Type of help:
+    HELP_TYPE : HelpType.ADDITIONAL_HELP,
+    # One line summary of this help.
+    HELP_ONE_LINE_SUMMARY : 'How to get Google Cloud Storage support',
+    # The full help text.
+    HELP_TEXT : _detailed_help_text,
+  }