aliyun-odps 0.1.0 → 0.4.0

data/lib/aliyun/odps/version.rb

@@ -1,5 +1,5 @@
 module Aliyun
   module Odps
-    VERSION = "0.1.0"
+    VERSION = '0.4.0'
   end
 end

data/wiki/error.md

@@ -0,0 +1,188 @@
+## ERROR Code
+
+In our library, `Aliyun::Odps::RequestError` is raised when Request fail, ***keep your code wrap with begin...rescue***. Below is a example.
+
+
+    begin
+      projects = Aliyun::Odps.list_projects
+      puts projects.map(&:name)
+    rescue Aliyun::Odps::RequestError => e
+      puts "Code: #{e.code}"
+      puts "Message: #{e.message}"
+      puts "Request ID: #{e.request_id}"
+    end
+
+ODPS define many error code, you can find more detail in this section.
+
+If your need help for your problem, search it via Google or Aliyun forum. If you still has trouble after that, contact ODPS Developer and provide your error code and request id, they are nice and rapid response.       
+
+
+Below is the ODPS ERROR Code:
+
+### ODPS Common ERROR Code
+
+| CODE | Message | HTTP STATUS CODE |
+|------|---------|------------------|
+|ServiceUnavailable	|Service is temporarily unavailable, Please try again later.	|	503 Service Unavailable|
+|InternalServerError|	Service internal error, please try again later.|	500 Internal Server Error|
+|MethodNotAllowed|	Unsupported request method.|	405 Method Not Allowed|
+|MissingAuthorization|	Authorization is required.|	401 Unauthorized|
+|InvalidAuthorization|	Invalid authorization.|401 Unauthorized|
+|MissingContentType	|Content-type is required.|	415 Unsupported Media Type|
+|InvalidContentType	|Invalid content type.	|	415 Unsupported Media Type|
+|MissingContentLength|	Content-length is required.|	411 Length Required|
+|ObjectAlreadyExists|	The "" has already exited.|409 Conflict|
+|NoSuchObject|	Object not found-''.|404 Not found|
+|NoPermission|	You haven't enough privilege to XX.|403 Forbidden|
+|InvalidParameter|	Invalid parameter XX|400 Bad Request|
+|InvalidStatusSetting|	Current status is . It is not allowed to set status to|409 Conflict|
+|DeleteConflict|Cannot delete object which is using|409 Conflict|
+|AccessDenied| Access Denied|403 Forbidden|
+
+
+### SQL ERROR Code
+
+In General, ODPS return ERROR Code with ODPS-MMCCCCX format.
+
+Below is a table list all SQL ERROR Code:
+
+|Code |Severity Level |ERROR Message |
+|-----|---------------|--------------|
+|ODPS-0110005|	5	|	Unknown exception from metadata operation|
+|ODPS-0110011|	1	|Authorization exception|
+|ODPS-0110021|	1	|	Invalid parameters|
+|ODPS-0110031|	1	|	Invalid object type|
+|ODPS-0110041|	1	|	Invalid meta operation|
+|ODPS-0110061|	1	|	Failed to run ddltask|
+|ODPS-0110071|	1	|	OTS initialization exception|
+|ODPS-0110081|	1	|	OTS transaction exception|
+|ODPS-0110091|	1	|	OTS filtering exception|
+|ODPS-0110101|	1	|OTS processing exception|
+|ODPS-0110111|	1	|OTS invalid data object|
+|ODPS-0110121|	1	|Unknown OTS exception|
+|ODPS-0110131|	1	|	StorageDescriptor compression exception|
+|ODPS-0110141|	1	|Data version exception|
+|ODPS-0110999|	9	|Critical! Internal error happened in commit |operation and rollback failed, possible breach of atomicity|
+|ODPS-0120005|	5	|	Unknown exception from processor|
+|ODPS-0120011|	1	|Authorization exception|
+|ODPS-0120021|	1	|the delimitor must be the same in wm_concat|
+|ODPS-0120031|	1	|Instance has been cancelled|
+|ODPS-0121011|	1	|	Invalid regular expression pattern|
+|ODPS-0121021|	1	|	Regexec call failed|
+|ODPS-0121035|	5	|Illegal implicit type cast|
+|ODPS-0121045|	5	|	Unsupported return type|
+|ODPS-0121055|	5	|	Empty argument value|
+|ODPS-0121065|	5	|Argument value out of range|
+|ODPS-0121075|	5	|Invalid number of arguments|
+|ODPS-0121081|	1	|Illegal argument type|
+|ODPS-0121095|	5	|Invalid arguments|
+|ODPS-0121105|	5	|Constant argument value expected|
+|ODPS-0121115|	5	|Column reference expected|
+|ODPS-0121125|	5	|Unsupported function or operation|
+|ODPS-0121135|	5	|Malloc memory failed|
+|ODPS-0121145|	5	|	Data overflow|
+|ODPS-0123019|	9	|Distributed file operation exception|
+|ODPS-0123023|	3	|Unsupported reduce type|
+|ODPS-0123031|	1	|Partition exception|
+|ODPS-0123043|	3	|buffer overflow|
+|ODPS-0123055|	5	|Script exception|
+|ODPS-0123065|	5	|Join exception|
+|ODPS-0123075|	5	|Hash exception|
+|ODPS-0123081|	1	|Invalid datetime string|
+|ODPS-0123091|	1	|Illegal type cast|
+|ODPS-0123105|	5	|Job got killed|
+|ODPS-0123111|	1	|format string does not match datetime string|
+|ODPS-0123121|	1	|Mapjoin exception|
+|ODPS-0123131|	1	|User defined function exception|
+|ODPS-0130005|	5	|Unknown exception from parser|
+|ODPS-0130013|	3	|	Authorization exception|
+|ODPS-0130025|	5	|Failed to I/O|
+|ODPS-0130031|	1	|Failed to drop table|
+|ODPS-0130041|	1	|Statistics exception|
+|ODPS-0130051|	1	|Exception in sub query|
+|ODPS-0130061|	1	|Invalid table|
+|ODPS-0130071|	1	|Semantic analysis exception|
+|ODPS-0130081|	1	|Invalid UDF reference|
+|ODPS-0130091|	1	|Invalid parameters|
+|ODPS-0130101|	1	|	Ambiguous data type|
+|ODPS-0130111|	1	|Subquery partition pruning exception|
+|ODPS-0130121|	1	|Invalid argument type|
+|ODPS-0130131|	1	|Table not found|
+|ODPS-0130141|	1	|Illegal implicit type cast|
+|ODPS-0130151|	1	|Illegal data type|
+|ODPS-0130161|	1	|Parse exception|
+|ODPS-0130171|	1	|Creating view exception|
+|ODPS-0130181|	1	|Window function exception|
+|ODPS-0130191|	1	|Invalid column or partition key|
+|ODPS-0130201|	1	|View not found|
+|ODPS-0130211|	1	|Table or view already exists|
+|ODPS-0130221|	1	|Invalid number of arguments|
+|ODPS-0130231|	1	|Invalid view|
+|ODPS-0130241|	1	|Illegal union operation|
+|ODPS-0130252|	2	|Cartesian product is not allowed|
+|ODPS-0130261|	1	|Invalid schema|
+|ODPS-0130271|	1	|Partition does not exist|
+|ODPS-0140005|	5	|	Unknown exception from planner|
+|ODPS-0140011|	1	|Illegal type cast|
+|ODPS-0140021|	1	|Illegal implicit type cast|
+|ODPS-0140031|	1	|Invalid column reference|
+|ODPS-0140041|	1	|Invalid UDF reference|
+|ODPS-0140051|	1	|Invalid function|
+|ODPS-0140061|	1	|Invalid parameters|
+|ODPS-0140071|	1	|	Unsupported operator|
+|ODPS-0140081|	1	|Unsupported join type|
+|ODPS-0140091|	1	|	Unsupported stage type|
+|ODPS-0140105|	5	|Invalid multiple I/O|
+|ODPS-0140133|	3	|Invalid structure|
+|ODPS-0140151|	1	|Can not do topologic sort, the stages is not a DAG|
+|ODPS-0140171|	1	|Sandbox violation exception|
+|ODPS-0140181|	1	|Sql plan exception|
+
+### ODPS Tunnel ERROR Code
+
+Since now, the Tunnel has Different ERROR Code, Below is the List:
+
+|Code |Message |
+|-----|--------|
+|AccessDenied|	Access Denied|
+|CorruptedDataStream	|The data stream was corrupted, please try again later	|
+|DataUnderReplication	|The specified table data is under replication and you cannot initiate upload or download at this time. Please try again later|
+|DataVersionConflict	|The specified table has been modified since the upload or download initiated and table data is being replicated at this time. Please initiate another download or upload later|
+|FlowExceeded	|Your flow quota is exceeded	|
+|InConsistentBlockList	|The specified block list is not consistent with the uploaded block list on server side	|
+|IncompleteBody|	You did not provide the number of bytes specified by the Content-Length HTTP header	|
+|InternalServerError	|Service internal error, please try again later|
+|InvalidArgument	|Invalid argument	|
+|InvalidBlockID	|The specified block id is not valid|
+|InvalidColumnSpec	|The specified columns is not valid|
+|InvalidRowRange	|The specified row range is not valid|	
+|InvalidStatusChange|	You cannot change the specified upload or download status	|
+|InvalidURI	|Couldn’t parse the specified URI	|
+|InvalidUriSpec	|The specified uri spec is not valid|	
+|MalformedDataStream	|The data stream you provided was not well-formed or did not validate against schema	|
+|MalformedHeaderValue	|An HTTP header value was malformed|	
+|MalformedXML	|The XML you provided was not well-formed or did not validate against schema	|
+|MaxMessageLengthExceeded	|Your request was too big	|
+|MethodNotAllowed	|The specified method is not allowed against this resource	|
+|MissingContentLength	|You must provide the Content-Length HTTP header|	
+|MissingPartitionSpec	|You need to specify a partitionspec along with the specified table	|
+|MissingRequestBodyError	|The request body is missing|	
+|MissingRequiredHeaderError	Your |request was missing a required header|	
+|NoPermission	|You do not have enough privilege to complete the specified operation	|
+|NoSuchData	|The uploaded data within this uploaded no longer exists|	
+|NoSuchDownload	|The specified download id does not exist|	
+|NoSuchPartition	|The specified partition does not exist|
+|NoSuchProject	|The specified project name does not exist|	
+|NoSuchTable	|The specified table name does not exist|	
+|NoSuchUpload	|The specified upload id does not exist|	
+|NoSuchVolume	|The specified volume name does not exist|	
+|NoSuchVolumeFile	|The specified volume file does not exist|	
+|NoSuchVolumePartition|	The specified volume partition does not exist|	
+|NotImplemented	|A header you provided implies functionality that is not implemented	|
+|ObjectModified	|The specified object has been modified since the specified timestamp	|
+|RequestTimeOut	|Your socket connection to the server was not read from or written to within the timeout period	|
+|ServiceUnavailable	|Service is temporarily unavailable, Please try again later	|
+|StatusConflict	|You cannot complete the specified operation under the current upload or download status	|
+|TableModified	|The specified table has been modified since the download initiated. Try initiate another download	|
+|Unauthorized	|The request authorization header is invalid or missing|	
+|UnexpectedContent	|This request does not support content|	

data/wiki/functions.md

@@ -0,0 +1,39 @@
+## Function
+
+We can define functions in odps, which can be used in ODPS SQL.
+
+Function Notes:
+
++ create a function need name, type, resource and other resource.
++ function name should unique in project.
++ after create, function can be used in SQL.
+
+
+### Basic
+
+To list all functions:
+
+    project.functions.list
+
+To get function:
+
+    project.functions.get('function_name')
+    
+    
+### Create Function
+
+To create a function, should create resource first:
+
+
+    resource1 = project.resources.get('resource1')
+    resource2 = project.resources.get('resource1')
+    
+    function = project.functions.create('function1', 'Path/to/Class', [resource1, resource2])
+   
+
+To delete it:
+
+    project.functions.delete('function1')
+
+
+Now, already complete, let's visit the most powerful part [Instances](./instances.md)    

data/wiki/get_start.md

@@ -0,0 +1,34 @@
+## Get Started
+
+First of all, config your environment.
+
+If you use Rails, you can place it in `config/initializers/aliyun_odps.rb`, for other framework, just place it before your other odps code.
+
+    Aliyun::Odps.configure do |config|
+      config.access_key = '<your-access-key>'
+      config.secret_key = '<your-secret-key>'
+      config.endpoint = '<odps-server-api>'  # "http://service.odps.aliyun.com/api"
+      config.project = '<your-default-project>'
+    end
+
+After that, you can get your project and start party now!
+
+    project = Aliyun::Odps.project
+
+    # Accesss tables
+    project.tables.list
+
+    # Access resources
+    project.resources.list
+
+    # Access instances
+    project.instances.list
+
+    # Access functions
+    project.functions.list
+
+
+    # For Tunnel
+    project.table_tunnels
+
+Next, Let's visit [Project](./projects.md)

data/wiki/installation.md

@@ -0,0 +1,15 @@
+## Installation
+
+It's a Ruby Gem, so you can install it like any Gem:
+
+    $ gem install aliyun-odps
+
+If you use Gemfile manage your Gems, Add below to your Gemfile.
+
+    gem "aliyun-odps"
+
+And run:
+
+    $ bundle install
+    
+Now, [Getting started](./get_start.md)    

data/wiki/instances.md

@@ -0,0 +1,32 @@
+## Instance
+
+Instance is use created job instance, it support: SQL, SQLPLAN, MapReduce, DT, PLSQL now.
+
+we can define different task to create instance.
+
++ SQL Task
+
+To create a SQL Task:
+
+    task = Aliyun::Odps::InstanceTask.new(name: 'SqlTask', comment: 'sql task', query: 'select * from  test_table1;', type: 'SQL')
+    instance = prj.instances.create([task])
+    
+    # wait for it terminate
+    instance.wait_for_terminated
+    
+    # puts the results
+    task_result = instance.task_results[task.name]
+    
+    puts task_result.result
+    
+    # write task result into csv file
+    File.open('test.csv', "w") {|f| f.write task_result.result['__content__'] }
+
+
+To create other task, just replace SQL to your expected type, note the supported list metioned before.
+
+
+Want to write more powerful SQL, visit [How to use ODPS SQL](https://help.aliyun.com/document_detail/odps/SQL/summary.html?spm=5176.docodps/SQL/ddl.3.2.LdRubj)
+
+
+After Instance, let's visit [Table Tunnel](./tunnels.md), which can help us upload and download data with ODPS table.

data/wiki/projects.md

@@ -0,0 +1,51 @@
+## Project
+
+[Project](https://docs.aliyun.com/#/pub/odps/basic/definition&project) is the basic unit to keep your system organized, it's very similar to `database` or `schema` in traditional relational database.
+
+Besides the way to get your project you already hear in Readme, here we list some more feature to access your projects.
+
+
+### LIST
+
+We could list all available projects by calling `Aliyun::Odps.list_projects`.
+
+
+    #list projects by options
+    projects = Aliyun::Odps.list_projects(
+                 owner: 'owner', #specify the project owner
+                 marker: 'marker', #specify marker for paginate
+                 maxitems: '1'
+               ) #specify maxitems in this request, default 1000
+
+
+It will support three options `owner`, `marker` and `maxitem`, see the usage by example.
+
+Besides, it has a alias `Aliyun::Odps.projects`, choose your favorite style!!
+
+
+### GET
+
+As you known, Aliyun::Odps.project can get your default project, but if you pass a name, you can get other project quickly.
+
+We could call `Aliyun::Odps.project` to fetch a project information
+
+
+    project = Aliyun::Odps.project                     #get the default project
+    project = Aliyun::Odps.project('my_other_project') #fetch a project by name
+
+If no project name was provide, it will return the default project which was configured by `Aliyun::Odps.configure`
+
+If you pass a noexist project name, it will raise `Aliyun::Odps::RequestError: NoSuchObject: ODPS-0420111`.
+
+By the way, in our library, all request to odps fail, will raise `Aliyun::Odps::RequestError`. For more detail, visit [Error](./error.md)
+
+
+## UPDATE
+
+Right now you are only able to update a project comment by update method, passing by options
+
+
+    #update project by options
+    Aliyun::Odps.update_project('my_project_name', comment: 'just for test')
+    
+Now, let's go to [Tables](./tables.md)    

data/wiki/resources.md

@@ -0,0 +1,62 @@
+## Resources
+
+Resources is a important part for ODPS.
+
+To List all resources:
+
+    project.resources.list
+    
+In ODPS, resource type can be: py, jar, archive, file and table.
+
+### Create Resource
+
+
+#### file resource
+
+py, jar, archive, file are file resource, now let's take a look for below example to create a file resource:
+
+    resource = project.resources.create('file', 'test_file_resource', file: "path/to/resource.rb")
+    
+    # or
+    
+    resource = project.resources.create('file', 'test_file_resource', file: "puts 'hello world'")
+
+
+#### Table resource
+
+To create a resource with exist table exist in project, pass table with table path:
+
+    resource = project.resources.create('table', 'test_table_resource', table: "test_table partition(partname='part1')")
+
+please note the table argument format: 
+
++ "tablename partition(col=value,[col=value])" for table with partition
++ "tablename" for table without any partitions
+
+### Get Resource
+
+To get resource content:
+
+    resource = project.resources.get('test_file_resource')
+    
+    File.open(resource.name, "w") do |f|
+      f.write resource.content
+    end
+
+To get resource information except content:
+
+    resource = project.resources.head('test_file_resource')
+
+
+### Others
+
+Besides, you can update resource:
+
+    resource = project.resources.update('file', 'test_file_resource', file: "puts 'hello rio'")    
+
+To delete a resource:
+
+    project.resources.delete('test_file_resource')
+    
+    
+After resource, let's visit [Functions](./functions.md)    

data/wiki/ssl.md

@@ -0,0 +1,7 @@
+## SSL Connection
+
+We support SSL connection in two ways:
+
+1. just encrypt request but server not verify cert, just config with https endpoint, eg: https://service.odps.aliyun.com/api'
+
+2. To enable server verify your request, config ssl_ca_file in addition, eg: config.ssl_ca_file = 'path/to/your/ssl/ca/file'