jgm-cloudlib 0.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (7) hide show
  1. data/LICENSE +340 -0
  2. data/README +91 -0
  3. data/bin/cloudlib +231 -0
  4. data/bin/cloudlib-web +207 -0
  5. data/cloudlib.gemspec +28 -0
  6. data/lib/cloudlib.rb +380 -0
  7. metadata +96 -0
data/LICENSE ADDED
@@ -0,0 +1,340 @@
1
+ GNU GENERAL PUBLIC LICENSE
2
+ Version 2, June 1991
3
+
4
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
5
+ 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
6
+ Everyone is permitted to copy and distribute verbatim copies
7
+ of this license document, but changing it is not allowed.
8
+
9
+ Preamble
10
+
11
+ The licenses for most software are designed to take away your
12
+ freedom to share and change it. By contrast, the GNU General Public
13
+ License is intended to guarantee your freedom to share and change free
14
+ software--to make sure the software is free for all its users. This
15
+ General Public License applies to most of the Free Software
16
+ Foundation's software and to any other program whose authors commit to
17
+ using it. (Some other Free Software Foundation software is covered by
18
+ the GNU Library General Public License instead.) You can apply it to
19
+ your programs, too.
20
+
21
+ When we speak of free software, we are referring to freedom, not
22
+ price. Our General Public Licenses are designed to make sure that you
23
+ have the freedom to distribute copies of free software (and charge for
24
+ this service if you wish), that you receive source code or can get it
25
+ if you want it, that you can change the software or use pieces of it
26
+ in new free programs; and that you know you can do these things.
27
+
28
+ To protect your rights, we need to make restrictions that forbid
29
+ anyone to deny you these rights or to ask you to surrender the rights.
30
+ These restrictions translate to certain responsibilities for you if you
31
+ distribute copies of the software, or if you modify it.
32
+
33
+ For example, if you distribute copies of such a program, whether
34
+ gratis or for a fee, you must give the recipients all the rights that
35
+ you have. You must make sure that they, too, receive or can get the
36
+ source code. And you must show them these terms so they know their
37
+ rights.
38
+
39
+ We protect your rights with two steps: (1) copyright the software, and
40
+ (2) offer you this license which gives you legal permission to copy,
41
+ distribute and/or modify the software.
42
+
43
+ Also, for each author's protection and ours, we want to make certain
44
+ that everyone understands that there is no warranty for this free
45
+ software. If the software is modified by someone else and passed on, we
46
+ want its recipients to know that what they have is not the original, so
47
+ that any problems introduced by others will not reflect on the original
48
+ authors' reputations.
49
+
50
+ Finally, any free program is threatened constantly by software
51
+ patents. We wish to avoid the danger that redistributors of a free
52
+ program will individually obtain patent licenses, in effect making the
53
+ program proprietary. To prevent this, we have made it clear that any
54
+ patent must be licensed for everyone's free use or not licensed at all.
55
+
56
+ The precise terms and conditions for copying, distribution and
57
+ modification follow.
58
+
59
+ GNU GENERAL PUBLIC LICENSE
60
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
61
+
62
+ 0. This License applies to any program or other work which contains
63
+ a notice placed by the copyright holder saying it may be distributed
64
+ under the terms of this General Public License. The "Program", below,
65
+ refers to any such program or work, and a "work based on the Program"
66
+ means either the Program or any derivative work under copyright law:
67
+ that is to say, a work containing the Program or a portion of it,
68
+ either verbatim or with modifications and/or translated into another
69
+ language. (Hereinafter, translation is included without limitation in
70
+ the term "modification".) Each licensee is addressed as "you".
71
+
72
+ Activities other than copying, distribution and modification are not
73
+ covered by this License; they are outside its scope. The act of
74
+ running the Program is not restricted, and the output from the Program
75
+ is covered only if its contents constitute a work based on the
76
+ Program (independent of having been made by running the Program).
77
+ Whether that is true depends on what the Program does.
78
+
79
+ 1. You may copy and distribute verbatim copies of the Program's
80
+ source code as you receive it, in any medium, provided that you
81
+ conspicuously and appropriately publish on each copy an appropriate
82
+ copyright notice and disclaimer of warranty; keep intact all the
83
+ notices that refer to this License and to the absence of any warranty;
84
+ and give any other recipients of the Program a copy of this License
85
+ along with the Program.
86
+
87
+ You may charge a fee for the physical act of transferring a copy, and
88
+ you may at your option offer warranty protection in exchange for a fee.
89
+
90
+ 2. You may modify your copy or copies of the Program or any portion
91
+ of it, thus forming a work based on the Program, and copy and
92
+ distribute such modifications or work under the terms of Section 1
93
+ above, provided that you also meet all of these conditions:
94
+
95
+ a) You must cause the modified files to carry prominent notices
96
+ stating that you changed the files and the date of any change.
97
+
98
+ b) You must cause any work that you distribute or publish, that in
99
+ whole or in part contains or is derived from the Program or any
100
+ part thereof, to be licensed as a whole at no charge to all third
101
+ parties under the terms of this License.
102
+
103
+ c) If the modified program normally reads commands interactively
104
+ when run, you must cause it, when started running for such
105
+ interactive use in the most ordinary way, to print or display an
106
+ announcement including an appropriate copyright notice and a
107
+ notice that there is no warranty (or else, saying that you provide
108
+ a warranty) and that users may redistribute the program under
109
+ these conditions, and telling the user how to view a copy of this
110
+ License. (Exception: if the Program itself is interactive but
111
+ does not normally print such an announcement, your work based on
112
+ the Program is not required to print an announcement.)
113
+
114
+ These requirements apply to the modified work as a whole. If
115
+ identifiable sections of that work are not derived from the Program,
116
+ and can be reasonably considered independent and separate works in
117
+ themselves, then this License, and its terms, do not apply to those
118
+ sections when you distribute them as separate works. But when you
119
+ distribute the same sections as part of a whole which is a work based
120
+ on the Program, the distribution of the whole must be on the terms of
121
+ this License, whose permissions for other licensees extend to the
122
+ entire whole, and thus to each and every part regardless of who wrote it.
123
+
124
+ Thus, it is not the intent of this section to claim rights or contest
125
+ your rights to work written entirely by you; rather, the intent is to
126
+ exercise the right to control the distribution of derivative or
127
+ collective works based on the Program.
128
+
129
+ In addition, mere aggregation of another work not based on the Program
130
+ with the Program (or with a work based on the Program) on a volume of
131
+ a storage or distribution medium does not bring the other work under
132
+ the scope of this License.
133
+
134
+ 3. You may copy and distribute the Program (or a work based on it,
135
+ under Section 2) in object code or executable form under the terms of
136
+ Sections 1 and 2 above provided that you also do one of the following:
137
+
138
+ a) Accompany it with the complete corresponding machine-readable
139
+ source code, which must be distributed under the terms of Sections
140
+ 1 and 2 above on a medium customarily used for software interchange; or,
141
+
142
+ b) Accompany it with a written offer, valid for at least three
143
+ years, to give any third party, for a charge no more than your
144
+ cost of physically performing source distribution, a complete
145
+ machine-readable copy of the corresponding source code, to be
146
+ distributed under the terms of Sections 1 and 2 above on a medium
147
+ customarily used for software interchange; or,
148
+
149
+ c) Accompany it with the information you received as to the offer
150
+ to distribute corresponding source code. (This alternative is
151
+ allowed only for noncommercial distribution and only if you
152
+ received the program in object code or executable form with such
153
+ an offer, in accord with Subsection b above.)
154
+
155
+ The source code for a work means the preferred form of the work for
156
+ making modifications to it. For an executable work, complete source
157
+ code means all the source code for all modules it contains, plus any
158
+ associated interface definition files, plus the scripts used to
159
+ control compilation and installation of the executable. However, as a
160
+ special exception, the source code distributed need not include
161
+ anything that is normally distributed (in either source or binary
162
+ form) with the major components (compiler, kernel, and so on) of the
163
+ operating system on which the executable runs, unless that component
164
+ itself accompanies the executable.
165
+
166
+ If distribution of executable or object code is made by offering
167
+ access to copy from a designated place, then offering equivalent
168
+ access to copy the source code from the same place counts as
169
+ distribution of the source code, even though third parties are not
170
+ compelled to copy the source along with the object code.
171
+
172
+ 4. You may not copy, modify, sublicense, or distribute the Program
173
+ except as expressly provided under this License. Any attempt
174
+ otherwise to copy, modify, sublicense or distribute the Program is
175
+ void, and will automatically terminate your rights under this License.
176
+ However, parties who have received copies, or rights, from you under
177
+ this License will not have their licenses terminated so long as such
178
+ parties remain in full compliance.
179
+
180
+ 5. You are not required to accept this License, since you have not
181
+ signed it. However, nothing else grants you permission to modify or
182
+ distribute the Program or its derivative works. These actions are
183
+ prohibited by law if you do not accept this License. Therefore, by
184
+ modifying or distributing the Program (or any work based on the
185
+ Program), you indicate your acceptance of this License to do so, and
186
+ all its terms and conditions for copying, distributing or modifying
187
+ the Program or works based on it.
188
+
189
+ 6. Each time you redistribute the Program (or any work based on the
190
+ Program), the recipient automatically receives a license from the
191
+ original licensor to copy, distribute or modify the Program subject to
192
+ these terms and conditions. You may not impose any further
193
+ restrictions on the recipients' exercise of the rights granted herein.
194
+ You are not responsible for enforcing compliance by third parties to
195
+ this License.
196
+
197
+ 7. If, as a consequence of a court judgment or allegation of patent
198
+ infringement or for any other reason (not limited to patent issues),
199
+ conditions are imposed on you (whether by court order, agreement or
200
+ otherwise) that contradict the conditions of this License, they do not
201
+ excuse you from the conditions of this License. If you cannot
202
+ distribute so as to satisfy simultaneously your obligations under this
203
+ License and any other pertinent obligations, then as a consequence you
204
+ may not distribute the Program at all. For example, if a patent
205
+ license would not permit royalty-free redistribution of the Program by
206
+ all those who receive copies directly or indirectly through you, then
207
+ the only way you could satisfy both it and this License would be to
208
+ refrain entirely from distribution of the Program.
209
+
210
+ If any portion of this section is held invalid or unenforceable under
211
+ any particular circumstance, the balance of the section is intended to
212
+ apply and the section as a whole is intended to apply in other
213
+ circumstances.
214
+
215
+ It is not the purpose of this section to induce you to infringe any
216
+ patents or other property right claims or to contest validity of any
217
+ such claims; this section has the sole purpose of protecting the
218
+ integrity of the free software distribution system, which is
219
+ implemented by public license practices. Many people have made
220
+ generous contributions to the wide range of software distributed
221
+ through that system in reliance on consistent application of that
222
+ system; it is up to the author/donor to decide if he or she is willing
223
+ to distribute software through any other system and a licensee cannot
224
+ impose that choice.
225
+
226
+ This section is intended to make thoroughly clear what is believed to
227
+ be a consequence of the rest of this License.
228
+
229
+ 8. If the distribution and/or use of the Program is restricted in
230
+ certain countries either by patents or by copyrighted interfaces, the
231
+ original copyright holder who places the Program under this License
232
+ may add an explicit geographical distribution limitation excluding
233
+ those countries, so that distribution is permitted only in or among
234
+ countries not thus excluded. In such case, this License incorporates
235
+ the limitation as if written in the body of this License.
236
+
237
+ 9. The Free Software Foundation may publish revised and/or new versions
238
+ of the General Public License from time to time. Such new versions will
239
+ be similar in spirit to the present version, but may differ in detail to
240
+ address new problems or concerns.
241
+
242
+ Each version is given a distinguishing version number. If the Program
243
+ specifies a version number of this License which applies to it and "any
244
+ later version", you have the option of following the terms and conditions
245
+ either of that version or of any later version published by the Free
246
+ Software Foundation. If the Program does not specify a version number of
247
+ this License, you may choose any version ever published by the Free Software
248
+ Foundation.
249
+
250
+ 10. If you wish to incorporate parts of the Program into other free
251
+ programs whose distribution conditions are different, write to the author
252
+ to ask for permission. For software which is copyrighted by the Free
253
+ Software Foundation, write to the Free Software Foundation; we sometimes
254
+ make exceptions for this. Our decision will be guided by the two goals
255
+ of preserving the free status of all derivatives of our free software and
256
+ of promoting the sharing and reuse of software generally.
257
+
258
+ NO WARRANTY
259
+
260
+ 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
261
+ FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
262
+ OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
263
+ PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
264
+ OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
265
+ MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
266
+ TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
267
+ PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
268
+ REPAIR OR CORRECTION.
269
+
270
+ 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
271
+ WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
272
+ REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
273
+ INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
274
+ OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
275
+ TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
276
+ YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
277
+ PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
278
+ POSSIBILITY OF SUCH DAMAGES.
279
+
280
+ END OF TERMS AND CONDITIONS
281
+
282
+ How to Apply These Terms to Your New Programs
283
+
284
+ If you develop a new program, and you want it to be of the greatest
285
+ possible use to the public, the best way to achieve this is to make it
286
+ free software which everyone can redistribute and change under these terms.
287
+
288
+ To do so, attach the following notices to the program. It is safest
289
+ to attach them to the start of each source file to most effectively
290
+ convey the exclusion of warranty; and each file should have at least
291
+ the "copyright" line and a pointer to where the full notice is found.
292
+
293
+ <one line to give the program's name and a brief idea of what it does.>
294
+ Copyright (C) <year> <name of author>
295
+
296
+ This program is free software; you can redistribute it and/or modify
297
+ it under the terms of the GNU General Public License as published by
298
+ the Free Software Foundation; either version 2 of the License, or
299
+ (at your option) any later version.
300
+
301
+ This program is distributed in the hope that it will be useful,
302
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
303
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
304
+ GNU General Public License for more details.
305
+
306
+ You should have received a copy of the GNU General Public License
307
+ along with this program; if not, write to the Free Software
308
+ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
309
+
310
+
311
+ Also add information on how to contact you by electronic and paper mail.
312
+
313
+ If the program is interactive, make it output a short notice like this
314
+ when it starts in an interactive mode:
315
+
316
+ Gnomovision version 69, Copyright (C) year name of author
317
+ Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
318
+ This is free software, and you are welcome to redistribute it
319
+ under certain conditions; type `show c' for details.
320
+
321
+ The hypothetical commands `show w' and `show c' should show the appropriate
322
+ parts of the General Public License. Of course, the commands you use may
323
+ be called something other than `show w' and `show c'; they could even be
324
+ mouse-clicks or menu items--whatever suits your program.
325
+
326
+ You should also get your employer (if you work as a programmer) or your
327
+ school, if any, to sign a "copyright disclaimer" for the program, if
328
+ necessary. Here is a sample; alter the names:
329
+
330
+ Yoyodyne, Inc., hereby disclaims all copyright interest in the program
331
+ `Gnomovision' (which makes passes at compilers) written by James Hacker.
332
+
333
+ <signature of Ty Coon>, 1 April 1989
334
+ Ty Coon, President of Vice
335
+
336
+ This General Public License does not permit incorporating your program into
337
+ proprietary programs. If your program is a subroutine library, you may
338
+ consider it more useful to permit linking proprietary applications with the
339
+ library. If this is what you want to do, use the GNU Library General
340
+ Public License instead of this License.
data/README ADDED
@@ -0,0 +1,91 @@
1
+ cloudlib is a system for maintaining a database of electronic papers
2
+ and books using Amazon's web services (for background, see
3
+ http://aws.amazon.com/). Think of it as an indefinitely extensible
4
+ personal library, accessible from anywhere in the world.
5
+
6
+ The papers and books themselves are stored in a S3 bucket. The
7
+ metadata are stored in a SimpleDB database, so they can be searched easily.
8
+ The cloudlib ruby library integrates these two components and insulates
9
+ the user from the S3- and SimpleDB-specific details.
10
+
11
+ Note that S3 and SimpleDB are pay services. You will pay Amazon
12
+ proportionally to your usage. As of this writing (December 2008),
13
+ SimpleDB is free for the kind of usage this library normally requires.
14
+ For S3, the fee in North America is $0.15/GB/month for storage.
15
+ (See http://aws.amazon.com/s3/#pricing for up-to-date figures, including
16
+ fees for data transfer.) At these rates, it will cost less than a tenth
17
+ of a cent to store an average journal article for a year, and less than
18
+ a penny to store a good-sized book.
19
+
20
+ In addition to a ruby library, two programs are provided:
21
+
22
+ - cloudlib offers a command-line interface for interacting with a library.
23
+
24
+ - cloudlib-web starts a web-server which can either be used locally
25
+ (so that the library can be controlled through a browser) or made available
26
+ on the open internet. HTTP authentication is used to password-protect
27
+ the web application.
28
+
29
+ Both programs assume that the following environment variables have been set.
30
+ (If they are not, they will prompt for these values.)
31
+
32
+ - CLOUDLIB_LIBRARY_NAME - a name, of your choosing, that will identify the
33
+ library. It will be used both for the S3 bucket and the SimpleDB domain.
34
+ S3 bucket names must be unique, so pick a name like
35
+ 'your-name-library', not something generic like 'my-library'.
36
+
37
+ - AWS_ACCESS_KEY_ID - the access key id you are provided when you sign up
38
+ for Amazon web services.
39
+
40
+ - AWS_SECRET_ACCESS_KEY - the secret key you are provided when you sign
41
+ up for Amazon web services.
42
+
43
+ cloudlib-web also assumes that the following environment variables have
44
+ been set:
45
+
46
+ - CLOUDLIB_WEB_USERNAME - a username of your choice, to be used to gain
47
+ access to the web interface.
48
+
49
+ - CLOUDLIB_WEB_PASSWORD - a password of your choice, to be used to gain
50
+ access to the web interface.
51
+
52
+ These environment variables can be set as follows:
53
+
54
+ export AWS_ACCESS_KEY_ID=xxxx-my-key-id-xxx
55
+ export AWS_SECRET_ACCESS_KEY=xxx-my-key-xxx
56
+ export CLOUDLIB_LIBRARY_NAME=xxx-my-library-name-xxx
57
+ export CLOUDLIB_WEB_USERNAME=xxx-username-xxx
58
+ export CLOUDLIB_WEB_PASSWORD=xxx-password-xxx
59
+
60
+ You may want to put these commands in a file, cloudlib-setup,
61
+ so that you can set all the variables at once:
62
+
63
+ source cloudlib-setup
64
+
65
+ After setting the environment variables, but before doing anything else,
66
+ you will need to create your library:
67
+
68
+ cloudlib new-library
69
+
70
+ If that succeeds, you can use either cloudlib or cloudlib-web to access
71
+ the library. By default, cloudlib-web will start a webserver on port 4567.
72
+ (This can be changed using the -p PORT option.) To use the web interface
73
+ after you've started the server, just point your browser at <http://localhost:4567>.
74
+
75
+ For instructions on the use of cloudlib, try
76
+
77
+ cloudlib --help
78
+
79
+ cloudlib by itself will start an interactive session. To add a new file,
80
+ use
81
+
82
+ cloudlib add /path/to/file
83
+
84
+ The commands 'dump' and 'restore' are also provided to allow creation of
85
+ a local backup of the database.
86
+
87
+ To install cloudlib:
88
+
89
+ gem sources -a http://gems.github.com # you only have to do this once
90
+ sudo gem install jgm-cloudlib
91
+
data/bin/cloudlib ADDED
@@ -0,0 +1,231 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'yaml'
4
+ require 'rubygems'
5
+ require 'digest/sha1'
6
+ require 'lib/cloudlib' # cloudlib gem
7
+ require 'optparse'
8
+ require 'highline/import' # highline gem
9
+
10
+ PROGNAME = "cloudlib"
11
+ CLOUDLIB_VERSION = "0.1"
12
+ NUMITEMS = 10
13
+
14
+ options = {}
15
+ opts = OptionParser.new do |opts|
16
+ opts.program_name = "#{PROGNAME} (c) 2008 John MacFarlane"
17
+ opts.version = "version #{CLOUDLIB_VERSION}"
18
+ opts.banner = "cloudlib -- a library of books and articles in the AWS `cloud'\n" +
19
+ "Usage: #{PROGNAME} - start interactive menu\n" +
20
+ " #{PROGNAME} add FILE - upload FILE to library, prompting for metadata\n" +
21
+ " #{PROGNAME} new-library - initialize new library\n" +
22
+ " #{PROGNAME} delete-library - delete library and all its entries\n" +
23
+ " #{PROGNAME} dump [PATH] - create a local backup of library in PATH\n" +
24
+ " #{PROGNAME} restore [PATH] - restore library from local backup in PATH\n" +
25
+ "Options:"
26
+ end
27
+ opts.on("-v", "--version", "Show version") do
28
+ puts opts.ver
29
+ exit 0
30
+ end
31
+ opts.on("-h", "--help", "Show usage message") do
32
+ puts opts.help
33
+ exit 0
34
+ end
35
+
36
+ begin
37
+ opts.parse!
38
+ rescue OptionParser::ParseError => e
39
+ puts e.message
40
+ puts opts.help
41
+ exit 1
42
+ end
43
+
44
+ # check that required environment variables are set, and prompt if they aren't
45
+ envvars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "CLOUDLIB_LIBRARY_NAME"]
46
+ envvars.each do |var|
47
+ unless ENV[var]
48
+ ENV[var] = ask("#{var}: ", String) { |q| q.echo = if var == "AWS_SECRET_ACCESS_KEY" then "*" else true end }
49
+ end
50
+ end
51
+
52
+ Cloudlib::Entry.connect(ENV['CLOUDLIB_LIBRARY_NAME'], ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
53
+
54
+ class Cloudlib::Entry
55
+ # Prompts for metadata for the entry, using existing metadata as defaults.
56
+ def ask_attributes
57
+ default_type = if self.show_attribute('entry_type').empty?
58
+ "article"
59
+ else
60
+ self.show_attribute('entry_type')
61
+ end
62
+ entry_types = ['article','book','chapter','incollection','unpublished']
63
+ type = ask("Type (#{entry_types.join(', ')})? ", entry_types) {|q| q.default = default_type; q.case = :downcase; q.readline = true }
64
+ self.attributes['entry_type'] = [type]
65
+ self.fields.each { |field| self.ask_attribute(field.to_s) }
66
+ return self
67
+ end
68
+
69
+ # Prompts for metadata for a particular field.
70
+ def ask_attribute(attribute)
71
+ default = self.show_attribute(attribute)
72
+ ans = ask(attribute.capitalize +
73
+ if attribute == 'editors' || attribute == 'authors' then " (name [and name...]) " else " " end, String) { |q| q.readline = true; q.default = default }
74
+ set_attribute(attribute, ans)
75
+ end
76
+
77
+ end
78
+
79
+ def show_items(items, more=false)
80
+ items.each_index do |i|
81
+ say "<%= color('[#{i}]', BOLD) %> #{items[i].to_s}\n"
82
+ end
83
+ if more
84
+ say "<%= color('Enter', BOLD)%> for more...\n"
85
+ end
86
+ end
87
+
88
+ def menu(items)
89
+ token = query = ""
90
+ while true
91
+ commands = ["find KEYWORDS", "quit"]
92
+ if items.length > 0
93
+ then commands = ["bib NUM", "get NUM", "del NUM", "mod NUM"] + commands
94
+ end
95
+ choices = commands.join(' | ')
96
+ ans = ask("#{choices} ? ", String) { |q| q.readline = true; q.case = :downcase }
97
+ if ans == ""
98
+ if token.empty?
99
+ items = []
100
+ else
101
+ token, items = Cloudlib::Entry.query(query, NUMITEMS, token)
102
+ end
103
+ show_items(items, more=(not token.empty?))
104
+ elsif ans =~ /^quit|q$/
105
+ exit 0
106
+ elsif ans =~ /^find *(.+)$/
107
+ query = $1
108
+ token, items = Cloudlib::Entry.query(query, NUMITEMS)
109
+ show_items(items, more=(not token.empty?))
110
+ elsif ans =~ /^(get|del|mod|bib) *(\d+)$/
111
+ num = $2.to_i
112
+ if (num < 0) || (num >= items.length)
113
+ next
114
+ else
115
+ item = items[num]
116
+ case $1
117
+ when "get":
118
+ destpath = ask("Save as: ", String) { |q| q.default = item.friendly_filename; q.readline = true }
119
+ item.download(destpath)
120
+ puts "Downloaded #{destpath}"
121
+ when "mod":
122
+ item.ask_attributes
123
+ item.save
124
+ when "del":
125
+ item.delete
126
+ when "bib":
127
+ puts item.to_bibtex
128
+ else
129
+ raise "Unknown command."
130
+ end
131
+ end
132
+ else
133
+ puts "Unknown command"
134
+ end
135
+ end
136
+ end
137
+
138
+ if ARGV.length >= 1
139
+ if ARGV[0] == "new-library"
140
+ print "Create new library `#{ENV['CLOUDLIB_LIBRARY_NAME']}' (y/n)? "
141
+ ans = STDIN.gets
142
+ if ans =~ /^[Yy]/
143
+ begin
144
+ Cloudlib::Entry.create_library
145
+ rescue AWS::S3::BucketAlreadyExists
146
+ STDERR.puts "The library name `#{ENV['CLOUDLIB_LIBRARY_NAME']}' is already taken by another user."
147
+ STDERR.puts "Please set CLOUDLIB_LIBRARY_NAME to something else and try again."
148
+ exit 1
149
+ end
150
+ end
151
+ exit 0
152
+ end
153
+ if ARGV[0] == "delete-library"
154
+ print "Delete `#{ENV['CLOUDLIB_LIBRARY_NAME']}' and ALL OF ITS CONTENTS (y/n)? "
155
+ ans = STDIN.gets
156
+ if ans =~ /^[Yy]/
157
+ Cloudlib::Entry.delete_library
158
+ end
159
+ exit 0
160
+ end
161
+ if ARGV[0] == "backup"
162
+ print "Delete `#{ENV['CLOUDLIB_LIBRARY_NAME']}' and ALL OF ITS CONTENTS (y/n)? "
163
+ ans = STDIN.gets
164
+ if ans =~ /^[Yy]/
165
+ Cloudlib::Entry.delete_library
166
+ end
167
+ exit 0
168
+ end
169
+ if ARGV[0] == "add"
170
+ ARGV[1..(ARGV.length - 1)].each do |target|
171
+ unless File.exists?(target)
172
+ puts "File not found: #{target}"
173
+ exit 1
174
+ end
175
+ item = Cloudlib::Entry.from_file(target)
176
+ puts "Please enter metadata for `#{target}':"
177
+ item.ask_attributes
178
+ item.save
179
+ puts "Uploaded #{target}"
180
+ end
181
+ exit 0
182
+ end
183
+ if ARGV[0] == "dump"
184
+ path = ARGV[1] || "."
185
+ new_entries = 0
186
+ dummy, entries = Cloudlib::Entry.query("", nil)
187
+ total_size = entries.inject(0) {|accum, e| accum + e.attributes['size'][0].to_i}
188
+ printf("Total size is %.2f megabytes. Make a local copy (y/n)? ", (total_size / (1024.0 * 1024.0)))
189
+ ans = STDIN.gets
190
+ if ans =~ /^[Yy]/
191
+ open("#{path}/#{ENV['CLOUDLIB_LIBRARY_NAME']}.db", 'w') do |file|
192
+ STDERR.puts "Backing up metadata..."
193
+ file.write(YAML.dump(entries))
194
+ end
195
+ STDERR.puts "Backing up files:"
196
+ entries.each do |entry|
197
+ if not File.exists?("#{path}/#{entry.name}")
198
+ entry.download("#{path}/#{entry.name}")
199
+ new_entries += 1
200
+ STDERR.puts entry.to_s
201
+ end
202
+ end
203
+ STDERR.puts "Backed up metadata and #{new_entries} new files."
204
+ end
205
+ exit 0
206
+ end
207
+ if ARGV[0] == "restore"
208
+ path = ARGV[1] || "."
209
+ entries = open("#{path}/#{ENV['CLOUDLIB_LIBRARY_NAME']}.db", 'r') { |file| YAML.load(file) }
210
+ entries.each do |entry|
211
+ filename = "#{path}/#{entry.name}"
212
+ Cloudlib::Entry.from_file(filename)
213
+ entry.save
214
+ STDERR.puts entry.to_s
215
+ end
216
+ STDERR.puts "Restored #{entries.length} entries."
217
+ exit 0
218
+ end
219
+ STDERR.puts "Unknown command #{ARGV[0]}."
220
+ puts opts.help
221
+ exit 1
222
+ else
223
+ begin
224
+ menu([])
225
+ rescue AwsSdb::NoSuchDomainError
226
+ STDERR.puts "The library `#{ENV['CLOUDLIB_LIBRARY_NAME']}' does not exist."
227
+ STDERR.puts "Use `#{PROGNAME} new-library' to create it."
228
+ exit 1
229
+ end
230
+ end
231
+
data/bin/cloudlib-web ADDED
@@ -0,0 +1,207 @@
1
+ #!/usr/bin/env ruby
2
+ require 'rubygems'
3
+ require 'digest/sha1'
4
+ require 'sinatra' # sinatra gem
5
+ require 'tempfile'
6
+ require 'cloudlib'
7
+ require 'highline/import' # highline gem
8
+
9
+ # Number of items to display per page after query
10
+ NUMITEMS = 10
11
+
12
+ # check that required environment variables are set
13
+ envvars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "CLOUDLIB_LIBRARY_NAME", "CLOUDLIB_WEB_USERNAME", "CLOUDLIB_WEB_PASSWORD"]
14
+ envvars.each do |var|
15
+ unless ENV[var]
16
+ ENV[var] = ask("#{var}: ", String) { |q| q.echo = if var == "AWS_SECRET_ACCESS_KEY" then "*" else true end }
17
+ end
18
+ end
19
+ Cloudlib::Entry.connect(ENV['CLOUDLIB_LIBRARY_NAME'], ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
20
+
21
+ use Rack::Auth::Basic do |username, password|
22
+ username == ENV['CLOUDLIB_WEB_USERNAME'] &&
23
+ password == ENV['CLOUDLIB_WEB_PASSWORD']
24
+ end
25
+
26
+ get '/stylesheet.css' do
27
+ content_type 'text/css', :charset => 'utf-8'
28
+ sass :stylesheet
29
+ end
30
+
31
+ get '/' do
32
+ @token, @entries = "", []
33
+ @query = ""
34
+ haml :index
35
+ end
36
+
37
+ post '/' do
38
+ if params[:query]
39
+ @query = params[:query]
40
+ @token, @entries = Cloudlib::Entry.query(@query, NUMITEMS, params[:token])
41
+ else
42
+ @token, @entries = "", []
43
+ @query = ""
44
+ end
45
+ haml :index
46
+ end
47
+
48
+ get '/upload' do
49
+ haml :upload
50
+ end
51
+
52
+ post '/upload' do
53
+ tempfile = params[:fileToUpload][:tempfile]
54
+ tempfilepath = tempfile.path
55
+ tempfile.close
56
+ entry = Cloudlib::Entry.from_file(tempfilepath, params[:fileToUpload][:filename])
57
+ set_attributes_from_form(entry)
58
+ entry.save
59
+ redirect "/"
60
+ end
61
+
62
+ get '/*/bibtex' do
63
+ @entry = Cloudlib::Entry.find_by_name(params[:splat][0])
64
+ content_type 'text/plain', :charset => 'utf-8'
65
+ @entry.to_bibtex
66
+ end
67
+
68
+ get '/*' do
69
+ name = "#{params[:splat][0]}"
70
+ @entry = Cloudlib::Entry.find_by_name(name)
71
+ haml :modify
72
+ end
73
+
74
+ post '/*' do
75
+ entry = Cloudlib::Entry.find_by_name(params[:splat][0])
76
+ set_attributes_from_form(entry)
77
+ entry.save
78
+ redirect '/'
79
+ end
80
+
81
+ delete '/*' do
82
+ entry = Cloudlib::Entry.find_by_name(params[:splat][0])
83
+ entry.delete
84
+ redirect '/'
85
+ end
86
+
87
+ def field_for_type?(field, type)
88
+ Cloudlib::Entry.fields(type).member?(field)
89
+ end
90
+
91
+ def show_fields(type)
92
+ cmds = Cloudlib::Entry.fields.map do |field|
93
+ "document.getElementById('#{field.to_s}').setAttribute('style', 'display: #{if field_for_type?(field, type) then 'all' else 'none' end}'); "
94
+ end
95
+ return cmds.join
96
+ end
97
+
98
+ def set_attributes_from_form(entry)
99
+ entry.attributes['entry_type'] = params['entry_type']
100
+ Cloudlib::Entry.fields.each do |field|
101
+ if field_for_type?(field, params['entry_type']) && params[field]
102
+ entry.set_attribute(field.to_s, params[field])
103
+ else
104
+ entry.set_attribute(field.to_s, '')
105
+ end
106
+ end
107
+ end
108
+
109
+ use_in_file_templates!
110
+
111
+ __END__
112
+
113
+ @@ layout
114
+ !!! Strict
115
+ %head
116
+ %link{:href => '/stylesheet.css', :type => 'text/css', :media => 'all', :rel => 'stylesheet'}
117
+ %title
118
+ = ENV['CLOUDLIB_LIBRARY_NAME']
119
+ %body
120
+ %h1
121
+ %a{:href => '/'}
122
+ = ENV['CLOUDLIB_LIBRARY_NAME']
123
+ %div#content
124
+ = yield
125
+ %div#footer
126
+ powered by
127
+ %a{:href => ''}cloudlib
128
+
129
+ @@ index
130
+ %div.queryform
131
+ %form{:method => 'POST', :action => '/'}
132
+ %input{:type => 'text', :name => 'query', :value => @query, :size => '30'}
133
+ %input{:type => 'submit', :value => 'Search'}
134
+ %ol
135
+ - @entries.each do |i|
136
+ %li
137
+ = i.to_s
138
+ %a{:href => "/#{i.name}/bibtex"}bibtex
139
+ %span.separator &bull;
140
+ %a{:href => "/#{i.name}"}modify
141
+ %span.separator &bull;
142
+ %a{:href => i.url}download
143
+ - if not @token.empty?
144
+ %form{:method => 'POST', :action => '/'}
145
+ %input{:type => 'text', :name => 'query', :value => @query, :style => 'display: none'}
146
+ %input{:type => 'text', :name => 'token', :value => @token, :style => 'display: none'}
147
+ %input{:type => 'submit', :value => 'More matches...'}
148
+ %a{:href => "/upload"}upload
149
+
150
+ @@ upload
151
+ %form{:method => 'POST', :action => '/upload', :enctype => 'multipart/form-data'}
152
+ %label Select file to upload:
153
+ %br
154
+ %input{:type => 'file', :name => 'fileToUpload', :size => 40}
155
+ = haml :metadata, :layout => false
156
+ %input{:type => 'submit', :value => 'Add file to library'}
157
+
158
+ @@ metadata
159
+ %table
160
+ %tr
161
+ %td
162
+ %label Type:
163
+ %td
164
+ %select{:name => 'entry_type'}
165
+ - ['','article','book','incollection','chapter','unpublished'].each do |type|
166
+ %option{:onClick => show_fields(type), :selected => (@entry && @entry.show_attribute('entry_type') == type) || type.empty?}
167
+ = type
168
+ - Cloudlib::Entry.fields.each do |field|
169
+ %tr{:style => (@entry && field_for_type?(field, @entry.show_attribute('entry_type'))) || 'display: none;', :id => field.to_s}
170
+ %td
171
+ %label
172
+ = field.to_s.capitalize + ':'
173
+ %td
174
+ %input{:type => 'text', :name => field.to_s, :value => (@entry && @entry.show_attribute(field.to_s)) || '', :size => 50}
175
+
176
+ @@ modify
177
+ %div.detail
178
+ %form{:method => 'POST', :action => "/#{@entry.name}"}
179
+ = haml :metadata, :layout => false
180
+ %p
181
+ %input{:type => 'submit', :value => 'Update metadata'}
182
+ %form{:method => 'POST', :action => "/#{@entry.name}"}
183
+ %p
184
+ %input{:type => 'text', :name => '_method', :value => 'delete', :style => 'display: none;'}
185
+ %input{:type => 'submit', :value => 'Delete this entry'}
186
+
187
+ @@ stylesheet
188
+ body
189
+ font-size: small
190
+ padding: 10px
191
+ h1
192
+ border-top: 1px solid gray
193
+ border-bottom: 1px solid gray
194
+ h1 a
195
+ color: #7a7a7a
196
+ text-decoration: none
197
+ &:visited
198
+ color: #7a7a7a
199
+ li
200
+ padding-bottom: 0.3em
201
+ #footer
202
+ border-top: 1px solid gray
203
+ margin-top: 1em
204
+ padding-top: 1em
205
+ font-size: x-small
206
+ text-align: center
207
+
data/cloudlib.gemspec ADDED
@@ -0,0 +1,28 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = "cloudlib"
3
+ s.version = "0.1"
4
+ s.date = "2008-12-25"
5
+ s.summary = "Tools for maintaining a library of books and articles in Amazon S3 and SimpleDB"
6
+ s.email = "jgm@berkeley.edu"
7
+ s.homepage = "http://github.com/jgm/cloudlib"
8
+ s.description = "Cloudlib is a ruby library and commands for maintaining a library of books and articles on the Amazon 'cloud': S3 and SimpleDB."
9
+ s.has_rdoc = true
10
+ s.authors = ["John MacFarlane"]
11
+ s.bindir = "bin"
12
+ s.executables = ["cloudlib", "cloudlib-web"]
13
+ s.default_executable = "cloudlib"
14
+ s.files = [ "README",
15
+ "LICENSE",
16
+ "cloudlib.gemspec",
17
+ "lib/cloudlib.rb",
18
+ "bin/cloudlib",
19
+ "bin/cloudlib-web" ]
20
+ s.test_files = []
21
+ s.rdoc_options = ["--main", "README", "--inline-source"]
22
+ s.extra_rdoc_files = ["README"]
23
+ s.add_dependency("aws-s3", [">= 0.5.1"])
24
+ s.add_dependency("aws-sdb", [">= 0.3.1"])
25
+ s.add_dependency("sinatra", [">= 0.3.2"])
26
+ s.add_dependency("highline", [">= 1.2.9"])
27
+ end
28
+
data/lib/cloudlib.rb ADDED
@@ -0,0 +1,380 @@
1
+ # This library provides the means for maintaining a database of
2
+ # documents on Amazon's S3 file store, with searchable metadata in
3
+ # Amazon's SimpleDB database. Think of it as a filing cabinet or library
4
+ # that can be extended indefinitely and accessed from anywhere in the
5
+ # world: a library that lives "in the cloud."
6
+
7
+ # In order to use this library, you need to sign up for
8
+ # Amazon's S3 and SimpleDB services:
9
+ #
10
+ # * Amazon SimpleDB: http://aws.amazon.com/simpledb/
11
+ # * Amazon S3: http://aws.amazon.com/s3/
12
+ #
13
+ # Simple usage example:
14
+ #
15
+ # require 'rubygems'
16
+ # require 'cloudlib'
17
+ # include Cloudlib
18
+ # Entry.connect('xxx_key_id_xxx', 'xxx_secret_access_key_xxx', 'my_aws_library')
19
+ # logic_entries = Entry.query('logic')
20
+ # logic_entries.each {|entry| puts entry.to_s}
21
+ #
22
+ # For more examples of the use of the library, see the programs cloudlib.rb
23
+ # and cloudlib-web.rb, included in the gem.
24
+
25
+ # Author:: John MacFarlane (jgm at berkeley dot edu)
26
+ # Copyright:: Copyright (c) 2008 John MacFarlane
27
+ # License:: GPL v2
28
+
29
+ require 'rubygems'
30
+ require 'readline'
31
+ require 'aws/s3' # aws-s3 gem
32
+ require 'aws_sdb' # aws-sdb gem
33
+ require 'open-uri'
34
+ require 'fileutils'
35
+
36
+ module Cloudlib
37
+
38
+ # A library entry, including content and metadata. An entry has a name
39
+ # (which is also the key of the associated S3 object) and an attributes
40
+ # hash. The name is of the form "sha1.ext", where sha1 is a SHA1 hash of
41
+ # the contents of the file, and ext is the file extension. This makes
42
+ # it impossible to have entries with duplicate contents. The attributes
43
+ # hash contains the following fields:
44
+ #
45
+ # * extension - file extension including .
46
+ # * size - size of contents (bytes)
47
+ # * date-added - date entry was added to library
48
+ # * entry_type - article, book, chapter, incollection, unpublished
49
+ # * authors - list of authors
50
+ # * editors - list of editors
51
+ # * title - title of entry
52
+ # * booktitle - title of book containing entry
53
+ # * year - publication year of entry
54
+ # * publisher - publisher of book
55
+ # * address - publication address
56
+ # * journal - journal containing entry
57
+ # * volume - volume number of journal
58
+ # * pages - page range of entry in book or journal
59
+ # * keywords - keywords
60
+ # * doi - DOI for entry
61
+ # * url - URL for entry
62
+ # * comments - miscellaneous comments
63
+ # * *_lowercase - lowercase version of *
64
+ # * *_words - lowercase version of *, split into a list of words
65
+ # * all_words - list of words in title, authors, editors, booktitle, keywords
66
+
67
+ class Entry
68
+
69
+ attr_accessor :name, :attributes
70
+
71
+ # Establish connections to the S3 file store and the SimpleDB database.
72
+ # If values are not supplied for the parameters, they will default to
73
+ # the values of the environment variables CLOUDLIB_LIBRARY_NAME,
74
+ # AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY. Note that library_name
75
+ # is the name of both the S3 bucket that will hold the contents of
76
+ # the entries and the SimpleDB domain that will hold the metadata.
77
+ def self.connect(library_name=ENV['CLOUDLIB_LIBRARY_NAME'],
78
+ aws_access_key_id=ENV['AWS_ACCESS_KEY_ID'],
79
+ aws_secret_access_key=ENV['AWS_SECRET_ACCESS_KEY'],
80
+ debug = false)
81
+ @@aws_access_key_id = aws_access_key_id
82
+ @@aws_secret_access_key = aws_secret_access_key
83
+ AWS::S3::Base.establish_connection!(:access_key_id => @@aws_access_key_id, :secret_access_key => @@aws_secret_access_key, :use_ssl => true)
84
+ @@bucket = library_name
85
+ logger = Logger.new(STDERR)
86
+ logger.level = if debug then Logger::DEBUG else Logger::WARN end
87
+ @@db = AwsSdb::Service.new(:access_key_id => @@aws_access_key_id, :secret_access_key => @@aws_secret_access_key, :use_ssl => true, :logger => logger)
88
+ end
89
+
90
+ # Creates a new entry object. To create an entry with contents,
91
+ # use Entry.from_file.
92
+ def initialize(name, attributes={'all_words' => []})
93
+ @name = name
94
+ @attributes = attributes
95
+ end
96
+
97
+ # Create the S3 bucket and SimpleDB domain that will store the library entries.
98
+ # This method should be run once to create the library.
99
+ def self.create_library
100
+ AWS::S3::Bucket.create(@@bucket)
101
+ @@db.create_domain(@@bucket)
102
+ end
103
+
104
+ # Delete the S3 bucket and SimpleDB domain that store the library entries.
105
+ # All data will be lost.
106
+ def self.delete_library
107
+ AWS::S3::Bucket.delete(@@bucket, :force => true)
108
+ @@db.delete_domain(@@bucket)
109
+ end
110
+
111
+ # Creates and saves an entry from a file, using attributes supplied.
112
+ # Returns the entry.
113
+ def self.from_file(path, filename=path, attributes={'all_words' => []})
114
+ sha1 = Digest::SHA1.file(path).hexdigest
115
+ ext = File.extname(filename)
116
+ name = "#{sha1}#{ext}"
117
+ attributes['size'] = File.size(path).to_s
118
+ attributes['date-added'] = Date.today.to_s
119
+ entry = Entry.new(name, attributes)
120
+ AWS::S3::S3Object.store(name, open(path), @@bucket)
121
+ @@db.put_attributes(@@bucket, name, attributes, replace=true)
122
+ return entry
123
+ end
124
+
125
+ # Return an entry with the specified name. Raises an error if not found.
126
+ def self.find_by_name(name)
127
+ attributes = @@db.get_attributes(@@bucket, name)
128
+ if attributes == {} then raise "Item not found." end
129
+ Entry.new(name, attributes)
130
+ end
131
+
132
+ # Queries the database and returns a list [token, entries]. entries is
133
+ # a list of up to numitems Entry objects that match the query. If
134
+ # there are more entries than numitems, token will be nonempty, and
135
+ # can be passed in on a subsequent calls for the remaining entries.
136
+ #
137
+ # The query string can contain one or more words. If a word is
138
+ # preceded by ti=, only entries that match it in the title will be
139
+ # returned. Similarly, au= searches authors, jo= journals, pu=
140
+ # publishers, ad= addresses, ed= editors, bo= booktitle (for collections),
141
+ # and ye= years. ye> and # ye< may also be used.
142
+ # The form ti='word1 word2' may also be used; entries will only match
143
+ # if their titles contain both word1 and word2.
144
+ def self.query(query_string, numitems=10, token=nil)
145
+ query_parts = query_string.downcase.scan(/((ti(?:title)|au(?:thor?s)|jo(?:urnal)|bo(?:ooktitle)|pu(?:blisher)|ad(?:ddress)|ed(?:itor?s)|ye(?:ar))[^<=>]*([<=>])('[^']*'|"[^"]*"|\S*)|\S+)\s*/)
146
+ query = query_parts.reject {|part| part[0] == '*'}.map do |part|
147
+ whole, key, comparison, val = part
148
+ if val then val = val.gsub(/^['"](.*)['"]$/, "\\1") end
149
+ if not val then val = whole end
150
+ key_full = case key
151
+ when 'ti'
152
+ 'title'
153
+ when 'au'
154
+ 'authors'
155
+ when 'jo'
156
+ 'journal'
157
+ when 'pu'
158
+ 'publisher'
159
+ when 'ad'
160
+ 'address'
161
+ when 'ed'
162
+ 'editors'
163
+ when 'ye'
164
+ 'year'
165
+ else 'all'
166
+ end
167
+ vals = val.split
168
+ vals.map do |v|
169
+ if key_full == 'year' # there is no year_words field
170
+ "['year' #{comparison} '#{v}']"
171
+ else
172
+ "['#{key_full}_words' = '#{v}']"
173
+ end
174
+ end.join(" intersection ")
175
+ end.join(" intersection ")
176
+ # note: query has to include year in order to sort by year
177
+ # hence this dummy search
178
+ if query.empty?
179
+ query = "['year' starts-with ''] sort 'year'"
180
+ else
181
+ query += " intersection ['year' starts-with ''] sort 'year'"
182
+ end
183
+ names, token = if token
184
+ @@db.query(@@bucket, query, numitems, token)
185
+ else
186
+ @@db.query(@@bucket, query, numitems)
187
+ end
188
+ entries = names.map do |name|
189
+ attributes = @@db.get_attributes(@@bucket, name)
190
+ Entry.new(name, attributes)
191
+ end
192
+ return token, entries
193
+ end
194
+
195
+ # Returns a human-friendly filename for the entry, constructed from
196
+ # authors and title.
197
+ def friendly_filename
198
+ authornames = self.attributes['authors'].map {|a| last_name(a)}.join('_')
199
+ title = self.show_attribute('title').gsub(/[,.\/[:space:]]+/,'_')
200
+ ext = File.extname(self.name)
201
+ return "#{authornames}_#{title}#{ext}"
202
+ end
203
+
204
+ # Deletes the entry.
205
+ def delete
206
+ AWS::S3::S3Object.delete(self.name, @@bucket)
207
+ @@db.delete_attributes(@@bucket, self.name)
208
+ end
209
+
210
+ # Saves the entry (metadata only; contents are saved by the from_file
211
+ # method).
212
+ def save
213
+ @@db.put_attributes(@@bucket, self.name, self.attributes, replace=true)
214
+ end
215
+
216
+ # Downloads the entry and saves as filename.
217
+ def download(path)
218
+ if File.exist?(path)
219
+ STDERR.puts "Backing up existing #{path} as #{path}~"
220
+ FileUtils.copy_file(path, "#{path}~", preserve=true)
221
+ end
222
+ open(path, 'w') do |outfile|
223
+ open(self.url, 'r') do |source|
224
+ FileUtils.copy_stream(source, outfile)
225
+ end
226
+ end
227
+ return path
228
+ end
229
+
230
+ # Returns a bibtex entry for the entry.
231
+ def to_bibtex
232
+ pairs = self.fields.map do |field|
233
+ if self.attributes[field.to_s]
234
+ sprintf(" %-15s: {%s}", field.to_s, self.show_attribute(field.to_s))
235
+ else
236
+ nil
237
+ end
238
+ end
239
+ pairs += [sprintf(" %-15s: {%s}", "file", self.name)]
240
+ authornames = self.attributes['authors'].map {|a| last_name(a)}.join('.')
241
+ year = self.attributes['year']
242
+ entry_type = self.show_attribute('entry_type') || 'unknown'
243
+ if entry_type == 'chapter' then entry_type = 'inbook' end
244
+ entry_key = "#{authornames}:#{year}"
245
+ "@#{entry_type.upcase}{#{entry_key},\n#{pairs.join(",\n")}\n}"
246
+ end
247
+
248
+ # Returns a string representation of the entry's metadata.
249
+ def to_s
250
+ authors = self.show_attribute('authors')
251
+ unless authors.empty?
252
+ authors = "#{authors}, "
253
+ end
254
+ title = "#{self.show_attribute('title')}"
255
+ year = self.show_attribute('year')
256
+ titleyear = if year.empty?
257
+ title + ". "
258
+ else
259
+ title + " (#{year}). "
260
+ end
261
+ pubaddr = [self.show_attribute('address'),
262
+ self.show_attribute('publisher')].reject {|x| x.empty?}.join(": ")
263
+ chapter = self.show_attribute('chapter')
264
+ pages = self.show_attribute('pages')
265
+ booktitle = self.show_attribute('booktitle')
266
+ editors = self.show_attribute('editors')
267
+ journal = self.show_attribute('journal')
268
+ volume = self.show_attribute('volume')
269
+ rest = case self.show_attribute('entry_type')
270
+ when 'article'
271
+ if journal.empty?
272
+ ""
273
+ else
274
+ "#{journal} #{volume}" +
275
+ if pages.empty? then "." else ", #{pages}." end
276
+ end
277
+ when 'book'
278
+ if pubaddr.empty? then "" else "#{pubaddr}." end
279
+ when 'chapter'
280
+ if pubaddr.empty? then "" else "#{pubaddr}." end +
281
+ if chapter.empty? then "" else " Chapter #{chapter}." end +
282
+ if pages.empty? then "" else " #{pages}." end
283
+ when 'incollection'
284
+ "In " +
285
+ if editors.empty? then "" else editors + " (eds.), " end +
286
+ booktitle +
287
+ if pubaddr.empty? then "" else " (#{pubaddr})." end +
288
+ if chapter.empty? then "" else " Chapter #{chapter}." end +
289
+ if pages.empty? then "" else " #{pages}." end
290
+ when 'unpublished'
291
+ " (unpublished)."
292
+ else ""
293
+ end
294
+ return authors + titleyear + rest
295
+ end
296
+
297
+ # Sets the specified metadata attribute to ans. ans is assumed to be a regular string.
298
+ # It will be split by " and " for authors and editors, or by spaces for keywords.
299
+ def set_attribute(attribute, ans)
300
+ index = ['title', 'authors', 'editors', 'booktitle'].member?(attribute)
301
+ if ans.nil? || ans.empty?
302
+ self.attributes[attribute] = nil
303
+ else
304
+ newval = if attribute == 'editors' || attribute == 'authors'
305
+ ans.split(" and ").map {|a| a.strip}
306
+ elsif attribute == 'keywords'
307
+ ans.split
308
+ else
309
+ [ans.strip]
310
+ end
311
+ self.attributes[attribute] = newval
312
+ unless ['url', 'doi', 'keywords'].member?(attribute)
313
+ self.attributes[attribute + "_lowercase"] = newval.map {|a| a.downcase}
314
+ self.attributes[attribute + "_words"] = self.attributes[attribute + "_lowercase"].map {|a| a.split(/[[:space:][:punct:]] */)}.flatten
315
+ end
316
+ # recalculate all_words
317
+ tit_auth_words = ['title', 'authors', 'editors', 'booktitle'].map {|att| self.attributes[att + "_words"] || []}.flatten
318
+ keywords = self.attributes['keywords'] || []
319
+ self.attributes['all_words'] = keywords + tit_auth_words
320
+ end
321
+ end
322
+
323
+ # Returns a string representation of an attribute.
324
+ def show_attribute(attribute)
325
+ value = self.attributes[attribute]
326
+ if value.nil?
327
+ ""
328
+ elsif attribute == 'keywords'
329
+ value.join(' ')
330
+ elsif attribute == 'editors' || attribute == 'authors'
331
+ value.join(' and ')
332
+ else
333
+ value[0]
334
+ end
335
+ end
336
+
337
+ # Returns an array of the field keywords appropriate for a type of entry.
338
+ def self.fields(entry_type='*')
339
+ fields = [:title, :authors, :year]
340
+ case entry_type
341
+ when 'article'
342
+ fields += [:journal, :volume, :pages]
343
+ when 'book'
344
+ fields += [:publisher, :address]
345
+ when 'chapter'
346
+ fields += [:booktitle, :chapter, :publisher, :address, :pages]
347
+ when 'incollection'
348
+ fields += [:booktitle, :chapter, :publisher, :address, :editors, :pages]
349
+ when '*'
350
+ fields += [:journal, :volume, :booktitle, :editors, :chapter,
351
+ :publisher, :address, :pages]
352
+ end
353
+ fields += [:keywords, :url, :doi, :comments]
354
+ return fields
355
+ end
356
+
357
+ # Returns the fields appropriate for an entry.
358
+ def fields
359
+ entry_type = self.show_attribute('entry_type')
360
+ Entry.fields(entry_type)
361
+ end
362
+
363
+ def url
364
+ AWS::S3::S3Object.find(self.name, @@bucket).url(:expires_in => 60 * 10) # expires in 10 min
365
+ end
366
+
367
+ private
368
+ # Returns the author's last name.
369
+ def last_name(author)
370
+ if author =~ /,/
371
+ author =~ /([^ ,]+),/
372
+ else
373
+ author =~ /([^ \t]+)$/
374
+ end
375
+ return $1
376
+ end
377
+
378
+ end
379
+
380
+ end
metadata ADDED
@@ -0,0 +1,96 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: jgm-cloudlib
3
+ version: !ruby/object:Gem::Version
4
+ version: "0.1"
5
+ platform: ruby
6
+ authors:
7
+ - John MacFarlane
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+
12
+ date: 2008-12-25 00:00:00 -08:00
13
+ default_executable: cloudlib
14
+ dependencies:
15
+ - !ruby/object:Gem::Dependency
16
+ name: aws-s3
17
+ version_requirement:
18
+ version_requirements: !ruby/object:Gem::Requirement
19
+ requirements:
20
+ - - ">="
21
+ - !ruby/object:Gem::Version
22
+ version: 0.5.1
23
+ version:
24
+ - !ruby/object:Gem::Dependency
25
+ name: aws-sdb
26
+ version_requirement:
27
+ version_requirements: !ruby/object:Gem::Requirement
28
+ requirements:
29
+ - - ">="
30
+ - !ruby/object:Gem::Version
31
+ version: 0.3.1
32
+ version:
33
+ - !ruby/object:Gem::Dependency
34
+ name: sinatra
35
+ version_requirement:
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: 0.3.2
41
+ version:
42
+ - !ruby/object:Gem::Dependency
43
+ name: highline
44
+ version_requirement:
45
+ version_requirements: !ruby/object:Gem::Requirement
46
+ requirements:
47
+ - - ">="
48
+ - !ruby/object:Gem::Version
49
+ version: 1.2.9
50
+ version:
51
+ description: "Cloudlib is a ruby library and commands for maintaining a library of books and articles on the Amazon 'cloud': S3 and SimpleDB."
52
+ email: jgm@berkeley.edu
53
+ executables:
54
+ - cloudlib
55
+ - cloudlib-web
56
+ extensions: []
57
+
58
+ extra_rdoc_files:
59
+ - README
60
+ files:
61
+ - README
62
+ - LICENSE
63
+ - cloudlib.gemspec
64
+ - lib/cloudlib.rb
65
+ - bin/cloudlib
66
+ - bin/cloudlib-web
67
+ has_rdoc: true
68
+ homepage: http://github.com/jgm/cloudlib
69
+ post_install_message:
70
+ rdoc_options:
71
+ - --main
72
+ - README
73
+ - --inline-source
74
+ require_paths:
75
+ - lib
76
+ required_ruby_version: !ruby/object:Gem::Requirement
77
+ requirements:
78
+ - - ">="
79
+ - !ruby/object:Gem::Version
80
+ version: "0"
81
+ version:
82
+ required_rubygems_version: !ruby/object:Gem::Requirement
83
+ requirements:
84
+ - - ">="
85
+ - !ruby/object:Gem::Version
86
+ version: "0"
87
+ version:
88
+ requirements: []
89
+
90
+ rubyforge_project:
91
+ rubygems_version: 1.2.0
92
+ signing_key:
93
+ specification_version: 2
94
+ summary: Tools for maintaining a library of books and articles in Amazon S3 and SimpleDB
95
+ test_files: []
96
+