jgm-cloudlib 0.1
Sign up to get free protection for your applications and to get access to all the features.
- data/LICENSE +340 -0
- data/README +91 -0
- data/bin/cloudlib +231 -0
- data/bin/cloudlib-web +207 -0
- data/cloudlib.gemspec +28 -0
- data/lib/cloudlib.rb +380 -0
- metadata +96 -0
data/LICENSE
ADDED
@@ -0,0 +1,340 @@
|
|
1
|
+
GNU GENERAL PUBLIC LICENSE
|
2
|
+
Version 2, June 1991
|
3
|
+
|
4
|
+
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
|
5
|
+
51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
|
6
|
+
Everyone is permitted to copy and distribute verbatim copies
|
7
|
+
of this license document, but changing it is not allowed.
|
8
|
+
|
9
|
+
Preamble
|
10
|
+
|
11
|
+
The licenses for most software are designed to take away your
|
12
|
+
freedom to share and change it. By contrast, the GNU General Public
|
13
|
+
License is intended to guarantee your freedom to share and change free
|
14
|
+
software--to make sure the software is free for all its users. This
|
15
|
+
General Public License applies to most of the Free Software
|
16
|
+
Foundation's software and to any other program whose authors commit to
|
17
|
+
using it. (Some other Free Software Foundation software is covered by
|
18
|
+
the GNU Library General Public License instead.) You can apply it to
|
19
|
+
your programs, too.
|
20
|
+
|
21
|
+
When we speak of free software, we are referring to freedom, not
|
22
|
+
price. Our General Public Licenses are designed to make sure that you
|
23
|
+
have the freedom to distribute copies of free software (and charge for
|
24
|
+
this service if you wish), that you receive source code or can get it
|
25
|
+
if you want it, that you can change the software or use pieces of it
|
26
|
+
in new free programs; and that you know you can do these things.
|
27
|
+
|
28
|
+
To protect your rights, we need to make restrictions that forbid
|
29
|
+
anyone to deny you these rights or to ask you to surrender the rights.
|
30
|
+
These restrictions translate to certain responsibilities for you if you
|
31
|
+
distribute copies of the software, or if you modify it.
|
32
|
+
|
33
|
+
For example, if you distribute copies of such a program, whether
|
34
|
+
gratis or for a fee, you must give the recipients all the rights that
|
35
|
+
you have. You must make sure that they, too, receive or can get the
|
36
|
+
source code. And you must show them these terms so they know their
|
37
|
+
rights.
|
38
|
+
|
39
|
+
We protect your rights with two steps: (1) copyright the software, and
|
40
|
+
(2) offer you this license which gives you legal permission to copy,
|
41
|
+
distribute and/or modify the software.
|
42
|
+
|
43
|
+
Also, for each author's protection and ours, we want to make certain
|
44
|
+
that everyone understands that there is no warranty for this free
|
45
|
+
software. If the software is modified by someone else and passed on, we
|
46
|
+
want its recipients to know that what they have is not the original, so
|
47
|
+
that any problems introduced by others will not reflect on the original
|
48
|
+
authors' reputations.
|
49
|
+
|
50
|
+
Finally, any free program is threatened constantly by software
|
51
|
+
patents. We wish to avoid the danger that redistributors of a free
|
52
|
+
program will individually obtain patent licenses, in effect making the
|
53
|
+
program proprietary. To prevent this, we have made it clear that any
|
54
|
+
patent must be licensed for everyone's free use or not licensed at all.
|
55
|
+
|
56
|
+
The precise terms and conditions for copying, distribution and
|
57
|
+
modification follow.
|
58
|
+
|
59
|
+
GNU GENERAL PUBLIC LICENSE
|
60
|
+
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
61
|
+
|
62
|
+
0. This License applies to any program or other work which contains
|
63
|
+
a notice placed by the copyright holder saying it may be distributed
|
64
|
+
under the terms of this General Public License. The "Program", below,
|
65
|
+
refers to any such program or work, and a "work based on the Program"
|
66
|
+
means either the Program or any derivative work under copyright law:
|
67
|
+
that is to say, a work containing the Program or a portion of it,
|
68
|
+
either verbatim or with modifications and/or translated into another
|
69
|
+
language. (Hereinafter, translation is included without limitation in
|
70
|
+
the term "modification".) Each licensee is addressed as "you".
|
71
|
+
|
72
|
+
Activities other than copying, distribution and modification are not
|
73
|
+
covered by this License; they are outside its scope. The act of
|
74
|
+
running the Program is not restricted, and the output from the Program
|
75
|
+
is covered only if its contents constitute a work based on the
|
76
|
+
Program (independent of having been made by running the Program).
|
77
|
+
Whether that is true depends on what the Program does.
|
78
|
+
|
79
|
+
1. You may copy and distribute verbatim copies of the Program's
|
80
|
+
source code as you receive it, in any medium, provided that you
|
81
|
+
conspicuously and appropriately publish on each copy an appropriate
|
82
|
+
copyright notice and disclaimer of warranty; keep intact all the
|
83
|
+
notices that refer to this License and to the absence of any warranty;
|
84
|
+
and give any other recipients of the Program a copy of this License
|
85
|
+
along with the Program.
|
86
|
+
|
87
|
+
You may charge a fee for the physical act of transferring a copy, and
|
88
|
+
you may at your option offer warranty protection in exchange for a fee.
|
89
|
+
|
90
|
+
2. You may modify your copy or copies of the Program or any portion
|
91
|
+
of it, thus forming a work based on the Program, and copy and
|
92
|
+
distribute such modifications or work under the terms of Section 1
|
93
|
+
above, provided that you also meet all of these conditions:
|
94
|
+
|
95
|
+
a) You must cause the modified files to carry prominent notices
|
96
|
+
stating that you changed the files and the date of any change.
|
97
|
+
|
98
|
+
b) You must cause any work that you distribute or publish, that in
|
99
|
+
whole or in part contains or is derived from the Program or any
|
100
|
+
part thereof, to be licensed as a whole at no charge to all third
|
101
|
+
parties under the terms of this License.
|
102
|
+
|
103
|
+
c) If the modified program normally reads commands interactively
|
104
|
+
when run, you must cause it, when started running for such
|
105
|
+
interactive use in the most ordinary way, to print or display an
|
106
|
+
announcement including an appropriate copyright notice and a
|
107
|
+
notice that there is no warranty (or else, saying that you provide
|
108
|
+
a warranty) and that users may redistribute the program under
|
109
|
+
these conditions, and telling the user how to view a copy of this
|
110
|
+
License. (Exception: if the Program itself is interactive but
|
111
|
+
does not normally print such an announcement, your work based on
|
112
|
+
the Program is not required to print an announcement.)
|
113
|
+
|
114
|
+
These requirements apply to the modified work as a whole. If
|
115
|
+
identifiable sections of that work are not derived from the Program,
|
116
|
+
and can be reasonably considered independent and separate works in
|
117
|
+
themselves, then this License, and its terms, do not apply to those
|
118
|
+
sections when you distribute them as separate works. But when you
|
119
|
+
distribute the same sections as part of a whole which is a work based
|
120
|
+
on the Program, the distribution of the whole must be on the terms of
|
121
|
+
this License, whose permissions for other licensees extend to the
|
122
|
+
entire whole, and thus to each and every part regardless of who wrote it.
|
123
|
+
|
124
|
+
Thus, it is not the intent of this section to claim rights or contest
|
125
|
+
your rights to work written entirely by you; rather, the intent is to
|
126
|
+
exercise the right to control the distribution of derivative or
|
127
|
+
collective works based on the Program.
|
128
|
+
|
129
|
+
In addition, mere aggregation of another work not based on the Program
|
130
|
+
with the Program (or with a work based on the Program) on a volume of
|
131
|
+
a storage or distribution medium does not bring the other work under
|
132
|
+
the scope of this License.
|
133
|
+
|
134
|
+
3. You may copy and distribute the Program (or a work based on it,
|
135
|
+
under Section 2) in object code or executable form under the terms of
|
136
|
+
Sections 1 and 2 above provided that you also do one of the following:
|
137
|
+
|
138
|
+
a) Accompany it with the complete corresponding machine-readable
|
139
|
+
source code, which must be distributed under the terms of Sections
|
140
|
+
1 and 2 above on a medium customarily used for software interchange; or,
|
141
|
+
|
142
|
+
b) Accompany it with a written offer, valid for at least three
|
143
|
+
years, to give any third party, for a charge no more than your
|
144
|
+
cost of physically performing source distribution, a complete
|
145
|
+
machine-readable copy of the corresponding source code, to be
|
146
|
+
distributed under the terms of Sections 1 and 2 above on a medium
|
147
|
+
customarily used for software interchange; or,
|
148
|
+
|
149
|
+
c) Accompany it with the information you received as to the offer
|
150
|
+
to distribute corresponding source code. (This alternative is
|
151
|
+
allowed only for noncommercial distribution and only if you
|
152
|
+
received the program in object code or executable form with such
|
153
|
+
an offer, in accord with Subsection b above.)
|
154
|
+
|
155
|
+
The source code for a work means the preferred form of the work for
|
156
|
+
making modifications to it. For an executable work, complete source
|
157
|
+
code means all the source code for all modules it contains, plus any
|
158
|
+
associated interface definition files, plus the scripts used to
|
159
|
+
control compilation and installation of the executable. However, as a
|
160
|
+
special exception, the source code distributed need not include
|
161
|
+
anything that is normally distributed (in either source or binary
|
162
|
+
form) with the major components (compiler, kernel, and so on) of the
|
163
|
+
operating system on which the executable runs, unless that component
|
164
|
+
itself accompanies the executable.
|
165
|
+
|
166
|
+
If distribution of executable or object code is made by offering
|
167
|
+
access to copy from a designated place, then offering equivalent
|
168
|
+
access to copy the source code from the same place counts as
|
169
|
+
distribution of the source code, even though third parties are not
|
170
|
+
compelled to copy the source along with the object code.
|
171
|
+
|
172
|
+
4. You may not copy, modify, sublicense, or distribute the Program
|
173
|
+
except as expressly provided under this License. Any attempt
|
174
|
+
otherwise to copy, modify, sublicense or distribute the Program is
|
175
|
+
void, and will automatically terminate your rights under this License.
|
176
|
+
However, parties who have received copies, or rights, from you under
|
177
|
+
this License will not have their licenses terminated so long as such
|
178
|
+
parties remain in full compliance.
|
179
|
+
|
180
|
+
5. You are not required to accept this License, since you have not
|
181
|
+
signed it. However, nothing else grants you permission to modify or
|
182
|
+
distribute the Program or its derivative works. These actions are
|
183
|
+
prohibited by law if you do not accept this License. Therefore, by
|
184
|
+
modifying or distributing the Program (or any work based on the
|
185
|
+
Program), you indicate your acceptance of this License to do so, and
|
186
|
+
all its terms and conditions for copying, distributing or modifying
|
187
|
+
the Program or works based on it.
|
188
|
+
|
189
|
+
6. Each time you redistribute the Program (or any work based on the
|
190
|
+
Program), the recipient automatically receives a license from the
|
191
|
+
original licensor to copy, distribute or modify the Program subject to
|
192
|
+
these terms and conditions. You may not impose any further
|
193
|
+
restrictions on the recipients' exercise of the rights granted herein.
|
194
|
+
You are not responsible for enforcing compliance by third parties to
|
195
|
+
this License.
|
196
|
+
|
197
|
+
7. If, as a consequence of a court judgment or allegation of patent
|
198
|
+
infringement or for any other reason (not limited to patent issues),
|
199
|
+
conditions are imposed on you (whether by court order, agreement or
|
200
|
+
otherwise) that contradict the conditions of this License, they do not
|
201
|
+
excuse you from the conditions of this License. If you cannot
|
202
|
+
distribute so as to satisfy simultaneously your obligations under this
|
203
|
+
License and any other pertinent obligations, then as a consequence you
|
204
|
+
may not distribute the Program at all. For example, if a patent
|
205
|
+
license would not permit royalty-free redistribution of the Program by
|
206
|
+
all those who receive copies directly or indirectly through you, then
|
207
|
+
the only way you could satisfy both it and this License would be to
|
208
|
+
refrain entirely from distribution of the Program.
|
209
|
+
|
210
|
+
If any portion of this section is held invalid or unenforceable under
|
211
|
+
any particular circumstance, the balance of the section is intended to
|
212
|
+
apply and the section as a whole is intended to apply in other
|
213
|
+
circumstances.
|
214
|
+
|
215
|
+
It is not the purpose of this section to induce you to infringe any
|
216
|
+
patents or other property right claims or to contest validity of any
|
217
|
+
such claims; this section has the sole purpose of protecting the
|
218
|
+
integrity of the free software distribution system, which is
|
219
|
+
implemented by public license practices. Many people have made
|
220
|
+
generous contributions to the wide range of software distributed
|
221
|
+
through that system in reliance on consistent application of that
|
222
|
+
system; it is up to the author/donor to decide if he or she is willing
|
223
|
+
to distribute software through any other system and a licensee cannot
|
224
|
+
impose that choice.
|
225
|
+
|
226
|
+
This section is intended to make thoroughly clear what is believed to
|
227
|
+
be a consequence of the rest of this License.
|
228
|
+
|
229
|
+
8. If the distribution and/or use of the Program is restricted in
|
230
|
+
certain countries either by patents or by copyrighted interfaces, the
|
231
|
+
original copyright holder who places the Program under this License
|
232
|
+
may add an explicit geographical distribution limitation excluding
|
233
|
+
those countries, so that distribution is permitted only in or among
|
234
|
+
countries not thus excluded. In such case, this License incorporates
|
235
|
+
the limitation as if written in the body of this License.
|
236
|
+
|
237
|
+
9. The Free Software Foundation may publish revised and/or new versions
|
238
|
+
of the General Public License from time to time. Such new versions will
|
239
|
+
be similar in spirit to the present version, but may differ in detail to
|
240
|
+
address new problems or concerns.
|
241
|
+
|
242
|
+
Each version is given a distinguishing version number. If the Program
|
243
|
+
specifies a version number of this License which applies to it and "any
|
244
|
+
later version", you have the option of following the terms and conditions
|
245
|
+
either of that version or of any later version published by the Free
|
246
|
+
Software Foundation. If the Program does not specify a version number of
|
247
|
+
this License, you may choose any version ever published by the Free Software
|
248
|
+
Foundation.
|
249
|
+
|
250
|
+
10. If you wish to incorporate parts of the Program into other free
|
251
|
+
programs whose distribution conditions are different, write to the author
|
252
|
+
to ask for permission. For software which is copyrighted by the Free
|
253
|
+
Software Foundation, write to the Free Software Foundation; we sometimes
|
254
|
+
make exceptions for this. Our decision will be guided by the two goals
|
255
|
+
of preserving the free status of all derivatives of our free software and
|
256
|
+
of promoting the sharing and reuse of software generally.
|
257
|
+
|
258
|
+
NO WARRANTY
|
259
|
+
|
260
|
+
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
|
261
|
+
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
|
262
|
+
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
|
263
|
+
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
|
264
|
+
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
265
|
+
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
|
266
|
+
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
|
267
|
+
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
|
268
|
+
REPAIR OR CORRECTION.
|
269
|
+
|
270
|
+
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
271
|
+
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
|
272
|
+
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
|
273
|
+
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
|
274
|
+
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
|
275
|
+
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
|
276
|
+
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
|
277
|
+
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
|
278
|
+
POSSIBILITY OF SUCH DAMAGES.
|
279
|
+
|
280
|
+
END OF TERMS AND CONDITIONS
|
281
|
+
|
282
|
+
How to Apply These Terms to Your New Programs
|
283
|
+
|
284
|
+
If you develop a new program, and you want it to be of the greatest
|
285
|
+
possible use to the public, the best way to achieve this is to make it
|
286
|
+
free software which everyone can redistribute and change under these terms.
|
287
|
+
|
288
|
+
To do so, attach the following notices to the program. It is safest
|
289
|
+
to attach them to the start of each source file to most effectively
|
290
|
+
convey the exclusion of warranty; and each file should have at least
|
291
|
+
the "copyright" line and a pointer to where the full notice is found.
|
292
|
+
|
293
|
+
<one line to give the program's name and a brief idea of what it does.>
|
294
|
+
Copyright (C) <year> <name of author>
|
295
|
+
|
296
|
+
This program is free software; you can redistribute it and/or modify
|
297
|
+
it under the terms of the GNU General Public License as published by
|
298
|
+
the Free Software Foundation; either version 2 of the License, or
|
299
|
+
(at your option) any later version.
|
300
|
+
|
301
|
+
This program is distributed in the hope that it will be useful,
|
302
|
+
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
303
|
+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
304
|
+
GNU General Public License for more details.
|
305
|
+
|
306
|
+
You should have received a copy of the GNU General Public License
|
307
|
+
along with this program; if not, write to the Free Software
|
308
|
+
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
|
309
|
+
|
310
|
+
|
311
|
+
Also add information on how to contact you by electronic and paper mail.
|
312
|
+
|
313
|
+
If the program is interactive, make it output a short notice like this
|
314
|
+
when it starts in an interactive mode:
|
315
|
+
|
316
|
+
Gnomovision version 69, Copyright (C) year name of author
|
317
|
+
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
318
|
+
This is free software, and you are welcome to redistribute it
|
319
|
+
under certain conditions; type `show c' for details.
|
320
|
+
|
321
|
+
The hypothetical commands `show w' and `show c' should show the appropriate
|
322
|
+
parts of the General Public License. Of course, the commands you use may
|
323
|
+
be called something other than `show w' and `show c'; they could even be
|
324
|
+
mouse-clicks or menu items--whatever suits your program.
|
325
|
+
|
326
|
+
You should also get your employer (if you work as a programmer) or your
|
327
|
+
school, if any, to sign a "copyright disclaimer" for the program, if
|
328
|
+
necessary. Here is a sample; alter the names:
|
329
|
+
|
330
|
+
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
|
331
|
+
`Gnomovision' (which makes passes at compilers) written by James Hacker.
|
332
|
+
|
333
|
+
<signature of Ty Coon>, 1 April 1989
|
334
|
+
Ty Coon, President of Vice
|
335
|
+
|
336
|
+
This General Public License does not permit incorporating your program into
|
337
|
+
proprietary programs. If your program is a subroutine library, you may
|
338
|
+
consider it more useful to permit linking proprietary applications with the
|
339
|
+
library. If this is what you want to do, use the GNU Library General
|
340
|
+
Public License instead of this License.
|
data/README
ADDED
@@ -0,0 +1,91 @@
|
|
1
|
+
cloudlib is a system for maintaining a database of electronic papers
|
2
|
+
and books using Amazon's web services (for background, see
|
3
|
+
http://aws.amazon.com/). Think of it as an indefinitely extensible
|
4
|
+
personal library, accessible from anywhere in the world.
|
5
|
+
|
6
|
+
The papers and books themselves are stored in a S3 bucket. The
|
7
|
+
metadata are stored in a SimpleDB database, so they can be searched easily.
|
8
|
+
The cloudlib ruby library integrates these two components and insulates
|
9
|
+
the user from the S3- and SimpleDB-specific details.
|
10
|
+
|
11
|
+
Note that S3 and SimpleDB are pay services. You will pay Amazon
|
12
|
+
proportionally to your usage. As of this writing (December 2008),
|
13
|
+
SimpleDB is free for the kind of usage this library normally requires.
|
14
|
+
For S3, the fee in North America is $0.15/GB/month for storage.
|
15
|
+
(See http://aws.amazon.com/s3/#pricing for up-to-date figures, including
|
16
|
+
fees for data transfer.) At these rates, it will cost less than a tenth
|
17
|
+
of a cent to store an average journal article for a year, and less than
|
18
|
+
a penny to store a good-sized book.
|
19
|
+
|
20
|
+
In addition to a ruby library, two programs are provided:
|
21
|
+
|
22
|
+
- cloudlib offers a command-line interface for interacting with a library.
|
23
|
+
|
24
|
+
- cloudlib-web starts a web-server which can either be used locally
|
25
|
+
(so that the library can be controlled through a browser) or made available
|
26
|
+
on the open internet. HTTP authentication is used to password-protect
|
27
|
+
the web application.
|
28
|
+
|
29
|
+
Both programs assume that the following environment variables have been set.
|
30
|
+
(If they are not, they will prompt for these values.)
|
31
|
+
|
32
|
+
- CLOUDLIB_LIBRARY_NAME - a name, of your choosing, that will identify the
|
33
|
+
library. It will be used both for the S3 bucket and the SimpleDB domain.
|
34
|
+
S3 bucket names must be unique, so pick a name like
|
35
|
+
'your-name-library', not something generic like 'my-library'.
|
36
|
+
|
37
|
+
- AWS_ACCESS_KEY_ID - the access key id you are provided when you sign up
|
38
|
+
for Amazon web services.
|
39
|
+
|
40
|
+
- AWS_SECRET_ACCESS_KEY - the secret key you are provided when you sign
|
41
|
+
up for Amazon web services.
|
42
|
+
|
43
|
+
cloudlib-web also assumes that the following environment variables have
|
44
|
+
been set:
|
45
|
+
|
46
|
+
- CLOUDLIB_WEB_USERNAME - a username of your choice, to be used to gain
|
47
|
+
access to the web interface.
|
48
|
+
|
49
|
+
- CLOUDLIB_WEB_PASSWORD - a password of your choice, to be used to gain
|
50
|
+
access to the web interface.
|
51
|
+
|
52
|
+
These environment variables can be set as follows:
|
53
|
+
|
54
|
+
export AWS_ACCESS_KEY_ID=xxxx-my-key-id-xxx
|
55
|
+
export AWS_SECRET_ACCESS_KEY=xxx-my-key-xxx
|
56
|
+
export CLOUDLIB_LIBRARY_NAME=xxx-my-library-name-xxx
|
57
|
+
export CLOUDLIB_WEB_USERNAME=xxx-username-xxx
|
58
|
+
export CLOUDLIB_WEB_PASSWORD=xxx-password-xxx
|
59
|
+
|
60
|
+
You may want to put these commands in a file, cloudlib-setup,
|
61
|
+
so that you can set all the variables at once:
|
62
|
+
|
63
|
+
source cloudlib-setup
|
64
|
+
|
65
|
+
After setting the environment variables, but before doing anything else,
|
66
|
+
you will need to create your library:
|
67
|
+
|
68
|
+
cloudlib new-library
|
69
|
+
|
70
|
+
If that succeeds, you can use either cloudlib or cloudlib-web to access
|
71
|
+
the library. By default, cloudlib-web will start a webserver on port 4567.
|
72
|
+
(This can be changed using the -p PORT option.) To use the web interface
|
73
|
+
after you've started the server, just point your browser at <http://localhost:4567>.
|
74
|
+
|
75
|
+
For instructions on the use of cloudlib, try
|
76
|
+
|
77
|
+
cloudlib --help
|
78
|
+
|
79
|
+
cloudlib by itself will start an interactive session. To add a new file,
|
80
|
+
use
|
81
|
+
|
82
|
+
cloudlib add /path/to/file
|
83
|
+
|
84
|
+
The commands 'dump' and 'restore' are also provided to allow creation of
|
85
|
+
a local backup of the database.
|
86
|
+
|
87
|
+
To install cloudlib:
|
88
|
+
|
89
|
+
gem sources -a http://gems.github.com # you only have to do this once
|
90
|
+
sudo gem install jgm-cloudlib
|
91
|
+
|
data/bin/cloudlib
ADDED
@@ -0,0 +1,231 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require 'yaml'
|
4
|
+
require 'rubygems'
|
5
|
+
require 'digest/sha1'
|
6
|
+
require 'lib/cloudlib' # cloudlib gem
|
7
|
+
require 'optparse'
|
8
|
+
require 'highline/import' # highline gem
|
9
|
+
|
10
|
+
PROGNAME = "cloudlib"
|
11
|
+
CLOUDLIB_VERSION = "0.1"
|
12
|
+
NUMITEMS = 10
|
13
|
+
|
14
|
+
options = {}
|
15
|
+
opts = OptionParser.new do |opts|
|
16
|
+
opts.program_name = "#{PROGNAME} (c) 2008 John MacFarlane"
|
17
|
+
opts.version = "version #{CLOUDLIB_VERSION}"
|
18
|
+
opts.banner = "cloudlib -- a library of books and articles in the AWS `cloud'\n" +
|
19
|
+
"Usage: #{PROGNAME} - start interactive menu\n" +
|
20
|
+
" #{PROGNAME} add FILE - upload FILE to library, prompting for metadata\n" +
|
21
|
+
" #{PROGNAME} new-library - initialize new library\n" +
|
22
|
+
" #{PROGNAME} delete-library - delete library and all its entries\n" +
|
23
|
+
" #{PROGNAME} dump [PATH] - create a local backup of library in PATH\n" +
|
24
|
+
" #{PROGNAME} restore [PATH] - restore library from local backup in PATH\n" +
|
25
|
+
"Options:"
|
26
|
+
end
|
27
|
+
opts.on("-v", "--version", "Show version") do
|
28
|
+
puts opts.ver
|
29
|
+
exit 0
|
30
|
+
end
|
31
|
+
opts.on("-h", "--help", "Show usage message") do
|
32
|
+
puts opts.help
|
33
|
+
exit 0
|
34
|
+
end
|
35
|
+
|
36
|
+
begin
|
37
|
+
opts.parse!
|
38
|
+
rescue OptionParser::ParseError => e
|
39
|
+
puts e.message
|
40
|
+
puts opts.help
|
41
|
+
exit 1
|
42
|
+
end
|
43
|
+
|
44
|
+
# check that required environment variables are set, and prompt if they aren't
|
45
|
+
envvars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "CLOUDLIB_LIBRARY_NAME"]
|
46
|
+
envvars.each do |var|
|
47
|
+
unless ENV[var]
|
48
|
+
ENV[var] = ask("#{var}: ", String) { |q| q.echo = if var == "AWS_SECRET_ACCESS_KEY" then "*" else true end }
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
Cloudlib::Entry.connect(ENV['CLOUDLIB_LIBRARY_NAME'], ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
|
53
|
+
|
54
|
+
class Cloudlib::Entry
|
55
|
+
# Prompts for metadata for the entry, using existing metadata as defaults.
|
56
|
+
def ask_attributes
|
57
|
+
default_type = if self.show_attribute('entry_type').empty?
|
58
|
+
"article"
|
59
|
+
else
|
60
|
+
self.show_attribute('entry_type')
|
61
|
+
end
|
62
|
+
entry_types = ['article','book','chapter','incollection','unpublished']
|
63
|
+
type = ask("Type (#{entry_types.join(', ')})? ", entry_types) {|q| q.default = default_type; q.case = :downcase; q.readline = true }
|
64
|
+
self.attributes['entry_type'] = [type]
|
65
|
+
self.fields.each { |field| self.ask_attribute(field.to_s) }
|
66
|
+
return self
|
67
|
+
end
|
68
|
+
|
69
|
+
# Prompts for metadata for a particular field.
|
70
|
+
def ask_attribute(attribute)
|
71
|
+
default = self.show_attribute(attribute)
|
72
|
+
ans = ask(attribute.capitalize +
|
73
|
+
if attribute == 'editors' || attribute == 'authors' then " (name [and name...]) " else " " end, String) { |q| q.readline = true; q.default = default }
|
74
|
+
set_attribute(attribute, ans)
|
75
|
+
end
|
76
|
+
|
77
|
+
end
|
78
|
+
|
79
|
+
def show_items(items, more=false)
|
80
|
+
items.each_index do |i|
|
81
|
+
say "<%= color('[#{i}]', BOLD) %> #{items[i].to_s}\n"
|
82
|
+
end
|
83
|
+
if more
|
84
|
+
say "<%= color('Enter', BOLD)%> for more...\n"
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
def menu(items)
|
89
|
+
token = query = ""
|
90
|
+
while true
|
91
|
+
commands = ["find KEYWORDS", "quit"]
|
92
|
+
if items.length > 0
|
93
|
+
then commands = ["bib NUM", "get NUM", "del NUM", "mod NUM"] + commands
|
94
|
+
end
|
95
|
+
choices = commands.join(' | ')
|
96
|
+
ans = ask("#{choices} ? ", String) { |q| q.readline = true; q.case = :downcase }
|
97
|
+
if ans == ""
|
98
|
+
if token.empty?
|
99
|
+
items = []
|
100
|
+
else
|
101
|
+
token, items = Cloudlib::Entry.query(query, NUMITEMS, token)
|
102
|
+
end
|
103
|
+
show_items(items, more=(not token.empty?))
|
104
|
+
elsif ans =~ /^quit|q$/
|
105
|
+
exit 0
|
106
|
+
elsif ans =~ /^find *(.+)$/
|
107
|
+
query = $1
|
108
|
+
token, items = Cloudlib::Entry.query(query, NUMITEMS)
|
109
|
+
show_items(items, more=(not token.empty?))
|
110
|
+
elsif ans =~ /^(get|del|mod|bib) *(\d+)$/
|
111
|
+
num = $2.to_i
|
112
|
+
if (num < 0) || (num >= items.length)
|
113
|
+
next
|
114
|
+
else
|
115
|
+
item = items[num]
|
116
|
+
case $1
|
117
|
+
when "get":
|
118
|
+
destpath = ask("Save as: ", String) { |q| q.default = item.friendly_filename; q.readline = true }
|
119
|
+
item.download(destpath)
|
120
|
+
puts "Downloaded #{destpath}"
|
121
|
+
when "mod":
|
122
|
+
item.ask_attributes
|
123
|
+
item.save
|
124
|
+
when "del":
|
125
|
+
item.delete
|
126
|
+
when "bib":
|
127
|
+
puts item.to_bibtex
|
128
|
+
else
|
129
|
+
raise "Unknown command."
|
130
|
+
end
|
131
|
+
end
|
132
|
+
else
|
133
|
+
puts "Unknown command"
|
134
|
+
end
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
|
+
if ARGV.length >= 1
|
139
|
+
if ARGV[0] == "new-library"
|
140
|
+
print "Create new library `#{ENV['CLOUDLIB_LIBRARY_NAME']}' (y/n)? "
|
141
|
+
ans = STDIN.gets
|
142
|
+
if ans =~ /^[Yy]/
|
143
|
+
begin
|
144
|
+
Cloudlib::Entry.create_library
|
145
|
+
rescue AWS::S3::BucketAlreadyExists
|
146
|
+
STDERR.puts "The library name `#{ENV['CLOUDLIB_LIBRARY_NAME']}' is already taken by another user."
|
147
|
+
STDERR.puts "Please set CLOUDLIB_LIBRARY_NAME to something else and try again."
|
148
|
+
exit 1
|
149
|
+
end
|
150
|
+
end
|
151
|
+
exit 0
|
152
|
+
end
|
153
|
+
if ARGV[0] == "delete-library"
|
154
|
+
print "Delete `#{ENV['CLOUDLIB_LIBRARY_NAME']}' and ALL OF ITS CONTENTS (y/n)? "
|
155
|
+
ans = STDIN.gets
|
156
|
+
if ans =~ /^[Yy]/
|
157
|
+
Cloudlib::Entry.delete_library
|
158
|
+
end
|
159
|
+
exit 0
|
160
|
+
end
|
161
|
+
if ARGV[0] == "backup"
|
162
|
+
print "Delete `#{ENV['CLOUDLIB_LIBRARY_NAME']}' and ALL OF ITS CONTENTS (y/n)? "
|
163
|
+
ans = STDIN.gets
|
164
|
+
if ans =~ /^[Yy]/
|
165
|
+
Cloudlib::Entry.delete_library
|
166
|
+
end
|
167
|
+
exit 0
|
168
|
+
end
|
169
|
+
if ARGV[0] == "add"
|
170
|
+
ARGV[1..(ARGV.length - 1)].each do |target|
|
171
|
+
unless File.exists?(target)
|
172
|
+
puts "File not found: #{target}"
|
173
|
+
exit 1
|
174
|
+
end
|
175
|
+
item = Cloudlib::Entry.from_file(target)
|
176
|
+
puts "Please enter metadata for `#{target}':"
|
177
|
+
item.ask_attributes
|
178
|
+
item.save
|
179
|
+
puts "Uploaded #{target}"
|
180
|
+
end
|
181
|
+
exit 0
|
182
|
+
end
|
183
|
+
if ARGV[0] == "dump"
|
184
|
+
path = ARGV[1] || "."
|
185
|
+
new_entries = 0
|
186
|
+
dummy, entries = Cloudlib::Entry.query("", nil)
|
187
|
+
total_size = entries.inject(0) {|accum, e| accum + e.attributes['size'][0].to_i}
|
188
|
+
printf("Total size is %.2f megabytes. Make a local copy (y/n)? ", (total_size / (1024.0 * 1024.0)))
|
189
|
+
ans = STDIN.gets
|
190
|
+
if ans =~ /^[Yy]/
|
191
|
+
open("#{path}/#{ENV['CLOUDLIB_LIBRARY_NAME']}.db", 'w') do |file|
|
192
|
+
STDERR.puts "Backing up metadata..."
|
193
|
+
file.write(YAML.dump(entries))
|
194
|
+
end
|
195
|
+
STDERR.puts "Backing up files:"
|
196
|
+
entries.each do |entry|
|
197
|
+
if not File.exists?("#{path}/#{entry.name}")
|
198
|
+
entry.download("#{path}/#{entry.name}")
|
199
|
+
new_entries += 1
|
200
|
+
STDERR.puts entry.to_s
|
201
|
+
end
|
202
|
+
end
|
203
|
+
STDERR.puts "Backed up metadata and #{new_entries} new files."
|
204
|
+
end
|
205
|
+
exit 0
|
206
|
+
end
|
207
|
+
if ARGV[0] == "restore"
|
208
|
+
path = ARGV[1] || "."
|
209
|
+
entries = open("#{path}/#{ENV['CLOUDLIB_LIBRARY_NAME']}.db", 'r') { |file| YAML.load(file) }
|
210
|
+
entries.each do |entry|
|
211
|
+
filename = "#{path}/#{entry.name}"
|
212
|
+
Cloudlib::Entry.from_file(filename)
|
213
|
+
entry.save
|
214
|
+
STDERR.puts entry.to_s
|
215
|
+
end
|
216
|
+
STDERR.puts "Restored #{entries.length} entries."
|
217
|
+
exit 0
|
218
|
+
end
|
219
|
+
STDERR.puts "Unknown command #{ARGV[0]}."
|
220
|
+
puts opts.help
|
221
|
+
exit 1
|
222
|
+
else
|
223
|
+
begin
|
224
|
+
menu([])
|
225
|
+
rescue AwsSdb::NoSuchDomainError
|
226
|
+
STDERR.puts "The library `#{ENV['CLOUDLIB_LIBRARY_NAME']}' does not exist."
|
227
|
+
STDERR.puts "Use `#{PROGNAME} new-library' to create it."
|
228
|
+
exit 1
|
229
|
+
end
|
230
|
+
end
|
231
|
+
|
data/bin/cloudlib-web
ADDED
@@ -0,0 +1,207 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
require 'rubygems'
|
3
|
+
require 'digest/sha1'
|
4
|
+
require 'sinatra' # sinatra gem
|
5
|
+
require 'tempfile'
|
6
|
+
require 'cloudlib'
|
7
|
+
require 'highline/import' # highline gem
|
8
|
+
|
9
|
+
# Number of items to display per page after query
|
10
|
+
NUMITEMS = 10
|
11
|
+
|
12
|
+
# check that required environment variables are set
|
13
|
+
envvars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "CLOUDLIB_LIBRARY_NAME", "CLOUDLIB_WEB_USERNAME", "CLOUDLIB_WEB_PASSWORD"]
|
14
|
+
envvars.each do |var|
|
15
|
+
unless ENV[var]
|
16
|
+
ENV[var] = ask("#{var}: ", String) { |q| q.echo = if var == "AWS_SECRET_ACCESS_KEY" then "*" else true end }
|
17
|
+
end
|
18
|
+
end
|
19
|
+
Cloudlib::Entry.connect(ENV['CLOUDLIB_LIBRARY_NAME'], ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
|
20
|
+
|
21
|
+
use Rack::Auth::Basic do |username, password|
|
22
|
+
username == ENV['CLOUDLIB_WEB_USERNAME'] &&
|
23
|
+
password == ENV['CLOUDLIB_WEB_PASSWORD']
|
24
|
+
end
|
25
|
+
|
26
|
+
get '/stylesheet.css' do
|
27
|
+
content_type 'text/css', :charset => 'utf-8'
|
28
|
+
sass :stylesheet
|
29
|
+
end
|
30
|
+
|
31
|
+
get '/' do
|
32
|
+
@token, @entries = "", []
|
33
|
+
@query = ""
|
34
|
+
haml :index
|
35
|
+
end
|
36
|
+
|
37
|
+
post '/' do
|
38
|
+
if params[:query]
|
39
|
+
@query = params[:query]
|
40
|
+
@token, @entries = Cloudlib::Entry.query(@query, NUMITEMS, params[:token])
|
41
|
+
else
|
42
|
+
@token, @entries = "", []
|
43
|
+
@query = ""
|
44
|
+
end
|
45
|
+
haml :index
|
46
|
+
end
|
47
|
+
|
48
|
+
get '/upload' do
|
49
|
+
haml :upload
|
50
|
+
end
|
51
|
+
|
52
|
+
post '/upload' do
|
53
|
+
tempfile = params[:fileToUpload][:tempfile]
|
54
|
+
tempfilepath = tempfile.path
|
55
|
+
tempfile.close
|
56
|
+
entry = Cloudlib::Entry.from_file(tempfilepath, params[:fileToUpload][:filename])
|
57
|
+
set_attributes_from_form(entry)
|
58
|
+
entry.save
|
59
|
+
redirect "/"
|
60
|
+
end
|
61
|
+
|
62
|
+
get '/*/bibtex' do
|
63
|
+
@entry = Cloudlib::Entry.find_by_name(params[:splat][0])
|
64
|
+
content_type 'text/plain', :charset => 'utf-8'
|
65
|
+
@entry.to_bibtex
|
66
|
+
end
|
67
|
+
|
68
|
+
get '/*' do
|
69
|
+
name = "#{params[:splat][0]}"
|
70
|
+
@entry = Cloudlib::Entry.find_by_name(name)
|
71
|
+
haml :modify
|
72
|
+
end
|
73
|
+
|
74
|
+
post '/*' do
|
75
|
+
entry = Cloudlib::Entry.find_by_name(params[:splat][0])
|
76
|
+
set_attributes_from_form(entry)
|
77
|
+
entry.save
|
78
|
+
redirect '/'
|
79
|
+
end
|
80
|
+
|
81
|
+
delete '/*' do
|
82
|
+
entry = Cloudlib::Entry.find_by_name(params[:splat][0])
|
83
|
+
entry.delete
|
84
|
+
redirect '/'
|
85
|
+
end
|
86
|
+
|
87
|
+
def field_for_type?(field, type)
|
88
|
+
Cloudlib::Entry.fields(type).member?(field)
|
89
|
+
end
|
90
|
+
|
91
|
+
def show_fields(type)
|
92
|
+
cmds = Cloudlib::Entry.fields.map do |field|
|
93
|
+
"document.getElementById('#{field.to_s}').setAttribute('style', 'display: #{if field_for_type?(field, type) then 'all' else 'none' end}'); "
|
94
|
+
end
|
95
|
+
return cmds.join
|
96
|
+
end
|
97
|
+
|
98
|
+
def set_attributes_from_form(entry)
|
99
|
+
entry.attributes['entry_type'] = params['entry_type']
|
100
|
+
Cloudlib::Entry.fields.each do |field|
|
101
|
+
if field_for_type?(field, params['entry_type']) && params[field]
|
102
|
+
entry.set_attribute(field.to_s, params[field])
|
103
|
+
else
|
104
|
+
entry.set_attribute(field.to_s, '')
|
105
|
+
end
|
106
|
+
end
|
107
|
+
end
|
108
|
+
|
109
|
+
use_in_file_templates!
|
110
|
+
|
111
|
+
__END__
|
112
|
+
|
113
|
+
@@ layout
|
114
|
+
!!! Strict
|
115
|
+
%head
|
116
|
+
%link{:href => '/stylesheet.css', :type => 'text/css', :media => 'all', :rel => 'stylesheet'}
|
117
|
+
%title
|
118
|
+
= ENV['CLOUDLIB_LIBRARY_NAME']
|
119
|
+
%body
|
120
|
+
%h1
|
121
|
+
%a{:href => '/'}
|
122
|
+
= ENV['CLOUDLIB_LIBRARY_NAME']
|
123
|
+
%div#content
|
124
|
+
= yield
|
125
|
+
%div#footer
|
126
|
+
powered by
|
127
|
+
%a{:href => ''}cloudlib
|
128
|
+
|
129
|
+
@@ index
|
130
|
+
%div.queryform
|
131
|
+
%form{:method => 'POST', :action => '/'}
|
132
|
+
%input{:type => 'text', :name => 'query', :value => @query, :size => '30'}
|
133
|
+
%input{:type => 'submit', :value => 'Search'}
|
134
|
+
%ol
|
135
|
+
- @entries.each do |i|
|
136
|
+
%li
|
137
|
+
= i.to_s
|
138
|
+
%a{:href => "/#{i.name}/bibtex"}bibtex
|
139
|
+
%span.separator •
|
140
|
+
%a{:href => "/#{i.name}"}modify
|
141
|
+
%span.separator •
|
142
|
+
%a{:href => i.url}download
|
143
|
+
- if not @token.empty?
|
144
|
+
%form{:method => 'POST', :action => '/'}
|
145
|
+
%input{:type => 'text', :name => 'query', :value => @query, :style => 'display: none'}
|
146
|
+
%input{:type => 'text', :name => 'token', :value => @token, :style => 'display: none'}
|
147
|
+
%input{:type => 'submit', :value => 'More matches...'}
|
148
|
+
%a{:href => "/upload"}upload
|
149
|
+
|
150
|
+
@@ upload
|
151
|
+
%form{:method => 'POST', :action => '/upload', :enctype => 'multipart/form-data'}
|
152
|
+
%label Select file to upload:
|
153
|
+
%br
|
154
|
+
%input{:type => 'file', :name => 'fileToUpload', :size => 40}
|
155
|
+
= haml :metadata, :layout => false
|
156
|
+
%input{:type => 'submit', :value => 'Add file to library'}
|
157
|
+
|
158
|
+
@@ metadata
|
159
|
+
%table
|
160
|
+
%tr
|
161
|
+
%td
|
162
|
+
%label Type:
|
163
|
+
%td
|
164
|
+
%select{:name => 'entry_type'}
|
165
|
+
- ['','article','book','incollection','chapter','unpublished'].each do |type|
|
166
|
+
%option{:onClick => show_fields(type), :selected => (@entry && @entry.show_attribute('entry_type') == type) || type.empty?}
|
167
|
+
= type
|
168
|
+
- Cloudlib::Entry.fields.each do |field|
|
169
|
+
%tr{:style => (@entry && field_for_type?(field, @entry.show_attribute('entry_type'))) || 'display: none;', :id => field.to_s}
|
170
|
+
%td
|
171
|
+
%label
|
172
|
+
= field.to_s.capitalize + ':'
|
173
|
+
%td
|
174
|
+
%input{:type => 'text', :name => field.to_s, :value => (@entry && @entry.show_attribute(field.to_s)) || '', :size => 50}
|
175
|
+
|
176
|
+
@@ modify
|
177
|
+
%div.detail
|
178
|
+
%form{:method => 'POST', :action => "/#{@entry.name}"}
|
179
|
+
= haml :metadata, :layout => false
|
180
|
+
%p
|
181
|
+
%input{:type => 'submit', :value => 'Update metadata'}
|
182
|
+
%form{:method => 'POST', :action => "/#{@entry.name}"}
|
183
|
+
%p
|
184
|
+
%input{:type => 'text', :name => '_method', :value => 'delete', :style => 'display: none;'}
|
185
|
+
%input{:type => 'submit', :value => 'Delete this entry'}
|
186
|
+
|
187
|
+
@@ stylesheet
|
188
|
+
body
|
189
|
+
font-size: small
|
190
|
+
padding: 10px
|
191
|
+
h1
|
192
|
+
border-top: 1px solid gray
|
193
|
+
border-bottom: 1px solid gray
|
194
|
+
h1 a
|
195
|
+
color: #7a7a7a
|
196
|
+
text-decoration: none
|
197
|
+
&:visited
|
198
|
+
color: #7a7a7a
|
199
|
+
li
|
200
|
+
padding-bottom: 0.3em
|
201
|
+
#footer
|
202
|
+
border-top: 1px solid gray
|
203
|
+
margin-top: 1em
|
204
|
+
padding-top: 1em
|
205
|
+
font-size: x-small
|
206
|
+
text-align: center
|
207
|
+
|
data/cloudlib.gemspec
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
Gem::Specification.new do |s|
|
2
|
+
s.name = "cloudlib"
|
3
|
+
s.version = "0.1"
|
4
|
+
s.date = "2008-12-25"
|
5
|
+
s.summary = "Tools for maintaining a library of books and articles in Amazon S3 and SimpleDB"
|
6
|
+
s.email = "jgm@berkeley.edu"
|
7
|
+
s.homepage = "http://github.com/jgm/cloudlib"
|
8
|
+
s.description = "Cloudlib is a ruby library and commands for maintaining a library of books and articles on the Amazon 'cloud': S3 and SimpleDB."
|
9
|
+
s.has_rdoc = true
|
10
|
+
s.authors = ["John MacFarlane"]
|
11
|
+
s.bindir = "bin"
|
12
|
+
s.executables = ["cloudlib", "cloudlib-web"]
|
13
|
+
s.default_executable = "cloudlib"
|
14
|
+
s.files = [ "README",
|
15
|
+
"LICENSE",
|
16
|
+
"cloudlib.gemspec",
|
17
|
+
"lib/cloudlib.rb",
|
18
|
+
"bin/cloudlib",
|
19
|
+
"bin/cloudlib-web" ]
|
20
|
+
s.test_files = []
|
21
|
+
s.rdoc_options = ["--main", "README", "--inline-source"]
|
22
|
+
s.extra_rdoc_files = ["README"]
|
23
|
+
s.add_dependency("aws-s3", [">= 0.5.1"])
|
24
|
+
s.add_dependency("aws-sdb", [">= 0.3.1"])
|
25
|
+
s.add_dependency("sinatra", [">= 0.3.2"])
|
26
|
+
s.add_dependency("highline", [">= 1.2.9"])
|
27
|
+
end
|
28
|
+
|
data/lib/cloudlib.rb
ADDED
@@ -0,0 +1,380 @@
|
|
1
|
+
# This library provides the means for maintaining a database of
|
2
|
+
# documents on Amazon's S3 file store, with searchable metadata in
|
3
|
+
# Amazon's SimpleDB database. Think of it as a filing cabinet or library
|
4
|
+
# that can be extended indefinitely and accessed from anywhere in the
|
5
|
+
# world: a library that lives "in the cloud."
|
6
|
+
|
7
|
+
# In order to use this library, you need to sign up for
|
8
|
+
# Amazon's S3 and SimpleDB services:
|
9
|
+
#
|
10
|
+
# * Amazon SimpleDB: http://aws.amazon.com/simpledb/
|
11
|
+
# * Amazon S3: http://aws.amazon.com/s3/
|
12
|
+
#
|
13
|
+
# Simple usage example:
|
14
|
+
#
|
15
|
+
# require 'rubygems'
|
16
|
+
# require 'cloudlib'
|
17
|
+
# include Cloudlib
|
18
|
+
# Entry.connect('xxx_key_id_xxx', 'xxx_secret_access_key_xxx', 'my_aws_library')
|
19
|
+
# logic_entries = Entry.query('logic')
|
20
|
+
# logic_entries.each {|entry| puts entry.to_s}
|
21
|
+
#
|
22
|
+
# For more examples of the use of the library, see the programs cloudlib.rb
|
23
|
+
# and cloudlib-web.rb, included in the gem.
|
24
|
+
|
25
|
+
# Author:: John MacFarlane (jgm at berkeley dot edu)
|
26
|
+
# Copyright:: Copyright (c) 2008 John MacFarlane
|
27
|
+
# License:: GPL v2
|
28
|
+
|
29
|
+
require 'rubygems'
|
30
|
+
require 'readline'
|
31
|
+
require 'aws/s3' # aws-s3 gem
|
32
|
+
require 'aws_sdb' # aws-sdb gem
|
33
|
+
require 'open-uri'
|
34
|
+
require 'fileutils'
|
35
|
+
|
36
|
+
module Cloudlib
|
37
|
+
|
38
|
+
# A library entry, including content and metadata. An entry has a name
|
39
|
+
# (which is also the key of the associated S3 object) and an attributes
|
40
|
+
# hash. The name is of the form "sha1.ext", where sha1 is a SHA1 hash of
|
41
|
+
# the contents of the file, and ext is the file extension. This makes
|
42
|
+
# it impossible to have entries with duplicate contents. The attributes
|
43
|
+
# hash contains the following fields:
|
44
|
+
#
|
45
|
+
# * extension - file extension including .
|
46
|
+
# * size - size of contents (bytes)
|
47
|
+
# * date-added - date entry was added to library
|
48
|
+
# * entry_type - article, book, chapter, incollection, unpublished
|
49
|
+
# * authors - list of authors
|
50
|
+
# * editors - list of editors
|
51
|
+
# * title - title of entry
|
52
|
+
# * booktitle - title of book containing entry
|
53
|
+
# * year - publication year of entry
|
54
|
+
# * publisher - publisher of book
|
55
|
+
# * address - publication address
|
56
|
+
# * journal - journal containing entry
|
57
|
+
# * volume - volume number of journal
|
58
|
+
# * pages - page range of entry in book or journal
|
59
|
+
# * keywords - keywords
|
60
|
+
# * doi - DOI for entry
|
61
|
+
# * url - URL for entry
|
62
|
+
# * comments - miscellaneous comments
|
63
|
+
# * *_lowercase - lowercase version of *
|
64
|
+
# * *_words - lowercase version of *, split into a list of words
|
65
|
+
# * all_words - list of words in title, authors, editors, booktitle, keywords
|
66
|
+
|
67
|
+
class Entry
|
68
|
+
|
69
|
+
attr_accessor :name, :attributes
|
70
|
+
|
71
|
+
# Establish connections to the S3 file store and the SimpleDB database.
|
72
|
+
# If values are not supplied for the parameters, they will default to
|
73
|
+
# the values of the environment variables CLOUDLIB_LIBRARY_NAME,
|
74
|
+
# AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY. Note that library_name
|
75
|
+
# is the name of both the S3 bucket that will hold the contents of
|
76
|
+
# the entries and the SimpleDB domain that will hold the metadata.
|
77
|
+
def self.connect(library_name=ENV['CLOUDLIB_LIBRARY_NAME'],
|
78
|
+
aws_access_key_id=ENV['AWS_ACCESS_KEY_ID'],
|
79
|
+
aws_secret_access_key=ENV['AWS_SECRET_ACCESS_KEY'],
|
80
|
+
debug = false)
|
81
|
+
@@aws_access_key_id = aws_access_key_id
|
82
|
+
@@aws_secret_access_key = aws_secret_access_key
|
83
|
+
AWS::S3::Base.establish_connection!(:access_key_id => @@aws_access_key_id, :secret_access_key => @@aws_secret_access_key, :use_ssl => true)
|
84
|
+
@@bucket = library_name
|
85
|
+
logger = Logger.new(STDERR)
|
86
|
+
logger.level = if debug then Logger::DEBUG else Logger::WARN end
|
87
|
+
@@db = AwsSdb::Service.new(:access_key_id => @@aws_access_key_id, :secret_access_key => @@aws_secret_access_key, :use_ssl => true, :logger => logger)
|
88
|
+
end
|
89
|
+
|
90
|
+
# Creates a new entry object. To create an entry with contents,
|
91
|
+
# use Entry.from_file.
|
92
|
+
def initialize(name, attributes={'all_words' => []})
|
93
|
+
@name = name
|
94
|
+
@attributes = attributes
|
95
|
+
end
|
96
|
+
|
97
|
+
# Create the S3 bucket and SimpleDB domain that will store the library entries.
|
98
|
+
# This method should be run once to create the library.
|
99
|
+
def self.create_library
|
100
|
+
AWS::S3::Bucket.create(@@bucket)
|
101
|
+
@@db.create_domain(@@bucket)
|
102
|
+
end
|
103
|
+
|
104
|
+
# Delete the S3 bucket and SimpleDB domain that store the library entries.
|
105
|
+
# All data will be lost.
|
106
|
+
def self.delete_library
|
107
|
+
AWS::S3::Bucket.delete(@@bucket, :force => true)
|
108
|
+
@@db.delete_domain(@@bucket)
|
109
|
+
end
|
110
|
+
|
111
|
+
# Creates and saves an entry from a file, using attributes supplied.
|
112
|
+
# Returns the entry.
|
113
|
+
def self.from_file(path, filename=path, attributes={'all_words' => []})
|
114
|
+
sha1 = Digest::SHA1.file(path).hexdigest
|
115
|
+
ext = File.extname(filename)
|
116
|
+
name = "#{sha1}#{ext}"
|
117
|
+
attributes['size'] = File.size(path).to_s
|
118
|
+
attributes['date-added'] = Date.today.to_s
|
119
|
+
entry = Entry.new(name, attributes)
|
120
|
+
AWS::S3::S3Object.store(name, open(path), @@bucket)
|
121
|
+
@@db.put_attributes(@@bucket, name, attributes, replace=true)
|
122
|
+
return entry
|
123
|
+
end
|
124
|
+
|
125
|
+
# Return an entry with the specified name. Raises an error if not found.
|
126
|
+
def self.find_by_name(name)
|
127
|
+
attributes = @@db.get_attributes(@@bucket, name)
|
128
|
+
if attributes == {} then raise "Item not found." end
|
129
|
+
Entry.new(name, attributes)
|
130
|
+
end
|
131
|
+
|
132
|
+
# Queries the database and returns a list [token, entries]. entries is
|
133
|
+
# a list of up to numitems Entry objects that match the query. If
|
134
|
+
# there are more entries than numitems, token will be nonempty, and
|
135
|
+
# can be passed in on a subsequent calls for the remaining entries.
|
136
|
+
#
|
137
|
+
# The query string can contain one or more words. If a word is
|
138
|
+
# preceded by ti=, only entries that match it in the title will be
|
139
|
+
# returned. Similarly, au= searches authors, jo= journals, pu=
|
140
|
+
# publishers, ad= addresses, ed= editors, bo= booktitle (for collections),
|
141
|
+
# and ye= years. ye> and # ye< may also be used.
|
142
|
+
# The form ti='word1 word2' may also be used; entries will only match
|
143
|
+
# if their titles contain both word1 and word2.
|
144
|
+
def self.query(query_string, numitems=10, token=nil)
|
145
|
+
query_parts = query_string.downcase.scan(/((ti(?:title)|au(?:thor?s)|jo(?:urnal)|bo(?:ooktitle)|pu(?:blisher)|ad(?:ddress)|ed(?:itor?s)|ye(?:ar))[^<=>]*([<=>])('[^']*'|"[^"]*"|\S*)|\S+)\s*/)
|
146
|
+
query = query_parts.reject {|part| part[0] == '*'}.map do |part|
|
147
|
+
whole, key, comparison, val = part
|
148
|
+
if val then val = val.gsub(/^['"](.*)['"]$/, "\\1") end
|
149
|
+
if not val then val = whole end
|
150
|
+
key_full = case key
|
151
|
+
when 'ti'
|
152
|
+
'title'
|
153
|
+
when 'au'
|
154
|
+
'authors'
|
155
|
+
when 'jo'
|
156
|
+
'journal'
|
157
|
+
when 'pu'
|
158
|
+
'publisher'
|
159
|
+
when 'ad'
|
160
|
+
'address'
|
161
|
+
when 'ed'
|
162
|
+
'editors'
|
163
|
+
when 'ye'
|
164
|
+
'year'
|
165
|
+
else 'all'
|
166
|
+
end
|
167
|
+
vals = val.split
|
168
|
+
vals.map do |v|
|
169
|
+
if key_full == 'year' # there is no year_words field
|
170
|
+
"['year' #{comparison} '#{v}']"
|
171
|
+
else
|
172
|
+
"['#{key_full}_words' = '#{v}']"
|
173
|
+
end
|
174
|
+
end.join(" intersection ")
|
175
|
+
end.join(" intersection ")
|
176
|
+
# note: query has to include year in order to sort by year
|
177
|
+
# hence this dummy search
|
178
|
+
if query.empty?
|
179
|
+
query = "['year' starts-with ''] sort 'year'"
|
180
|
+
else
|
181
|
+
query += " intersection ['year' starts-with ''] sort 'year'"
|
182
|
+
end
|
183
|
+
names, token = if token
|
184
|
+
@@db.query(@@bucket, query, numitems, token)
|
185
|
+
else
|
186
|
+
@@db.query(@@bucket, query, numitems)
|
187
|
+
end
|
188
|
+
entries = names.map do |name|
|
189
|
+
attributes = @@db.get_attributes(@@bucket, name)
|
190
|
+
Entry.new(name, attributes)
|
191
|
+
end
|
192
|
+
return token, entries
|
193
|
+
end
|
194
|
+
|
195
|
+
# Returns a human-friendly filename for the entry, constructed from
|
196
|
+
# authors and title.
|
197
|
+
def friendly_filename
|
198
|
+
authornames = self.attributes['authors'].map {|a| last_name(a)}.join('_')
|
199
|
+
title = self.show_attribute('title').gsub(/[,.\/[:space:]]+/,'_')
|
200
|
+
ext = File.extname(self.name)
|
201
|
+
return "#{authornames}_#{title}#{ext}"
|
202
|
+
end
|
203
|
+
|
204
|
+
# Deletes the entry.
|
205
|
+
def delete
|
206
|
+
AWS::S3::S3Object.delete(self.name, @@bucket)
|
207
|
+
@@db.delete_attributes(@@bucket, self.name)
|
208
|
+
end
|
209
|
+
|
210
|
+
# Saves the entry (metadata only; contents are saved by the from_file
|
211
|
+
# method).
|
212
|
+
def save
|
213
|
+
@@db.put_attributes(@@bucket, self.name, self.attributes, replace=true)
|
214
|
+
end
|
215
|
+
|
216
|
+
# Downloads the entry and saves as filename.
|
217
|
+
def download(path)
|
218
|
+
if File.exist?(path)
|
219
|
+
STDERR.puts "Backing up existing #{path} as #{path}~"
|
220
|
+
FileUtils.copy_file(path, "#{path}~", preserve=true)
|
221
|
+
end
|
222
|
+
open(path, 'w') do |outfile|
|
223
|
+
open(self.url, 'r') do |source|
|
224
|
+
FileUtils.copy_stream(source, outfile)
|
225
|
+
end
|
226
|
+
end
|
227
|
+
return path
|
228
|
+
end
|
229
|
+
|
230
|
+
# Returns a bibtex entry for the entry.
|
231
|
+
def to_bibtex
|
232
|
+
pairs = self.fields.map do |field|
|
233
|
+
if self.attributes[field.to_s]
|
234
|
+
sprintf(" %-15s: {%s}", field.to_s, self.show_attribute(field.to_s))
|
235
|
+
else
|
236
|
+
nil
|
237
|
+
end
|
238
|
+
end
|
239
|
+
pairs += [sprintf(" %-15s: {%s}", "file", self.name)]
|
240
|
+
authornames = self.attributes['authors'].map {|a| last_name(a)}.join('.')
|
241
|
+
year = self.attributes['year']
|
242
|
+
entry_type = self.show_attribute('entry_type') || 'unknown'
|
243
|
+
if entry_type == 'chapter' then entry_type = 'inbook' end
|
244
|
+
entry_key = "#{authornames}:#{year}"
|
245
|
+
"@#{entry_type.upcase}{#{entry_key},\n#{pairs.join(",\n")}\n}"
|
246
|
+
end
|
247
|
+
|
248
|
+
# Returns a string representation of the entry's metadata.
|
249
|
+
def to_s
|
250
|
+
authors = self.show_attribute('authors')
|
251
|
+
unless authors.empty?
|
252
|
+
authors = "#{authors}, "
|
253
|
+
end
|
254
|
+
title = "#{self.show_attribute('title')}"
|
255
|
+
year = self.show_attribute('year')
|
256
|
+
titleyear = if year.empty?
|
257
|
+
title + ". "
|
258
|
+
else
|
259
|
+
title + " (#{year}). "
|
260
|
+
end
|
261
|
+
pubaddr = [self.show_attribute('address'),
|
262
|
+
self.show_attribute('publisher')].reject {|x| x.empty?}.join(": ")
|
263
|
+
chapter = self.show_attribute('chapter')
|
264
|
+
pages = self.show_attribute('pages')
|
265
|
+
booktitle = self.show_attribute('booktitle')
|
266
|
+
editors = self.show_attribute('editors')
|
267
|
+
journal = self.show_attribute('journal')
|
268
|
+
volume = self.show_attribute('volume')
|
269
|
+
rest = case self.show_attribute('entry_type')
|
270
|
+
when 'article'
|
271
|
+
if journal.empty?
|
272
|
+
""
|
273
|
+
else
|
274
|
+
"#{journal} #{volume}" +
|
275
|
+
if pages.empty? then "." else ", #{pages}." end
|
276
|
+
end
|
277
|
+
when 'book'
|
278
|
+
if pubaddr.empty? then "" else "#{pubaddr}." end
|
279
|
+
when 'chapter'
|
280
|
+
if pubaddr.empty? then "" else "#{pubaddr}." end +
|
281
|
+
if chapter.empty? then "" else " Chapter #{chapter}." end +
|
282
|
+
if pages.empty? then "" else " #{pages}." end
|
283
|
+
when 'incollection'
|
284
|
+
"In " +
|
285
|
+
if editors.empty? then "" else editors + " (eds.), " end +
|
286
|
+
booktitle +
|
287
|
+
if pubaddr.empty? then "" else " (#{pubaddr})." end +
|
288
|
+
if chapter.empty? then "" else " Chapter #{chapter}." end +
|
289
|
+
if pages.empty? then "" else " #{pages}." end
|
290
|
+
when 'unpublished'
|
291
|
+
" (unpublished)."
|
292
|
+
else ""
|
293
|
+
end
|
294
|
+
return authors + titleyear + rest
|
295
|
+
end
|
296
|
+
|
297
|
+
# Sets the specified metadata attribute to ans. ans is assumed to be a regular string.
|
298
|
+
# It will be split by " and " for authors and editors, or by spaces for keywords.
|
299
|
+
def set_attribute(attribute, ans)
|
300
|
+
index = ['title', 'authors', 'editors', 'booktitle'].member?(attribute)
|
301
|
+
if ans.nil? || ans.empty?
|
302
|
+
self.attributes[attribute] = nil
|
303
|
+
else
|
304
|
+
newval = if attribute == 'editors' || attribute == 'authors'
|
305
|
+
ans.split(" and ").map {|a| a.strip}
|
306
|
+
elsif attribute == 'keywords'
|
307
|
+
ans.split
|
308
|
+
else
|
309
|
+
[ans.strip]
|
310
|
+
end
|
311
|
+
self.attributes[attribute] = newval
|
312
|
+
unless ['url', 'doi', 'keywords'].member?(attribute)
|
313
|
+
self.attributes[attribute + "_lowercase"] = newval.map {|a| a.downcase}
|
314
|
+
self.attributes[attribute + "_words"] = self.attributes[attribute + "_lowercase"].map {|a| a.split(/[[:space:][:punct:]] */)}.flatten
|
315
|
+
end
|
316
|
+
# recalculate all_words
|
317
|
+
tit_auth_words = ['title', 'authors', 'editors', 'booktitle'].map {|att| self.attributes[att + "_words"] || []}.flatten
|
318
|
+
keywords = self.attributes['keywords'] || []
|
319
|
+
self.attributes['all_words'] = keywords + tit_auth_words
|
320
|
+
end
|
321
|
+
end
|
322
|
+
|
323
|
+
# Returns a string representation of an attribute.
|
324
|
+
def show_attribute(attribute)
|
325
|
+
value = self.attributes[attribute]
|
326
|
+
if value.nil?
|
327
|
+
""
|
328
|
+
elsif attribute == 'keywords'
|
329
|
+
value.join(' ')
|
330
|
+
elsif attribute == 'editors' || attribute == 'authors'
|
331
|
+
value.join(' and ')
|
332
|
+
else
|
333
|
+
value[0]
|
334
|
+
end
|
335
|
+
end
|
336
|
+
|
337
|
+
# Returns an array of the field keywords appropriate for a type of entry.
|
338
|
+
def self.fields(entry_type='*')
|
339
|
+
fields = [:title, :authors, :year]
|
340
|
+
case entry_type
|
341
|
+
when 'article'
|
342
|
+
fields += [:journal, :volume, :pages]
|
343
|
+
when 'book'
|
344
|
+
fields += [:publisher, :address]
|
345
|
+
when 'chapter'
|
346
|
+
fields += [:booktitle, :chapter, :publisher, :address, :pages]
|
347
|
+
when 'incollection'
|
348
|
+
fields += [:booktitle, :chapter, :publisher, :address, :editors, :pages]
|
349
|
+
when '*'
|
350
|
+
fields += [:journal, :volume, :booktitle, :editors, :chapter,
|
351
|
+
:publisher, :address, :pages]
|
352
|
+
end
|
353
|
+
fields += [:keywords, :url, :doi, :comments]
|
354
|
+
return fields
|
355
|
+
end
|
356
|
+
|
357
|
+
# Returns the fields appropriate for an entry.
|
358
|
+
def fields
|
359
|
+
entry_type = self.show_attribute('entry_type')
|
360
|
+
Entry.fields(entry_type)
|
361
|
+
end
|
362
|
+
|
363
|
+
def url
|
364
|
+
AWS::S3::S3Object.find(self.name, @@bucket).url(:expires_in => 60 * 10) # expires in 10 min
|
365
|
+
end
|
366
|
+
|
367
|
+
private
|
368
|
+
# Returns the author's last name.
|
369
|
+
def last_name(author)
|
370
|
+
if author =~ /,/
|
371
|
+
author =~ /([^ ,]+),/
|
372
|
+
else
|
373
|
+
author =~ /([^ \t]+)$/
|
374
|
+
end
|
375
|
+
return $1
|
376
|
+
end
|
377
|
+
|
378
|
+
end
|
379
|
+
|
380
|
+
end
|
metadata
ADDED
@@ -0,0 +1,96 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: jgm-cloudlib
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: "0.1"
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- John MacFarlane
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
|
12
|
+
date: 2008-12-25 00:00:00 -08:00
|
13
|
+
default_executable: cloudlib
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: aws-s3
|
17
|
+
version_requirement:
|
18
|
+
version_requirements: !ruby/object:Gem::Requirement
|
19
|
+
requirements:
|
20
|
+
- - ">="
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: 0.5.1
|
23
|
+
version:
|
24
|
+
- !ruby/object:Gem::Dependency
|
25
|
+
name: aws-sdb
|
26
|
+
version_requirement:
|
27
|
+
version_requirements: !ruby/object:Gem::Requirement
|
28
|
+
requirements:
|
29
|
+
- - ">="
|
30
|
+
- !ruby/object:Gem::Version
|
31
|
+
version: 0.3.1
|
32
|
+
version:
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: sinatra
|
35
|
+
version_requirement:
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 0.3.2
|
41
|
+
version:
|
42
|
+
- !ruby/object:Gem::Dependency
|
43
|
+
name: highline
|
44
|
+
version_requirement:
|
45
|
+
version_requirements: !ruby/object:Gem::Requirement
|
46
|
+
requirements:
|
47
|
+
- - ">="
|
48
|
+
- !ruby/object:Gem::Version
|
49
|
+
version: 1.2.9
|
50
|
+
version:
|
51
|
+
description: "Cloudlib is a ruby library and commands for maintaining a library of books and articles on the Amazon 'cloud': S3 and SimpleDB."
|
52
|
+
email: jgm@berkeley.edu
|
53
|
+
executables:
|
54
|
+
- cloudlib
|
55
|
+
- cloudlib-web
|
56
|
+
extensions: []
|
57
|
+
|
58
|
+
extra_rdoc_files:
|
59
|
+
- README
|
60
|
+
files:
|
61
|
+
- README
|
62
|
+
- LICENSE
|
63
|
+
- cloudlib.gemspec
|
64
|
+
- lib/cloudlib.rb
|
65
|
+
- bin/cloudlib
|
66
|
+
- bin/cloudlib-web
|
67
|
+
has_rdoc: true
|
68
|
+
homepage: http://github.com/jgm/cloudlib
|
69
|
+
post_install_message:
|
70
|
+
rdoc_options:
|
71
|
+
- --main
|
72
|
+
- README
|
73
|
+
- --inline-source
|
74
|
+
require_paths:
|
75
|
+
- lib
|
76
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
77
|
+
requirements:
|
78
|
+
- - ">="
|
79
|
+
- !ruby/object:Gem::Version
|
80
|
+
version: "0"
|
81
|
+
version:
|
82
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
83
|
+
requirements:
|
84
|
+
- - ">="
|
85
|
+
- !ruby/object:Gem::Version
|
86
|
+
version: "0"
|
87
|
+
version:
|
88
|
+
requirements: []
|
89
|
+
|
90
|
+
rubyforge_project:
|
91
|
+
rubygems_version: 1.2.0
|
92
|
+
signing_key:
|
93
|
+
specification_version: 2
|
94
|
+
summary: Tools for maintaining a library of books and articles in Amazon S3 and SimpleDB
|
95
|
+
test_files: []
|
96
|
+
|