Amazon S3 REST Wrapper

Posted At : September 7, 2006 4:17 PM | Posted By : Joe Danziger
Related Categories: Code, ColdFusion

As my first foray into open-source code, I'm releasing a wrapper for interacting with Amazon's Simple Storage Service (S3) via REST. The wrapper is packaged as a CFC and has the following methods:

init(accessKeyID, secretAccessKey) - initialize CFC (both parameters required).
getBuckets() - List all buckets.
putBucket(bucketName) - create a new bucket.
getBucket(bucketName, prefix, marker, maxKeys) - get contents of a bucket (prefix is optional and matches on the beginning of a key, marker is optional and results start from there, maxKeys is optional and restricts the number of objects returned).
deleteBucket(bucketName) - delete a bucket (bucket must be empty).
putObject(bucketName, fileKey, contentType, HTTPtimeout) - puts an object into a bucket (HTTPtimeout is in seconds).
getObject(bucketName, fileKey, minutesValid) - get link to an object (minutesValid is optional and defaults to 60).
deleteObject(bucketName, fileKey) - delete an object from a bucket.

A simple test script is included which demonstrates the use of the CFC. You must insert your Amazon S3 access keys in the first 2 lines in s3test.cfm, then just pull it up in a browser.

This is an initial release. Future plans include support for Access Control Lists. If you need something else added, let me know.

This script should run on both ColdFusion MX 6 and 7, let me know if you run into any problems.

The current version is 1.1 and you can visit the project page and download here.

Comments (56) |

Send |

del.icio.us |

Digg It! |

Linking Blogs | 23279 Views

Related Blog Entries

Amazon S3 & Smartermail REST Wrappers v1.2 (October 4, 2006)

TweetBacks

Comments

This looks interesting! Thank you. I've been intruiged by S3 since it was released. I'll have to give this a try if I can come up with a need to use S3 on a future project.

# Posted By Richard Davies | 9/7/06 7:27 PM

Rockin! good job Joe!

# Posted By Pete Freitag | 9/8/06 8:21 PM

Man I love this wrapper. I have 2 projects where I'm storing all the files on S3 and this is exactly what I needed. It's kind of funny. I did a Google search for Amazon S3 and Coldfusion and this was one of the top sites that came up. So the same day I downloaded the wrapper I subscribed to the blog's RSS feed and added it to del.icio.us so I can stay on top of any new posts you create.

# Posted By Rich | 10/5/06 9:34 AM

Thanks for the positive comments and glad you're getting some use out of the CFC!

S3 is definitely a great service and I've recently been able to move to a hosting provider after 10+ years of running my own servers now that I don't have to worry about the storage equation and scaling up. Stay tuned for an article on this wrapper in the next CFDJ..

# Posted By Joe Danziger | 10/5/06 4:22 PM

Thank you very much for all your work, this is great! I have a question though. I am able to use the getObject to view pictures I put on S3, but I can't get it to work for other things like pdfs. The link is the only thing that shows up in the browser.

Do you know if you have to put the link within a cfcontent and or cfheader tag?

Thank you again.

# Posted By Charles | 10/18/06 10:30 PM

Hey Charles, glad you're getting some use out of the CFC. The link that gets returned from getObject is a standard HTTP URL. You can access the file directly, or if you need to process it with ColdFusion, you would probably need to do a <CFHTTP> to that link and then you could return the file directly using <cfcontent> and <cfheader>, but for simple uses you shouldn't need to do all that..

# Posted By Joe Danziger | 10/23/06 6:17 PM

Is there something weird with S3 object URLs? The reason I ask is that I am using the S3 Wrapper in a Model-Glue Unity site I'm building. I instantiate the wrapper with Coldspring. But when I use the getObject method I always get a URL but sometimes the image that it points to shows and sometimes it doesn't. Is it something possibly on my end that I could be doing wrong. All I'm trying to do is use the generated URL in an img tag. Any help would be greatly appreciated.

# Posted By R. | 11/16/06 9:30 AM

I figured out the problem (I think). Sometimes the signature of the link has characters that need to be url encoded or else it gives me an error that the signature is not valid. So I changed the timedAmazonLink variable to have Signature=#URLEncodedFormat(signature)# instead of Signature=#signature#.

# Posted By R. | 11/16/06 6:30 PM

R,

Thanks for your solution. It cleared up the issue for another user as well and I've added made a 1 line fix to the CFC which clears everything up. Basically, line 276 now includes the URLEncodedFormat, like such: <cfset signature = URLEncodedFormat(ToBase64(Hex2Bin("#digest#")))>

Thanks for Charles K. for verifying the fix. Hopefully that's the last bug! :)

# Posted By Joe Danziger | 11/20/06 11:58 PM

When I run the s3test.cfm I get an error "Prefix must resolve to a namespace: " What am I doing wrong?

# Posted By Ron | 1/22/07 8:40 AM

Hmmmm. I seem to be getting the same "Prefix must resolve to a namespace:"

Any suggestions?

Thanks!

# Posted By DavidR | 1/31/07 5:37 PM

Okay... i'm an idiot. I wasn't completely signed-up to use Amazon S3. Now that I am, your Amazon S3 REST Wrapper works like a charm!!!

Thanks,
David.

# Posted By DavidR | 1/31/07 6:30 PM

Anyone have any issues with the terms of service that you can't call the API more than once a second?

How would you keep this from happening? It seems if you were retrieving the information it could be several times a second if multiple users hit different pages.

# Posted By jlr | 2/6/07 9:54 AM

Is that a restriction? I can't see why Amazon would restrict how much you could call the service since it is pay-per-use.

# Posted By Joe Danziger | 2/6/07 11:34 AM

I got worried :). Although if you look in the terms of service for AS3 they amended it to be unlimited. There other web services have limitations. Sorry for the incorrect post I was just trying to figure out how you would get around such a limitation.

Great Library by the way seems to work wonderfully. Have you done anything with private objects?

# Posted By jlr | 2/6/07 2:32 PM

jlr: Glad to hear you got it working! What did you have in mind in regards to private objects?

# Posted By Joe Danziger | 2/6/07 4:05 PM

It works 75% of the time for me. The other 25% of the time I get an error:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
Anyone else have this problem?

# Posted By Dominic | 2/7/07 4:52 PM

Dominic: you should download the latest version and try that. There was a small issue with URL encoding which has been fixed.

# Posted By Joe Danziger | 2/7/07 4:54 PM

Joe-
Thanks so much for your prompt reply. The readme file I have says version 1.2, is that not the latest release?
Also, I don't plan on doing any deleting, but just so you know, BinaryDecode isn't available in MX6.
Thanks,
Dominic

# Posted By Dominic | 2/7/07 5:40 PM

Dominic: there was actually a tiny change since 1.2, which was just wrapping a variable in URLEncodedFormat(). I really should have probably made that 1.21 or something. I would try updating the CFC if you are having those problems, it sounds like that's what it is.

As for MX6, there is a Hex2Bin function that can replicate the functionality. It may just be commented out in the released version, but let me know if you need it and it's not and I can probably find it somewhere.

# Posted By Joe Danziger | 2/7/07 6:43 PM

Hi Joe,
I just downloaded the package from the link on this page for the first time yesterday, and updated this morning as well, but I don't see any URLEncodedFormat(). Obviously it's just as easy for me to add it myself, but I thought I'd let you know.
Also, in the CFC I have now, all the functions use Hex2Bin (which is included) except for deleteObject, which uses binaryDecode. Again, I can make the change myself, but I thought you'd like to know. FWIW, the CFC I have says
Version 1.2 - Released: October 4, 2006
Thanks again,
Dominic

# Posted By Dominic | 2/8/07 11:34 AM

Joe,

This looks terrific. Everything works for me, except one thing: when I click on a link the file is downloaded, but it's still in an encrypted format. (I made the change for URLEncodedFormat -- that's not the problem).

I can see my files, I can get to the link, but when it downloads, the file is still encrypted. Any thoughts? I have all the hmac files in the right place. (I'm using BlueDragon 6, but that shouldn't be a problem, I don't think).

Appreciate any help. This looks like a terrific wrapper. Thanks!

# Posted By Jack | 2/16/07 3:45 PM

OK, so I figured out my problem. I had previously used another service to put files into S3, and apparently they were encrypted -- so when I tried to download them with your CFC, they showed up encrypted (what a surprise!) Of course, when I uploaded files via your CFC, then downloaded, everything is fine.

I did make 2 changes to the CFC that may be helpful.

1) Changed binaryDecode to Hex2Bin on about line 303 (as mentioned above), and
2) Also needed to add "ToString" on about line 249, as follows:

<cfhttpparam type="body" value="#ToString(binaryFileData)#">

Otherwise it was throwing an error, "Cannot convert binary data to string". Might be BlueDragon specific, but now everything is working perfectly.

Great CFC, Joe. Thank you!

# Posted By Jack Welde | 2/16/07 8:43 PM

Trying to upload a file to a subdirectory of bucket. Tried editing PutObject line in s3.cfc:
<cfhttp method="PUT" url="http://s3.amazonaws.com/#arguments.bucketName#/tes...;
timeout="#arguments.HTTPtimeout#">

The subdirectory /test/ does exist, file upload seems to go thru but file never gets there.

# Posted By cu | 2/18/07 7:39 PM

hello.
one question.
will this work on ColdFusion MX 7 Standard edition (one server) ?
Or you need a enterprise edition in order to access amazon S3 service ?
let me know
poz
ob1

# Posted By ob1 | 2/19/07 4:03 AM

Answered my own question in regards to putting an object into a subdirectory of a bucket:\
Also need to edit this line with subdirectory name:
<cfset var cs = "PUT\n\n#arguments.contentType#\n#dateTimeString#\nx-amz-acl:public-read\n/#arguments.bucketName#/#arguments.fileKey#">

I've created a var that I update and have inserted in the line above:
<cfset subPath = "test/">
<cfset var cs = "PUT\n\n#arguments.contentType#\n#dateTimeString#\nx-amz-acl:public-read\n/#arguments.bucketName#/#subPath##arguments.fileKey#">
Use trailing slash in case subPath is blank, so will not break if no sub path specified.
Quick and dirty.

# Posted By cu | 2/19/07 3:00 PM

I'm using CFMX7 Standard version and interfaces with Amazon S3 with no problems.

# Posted By cu | 2/19/07 3:11 PM

@Jack: I had one reference to the binaryDecode left in that one function which needs to be replaced like you did. That makes it more backwards compatible.

@bhbidder: Be careful with your bucket names - bucket names are unique across the enture Amazon S3 platform, so that is why test would be a problem - someone already created one with that name.

# Posted By Joe Danziger | 2/21/07 4:48 PM

Thanks for the info. Already aware of that - test referred to a subdirectory of a unique bucket name. I'm designing an interface in CF where I can move files around via S3 instead of having to do separate get, put, delete, etc.

# Posted By cu | 2/25/07 1:57 PM

Great script Joe! What is the eta for ACL?

# Posted By Perry | 3/4/07 2:43 PM

Hmm.. hadn't really heard anyone else ask for the ACL stuff really, so didn't think it was too critical. Kinda wrapped up at the moment but I will try and find some time over the next few weeks or months. Hopefully not too pressing right now!

# Posted By Joe Danziger | 3/6/07 3:47 PM

Found that routine chokes on subdirectory of buckets that are only one character in length.
ex. \mybucket\a\

# Posted By cu | 3/9/07 10:14 AM

love the script...it works great.

just curoius if you have thought about adding Logging functionality to it?
the Amazon documentation for enabeling logging is pretty wacky...i might take a stab at it if you don't have any plans to do it...

# Posted By hungry | 3/18/07 6:24 PM

In wanting to retrieve a file from a protected bucket, I ran into a problem with the epochTime parameter. Sometimes the file would render, other times not with a signature error message. I was not concerned with the expire time. Just wanted to access the file from the protected bucket. I made this change to the code to add one year to epochTime, and it seems to work so far:

<cfset epochTime = DateDiff("s", DateConvert("utc2Local", "January 1 1970 00:00"), CreateDateTime(Year(now())+1,"12","20","00","00","00") ) >

Thought it might help others.

# Posted By cu | 3/18/07 8:30 PM

Hi All,

I have enhanced this cfc with some new features.

-The ability to set an ACL when you put a bucket or object.
-A getAcl function to return groups and user Acl's.
-A putBucketLogging to enable and disable logging.
-A getBucketLogging to retrieve the current logging status.

I'll email Joe to see if he is happy to add it to his CFC to save creating multple versions.

# Posted By Robert McCarthy | 3/23/07 12:21 PM

I'd really like a copy Robert. Can you send or give a link to what you have?

Much thanks!

Perry

# Posted By Perry | 4/7/07 10:28 PM

Anyone able to tell be how i can create subdirectories and add files to the them using REST?

Any help greatley appreciated

# Posted By Richard | 4/18/07 5:47 AM

Scratch my last post... figured my problem out. Thanks to Robert Mccarthy from Goss Interactive for his help. Thanks aslo do Joe for the CFC.. had made my life easier today. Now that I got this issues done and dusted ahead of schedule I'm off the to golf course for some R&R.

# Posted By Richard | 4/18/07 8:11 AM

Prefix must resolve to a namespace: - I'm getting this too, and I'm sure I'm properly signed up, as I can use the JungleDisk utilities.

# Posted By mac jordan | 4/25/07 9:31 AM

Great script Joe. Just what I needed to automate my uploads to Amazon S3 from the site. Keep up the fantastic work.

# Posted By Steve | 5/11/07 6:11 PM

Thanks for this amazing script!! I was just wondering if the custom tag cf_hmac could be more updated.. i noticed that it's code from 2003 and all these algo thingies are in MX6 and 7 aren't they?? I have no clue how that tag works but maybe someone could update it.. and make it into a CFC??
Thanks again for this amazing script.. i'm stoked at the moment!!

By the way.. you guys did know you can also reach your files thru the following construction... http://bucketnamet.s3.amazonaws.com/name-of-the-fi...

# Posted By Tjarko Rikkerink | 5/26/07 7:32 AM

Hey Everyone.
While recently working on a project using Amazon S3, I developed the need to be able to edit the ACL of the buckets I was adding to my S3 account.
Following on from the great work by Joe, I have added ACL compatibility to the putBucket method.
For those of you who are interested, you can download the edited version here - http://www.stevehicksonline.com/2007/06/04/amazon-...
Steve

# Posted By Steve | 6/4/07 5:27 PM

I followed the readme and got everything installed, but when I hit the test page I get: An error occured while Searching an XML document.
Prefix must resolve to a namespace:

Has anyone ever seen this before? Id really like to start using s3

# Posted By anthony | 6/7/07 2:29 AM

Nevermind, I hadnt clicked the link in my email to verify my account, working great now!

As for these "folders", anyone know if you somehow have subfolders within a folder?

# Posted By anthony | 6/7/07 2:50 AM

It would be nice to have full support to put to and get from "subdirectories" see: http://developer.amazonwebservices.com/connect/mes...

# Posted By anthony | 6/7/07 4:06 AM

Hi, I actually changed the complete use of the md5 custom tags and such.. you can read that on my blog. With ColdFusion MX 7 you don't need those customtags anymore to hash the keys.

# Posted By Tjarko Rikkerink | 6/21/07 4:16 PM

Looks interesting. Thanks for sharing your work. My only suggestion would be to add some error handling for when authorization fails. It currently just returns an xml search error, which isn't very intuitive.

# Posted By Chris S | 8/30/07 9:22 AM

As a simple test, I just tried storing s3.zip in a bucket. Uploading seemed to work fine, but when I tried to download it using the link generated by s3test.cfm I get the error "The request signature we calculated does not match the signature you provided. Check your key and signing method."

# Posted By Chris S | 8/30/07 9:34 AM

I'm getting that error about half the time. It happens for both ascii and binary files. I don't see a pattern in the errors, since I'll get the error for one link, then I'll refresh the link and it'll work, then a minute later it'll be broken again. I'd really like to start using S3 but if it's only going to work 50% of the time, it's not going to be worth it. Any thoughts?

# Posted By Chris S | 8/30/07 10:20 AM

Double-checked R's solution, and found that that bug was still not fixed in the posted code.

@Joe, could you please update the code? You said you fixed it in 1.2, but the only link I can find is to the older version.

# Posted By Chris S | 8/30/07 11:44 AM

@Chris S -- not sure if it's the same issue, but I've found that URLEncoding the signature in the getObject function eliminated the problem of links that only work half the time. I changed this line:

<cfset timedAmazonLink = "http://s3.amazonaws.com/#arguments.bucketName#/#ar...(signature)#">
Hope that helps...

# Posted By Jack Welde | 9/23/07 1:47 AM

I've also found that uploading is extremely slow, that this wrapper is next to useless for large files. For example, uploading a 3MB image consistently times out after 5 minutes. I installed Jungledisk, an S3 wrapper for the desktop, and was able to upload the exact same file in 38 seconds.

# Posted By Chris S | 11/8/07 11:07 AM

Everything works perfectly except I cant delete a bucket....I am getting a "403 Forbidden" . I can put an object into the bucket view it and delete it. I have tried URLEncoding the Hex2Bin as well.

Here is the function:
<cffunction name="deleteBucket" access="public" output="false" returntype="string"
            description="Deletes a bucket.">
      <cfargument name="bucketName" type="string" required="yes">

      <cfset var signature = "">
      <cfset var dateTimeString = GetHTTPTimeString(Now())>

      
      <cfset var cs = "DELETE\n\n\n#dateTimeString#\n/#arguments.bucketName#">

      
      <cfset var fixedData = replace(cs,"\n","#chr(10)#","all")>

      
      <cf_hmac hash_function="sha1" data="#fixedData#" key="#variables.secretAccessKey#">

      
      <cfset signature = URLEncodedFormat(ToBase64(Hex2Bin("#digest#")))>


      
      <cfhttp method="DELETE" url="http://s3.amazonaws.com/#arguments.bucketName#&quo...; charset="utf-8">
         <cfhttpparam type="header" name="Date" value="#dateTimeString#">
         <cfhttpparam type="header" name="Authorization" value="AWS #variables.accessKeyId#:#signature#">
      </cfhttp>

      <cfreturn cfhttp.header>
   </cffunction>

Any suggestions welcome,

Andrew.

# Posted By Andrew Grosset | 12/2/07 11:16 PM

I should have added that I can delete the bucket using "S3Fox" (firefox addition) and "S3Backup" so its not vital that I use the S3 Rest Wrapper - but it bugs me that I cant!

# Posted By Andrew Grosset | 12/4/07 1:21 AM

Here you can take HMAC SHA1 function (quite simple) to be used with S3 library:
http://www.coldfusiondeveloper.com.au/go/blog/2008...

and then:
<cfset signature = ToBase64(HMAC_SHA1(variables.secretAccessKey,fixedData)) />

# Posted By Dmitry Yakhnov | 2/10/08 6:16 AM

I've been hacking away on the CFC and I have fixed numerous bugs and added support
for compression, text uploads, folders, logging, using a proxies (aka fidler2) and a lot of other little things here and there

I also changed the behaviour to return the cfhttp result so it can be properly inspected
for the correct status codes and a dev level response status code checker

# Posted By Zac Spitzer | 2/26/09 12:18 AM

RSS

Calendar

Sun	Mon	Tue	Wed	Thu	Fri	Sat
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30

Enter your email address to subscribe to this blog.