Using AWS S3 object storage with a Paperspace GPU cloud server

Paperspace GPU cloud server

I was nearing the end of a demo for a prospective customer whose company was dealing with large amounts of 3D geospatial data that they wanted to store and directly access within S3. Can LucidLink do it? No problem! I replied and walked him through it.
He then landed on the question everyone inevitably arrives upon – could I use this for my personal storage solution? “I’ve got data on different drives, some in Dropbox, and some in a cloud server called Paperspace. It would be great if I could consolidate everything, not consume storage everywhere, but still have immediate access to whatever I need. Would a Paperspace cloud server be able to use LucidLink?”
Honestly, I’d never heard of Paperspace until then (2 days ago), but if it was just a virtual machine with an internet connection and a standard OS, then it should work just fine. After all, one of the most valuable capabilities of LucidLink is the ability to inject a universal filesystem anywhere, present S3 compatible object storage as if it is local, and only stream what the application requires on demand.
So I decided to give it a try.

Setting it all up

I navigated over to Paperspace to check it out, and it looked really straightforward – a couple of clicks and within 5 minutes I’d have a server all my own, running within my browser. Cool!
Next up – the storage aspect. You first had to choose pre-sized SSD drives for a monthly fee, per individual machine. What about teams? You can have team drives (for a modest additional fee), after setting up a private network, and mapping a network drive. How would I get files from my computer into Paperspace? Their FAQ suggests drag and drop for small files (and Windows only), and BTsync for large files. But what if you were trying to collaborate both from a Paperspace machine and an edge machine? Didn’t say, but I imagine the above in reverse, and take care of version control yourself or use a sync service. There are FAQ’s about running out of storage, and expanding your drive. Whew.
What struck me was the acute contrast between the simplicity of the server and the complexity of the storage. That’s not strange – storage is hard. Now I started getting excited. This was a perfect application for LucidLink – elastic grow on demand, persistent and accessible from anywhere, converge edge and cloud data, fast, reliable, cheap…
I set up an account. Very quick and easy as expected. Next I began selecting the specs of the lightest weight Windows 10 machine they had. (Cheapskate, I know.) Then impulsively (optimistically?) decided to  record the whole exercise live. (Spoiler alert – it worked perfectly. If you want to skip the rest of the post and just watch the video, scroll to the bottom.)

The process in pictures

For you stalwart masochists, here are some screenshots of the process.

Choose your region

Choose your OS

Choose specs and billing meter

Choose your storage option. Since we would be using LucidLink as our primary data repository, I elected the smallest drive possible – just to host the OS.

Snapshot? Nah, this is a burner! LucidLink is persistent, is protected to 13x9s by S3, and can be injected anywhere.

Hit “create your space”, and voila – a GPU optimized cloud machine designed to tackle those kinds of workloads.

The process was simple and elegant, a great user experience. I haven’t put any stress on the machine itself, so cannot vouch for that, but at this point I don’t have any reason to doubt the claims on the website.
Now, wouldn’t it be great to match that smooth server setup experience with an equally powerful and user friendly storage experience? Let’s add a LucidLink mount point! To set up the shared filespace, I chose the AWS us-west-1 datacenter as it would be closest to to the Paperspace datacenter.

Conveniently, Chrome is already installed and on the desktop, so I simply navigated to the LucidLink download page, and installed the client, joined the file space I created, and immediately had a production ready “drive” at my disposal, which would grow on demand, and could be exposed to any other machine, within Paperspace, another cloud, or on the edge!

The entire process to under 20 minutes, with the longest time being spent waiting for provisioning the new server. (The video is 11 minutes as I sped that portion up.)

That was easy. Almost too easy. I really like Paperspace, but I like it a whole lot more with a LucidLink file system!

*screenshots are courtesy of the Paperspace website