Full native S3 integration

July 2017, Peter Thompson

As we’ve stated before, we believe LucidLink is the easiest, highest performing way for any application on any OS to mount and use any object storage as local disk.
Our vision to provide seamless access to data stored in object storage in distant locations (cloud to edge, edge to cloud, cloud to cloud, compute to storage, etc), took a pretty major step forward with our recent build – LucidLink can now stream objects as files natively with Amazon S3.

Distributed file system, streaming objects as data

In order to realize our goal, we needed to be able to do 2 things: split the file into 2 planes (syncronizing the metadata and efficiently streaming the file data), and write this file data as objects, natively in and out of S3.

  • The first step was achieved before we officially formed LucidLink. (For a look at some of that history, check out George’s post about how it all got started.) Essentially, we would sync the metadata, which takes little space and is fast to do, and stream the file data. On the surface it sounds like an easy concept, but in practice there are a lot of moving parts – especially when you add a shared namespace / collaborative element into it.
  • The second step was also a non-trivial technical challenge – to integrate directly into Amazon S3, not as a call to their put/get commands, but writing data directly to objects. Not only does this greatly simplify the experience of setting up and using a bucket, but it also allows users to simply mount and object store, and use it as local disk, with a user experience to match.

Let’s dive into this in greater detail below.

Lucid Coordination Service

Below are a series of screenshots of the process of setting up a cloud, followed by a demo video.


The LucidHub functions to create a bucket in Amazon S3 with the customer’s own credentials, initialize the bucket, and associate it with any user running the Lucid App with access to the cloud name and credentials. This is a one-time set-up per cloud. In this current iteration for beta testing purposes, the only information required will be to enter AWS credentials, choose a cloud name, and an AWS region. As we implement authentication into the build, other fields will become required as well.

This is the LucidHub or the Lucid Coordination service. It can be configured to run anywhere, will be hosted and managed by LucidLink, and will be completely transparent to the user. I’m posting it here to provide a glimpse into what we are doing in the background, and to show that we are not just using API calls, but rather it is a true, integration.

Note the bucket which was created in my AWS account by the Lucid Coordination Service.

 Lucid App on client

Now simply log into the cloud you just created with the LucidApp and you are done!

LucidApp creates a virtual folder showing up in the user directory in Explorer (or Mac Finder etc.). The mount point can also be user configured. To use the cloud, just point your application at this folder, and anything saved here will be streamed and written as objects. There is nothing else to configure or set up – simple!

Here is the end-to-end video

The last storage upgrade you'll ever need

Get Started