As we’ve stated before, we believe LucidLink is the easiest, highest performing way for any application on any OS to mount and use any object storage as local disk.
Our vision to provide seamless access to data stored in object storage in distant locations (cloud to edge, edge to cloud, cloud to cloud, compute to storage, etc), took a pretty major step forward with our recent build – LucidLink can now stream objects as files natively with Amazon S3.
Distributed file system, streaming objects as data
In order to realize our goal, we needed to be able to do 2 things: split the file into 2 planes (syncronizing the metadata and efficiently streaming the file data), and write this file data as objects, natively in and out of S3.
- The first step was achieved before we officially formed LucidLink. (For a look at some of that history, check out George’s post about how it all got started.) Essentially, we would sync the metadata, which takes little space and is fast to do, and stream the file data. On the surface it sounds like an easy concept, but in practice there are a lot of moving parts – especially when you add a shared namespace / collaborative element into it.
- The second step was also a non-trivial technical challenge – to integrate directly into Amazon S3, not as a call to their put/get commands, but writing data directly to objects. Not only does this greatly simplify the experience of setting up and using a bucket, but it also allows users to simply mount and object store, and use it as local disk, with a user experience to match.
Let’s dive into this in greater detail below.
Lucid Coordination Service
Below are a series of screenshots of the process of setting up a cloud, followed by a demo video.
Lucid App on client