LucidLink Service Status

Past incidents

May 17, 2024

No incidents reported.

May 16, 2024

No incidents reported.

May 15, 2024

No incidents reported.

May 14, 2024

No incidents reported.

May 13, 2024

No incidents reported.

May 12, 2024

No incidents reported.

May 11, 2024

No incidents reported.

May 10, 2024

No incidents reported.

May 9, 2024

No incidents reported.

May 8, 2024

No incidents reported.

May 7, 2024

No incidents reported.

May 6, 2024
7pm UTC

As promised, we are sharing the root cause analysis publicly, which includes details on what led to the outage, the steps taken to reconnect service for all customers and the mitigation actions for the future.

We are extremely sorry for the disruption the outage had on our customers. While we are proud that our zero-knowledge encryption model performed as intended and our customer data remained secure throughout this malicious attack, the entire company has learned and grown from this experience. As a company made of both enterprise storage technologists and former content creators, our superpower has always been understanding the workflows and experiences of content creators. Because of this, we deeply felt the pain, anxiety, and frustration of our customers’ downtime.

Please know, we remain:

Committed to ensuring that the security of our infrastructure remains appropriate and limiting the potential that another outage like this is experienced again.

Devoted to our customers and continuing with your trust in the system that has been a part of your everyday workflows.

Intent on providing ongoing and transparent communication regarding relevant findings and outage resolution.

Dedicated to supporting the talented creatives that have been in our DNA since the beginning.

May 5, 2024

No incidents reported.

May 4, 2024

No incidents reported.

May 3, 2024
10:30pm UTC

Now that service has been restored for all customers, we are working on the root cause analysis and will distribute this document publicly once completed early next week. It bears repeating that this was not a ransomware attack, and no customer data was compromised. Although users couldn’t access their data during the outage, no personal or corporate data was leaked or otherwise affected due to LucidLink’s Zero-Knowledge security model. Given the severity and maliciousness of the attack, LucidLink has engaged with law enforcement, legal, and external cyber forensic experts. 

We also want to acknowledge that there are customers with metadata synchronization issues since regaining access to their filespaces. As we reported during the outage, this is likely a result of users who still had access to their filespace and continued working while the service was offline. If you’re experiencing these synchronization issues, please create a ticket in our support system so we can help you address this problem.

As a precaution, if you currently have access to work that was done while our service was offline, please make sure to copy this data to a location outside of your LucidLink filespace.

Thank you for your patience. We are committed to helping all of our customers address any remaining disruption from this outage.

May 2, 2024

No incidents reported.

May 1, 2024

7:30pm UTC

We are happy to announce that the manual intervention to restore all filespace metadata services for the few remaining accounts is complete. All customers regained access to their filespaces, and we became fully operational as of mid-day UTC.

We continue to monitor the situation. If you have any further issues, please create a ticket in our support system to allow us to support you in the best way possible.

The past couple of days have been pretty crazy. It’s the first time we’ve been in this situation since founding the company 8 years ago. We have learned a lot and grown from the experience. We are proud that our Zero-Knowledge architecture performed as intended, preventing any data breaches. We also want to acknowledge the downtime experienced by our customers during this outage. We fully understand that you rely on our technology and take this seriously.

Now that service has been restored for all customers, we will transition efforts to root cause analysis. As stated, we will share our findings once we have all the facts.

The support and encouragement we received from our customers has been humbling. It makes the efforts of the past 48 hours meaningful and reaffirms our commitment to openness and transparency.

05:15am UTC

We are happy to announce the restoration of filespace metadata service is now complete, and access to the vast majority of filespaces has been fully restored. As we expected, a small number of metadata instances failed to initialize properly and will require manual intervention. This is a time-consuming process, and our team will continue working until all customers are fully restored. 

If you are one of the customers who still has a filespace in a disconnected state, please create a ticket in our support system. We will proactively create tickets on behalf of affected accounts as we identify those without open tickets. Moving forward, we will switch our communications from general mass updates to individual tracking in our support ticket system. 

We understand that your organizations require additional in-depth accounting of what took place and why, as well as the steps being taken to mitigate the possibility of future occurrences. Now that we have brought the majority of our customers back online and reinforced our systems, we will continue implementing measures and will provide a report for you and your management. I do not yet have a timeline for this, but we are committed to sharing our findings.

Thank you to everyone who has supported, encouraged, and provided moral support these past days. It is the energy of our customers that fuels the passion of our entire team.  

00:00 UTC

We are about to launch the final and largest batch of filespace metadata service restorations. We have further increased the build infrastructure to expedite the process, which we estimate with a high degree of confidence to take 4 hours to complete. This is a rolling restoration, so filespaces will continuously become available during this period.

Once again, users may need to “connect” or “disconnect” and “connect” from the LucidLink App for the connection to resolve. Remember that if you have had access to your files during the outage, be sure to copy them to a local disk before completing the above.

We know there will be some edge cases where the service instance will require manual intervention, which we must address once the batch run has finished. Our teams will work with the individual customers owning these filespaces as soon as they are identified.

Thank you again for your patience and for hanging in there with us. We will provide an update when the next batch is completed.

Apr 30, 2024

9:00pm UTC

We now have about 20% of affected filespaces back online, and the restoration velocity is increasing.

Many of the early restores required manual intervention, taking longer than anticipated to get to this point. We’re now increasing the restoration batch size and expect to be through the process within the next 6 to 8 hours. Again, this will be done on a rolling basis, and the filespaces will be available as soon as each restoration is completed.

Note that you may need to manually click “connect” in your Lucid App or “disconnect” and then “connect” for your filespace to resolve as quickly as possible. Remember that if you have had access to your files during the outage, be sure to copy them to a local disk before completing the above.

Our next update will be in approximately three hours from now by 0:00 UTC.

6:30pm UTC

We rebuilt and scaled the discovery service to accommodate the additional load of the disconnected filespaces trying to reconnect to the offline metadata service. Based on these updates, a number of our customers have already regained access to their filespaces. You can confirm your connection status through the LucidLink App, which will show a status as “connected.”

We have confirmed that the backup restoration process is working as expected. The first restored filespaces are back online. As we complete more restorations, we’ll have a better estimate of how long the entire process will take to restore all of our customers affected. We realize this is the most important point for you, and as soon as we have more visibility, we’ll update you. 

You may have seen a new error message saying, “AuthProxy: Call to the LucidLink filespaces service timed out.” This indicates your client is now getting through to discovery but timing out because your filespace's metadata service is offline. Once the metadata service is restored for your filespace, you will be able to reconnect.

We’re receiving a number of questions regarding the scenario where customers remained connected to their filespace and continued to work during the outage. We have created a Knowledge Base article hosted on our support site detailing various scenarios, but because there are variations depending on circumstances not readily apparent, we recommend that customers save any work done during the outage to a local disk to ensure your work is saved.

3:30pm UTC

No file data prior to the last backup before the outage was lost and no personal or corporate information was leaked. However, the metadata service for each filespace was damaged during this attack, which is causing the connection outage. We’re fixing the metadata service, which will restore access to your data. Each filespace's metadata service is restored individually.

We’ve identified some filespaces with metadata service that are unaffected but are unable to connect to the discovery service due to the metadata infrastructure attack. We have allocated additional resources to focus on that subset and will manually reconnect these filespaces to the discovery service, bringing them online. This should happen within the next hour, by 16:00 UTC.

In addition, we are now in step two of the process, restoring the metadata state for all individual filespaces, and are on track to restore access to filespaces in the next 4-6 hours. We will do this on a rolling basis to get customers back online faster. During this process, we will not be able to notify customers individually when their filespace is back online. Customers will be able to monitor their connection status on the LucidLink app.

12:15pm UTC

To restore filespace access from our backup systems, we are in the process of rebuilding the metadata infrastructure. This infrastructure consists of a discrete instance for each filespace. The recovery process is a 2-step process: 1) build the infrastructure and 2) restore each individual instance from a specific filespace backup. Upon completing step two of this process, we will connect the new infrastructure to the discovery service, and customers will regain full access to their filespaces. Our estimate for this process is 6 to 8 hours (18:00 to 20:00 UTC). We will do everything we can to accelerate this timeline.

In the last update, we referred to the backup window. Here is additional information regarding the timing of this incident. The service went down at 12:55 UTC. As I noted above, our systems backup each filespace individually on a rolling basis every 6 hours. We cannot provide your individual backup time, but the worst case would be 6 hours prior to the incident or 6:55 UTC.

Apr 29, 2024

4:15am UTC

We have confirmed a malicious attempt against our core metadata service. No file data prior to the last backup before the outage was lost and no personal or corporate information was leaked. The metadata service now needs to be restored from our backups. As a result, there is a slight chance that a few hours of work from the point of the last backup to when the attack shut down the service will be unrecoverable. 

9:27pm UTC

We have confirmed the backup systems have not been impacted and will be using these systems to restore full access to your filespaces. This will take up to a day and could result in some loss of work done since our last system backups, which are scheduled to run every 6 hours.

4:20pm UTC

We have identified a system outage that may be affecting access to filespaces. Our team is working on identifying and resolving the issue as soon as possible.

Apr 28, 2024

No incidents reported.

Apr 27, 2024

No incidents reported.

Apr 26, 2024

No incidents reported.

Apr 25, 2024

No incidents reported.

Apr 24, 2024

No incidents reported.

Apr 23, 2024

No incidents reported.

Apr 22, 2024

No incidents reported.

Apr 21, 2024

No incidents reported.

Apr 20, 2024

No incidents reported.

Apr 19, 2024

No incidents reported.

Apr 18, 2024

No incidents reported.

Apr 17, 2024

No incidents reported.

Apr 16, 2024

No incidents reported.

Apr 15, 2024

No incidents reported.

Apr 14, 2024

No incidents reported.

Apr 13, 2024

No incidents reported.

Apr 12, 2024

No incidents reported.

Apr 11, 2024

No incidents reported.

Apr 10, 2024

No incidents reported.

Apr 9, 2024

No incidents reported.

Apr 8, 2024

No incidents reported.

Apr 7, 2024

No incidents reported.

Apr 6, 2024

No incidents reported.

Apr 5, 2024

No incidents reported.

Apr 4, 2024

No incidents reported.

Apr 3, 2024

No incidents reported.

Apr 2, 2024

No incidents reported.

Apr 1, 2024

No incidents reported.