Sorry, we had a problem

Hi, everyone! Servers are online now. An app update is required to restore the communication. Even though most data are fine, there may have been a partial loss of data ranging between January 24th, 2017 and today.

Your password may automatically change upon the first sync: in this case, we will send you an email with the new password and a link to reset it.

Affected users

The most affected users are the ones signed up after the 24th of January, 2017. If you are one of these users, don’t panic: if you’re logged in the app, a simple synchronization should work, and all your data will be automatically restored.

If you are not logged in on any Android device, your account could’ve been lost. To restore it, please sign up again from the Android device which contains your TV show collection. Your data should be recovered from it, and all will be synced to the server.

If you signed up before the 24th of January, a simple synchronization will work (if you’re not logged in, you can login into the device as usual).

Trakt.tv

If you use trakt.tv, you will need to open the account page in the app and update trakt.tv credentials and make a full sync to make it work again.

What went wrong

TLDR: a long chain of unlikely events happened, and as result, we lost some of the data created after the 24th of January.

This may get technical, so you can skip this paragraph if you like.

As you can imagine, one of our top priorities is to maintain a consistent and reliable service. To do so, we invest thousands of euros every year in hardware, software, guaranteed bandwidth, guaranteed electricity and so on. But this isn’t enough to protect from accidental data loss. To solve this, we do periodical backups on-site and continuous backup off-site to a remote server. This remote server had recently been down, so we temporary disabled the backups there. And this is when things started to collapse: a week ago we had an unusual number of requests (~10 million more than an average day), and, as a result, our main server was overloaded. This led to a hardware error which resulted in this data loss.

What we are changing

We took further measures to prevent anything like this to ever happen again.

In the last days, we upgraded all of our infrastructures. We have added a new server. Now all the data is replicated continuously across two different machines. Moreover, now we have three different offsite daily backups.

In the next months, we will further improve the reliability of the service by adding more servers, increasing the frequency of the backups and increase the number of offline sites.

 

If you have any question, please contact us at support@femastudios.net, we’ll try to answer all of them (please be patient, since we received thousands of emails in the last few days). If you have lost any of your data, we may have a gift for you 🙂

We are deeply sorry for the inconvenience,

 

FEMA Team

Leave a Reply