Service outage, 14-16 March 2023
Oslo, 14 March 2023 (updated 16 March)
Starting on Monday, we have experienced severe issues with many of our email services. Below you can read more about what happened and the impact. We deeply apologize to all of our customers!
We know that your email account is important to you. It can contain personal data and critical information that you need access to in your day-to-day life. Therefore, we take our mission extremely serious, and our mission is to provide a flawless and always-available service for our customers. This week, we failed to deliver on this mission for many of our customers. We apologize for all of our customers for any inconvenience caused by our service being periodically unavailable!
During the night and early morning Tuesday, we started serving many new email accounts from the Italian email service Email.it. This increase in volume has been planned for a long time, and we have scaled up our servers and systems to cope with the new email traffic. Still, we learned on Tuesday that there was one storage server in our environment that was hit too hard with the volume increase this time. Normally, we have fully duplicated our servers, but this one was an unfortunate exception, and it was just partly replicated on another server. This led to a long list of errors, slow service and (from time to time) unavailable service during the day. The issue was first picked up around 9am CET today, and it took until 10pm to identify the root cause and implement a solution. After Tuesday, the service is available but still quite slow, in particular during peak hours. We have now installed many new machines, and we are moving a lot of data to these, mainly during night time. Therefore, we expect to see gradually improving performance.
The impact for our customers is two-fold:
- Partly unavailable service throughout the day on Tuesday
- Delay in email deliveries
- Slow service, in particular impacting our webmail
We believe we have stabilized the situation now, and the email service is up and running although being slow. Outgoing (sending) emails should be working as normal, and ingoing emails may sometimes be delivered with a delay.
You can find updated status under “Status” on our email services’ websites.
Please note that there has been no loss of emails / data.
We acknowledge that our email services are important for our customers, and our ambition remains to be up and running 24/7 with high performance. Therefore, we are doing everything we can to prevent a similar situation in the future.
So once again, please accept our deepest apologies! Should you have any question to this, please contact the support staff of your email service. You can find the contact details on the email service’s website.
The Fjordmail team