We are giving our analysis of what happened a couple of weeks ago regarding the fire in the datacenter in Strasbourg. We want to give you some insight on our perspective of what went right, what didn't go so well, and most importantly on what we learned.
First of all: a big thanks goes to YOU, our clients! We were very grateful to see that our clients were understanding and patient of the situation. We know that you rely on us to provide you with the best service and that your success is closely connected with your website, consequently any minute of downtime causes stress and discomfort on your side. Nonetheless you allowed us to work intensively without interruptions, to get the sites up and running as fast as possible. I we not say it enough, thank you!
Let's now address the nature of this incident: what happened in Strasbourg with our server provider is of exceptional nature and unprecedented; as stated by multiple experts, it will have an impact on the datacenter industry. It is the absolutely worst that can happen to a datacenter provider. The news about the fire went global in hundreds of news sites, as well as national in where Ramsalt was mentioned in half a dozen of them. After the fire OVH ordered 15 000 new servers. The server assembly line capacity globally has tripled to attempt to meet the high demand after the fire.
To put this in context, the estimated impact is 3,6 million websites as well as email and similar services for smaller and larger organizations were affected. Also governmental sites from several countries were affected by the fire. One datacenter was burnt down completely, another one partly damaged, and then another two were shut down as a safety measure due to damaged power lines. Altogether causing over half of Ramsalt’s production and development servers inaccessible, bringing with it also most of the hot backups.
Here is a rough timeline of events from our perspective, starting from the night of 10 Mar 2021 (all times are in expressed in Europe/Oslo timezone):
From the next day (11 Mar) we continued working to get hold of the remaining sites which required DNS changes and except a couple of outliers we have all sites back within 48h since the fire did spread in our datacenter.
This was a large-scale event that destroyed the production and development data, servers and most hot-backups for the majority of our clients. We are in general pleased that our Disaster Recovery Plan (DRP) worked and we managed to restore our sites. The communication about our progress went mainly through status.ramsalt.com for frequent updates and with mail when we had bigger news.
As we mentioned, we are quite pleased with how we came out of a fire that took out four datacenters. What follows is what we believe to be the highlights.
Considering this a disaster of unprecedented size, there are also things which we would rather not think about, but we see it as our duty to be transparent with our clients who put a lot of trust in us and most of all we can learn from our mistakes by analyzing our mistakes. What follows is the list of what we believe to be the worst defeats we encountered and which we will work to improve.
In the previous section we highlighted what went right and what went wrong, but how do we go from here?
After the incident we had several internal discussions about what happened and most importantly where and how we could improve, what follows here is the list of the most important areas we will focus to improve on.
There are high availability solutions you can choose that would keep your services and websites running without interruption, even if a datacenter is put out of a fire. By having a load balancer and another mirrored server in a different location and datacenter. High availability clusters are more costly, but might be worth the investment if you can’t afford the risk of not having your services available. Get in touch with us for more information.
We would like to thank all our patient clients that believe in our services. We do not take lightly on this, and will continue to improve our services to ensure better stability and routines to keep your sites and services up and running.
If you have any questions please do not hesitate to contact us.
Cookies have played a significant role in the digital landscape for quite some time, offering various benefits. However, as online information becomes more abundant, the search giant Google is taking a decisive step by phasing out third-party cookies in its Chrome browser.
Drupal 9 reached its end of life (EOL) on November 1, prompting the need for an upgrade plan. Upgrading from Drupal 9 to Drupal 10 is expected to be smoother, thanks to smaller, incremental updates and faster iterations. Drupal recommends fully updating your Drupal 9 site with the latest module and theme versions before transitioning to Drupal 10.
Drupal has been a robust choice for building and maintaining websites for decades. Like all other CMS systems, technological advancements and security requirements continually evolve at rapid speed and older software versions will be taken off the market.
The fastest way to resolve a problem is to write a good support ticket. Learn how to write to get your problem fixed as soon as possible.