by 23:00 - 08:00 BST Tuesday 10th - Wed 11th July
For anyone interested, had the following message from our server company:
Yesterday we upgraded the capacity on our server storage system. Unfortunately this has resulted in an unexpected degradation of performance while the storage RAID controller re-stripes data over the new disks.
To ensure data integrity the storage has essentially gone into a type of “safe mode”. It has turned off write buffering and it's SSD caching, both of which degrade performance, especially on disk writes. The time it takes to re-stripe the data is dependant on how many other things are competing to access the data at the same time.
We have been talking to the manufacturers and working hard to try and speed up the the re-striping. Unfortunately, no measures we've been able to take while the system is online have helped. We have therefore decided to take the server platform offline this evening in an attempt to get this resolved as quickly as possible. This will allow the storage system to re-stripe whilst no data is still being written to the disks and to use the full system resources. We hope this will speed up the process but at this stage do not know by how much.
We will be performing this emergency maintenance between:
23:00 - 08:00 BST Tuesday 10th - Wed 11th July
All services will experience an outage during this window. We're really sorry for any inconvenience caused and will follow up with more detail about why this happened and what we're going to do to make sure it doesn't happen again.
