There was a first 48 min outage this morning around 11am. It’s down again at 3pm. So if you are experiencing issues with watching videos hosted at vpod.tv or trying to use the back office, our machines are down, among I guess all the clients there.
It’s really not the first time Redbus is having electrical issues in Paris. They’ve had them in London as well. I can only qualify this as professional incompetence. They are supposed to be a first rate data center with multiple layers of backup equipment and processes. This is one too many. We’re moving out ASAP.
Update: my friend Tariq Krim of netvibes.com is down too, and so are the 200,000 blogs at over-blog.com… way to go Redbus!
Update2: 20:48, our French provider iKoula for our test servers, hosted at Redbus has just sent us an update:
“Dear customer,
Redbus Interhouse, Data Center where are located our equipments including your hosting solution, faced today a major power crisis.
- 10h55 : A power failure caused by EDF touched all the West of Paris including Redbus Interhouse building in Courbevoie. The blackout is global.
- 10h15 : The Ikoula team on site is reinforced. Our technical staff is at 100% ready to face any situation
- 11h54 : Power is back. All Ikoula equipments are immediately up. Everything works at 100%
Redbus hsa not yet explained the abnormal outage of one hour.
- 14h32 : 2nd global power failure
- 15h16 : Power is back but the situation is not stable
Consequences of these incidents:
- The building is working on two generators (apprently the 3rd is out of order)
- All the UPS are empty
- The main dijonctor (EDF) is broken, thus, the switch to EDF power is not possible
- The switch to EDF network will involve a new outage
- The situation is highly unstable
Ikoula took the decision not to switch back on its equipments. Machines do not stand multiple brutal electrical cut. The damage risk is high. A broken hard drive can mean the loss of data and hours of work for you to get the service back online at 100%.
We took the decision to secure your data, your work rather than getting back online a few minutes before.
Update 3: monday 21:30. This is an email I got earlier today. it’s already been 1,5 hours. Not yet back online.
“
Dear Customers,
Redbus is planning a maintenance opreation on Redbus Interhouse which will consist in :
- A corrective maintenance of the Main incoming circuit breaker
- The Reconnection of the UPS to the generators
- And the Connection of Customer load to the UPS
In order to prevent any damage on your servers due to accidental power
restarts : We ask you to turn off all your equipments before 7:30 PM.
At 7:45 PM Ikoula technical staff will cut off all electrical connexions
between our equipment and Rebus.
The main electrical supply will be turn back when the Redbus'maintenance
will be over.
As we said previously this operation is made to prevent any damage to your equipment and your datas.
Regards, ”
We were right.
- 17h01 : 3rd global power failure
- 17h05 : Power is back
- 18h00 : Redbus inform us that the intervention on the main dijonctor will not take place tonight. But, the situation is more stable
- 18h30 : We switch on all our equipements, including your hosting solution.
All the Ikoula team is still at work to help you getting back your services if needed.
WARNING:
The intervention on the main dijonctor will take place tomorrow. In order to preserve the machines, your data and your work, we will turn off all our equipments before it. We will warn you before of course. Since then, we highly recommand that you back up all your data.
We apologize for the inconvenience and thank you for your confidence.
We will get back to you as soon as we get news. “
update 4: tuesday 3:20am – servers are back up. pfff…
