- 注册
- 2002-01-16
- 消息
- 28,466
- 荣誉分数
- 5,529
- 声望点数
- 393
5月31日下午, CFC 服务器所在的数据中心由于电线短路, 致使机房配电室起火爆炸, 三面墙倒塌, 工作人员全部撤离. 现场状况十分严重,消防部门禁止使用备用电源, 防止再有火灾发生. 9000 多台服务器停电, 70万个网站无法访问. 经过2天两夜的抢修, 终于在今天下3点左右机房恢复供电.
我们对由于此次CFC停机所造成不便,表示深深的歉意.
技术性讨论:
为什么CFC没有启动备份方案?
CFC的WEB, Database, Mail, DNS, Backup服务器分别存放在4各不同的数据中心, 具有很好的灾难恢复方案. 这次停机的是WEB 和DATABase服务器. 但是灾难恢复的DNS修改需要12-48小时的Refresh的时间, 如果修复时间大于48小时,我们才能启动备份服务器. 否则由于DNS更新延迟造成的服务器无法访问的时间会更长. 我们一直在等待数据中心的具体修复时间,数据中心昨天晚上才给出具体修复时间,那时距离供电时间只有20个小时了. 所以我们没有启动备份服务器.
下面是火灾当天数据中心发出的通知.
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
2008.5.31
CFC 的临时域名,如果下次遇到同样问题,大家可以访问这个地址:
www.comefromchina.org
我们对由于此次CFC停机所造成不便,表示深深的歉意.
技术性讨论:
为什么CFC没有启动备份方案?
CFC的WEB, Database, Mail, DNS, Backup服务器分别存放在4各不同的数据中心, 具有很好的灾难恢复方案. 这次停机的是WEB 和DATABase服务器. 但是灾难恢复的DNS修改需要12-48小时的Refresh的时间, 如果修复时间大于48小时,我们才能启动备份服务器. 否则由于DNS更新延迟造成的服务器无法访问的时间会更长. 我们一直在等待数据中心的具体修复时间,数据中心昨天晚上才给出具体修复时间,那时距离供电时间只有20个小时了. 所以我们没有启动备份服务器.
下面是火灾当天数据中心发出的通知.
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
2008.5.31
CFC 的临时域名,如果下次遇到同样问题,大家可以访问这个地址:
www.comefromchina.org
后来问了好多朋友才知道大家都上不了网～！
