At this time, we are performing a disk replacement on cPanel server cp18. This machine’s onboard sensors have indicated possible bad sectors on the drive, and as part of our ongoing upgrades to improve reliability, we are replacing the suspect drive with a brand new one. Thank you for your patience during this procedure, which will take about 2 hours to complete.
Update 1:21pm: We apologize for the long delay; due to the bad sectors on the disk, filesystem checks and data copying is taking longer than expected. We are working to restore service to this machine as quickly as possible. Thanks for your patience.
Update 2:44pm: This server is now back online and the faulty disk has been replaced. Normally the process of drive replacement takes approximately 90 minutes, during which we perform filesystem checks, copy all data to a replacement drive, and physically replace the disk in the server. In this case, due to the number of bad sectors on the disk, the filesystem checks and copy process took far longer than expected. Now that the faulty drive has been replaced, we do not expect any further issues. Thanks again for your continued patience.
Earlier this evening, Red Hat released a new kernel version which includes important security fixes that apply to the version of Red Hat Enterprise Linux used in our hosting environment. Over the next few hours, we will be rebooting each of our cPanel and Plesk servers to install this update. We expect most reboots to take only a few minutes, and appreciate your patience during this important maintenance. Thank you.
Update 1:30AM: All updates are now complete.
Late last week, Red Hat released a new kernel version which includes important security fixes that apply to the version of Red Hat Enterprise Linux used in our hosting environment. Over the next 2 hours, we will be rebooting each of our Sphera hosting servers to install this update. We expect most reboots to take only a few minutes, and appreciate your patience during this important maintenance. Thank you.
Update 3:45pm: All reboots are now complete. Thank you!
Sphera server sd06 became unresponsive a few moments ago due to extremely high disk I/O causing many processes to timeout. We have rebooted this server and are performing a filesystem check now to verify that there are no inconsistencies in the disk; we expect the server to be back online shortly. Thank you for your patience.
Update 4:47pm: Filesystem checks are now complete and all websites will be back online within a few moments. Thank you!
Earlier this afternoon, Red Hat released a new kernel version which includes important security fixes that apply to the version of Red Hat Enterprise Linux used in our hosting environment. Over the next hour, we will be rebooting each of our cPanel and Plesk servers to install this update. We expect most reboots to take only a few minutes, and appreciate your patience during this important maintenance. Thank you.
Update 3:40pm: All servers are now back online or will be back online within a few moments. Thank you!
Over the next few hours we will be performing an upgrade of our power infrastructure which includes installation of A/B transfer switches and larger power distribution units. To complete this upgrade, we will need to shut down and move the power source for many servers in our cPanel environment. You may notice a short (expected to be less than 10 minute) interruption in service as servers are moved to the new power; we appreciate your patience during this upgrade.
Please note that this is our final scheduled power upgrade event; once the cPanel environment is complete, no further power adjustments should be needed.
Update 11:10am: All power updates are complete; there is one machine (cp32) currently running a filesystem check. Thanks for your continued patience.
Update 11:45am: All machines are back online and no further work is required. Thank you!
Over the next few hours we will be performing an upgrade of our power infrastructure which includes installation of A/B transfer switches and larger power distribution units. To complete this upgrade, we will need to shut down and move the power source for many servers in our Plesk environment. You may notice a short (expected to be less than 10 minute) interruption in service as servers are moved to the new power; we appreciate your patience during this upgrade.
Update 4:30pm: These upgrades are complete; servers ps17, ps18, and ps29 are in the process of running filesystem checks but should be back online shortly. All other servers are online. Thank you!
Update 4:55pm: ps29 is now back online; ps17 and ps18 are still running filesystem checks that are taking longer than expected. Thanks for your patience.
Update 5:15pm: ps17 and ps18 are done with their filesystem checks and are back online. This completes all planned upgrades to the Plesk environment. Thank you!
Our Email Defense provider is currently experiencing intermittent mail delivery issues. They are currently working on the issue and we are in contact with them to get any additional information. We will continue to post updates here as we get them. You can check the status by clicking the URL below.
http://www.postini.com/support/systemstatus_ext.php?system=7
Update 5:30pmThe Email Defense provider has updated the status page to show mail is working as it should. There still may be some delays as the mail continues to get flushed off of the server. We will continue to monitor the filtering and update this post if needed.
Thank you for your patience.
Update 10/16/09 8:30am Below is the full incident report made available by our Email Defense provider.
http://postini.com/webdocs/rel_notes/announce/reports/PIR_10_Oct_13_Mail.pdf
At this time, Sphera server sd41 is undergoing a drive replacement in response to alerts from our hardware monitoring system. The purpose of this replacement procedure is to prevent the possibility of data loss. We expect that this maintenance will take approximately 1 hour. Thank you for your patience while we continue to insure the reliability and stability of our hosting services.
Update 4:20pm: Data has been moved off the drive; we are performing a final filesystem check now and the server should be back online within a few minutes. Thank you for your patience.
Update 4:33pm: This work is now complete and all websites are starting back up now. Thanks!
We just had to perform a quick reboot of Sphera server sd02 to complete a database operation; this is now complete. Thank you!