2006.11.28 11:27 AM

CrystalTech Smashed Yesterday

On Black Monday, no less. Everything at CrystalTech disappeared yesterday around 9AM and didn't fully come back (counting delayed email deliveries from external servers on ever-longer retry delays) until this morning. Here's a forum thread that captures the chaos. And this is the summary report from support (sent yesterday evening, but received this morning):

Dear CrystalTech Customer,

We wanted to provide you with some more detail surrounding the server issues that occurred earlier today.

This morning, we experienced a large Distributed Denial of Service attack (DDoS) on our main network. The attack was against the shared segment of our network and our Domain Name Servers (DNS) specifically. CrystalTech blocks this kind of malicious activity on a regular basis as our Network and Server Operations Teams are well equipped and trained to keep it from affecting the services we provide. However, this was a large scale and targeted DDoS attack on our DNS which caused the 3 – 4 hour window of intermittent availability you may have experienced.

Given the scope of this issue, it was resolved as efficiently as possible. Our Server and Network teams reduced traffic into our network by blocking thousands of offending IP addresses which were causing websites, email and shared databases to become unavailable. Then, traffic to our DNS servers was rerouted to mitigate the attack and to reduce any further activity.

While we take every effort to resolve network issues as quickly as possible, this was a very different and unique circumstance which required more time to resolve. While we are not satisfied with the length of time our shared network was effected, it easily could have taken longer to contain if not for the skill set of our Support, Network and Server Teams and the equipment and technology utilized within our current infrastructure.

Should you notice any further issues or discrepancies with how the service is performing, please let us know right away. Thank you for your patience while this issue was resolved, and we sincerely apologize for any inconvenience this may have caused.

CrystalTech
www.crystaltech.com

All and all I'd say it could have been worse. But as a lot of folks point out in that forum thread, it could have been better. Minimally there should have been a better communication plan. It was impossible to get through to CrystalTech for any info.

Anyhow, what it means for me is that if you were here yesterday trying to access various items linked from this site to ewbi.com, like the Sparklines source, you didn't get it. And, if you sent me an email, I didn't get it. Sorry. Looks like things are good now, though, so give it another try and let me know if you have any problems using the email address at right.


Comments

alles gut

infusion | 2006.11.29 10:22 AM


TrackBack

TrackBack URL:  http://www.typepad.com/services/trackback/6a00d8341c7bd453ef00d834ca21cb53ef

Listed below are links to weblogs that reference CrystalTech Smashed Yesterday: