Marketrends Status Updates Blog

Around 1:30am (pst) this morning we started experiencing some issues with the firewall that may have caused sites to not come up.  Around 2:15am (pst) we resolved the issue.  We’re looking now at what caused the issue to begin with.

We just received a large DDOS attack that slowed down the network, things appears to be back to normal now.  With the virtually unlimited bandwidth at the data center, things were still up and running, but really really slow.  Here’s a graph showing the spike:

Svrcp1 was down for about 30 minutes today.  It appears to be a memory issue, and has happened twice, but we’re looking into it further and will have it resolved.

This morning we received a distributed attack on a few servers…this caused network slowness on all servers.  A distributed attack is when the attack comes from multiple locations making it harder to block.  The majority of the attack was spam, but some was scripting.  Once we tracked down most of the sources and blocked them, service was returned to normal.  We also had to remove a few rogue scripts.

Scripting attacks (and rogue scripts) are when the attackers look for security weaknesses in your website, and if they find any, they use it and install their scripts that do various things, including sending spam, attacking others, etc.  Security weaknesses include security holes in third party software (wordpress, drupal, cms software, mailer software, database software, etc.), bad coding techniques and insecure folder permissions.  This is a good time to remind you to make sure your site is secure.  You do this by 1) make sure third party software is up-to-date and patched with any recent patches, 2) verify your web developer/coder is using secure coding techniques (ask them specifics), and 3) make sure folder permissions are secure.  When your account/server is first setup, it’s set with the default secure permissions of 755, anything other than that is less secure.

Most of the attacks we see come from Europe…this one was no different, they came from Sweden, Austria, Russia, Turkey, some also came from Brazil.

Thanks for your patience while we worked on blocking and cleaning up from the attacks.

svr12 went down this morning around 5am (pst), after troubleshooting we were able to get it back up around 6am (pst).  We are still working to determine the cause of the outage.

If you noticed any slowness in email delivery today, it was due to a Spam Attack we received this morning on a few servers.  It lasted for a few hours.  Between working to block the attack and it subsiding, it appears to be over with now.

Due to the amount of junk/spam that comes in, legitimate emails are delayed, thus the “slowness” issue.  Things are looking good now. 

Last night’s raid report showed a degraded hard drive in one of the arrays on svr12.  Tonight we replaced the degraded hard drive and the raid is currently rebuilding.  The server was down for for about 30 mins.

We just got hit with a major spam bomb and most of the shared hosted servers were affected.  The infrastructure and network handled it fine, but email delivery was slow and/or delayed because of it.  We’ve just finished blocking the attack and clearing out the spam.  Email delivery should be back to normal.  Let us know if not.    Have a great rest of your day!

Around 9:30pm (pst) tonight we received alerts that servers were down, due to the type of alerts we were quickly able to determine our firewall was down.  We called the data center and had our techs reboot the firewall.  It appears it has a power supply that is going bad.  We will be working on replacing it as well as getting a second, redundant firewall.  Outage time was about 20 mins.  Thanks for your patience.

On Tuesday 2/22/11, svr6 crashed for an unknown reason, however it appeared to be a power supply issue.  svr6 hard drives were moved to our emergency standby server (“builder”) while we investigated svr6 hardware.  On 2/23/11, while in the emergency “builder” machine, svr6 crashed again due to damaged data on the hard drives.  The hard drives were scaned and cleaned and the server brought back online.  Tonight 2/24/11, after replacing the power supply in the original svr6 hardware, the hard drives were moved back to their original svr6 box and the server brought back online.  svr6 repairs are now complete.