Twitter in 'Combat Mode' As Attacks Continue

twitter-cat

One of the famous Twitter error graphics (pre "Fail Whale"). Twitter has been enhancing its defenses against attacks.

Twitter has been the target of additional denial of service attacks in the wake of a major outage on Aug. 6, but enhanced defenses implemented by Twitter and NTT America have reduced their impact.

"We've been continually seeing attacks, but I think the public has not noticed because we've been mitigating them," said Kazuhiro Gomi, Chief Operating Officer of NTT America, which hosts Twitter's infrastructure. "We are definitely in combat mode, but handling it."

Twitter was badly hobbled by the Aug. 6 attack, which also targeted Facebook and several other social media sites. In a recent interview, NTT America addressed the steps taken to harden Twitter's defenses against these attacks, and addressed some criticisms of Twitter's performance.

The Aug. 6 attack was a distributed denial of service (DDoS), in which "bot networks" of thousands of compromised computers are used to attack a site and overload its infrastructure - essentially clogging the pipes with too many requests at once. Network tools can be used to filter incoming traffic and deflect suspicious packets while forwarding "good" requests.

Defense Tools Turned On
These DDoS defense tools were available from NTT America, but Twitter did not have them in place at the time of the Aug. 6 attack. Once they were turned on, Twitter gradually improved its perfromance and returned to service. NTT America has since put some new defense mechanisms in place to address the unusual scope of recent attacks. These measures are not a customized solution for Twitter, but a broader hardening to protect all of NTT's customers, the company said.

"We have fortified our infrastructure so we can better handle these attacks," said Gomi. "We saw a couple of issues we needed to address, I'll admit, and I believe Twitter will admit that they have identified some issues on their side as well."

Independent analyses of Twitter's infrastructure have suggested that its load balancers weren't optimized to defend against a large-scale DDoS. The company says the attack has been a learning experience that has helped it refine its architecture.  

"What we learned from this is that you've got to tune your systems to be able handle this scale of assault," said Twitter co-founder Biz Stone in an interview with Tavis Smiley. "We learned a lot from it. We worked behind the scenes with folks from Google and other companies to figure out how to stop the attacks, and better deal with them in the future."

Performance Issues Persist
It's important to note that NTT America and Twitter are not claiming victory, only that these new measures have helped improve the site's performance during DDoS attacks. Since the Aug. 6 attacks, Twitter has had downtime of 25 minutes or longer on at least four occasions, according to uptime tracking at Pingdom.

Some analysts have noted that Facebook appeared to manage the attack more effectively,  suffering slowdowns but not the extended outages seen by Twitter users. "Facebook has a longer history with these attacks (than Twitter), and their infrastructure is much bigger," said Gomi.

NTT America also disputed the notion that Twitter's decision to use a single provider for its connectivity (a practice known as single-homing) was a factor in its Aug. 6 downtime. "There's a misconception that Twitter was more vulnerable," sadi Gomi. "In reality, it's easier and quicker to provide the filter with one provider. If you are in a multi-homing environment, you have to put the same kind of filter in all of those network providers. If you have a single homed environment, just one network provider can provide the filtering."