Data Center Traffic Highlights WAN Optimization

Haseeb Budhani is Vice President, Products, Infineta.

Haseeb Budhani

It should come as no surprise to anyone that enterprises are continuing to push increasingly more traffic between their data centers. The drivers for this huge uptake in inter-data center WAN utilization range from the need to protect data (e.g. through replication and backup), to making data available ubiquitously (e.g. through active-active storage or live migration).

Regardless of the driver, one thing is certain: enterprises either need to upgrade their WANs (too costly and laborious) or deploy a WAN Optimization solution to mitigate the WAN bandwidth upgrade. Particularly when data centers are more than a few hundred miles apart, it is prudent to first evaluate WAN Optimization technologies to understand how these technologies can effectively reduce the footprint of traffic flowing between them.

Back to the beginning

Technically, the first real WAN Optimization solutions for inter-data center traffic were productized by storage and networking vendors, who built compression features into their various product lines. Compression is a potent reduction technique that can work on relatively small blocks of data (say a 4KB chunk) within a data stream and recognize repetitive patterns ranging in size from a few bytes to tens of kilobytes. In the majority of cases, using compression can result in about 25-50% reduction in traffic footprint on inter-data center WANs.

The next incarnation of WAN Optimization occurred when a whole pack of companies started selling dedicated appliances, labeled WAN Optimization Controllers (WOCs), to reduce WAN traffic footprints. WOCs introduced the notion of in-motion data deduplication, a concept akin to caching, whereby repetitive patterns could be identified and minimized across gigabytes of data, both within and across data streams. Much like caching engines, a cache (called a ‘dictionary’ by many in the WOC space) is maintained where data patterns are stored. Deduplication works extremely well in cases where large blocks of data are being transmitted repetitively over the WAN; in these cases, WOCs can do an extremely good job of reducing traffic footprints.

Such scenarios are prevalent over branch WANs, where, as an example, it is not uncommon for the same email attachment to be sent to multiple employees in a remote location. In this scenario, once the first email with the attachment has traversed the WAN, the WOCs will correctly identify all subsequent transmissions of the attachment and replace them with a reference that can be used to recreate the attached in the remote location.

Bridging Between Facilities

When it comes to inter-data center WANs, however, applications (carrying out, say, replication or backup) tend to only send changed blocks of data to the remote data center. In these scenarios, deduplication implementations must be ready to look for patterns that are smaller in size than the blocks being used by the applications. Consider a storage array that addresses data in 4KB blocks on disk, and only sends changed blocks to the remote storage array. If the WOCs in use only look for patterns that are 4KB or larger, the WOC will find NO repetitions. Even if the WOC works at 2KB granularity, the best-case scenario for reduction is 50%, which may be too low a reduction ratio to justify a WOC purchase.

WOC vendors are quick to claim that their deduplication engines make use of 32-byte chunks. This is probably true when WOCs are processing traffic at sub-10Mbps rates. However, when data is traversing the WAN at even a few hundred Mbps, the sheer amount of effort required to identify repetitions across gigabytes or terabytes of dictionaries forces WOCs to work on increasingly larger chunks. It is not uncommon to see 4KB chunks in use when WOCs are processing upwards of 500Mbps of traffic.

Some vendors subsequently packaged compression into WOCs. The idea was that deduplication would find the larger repetitions within and across streams, while compression would find the smaller repetitions within streams that deduplication happened to miss. This model certainly makes sense, but the critical question is whether deduplication is providing any meaningful benefit by itself. Customers should be careful, lest they find themselves paying a lot of money for a compression engine that they could just as easily have acquired directly from their storage or networking vendor, almost always for a lot less money.

Reduction Utopia

Deduplication for inter-data center WAN traffic must be entirely speed independent. Regardless of whether the traffic is running at 10Mbps or 10Gbps, the deduplication efficacy must remain unchanged. In talking to WAN Optimization vendors, customers should ask how the vendor in question is guaranteeing this. Ideally, the deduplication implementation must be able to find repetitions all the way down to sub-10 bytes within and across streams for optimal reduction benefits.

Compression is a key ingredient to the ideal reduction system. Keeping the non-commutative nature of deduplication and compression in mind, WAN Optimization systems should first deduplicate incoming traffic and then compress it. Compression is equally useful both when the deduplication engine is identifying a significant number of repetitions, and when the deduplication engine finds itself identifying and learning new patterns. In the former case, compression can opportunistically process multiple small packets exiting the deduplication engine in one fell swoop after each has been deduplicated with very high efficiency. In the latter case, compression can reduce the overhead of informing the remote WAN Optimization solution of a new pattern that needs to be “learned.”

If your organization is struggling with growing inter-data center WAN bandwidth demands, WAN Optimization is certainly the right place to look. The key is to select a vendor that is truly building systems for the speeds and traffic types prevalent between data centers. Don’t settle for a solution that was originally designed for a wholly different problem set (e.g. the branch WAN) and has been jerry-rigged to appear to solve the inter-data center WAN problem.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.