Object Storage: Why Organizations Are Making the Switch

More organizations than ever are turning to object storage to help store, manage, analyze and collaborate on their fast-growing stores of unstructured data.

At its core, object storage is a non-hierarchical, flat namespace that is addressed through a simple key-style addressing scheme. At a simple level, users can read, write or delete to the object if they have the "key." Object storage is often used in situations where organizations need to be able to access, share and identify characteristics of billions of things. Ideal for unstructured data such as documents, images, log files and media, it uses metadata to describe and help identify what is in each object.

"An object is just a 'blob' of data that doesn't change. Object stores are meant for storing massive quantities of blobs that applications can access from any operating system or device, from anywhere around the globe," said AB Periasamy, CEO of MinIO, an open-source cloud storage software vendor.

Rob Lee, chief architect at Pure Storage, used a parking lot analogy to explain how object storage differs from file and block storage.

"Block storage is like parking cars in a massive parking lot. It's very efficient, but it's not very flexible or easy to find your car," he said. "File storage is more like a multi-story parking garage where you have to remember that you parked in section C4. There is structure, so you know where to find your car, but it is limited in flexibility. If more people are parking their cars in certain areas of the garage, it's not like the parking garage can automatically expand and balance those cars and store them elsewhere. But it's easier for you to share data."

Object storage is more like valet parking, he added. "You give your car to the valet with the keys, and they give you a ticket. You have no idea where they are parking the car, and you don't care. All you know is that when you want your car back, you can bring that ticket to the valet." This model creates more flexibility; if more people are parking their cars and the valets have to move the cars around, they can do so.

Flexibility is key in today's business world. For example, if Facebook notices that a particular video is very popular, it can store an extra copy. If it notices increased demand on the East Coast, it can serve the video via its East Coast hub, and it could do the same with any of its other hubs around the world.

In addition to flexibility, object storage tends to be particularly available and durable.

"We're used to talking about enterprise IT equipment being available to five or six nines, but with object storage, we're often talking about 15 or 16 nines for scale-out, shared nothing clusters," said Duncan Moore, director of object storage at NetApp. "And you can lose networks, entire data centers or nodes, and you never lose the data because the data is protected through other distributed protection mechanisms like erasure coding."

These characteristics make object storage attractive to all types of organizations, for all types of use cases. That's why it's on the upswing. According to an IHS Markit survey of IT professionals, while current storage capacity comprises 38% file, 35% object and 27% block, those numbers are expected to shift in favor of more object storage over the next several years. The survey found that more than half of the respondents plan to increase their investment in object storage, especially for big data repositories and video distribution.

Why Now?

There are several reasons why, despite the fact that object storage has been around for more than a decade, it's experiencing significant growth today. One obvious reason is the exponential and continued growth of unstructured data—not only traditional types, but social media data, search engine queries, geographic data, and data from sensors and other smart devices.

Another major driver is the growth of cloud computing. While object storage works well in on-premises environments, there is no doubt that much of its use is in the cloud.

Here's why: The cloud is infinitely scalable, and object storage requires virtually unlimited scalability. In addition, most applications today are built for use in the cloud, and all modern, cloud-native applications rely on object storage. Much of the credit for that goes to Amazon's S3, which has become the de facto API for object storage in the cloud. The non-proprietary API is used by virtually all modern applications today.

The increased focus on collaboration also is a good fit for object storage.

"With object storage, you have a massive shared repository where people involved with different parts of a workflow using different applications can have access to the same data," Moore said. "It doesn't matter whether it's a media company where people need to collaborate on visual effects, raw camera data and outtakes, or a life sciences company where people are collaborating on data from genetic sequences."

While object storage isn't ideal for every situation—file or block storage is a better fit for transactional workloads, for example—there are plenty of use cases. Traditionally, it has been used for backup and archive, as well as for content storage and management. Cloud services, from backup as a service to Dropbox-type applications for file sharing, also naturally gravitate to public cloud object stores. Enterprises also are using object storage on-premises, especially for archiving large amounts of data.

While those use cases have been around for some time, the ways object storage is being used today have broadened greatly. It is being used in many vertical markets, from financial services to media and healthcare.

One fast-growing area for object storage centers around Internet of Things (IoT) and edge devices.

"Think about kiosks in airports, or sports stadiums that have video delivery to their seats and video surveillance, or cruise ships, which are basically small floating data centers," said Paul Speciale, chief product officer at Scality. "All of that is data on the edge and needs to be managed as part of a global namespace. What if you have 100,000 users in a stadium? That's a lot of data that needs to be collected or delivered on the endpoints. Or medical scanners from a hospital network, which create a lot of video and image data."

File and Block Storage Are Still Relevant

Even as your organization moves workloads to storage, you'll probably still need file and block storage to some extent. Not only are these other storage types better for structured data, but applications that use object storage sometimes rely on some file and/or block storage as well. And as organizations move away from monolithic applications to a larger number of microservices, IT organizations will need to provide much more agile and flexible block, file and object storage to those applications.

"Modern technology is quite shareable and interoperable, so you need to be able to share data between files and objects," Speciale said. "Take media: The applications in that world tend to create images and videos and store them as files, but they may want to serve them out through cloud and object protocols. You may want to store as a file and read as an object, or write as an object and read as a file."

To enable all of these types of storage to coexist within an organization, IT professionals need to understand at least the basics of all three. While there are many similarities in terms of management and monitoring, there are significant differences. For example, managing 20 petabytes of block storage inside an enterprise may take several people, but managing the same amount of enterprise object storage would require only a fraction of a person. The same goes for capacity planning; instead of worrying about adding a disk drive for block or file storage, capacity planning for object storage means simply provisioning a new server to the storage pool.

The way data is controlled and accessed is another differentiator. With object storage, both of these tasks are merged through a common API. For example, an IT administrator would provision storage by creating an S3 container. In that way, storage administration becomes somewhat merged with the role of application API developers.

While object storage has been around for some time, use cases and the technologies surrounding it continue to change. Many of these new use cases, which revolve around technologies like IoT, require massive concurrency, very tiny objects and high performance. Moore points to his company's introduction of an all-flash object storage appliance last year. "If you went back in time and told someone you'd be building something like that, they would think you were crazy because who needs that kind of performance? But today they do."