The Hybrid Cloud Integration Challenge

Eddie Cole, vice president of engineering, Scribe Software.

While moving to the cloud is becoming a common business decision, many organizations are still trying to figure out the best path to meet their business needs and IT infrastructure goals. The resulting discussions often add up to a hybrid approach where organizations tap the value of both public and private clouds for different services, which can lead to complex cloud silos that trap important data.

When moving to a hybrid cloud, integration professionals must take four new considerations into account including the need to understand SaaS data policies and API limitations, the implementation of new strategies for moving data, mitigating identity and licensing challenges, and planning for new security risks in the cloud.

No Backdoor to SaaS Applications

Before beginning a hybrid cloud integration initiative, you will need to understand the data policies and API limitations of each SaaS vendor that you work with. No two are created equal, but two new challenges to plan around are API rate limiting and a lack of full CRUD operations on all entities.

Unlike on-premise solutions, most SaaS applications implement some form of rate limiting, either through well-defined rules and policies or through poor performance exhibited under load. Your best bet for both cases is to discover the limitations as soon as possible and plan accordingly. The first case is usually easy to research, while the second can only be discovered by exercising the API.

The other new big blocker is that not all APIs offer full CRUD operations on all of their entities. In fact, more often than not the API will only expose a subset of the data model you previously had access to from the backend. Even if the API exposes the entity you are looking for and the operation you want, you still need to make sure the entity contains all of the properties you need to integrate. Very often we find APIs that expose entities missing properties that are available in the application’s UI. These two issues are especially true for user-defined entities and properties. Many APIs either do not expose these custom types, or do so through an entirely different set of calls. Always be sure to know what data entities are available and what operations you can perform on them before you commit to any hybrid integration project.

Limitations to Moving Data

Especially when it comes to large data migrations, you will likely need to come up with new strategies for moving on-premise data to public/private cloud stores or SaaS applications. There are typically three strategies to choose from.

Parallel Processing. This is usually the simplest strategy. As long as you are not running into a rate limiting issue, very often you can break large data migrations into separate processes that can be run in parallel. The trick is to understand how the entities you are migrating are related, and how your data itself can be segregated.

Incremental Loading. Incremental loading is all about the “slow and steady wins the race” approach. Usually the time available for a given data migration is based on how long you can have users off the system. By taking an incremental loading approach, data is moved based on change date from oldest to newest so that the two systems are eventually synced up to the point that the down time for cut over is only based on the data change rate, not the size of your data.

External Key Cross Referencing. This last strategy is perhaps the best way to optimize regardless of your approach. The concept involves building a cross reference of keys between the systems on a fast local data store, meaning that the integration never needs to perform costly searches on the slow and/or rate limited SaaS application to relate entities.

Identity and Ownership Challenges

In the on-premise world, you could simply add additional users as part of your migrations or integrations, so that the proper data could always be related to the proper principles on each side. Unfortunately SaaS applications often pose licensing issues around adding users. My favorite solution is to add all the real SaaS users first, and then build an external cross reference as mentioned above, which you can use to map your users between the systems. This approach preforms very well and also supports mapping many users into a single SaaS user if needed – not that I think you might be looking to skimp on a few seats.

Security is Top Concern

Integration by its very nature is security anti-pattern, as its ultimate goal is to expose secure data to other systems for consumption. Of course, any well-designed integration approach is only going to expose data to trusted systems in a secure manner. However, as data is moved to the public cloud where there is more risk, integration is necessarily faced with more obstacles to reach that secure data and to keep it secure once it’s in process.

Many SaaS applications allow both credential checking and IP whitelisting. If available, use both. If possible, avoid storing credentials and if nothing else, store them encrypted. For encryption of either credentials or any data you might put across the wire, remember that the encryption gets old fast – always keep your approach up-to-date.

The hybrid cloud offers enormous advantages for companies seeking the best of private and public clouds for scalability, price, control and flexibility. While companies with sufficient manpower and expertise can follow the steps above to successfully navigate these new integration challenges, any business can achieve their hybrid cloud goals by enlisting systems integrators and/or leveraging third party integration tools specializing in cloud integration. Either approach works, but be sure to plan for the new challenges and realistically assess your company’s available resources and timeline before embarking on a journey to the hybrid cloud.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission processfor information on participating. View previously published Industry Perspectives in our Knowledge Library.