Data Integration Tools
Data integration tools overview
Data integration is the process of combining data from different sources with the goal of providing a unified view of the combined data. This lets you query and manipulate all of your data from a single interface, perform analytics, and generate statistics.
Of course, your data sources will not integrate themselves. For that, you’ll need to use a data integration tool or platform, preferably one designed to handle your specific data needs. These tools often include functionality aimed at cleansing, transforming, and mapping the data, as well as monitoring the integration flow itself (error handling, reporting, etc.).
With data coming from local, software-based “batch” sources or from web-based streaming sources, data integration is a critical component of a larger data analytics strategy.
On-premise data integration tools
These tools excel at integrating data from various on-premise or local data sources. Typically these tools are installed in the local network or private cloud and include optimized native connectors for batch loading from various common data sources. On-premise data sources tend to include larger or legacy databases.
Here’s a list of common on-premise data integration tools:
- Centerprise Data Integrator
- IBM InfoSphere
- Informatica PowerCenter
- Microsoft SQL
- Oracle Data Service Integrator
- Talend Data Integration
- webMethods
Open source data integration tools
If you have the expertise in house, you might want to consider open source solutions to your data integration needs. Open source can be a good option if you’re trying to avoid using proprietary, potentially expensive enterprise solutions or if you want to have complete control over your data in-house. Keep in mind, though, that internal open source projects often have hidden or unexpected costs (servers/hardware, network throughput, training, etc.). And, depending on your situation, you may also have to handle data security and privacy compliance.
Here’s a list of common open source data integration tools:
- CloverETL
- Karma
- Myddleware
- Pentaho
- Pimcore
- Skool
- Talend Open Studio
Cloud-based data integration tools
Many cloud-based tools are integration platforms as a services (iPaaS) that help integrate data from various sources, often (but not only) into a cloud-based data warehouse. These services are usually “born of the web” and designed to handle newer, web-based streaming data sources as well as the common databases. As new web-based data sources tend to come online frequently, a key component of cloud-based services is the ability to integrate them quickly, sometimes via APIs/SDKs/Webhooks.
Here’s a list of some of the more common cloud-based data integration services and tools:
- Dell Boomi AtomSphere
- Informatica Cloud Data Integration
- Jitterbit
- MuleSoft Anypoint Platform
- Oracle Integration Cloud Service
- Salesforce Platform: Salesforce Connect
- SnapLogic
- Talend Cloud Integration
How to select the right data integration tool
That’s a long list of candidates, and there are other, smaller solutions not present. What’s the best way to select the right data integration tool to use?
Consider these factors in your decision:
- Enterprise size — as your data needs grow, so too will the complexity of your data integration strategy. Know that there are more and more streams and web-based data sources being created every day — selecting a tool or service that can grow to accommodate your expanding data is paramount.
- New data sources and throughput — remember, you’ll need more than just additional storage. You’ll need a solution that can connect to the various new streaming and web-based data sources. Some legacy/on-prem tools are not able to handle streaming data sources, or do so sub-optimally.
- Your integration use-case — a fully on-premise solution can be the right call, if you’re sure that your plans for data analysis won’t involve a full-scale move to the cloud and that you have data growth in check. There are also open source/”roll your own” approaches, though take care before attempting those: you’ll want to be sure you have the proper expertise and resources in-house.
- Security and compliance — make sure that your solution (or in-house team) has the expertise and resources to ensure you’re covered when it comes to security/privacy and compliance.
