Improving Data Streaming, CDC, ETL, and Replication

by Caroline Maier

Hero Image
April 21, 2021 8:57am

Blog

Managing data streaming workloads, along with other large-scale data management tasks like Change Data Capture (CDC), Extract, Transfer, Load (ETL), and replication can be time and resource intensive activities for an IT organization. As a result, data managers have long sought solutions that can speed up the processes while requiring fewer person-hours for their execution. 3rd Party, online reviewing platform - IT Central Station - asked its members to dive into this issue, with users of Equalum discussing how the solution enables them to improve the way they handle data streaming workloads, CDC, and more.





Equalum Use Cases

Equalum is being put to use in a variety of use cases. The end to end platform helps Joel B., a director of enterprise architecture at a pharma/biotech company with more than 10,000 employees, with CDC and data streaming.

A managing director at a small consultancy uses Equalum to take legacy data, siloed data, and information in the enterprise and integrate it into a consolidated database. From there, the data is used for transactions and their BI dashboard.

Shirin K., a database administrator at an energy/utilities company, employs Equalum for replication. She said, “We have databases and [Microsoft] SQL Server. There is some data that needs to go to Oracle for the application team because the application is connected to Oracle Databases, but the back-end application is connected to SQL Server. Then we create workflows where SQL Server is the source, Oracle is the target, and all the tables in SQL Server replicate to Oracle.” In total, she handles 59 workflows for five databases.

A senior software engineer at an online retailer with 500+ employees uses it for "micro-batching of Kafka topics, which is like small, little bits of clickstream data. For almost all our use cases, the target of the data goes into our data warehousing solution, Snowflake. We also take large XML files from multiple parties, transform them, and put them into our Snowflake."







How Equalum Benefits these Organizations

Equalum users are finding that the solution delivers benefits on multiple fronts. For a software engineer specialist at an energy/utilities company whose organization writes data from one source to target using 5,000 websites, the effect of Equalum has been to improve data accuracy. As he explained, “It tells you where things are not matching. For example, bad dates were coming in and the target database would not accept this format. So Equalum will tell me if there is a problem over there. For error logging and error messaging, it is very efficient. It tells you what the problem is, e.g., your data type is not long enough on the targets. The logging is efficient, very detailed, and will also tell you where the problem is. You can fix the data, transform it, or change the target to accept that type of data. The accuracy is 100 percent. I have not seen any data anomalies.”

“We have been able to create features in some of our analytical environments which probably would have not been possible without Equalum,” said Joel B. He added that “the ability for Equalum to stream data in real time is very powerful. You can stream or you can batch with Equalum. Before using Equalum, our batch processes caused operation reports and our analytics environments to be in slight disagreement. Equalum has allowed us to close this gap. Data in different environments now match.”

The managing director at the small consultancy found the advantage of using Equalum came from its no-code UI, with Kafka and Spark fully managed in the platform engine. For him, it was a matter of human resources. He revealed that “the number of engineers that it takes to both develop something of this nature yourself, and then maintain it, is significant. It's not easy. Anybody who tries to deal with Kafka as well as Spark, and then tries to use the technologies, is quite blown away by how quick and easy it is to get moving. You can realize ROI much faster with Equalum.”

Shirin K. elected to work with Equalum because some of her applications have stopped supporting Oracle. However, even though some applications migrated to Microsoft SQL Server, others are still linked to Oracle data. These applications still need Oracle. To fix this, her team initially wrote Python scripts to pull data from SQL Server to Oracle. This required a great deal of maintenance and development. Now, since beginning to work with Equalum, she noted that “the data has been flowing very fast.” She also remarked, “Overall, Equalum has resulted in a lot of system performance improvements in our organization. It has helped us out by keeping the source and target synchronized at all times.”







Value Points

Equalum users discussed specific points of value in their reviews on IT Central Station. These included performance monitoring, which stood out to the energy company’s software engineer specialist. As he shared, “If something is failing, you can see it in there at the higher level. E.g., your sources are down, your agent is down, or your flow is not running. All those kinds of things are very visual. You just log into the tool and can see what is happening.” The solution helped his team work faster. He said, “It takes me an hour to transition the solution's knowledge to somebody else. It is really efficient that way. I haven't seen any complications.”

“For consultants, it's a dream tool,” said the managing director at the small consultancy. In his case, what mattered was ease of use. He noted that “it's super-simple to use. It has its own Excel-type language, so as long as you know how to use Excel, in terms of data transformation, you can use this tool. And we're talking about being able to do massive data integration and transformation. And that's not referring to the drag-and-drop capabilities, which are for people who have zero skills. Even for them, it's that easy.”

To learn more about what IT Central Station members think about Equalum, visit https://www.itcentralstation.com/products/equalum-reviews.

Ready to Get Started?

Experience Enterprise-Grade Data Ingestion at Infinite Speed.