The first stage of every good pipeline is to perform data integration. With the increasing pace of change and the need for up to date analytics the need to integrate that data in near real time is growing. With the improvements and increased variety of options for streaming data engines and improved tools for change data capture it is possible for data teams to make that goal a reality. However, despite all of the tools and managed distributions of those streaming engines it is still a challenge to build a robust and reliable pipeline for streaming data integration, especially if you need to expose those capabilities to non-engineers.
In this episode Ido Friedman, CTO of Equalum, explains how Equalum built its no-code platform to make integration of streaming data and change data capture feeds easier to manage. He discusses the challenges that are inherent in the current state of CDC technologies, how they have architected their system to integrate well with existing data platforms, and how to build an appropriate level of abstraction for such a complex problem domain. If you are struggling with streaming data integration and change data capture then this interview is definitely worth a listen.
ABOUT THE DATA ENGINEERING PODCAST
The Data Engineering Podcast tackles a new approach to data management every week. Each new episode provides useful and informative insights into the projects, platforms, and practices that data engineers, team leaders, and data scientists need to know about to learn and grow in their career.
By digging deep into the technical platforms, edge cases, and lessons learned, the host ensures that you get the most value for your time spent listening. By following along you will learn more than you thought possible about diverse and important topics in the area of scaling processing pipelines, deploying databases, maintaining your data lake, and establishing proper governance practices. Exploring data management at every scale and across diverse industries brings you the perspective that you need to excel at your job.