With the emphasis on moving to a Streaming Data Architecture, it might feel like a rip and strip of your current architecture is the only way to ensure success, but that’s hardly the case. Trying to replace complex, existing systems is not the first move, much less is it something that your team can culturally adopt. Bottom line, change is hard, so smaller steps can often lead to quicker adoption and long term gains.
So often cultural adoption is what keeps organizations from embarking on a streaming journey. The Data Giants (or sometimes dinosaurs) are reticent to break from what is known and functional. At its core, new technology could be a threat to their understanding and even job security. Proceed knowing that as your Architecture becomes streamlined, and the lean on the tech team lightens, there will be new room for innovation, collaboration and future projects where experience and enthusiasm from all players can be fully utilized.
Take a good look at the business and identify where you see bottlenecks that, with real-time data instead of batch, could be a quick win and even generate new revenue streams. Give your team and business leaders a chance to see true value in what streaming can achieve. Once you demonstrate success and the viability of the new revenue channel, this same architecture can be brought to other areas of your organization more quickly. You’ll not only have proof of concept, but infrastructure in place and buy in from those who will need to learn and work with your new system.
For years, many companies had ETL tools in place and are now transitioning to open source technologies like Kafka, Spark or Hadoop as they try to embrace streaming on a larger scale. But as people who have walked the DIY path know all too well, this tech can be hard to maintain, manage and to build on. Sure, open source frameworks are free, but come loaded with required tech expertise, coding and a lack of support when things go wrong. You will invest countless hours patching your system in the gray area between platforms, and lose valuable ground when a team member leaves, taking coding and expertise along with them. Before you know it, you will be trying to maintain and manage an unwieldy monster of a system - recreating the wheel when you don’t have to. It might work, but you will spend more time on that than anything else in IT.
Ultimately, we all want to avoid risk as much as possible. If you want to stream data from data sources, querying a billion records every second is not going to work and neither will rewriting the application. Change Data Capture is your bridge - capturing data changes at the source. CDC will capture incremental changes on the data by listening to change mechanisms off the source - i.e. database transaction logs, journaling mechanisms within messaging services, rest APIs, etc. Each source is very unique and very different. CDC is how you will solve the first and hardest problems of a streaming architecture - capturing data from the source in a seamless way without making any changes to the application.
Replication Groups allow a streaming-first data platform to process non-stop changes to groups of tables in one shot. If multiple tables are updated in one transaction, that would be captured altogether. By using replication groups, you can select tables for replication by name or name patterns. If replication groups are not supported, then the user would be forced to create data flows for each table with independent flow execution on each - a laborious process and error prone to say the least.
Schema Evolution is a common feature of replication groups, providing full support for database schema evolution (schema changes) in an automated manner with options for the customer to determine how he/she wants to propagate schema changes. When a new table appears at the source and its name matches the replication pattern, it will create a table in the target database and start replicating it immediately. When considering streaming-first data platforms, replication groups and schema evolution should radically simplify massive data streaming processes.
Everyone talks about simplifying their architecture, but achieving a truly streamlined approach is another story. Here are few key components that you should look for:
Streaming data will change the way that your business operates. It’s an exciting new chapter in operating, but one that needs a thoughtful and strategic approach for implementation. When the right steps are put into place from the start, you and your organization can reap the many benefits to follow.
Interested in learning more?
Download our Whitepaper in collaboration with Eckerson: The How & Why of Streaming First Data Architectures