Flexible Data Pipelines | Balancing Standard and Custom Approaches by Eckerson

Hero Image
White Paper

Enterprises use both standard and custom data pipelines to process raw data into analytics-ready data sets. They must apply the right mix, creating efficiency and scale via standardization and automation where possible, but still accommodating customization in order to innovate. But many get the balance wrong and standardize too slowly, limiting the value they derive from analytics.

This report examines the impact of standardization and customization on data pipelines, with a focus on design, building, testing, rollout, operations, and adaptation. It seeks to help architects, data engineers, application developers, and data scientists strike the right balance in their environments. Enterprises can start by standardizing overly customized data pipelines and demonstrating clear ROI with bite-size projects. As they standardize more and regain the right balance, they will drive data democratization, increased productivity, and reduced risk. As they standardize, enterprises can also free up and redirect resources to custom work, fostering innovation and increasing analytics value.


  • Architectural Approaches to Data Pipelines
  • Why Customize?
  • Why Standardize?
  • The Data Pipeline Lifecycle
  • Why Enterprises Lose Balance
  • Adoption Patterns & Striking the Balance


Kevin Petrie has spent 25 years deciphering what technology means for practitioners as an industry analyst, writer, instructor, marketer, and services leader. Kevin launched, built, and led a profitable data services team for EMC Pivotal in the Americas and Europe. He also ran field training for the data integration software provider Attunity (now part of Qlik). Kevin is a frequent public speaker, the author of two books on data streaming, and a data management instructor at eLearningCurve.

Wayne W. Eckerson has been a thought leader in the data and analytics field since the early 1990s. He is a sought-after consultant, noted speaker, and expert educator who thinks critically, writes clearly, and presents persuasively about complex topics. Eckerson has conducted many groundbreaking research studies, chaired numerous conferences, written two widely read books on performance dashboards and analytics, and consulted on BI, analytics, and data management topics for numerous organizations. Eckerson is the founder and principal consultant of Eckerson Group.

Flexible Data Pipelines | Balancing Standard and Custom Approaches

Whitepaper by Eckerson & Equalum

Ready to Get Started?

Experience Enterprise-Grade Data Integration + Real-Time Streaming