cnfl.io/podcast-episode-193 | Getting data from a database management system (DBMS) into Apache Kafka® in real time is a subject of ongoing innovation. John Neal (Principal Solution Architect, Qlik) and Adam Mayer (Senior Technical Producer Marketing Manager, Qlik) explain how leveraging change data capture (CDC) for data ingestion into Kafka enables real-time data-driven insights.
It can be challenging to ingest data in real time. It is even more challenging when you have multiple data sources, including both traditional databases and mainframes, such as SAP and Oracle. Extracting data in batch for transfer and replication purposes is slow, and often incurs significant performance penalties. However, analytical queries are often even more resource intensive and are prohibitively expensive to run on production transactional databases. CDC enables the capture of source operations as a sequence of incrementing events, converting the data into events to be written to Kafka.
Once this data is available in the Kafka topics, it can be used for both analytical and operational use cases. Data can be consumed and modeled for analytics by individual groups across your organization. Meanwhile, the same Kafka topics can be used to help power microservice applications and help ensure data governance without impacting your production data source. Kafka makes it easy to integrate your CDC data into your data warehouses, data lake, NoSQL database, microservices, and any other system.
Adam and John highlight a few use cases where they see real-time Kafka data ingestion, processing, and analytics moving the needle-including real-time customer predictions, supply chain optimizations, and operational reporting. Finally, Adam and John cap it off with a discussion on how capturing and tracking data changes are critical for your machine learning model to enrich data quality.
EPISODE LINKS
► Fast Track Business Insights with Data in Motion: cnfl.io/confluent-and-qlik-cd...
► Join the Confluent Community: cnfl.io/join-community-episod...
► Learn Apache Kafka on Confluent Developer: cnfl.io/confluent-developer-e...
► Demo: Event-Driven Microservices with Confluent: cnfl.io/event-driven-microser...
► Use PODCAST100 to get $100 of free Confluent Cloud usage: cnfl.io/podcast100-for-cloud-...
► Promo code details: cnfl.io/podcast100-code-detai...
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#changedatacapture #datapipeline #apachekafka #kafka #confluent
5 июл 2024
