Session Title: Managing Data Pipelines with Conductor
Speaker(s): Doug Sillars
Abstract: Data pipelines are a challenging part of working in data science. Netflix Conductor is a battle tested workflow orchestration tool invented at Netflix to automate millions of microservice workflows. Using a workflow tool like Conductor as a part of your data analysis pipeline can help streamline and improve the reproducibility of your results. In this presentation we will use Conductor to automate a set of microservices to build an automated workflow for a data pipeline workflow, automating the data acquisition, cleaning and analysis.
500+ sessions are now available on-demand from Data Platform Summit 2022, 2021 & 2020 at no cost. Browse all sessions.
Stay tuned, more learning coming your way.