Building Your First Data Pipeline in Apache Spark by Kevin Feasel

Session Title: Building Your First Data Pipeline in Apache Spark
Speaker(s): Kevin Feasel

Abstract: As a data engineer, the Apache Spark platform provides a great deal of functionality designed to solve common problems around data movement and processing, particularly in the cloud. In this session, we will learn how to use Apache Spark in Microsoft Azure. We will see which Azure services provide Apache Spark integration points, look at use cases in which Apache Spark is a great choice, and use the metaphor of the data pipeline to perform data movement and transformation in the cloud. We will additionally learn how to use notebook workflows in Azure Databricks to simplify the process.