Quick info

Duration: Custom

Schedule: Custom

Level: Beginner to Intermediate

Pricing: Custom

Course overview

Introduction to Spark Programming

This course introduces the Apache Spark distributed computing engine, and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner. It is based on the Spark 2.x release.

The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataSets/DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL, DataFrames, and DataSets, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization.

The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data and integrating with the Kafka server.

The course is very hands-on, with many labs. Participants will interact with Spark through the Spark shell (for interactive, ad-hoc processing) as well as through programs using the Spark API.

Supported Platforms: Spark 2.1+

Objectives:

Understand the need for Spark in data processing

Understand the Spark architecture and how it distributes computations to cluster nodes

Be familiar with basic installation / setup / layout of Spark

Use the Spark shell for interactive and ad-hoc operations

Understand RDDs (Resilient Distributed Datasets), and data partitioning, pipelining, and computations

Understand/use RDD ops such as map(), filter() and others.

Understand and use Spark SQL and the DataFrame/DataSet API.

Understand DataSet/DataFrame capabilities, including the Catalyst query optimizer and Tungsten memory/cpu optimizations.

Be familiar with performance issues, and use the DataSet/DataFrame and Spark SQL for efficient computations

Understand Spark's data caching and use it for efficient data transfer

Write/run standalone Spark programs with the Spark API

Use Spark Streaming / Structured Streaming to process streaming (real-time) data

Ingest streaming data from Kafka, and process via Spark Structured Streaming

Understand performance implications and optimizations when using Spark

Topics

Module 1

Scala Ramp Up

Module 2

Introduction to Spark

Module 3

RDDs and Spark Architecture

Module 4

Spark SQL, DataFrames, and DataSets

Module 5

Shuffling Transformations and Performance

Module 6

Performance Tuning

Module 7

Creating Standalone Applications

Module 8

Spark Streaming

Join this course now

Need help? Contact us