An efficient way of optimizing spark jobs Coalescing What Is Coalescing? The coalesce method reduces the number of partitions in a DataFrame. Coalescing avoids a full shuffle. Instead of creating new partitions, it shuffles the data using the Hash Partitioner (default), and adjusts into existing partitions. …

Memory Management in Spark Following diagram shows how the memory is segregated in Spark executor Spark Executor Memory Distribution For the sake of understanding, we will take an example of 4GB Memory allocated to an executor and leave the default configuration and see how much memory each segment gets. Two main configurations that control…

1. Internal working of Spark It has small code base but system is divided into many layers. Each layer has responsibility. The layers are independent to each others. Spark driver contains many components to translate user program into actual jobs executed on cluster. Spark Context: It represent the connection to spark cluster, it is used…

What’s new in Apache Spark 3.0? In this article, will explain Apache Spark 3.0 new features and Adaptive Query Execution, Why it has become so popular. There are many new and notable features in Apache Spark 3.0; …

Join operation is a very common data processing operation. As a unified big data processing engine, spark provides very rich join scenarios.This article covers the different join strategies employed by Spark to perform the join operation. 1. Broadcast hash join When the dimension table and the fact table for the…

History Let’s begin by a bit of history. RDD was the first API. Then to make things easier, DataFrame was added. It is built upon RDD. It was inspired from SQL. You can deal with data by selecting columns, grouping them, etc. Dataset is an improvement of DataFrame with type-safety. It…

What is optimization : “The term optimization refers to a process in which a system is modified in such a way that it work more efficiently or it uses fewer resources.” Why optimization is important: In production environment Spark is running on distributed machines, and the distributed system may be used by some other applications too. Which means…

1. What is Spark SQL? Spark SQL integrates relational processing with Spark’s functional programming. It provides support for various data sources and makes it possible to weave SQL queries with code transformations thus resulting in a very powerful tool. 2. Why is Spark SQL used? Spark SQL originated as Apache Hive to run on top of Spark and is now integrated…

The Spark architecture depends upon two abstractions: Resilient Distributed Dataset (RDD) Directed Acyclic Graph (DAG) 1. What is RDD (Resilient Distributed Dataset)? RDD is a fundamental data structure of Spark and it is the primary data abstraction in Apache Spark and the Spark Core. RDDs are fault-tolerant, immutable distributed collections of objects, which means once you create…