Spark Interview Questions

Are you planning to move your career into Apache spark? Well, it’s a lucrative career option in today’s IT world. Major companies like Amazon, eBay, JPMorgan, and more are also adopting Apache Spark for their big data deployments.

However, due to heavy competition in the market, it’s essential for us to know each and every concept of Apache Spark to clear the interview. To help you out, we have collected the top Apache Spark Interview Questions and Answers for both freshers and experienced. All these questions are compiled after consulting with Apache Spark training experts.

So utilize our Apache spark Interview Questions to maximize your chances in getting hired.

If you want to enrich your career as an Apache Spark Developer, then go through our Spark Training

Apache Spark Interview Questions and Answers

Q1. What is Apache Spark?

Ans: Spark is an open-source and distributed data processing framework. It provides an advanced implementation engine which includes in-memory computation and periodic data flow. Apache Spark runs automatically both in Hadoop or cloud and able to access various data sources, including Hbase, HDFS, and Cassandra.

Q2. What are the main features of Apache Spark?

Ans: Following are the main features of Apache Spark:

• Integration with Hadoop.
• Includes an interactive language shell called Scala in which spark is written.
• Robust Distributed Data sets are cached between compute nodes in a cluster.
• Offers various analytical tools for real-time analysis, graphic processing, and interactive query analysis.

Q3. Define RDD.

Ans: 

Resilient Distribution Datasets (RDD) represents a fault-tolerant set of elements that operate in parallel. The data in the RDD section is distributed and immutable. There are mainly two types of RDDs.

• Parallelized collections: The existing RDDs operating parallel to each other.
• Hadoop datasets: The dataset that performs a function for each file record in HDFS or other storage systems.

Q4. What is the use of the Spark engine?

Ans: The objective of the Spark engine is to plan, distribute, and monitor data applications in a cluster.

Q5. What is the Partition?

Ans: Partition is a process of obtaining logical units of data for speeding up data processing. In simple words, partitions are smaller and logical data separation is similar to a ‘split’ in MapReduce.

Q6. What type of operations are supported by RDD support?

Ans: Transformations and actions are the two types of operations supported by RDD.

Q7. What do you mean by transformations in Spark?

Ans: In simple words, transformations are functions implemented in RDD. It will not work until action is performed. map() and filter() are some examples of transformations.

While map() function is repeated on each RDD line and splits into a new RDD, the filter function () creates a new RDD by selecting the elements that pass the function argument from the current RDD.

Q8. Explain Actions.

Ans: Actions in Spark makes it possible to bring data from RDD to the local machine. Reduce () and take () are the functions of Actions. Reduce() function is performed only when action repeats one by one until one value lefts. The take () accepts all RDD values ​​to the local key.

Q9. Explain the functions supported by Spark Core.

Ans: Various functions are supported by Spark Core like job scheduling, fault-tolerance, memory management, monitoring jobs and much more.

To gain more knowledge of Apache Spark, check out Apache Spark Tutorial

Q10. Define RDD Lineage?

Ans: Spark does not support data replication, so if you have lost information, it is reconstructed using RDD Lineage. RDD generation is a way to reconstruct lost data. The best thing to do is always remember how to create RDDs from other datasets.

Q11. What does Spark Driver do?

Ans: The Spark driver is a program that runs on the main node of the device and announces transformations and actions on the data RDD. In a nutshell, the driver in Spark makes SparkContext in conjunction with the given Spark Master. It also provides RDD graphs to the master, where the cluster manager operates.

Q12. What is Hive?

Ans: 

By default, Hive supports Spark on YARN mode.

HIVE execution is configured in Spark through:

hive> set spark.home=/location/to/sparkHome;
hive> set hive.execution.engine=spark;

Q13. List the most frequently used Spark ecosystems.

Ans: 

• For developers, Spark SQL (Shark).
• To process live data streams, Spark Streaming.
• To generate and compute, GraphX.
• MLlib (Machine Learning Algorithms)
• For promoting R programming in the Spark Engine, SparkR.

Q14. Explain Spark Streaming.

Ans: Stream processing is an extension to the Spark API, which allows live data streaming. Data from various sources such as Kafka, Flume and Kinesis are processed and sent to the file system, live dashboards and databases. In terms of input data, its similar to batch processing for dividing data into streams like batches.

Q15. What is GraphX?

Ans: Spark uses GraphX for graphics processing and building. The GraphX lets programmers think about big data.

Q16. What does MLlib do?

Ans: Spark supports MLlib, which is a scalable Machine Learning library. Its objective is to make Machine Learning easy and scalable with common learning algorithms and use cases like regression filtering, clustering, dimensional reduction, and the like.

Q17. Define Spark SQL?

Ans: Spark SQL is also known as Shark, is used for processing structured data. Using this module, relational queries on data are executed by Spark. It supports SchemaRDD, which consists of schema objects and row objects representing the data type of each column in a row. This is just like a table in relational databases.

Q18. What is a parquet file?

Ans: The columnar format file supported for other data processing systems is called a Parquet file. Both read and write operations on Parquet files are executed by Spark SQL, making it one of the best big data analytics formats.

Q19. Which file systems are supported by Apache Spark?

Ans: 

• Hadoop file distribution system (HDFS)
• Amazon S3
• Local File system

Q20. Define YARN?

Ans: Like Hadoop, YARN is one of the main features of Spark, which provides resource management platform to produce scalable operations across the entire cluster. 

Q21. Name a few functions of  Spark SQL.

Ans: Following are the functions of Spark SQL:

• Loads data from various structured sources.
• Query data using SQL elements.
• Provides advanced integration between regular Python/Java/Scala code and SQL.

Q22. What are the advantages of using Spark over MapReduce?

Ans: 

• Spark implements 10-100X times faster data processing than MapReduce due to the availability of in-memory processing. MapReduce uses persistence storage for data processing tasks. 
• Spark offers in-built libraries to execute multiple tasks using machine learning, steaming, batch processing, and more. Whereas, Hadoop supports only batch processing. 
• Spark supports in-memory data storage and caching, but Hadoop is highly disk-dependent.

Q23. Is there any benefit of learning MapReduce?

Ans: Yes, MapReduce is a standard used by many big data tools, including Apache Spark. As data grows, it becomes extremely important to use MapReduce. Many tools, such as Pig and Hive, convert queries to the MapReduce phases for optimizing them.

Q24. Describe Spark Executor.

Ans: When SparkContext joins the Cluster Manager, it receives the executor on nodes in the cluster. The executors are the Spark processes that perform predictions and save data on workplaces. The final tasks from SparkContext are sent to the executor for executions.

Q25. Can we use Spark for accessing and analyzing the data stored in Cassandra Database?

Ans. Yes, we can use Spark for accessing and analyzing the data stored using the Spark Cassandra Connector. For connecting Spark to a Cassandra cluster, we need to add the Cassandra Connector to the Spark project.

Q26. How to connect Spark with Apache Mesos?

Ans: 

1. First, configure the Spark driver program to connect to Mesos.
2. Next, Spark binary package must be in a location available to Mesos.
3. By installing Apache Spark in the exact location of Apache Mesos and configure the property  ‘spark.mesos.executor.home’ to point to the location where it is installed.

Q27. Define broadcast variables.

Ans: Broadcast variables enable the developers to have a read-only variable cached on each machine instead of copying it with tasks. In an efficient way, it allows every node to copy a large input dataset. Using efficient broadcast algorithms, Spark tries to share broadcast variables.

Q28. What is the use of Akka in Spark?

Ans: Akka is used for scheduling in Spark. All workers are allowed to task to master after registration. The master assigns only tasks. Here, Spark uses Akka to send messages between workers and teachers.

Q29. How is Spark SQL different from HQL and SQL?

Ans: Spark SQL supports SQL and the Hive query language in the Spark Core engine without changing any syntax. In Spark SQL, you can combine an SQL table and an HQL table.

Q30. What are the different data sources supported by Spark SQL?

Ans:

1. Parquet file
2. JSON datasets
3. Hive tables