PySpark Interview questions and answers

1. Explain how an object is implemented in python?

Ans:  In Python, an object shows a particular instance of a class. A class is like a blueprint; a thing is a concrete realization of this blueprint. When a class is instantiated, it results in an object. The instantiation process involves calling the class using its name. For example, consider a class Student with attributes like id, name, and estb. The instantiation of this class and the method to display its attributes can be demonstrated as follows:

Syntax 

= ()


Ex:

class Student:

    id = 25

    name = "HKR Trainings"

    estb = 10

    def display(self):

        print("ID: %d \n Name: %s \n Estb: %d " % (self.id, self.name, self.estb))

stud = Student()

stud.display()




This would output:

ID: 25

Name: HKR Trainings

Estb: 10

2. Explain Methods in Python

Ans: Methods in Python are essentially functions that are defined inside a class. These are used to define the conduct of a thing/ object. Unlike regular functions, methods are called on objects and can access and modify the state of the object. For example, the Student class may have a method named display to show student details:

class Student:

    roll = 17

    name = "gopal"

    age = 25

    def display(self):

        print(self.roll, self.name, self.age)

Here, the display is a method that prints a Student object's roll no, name, and age.

Become a Pyspark Certified professional by learning this HKR Pyspark Training 

3. What is encapsulation in Python?

Ans: 

Encapsulation in Python is a fundamental concept that involves bundling data and methods that work on that data within a single unit, such as a class. This mechanism restricts direct access to some of the object's components, preventing accidental interference and misuse of the methods and data. An example of encapsulation is creating a class with private variables or methods. For instance, in a Product class, the maximum price is encapsulated and cannot be changed directly from outside the class:

class Product:

    def __init__(self):

        self.__maxprice = 75

    def sell(self):

        print("Sale Price: {}".format(self.__maxprice))

    def setMaxPrice(self, price):

        self.__maxprice = price

p = Product()

p.sell()

p.__maxprice = 100

p.sell()




This will output:

Selling Price: 75

Selling Price: 75

Despite attempting to modify __maxprice, the encapsulated design prevents this change.

4. Explain the concept of Python Inheritance

Ans:  In Python, inheritance enables a class, known as a child class, to inherit elements and methods from another class, called the parent class. This concept is a basic pillar of OOPs and helps in code reusability. Further, a child class inherits from a parent class by mentioning the parent class name in parentheses after the child class name. For example:

class ParentClass:

    pass

class ChildClass(ParentClass):

    pass

This syntax enables ChildClass to inherit attributes and methods from ParentClass.

5. What is a python for loop?

Ans:  A for loop in Python is helpful for iterating over a series, which could be a list, tuple, string, or any other iterable thing/object. Also, it repeats a block of code for each attribute in the series (sequence). The for loop syntax in Python is very simple:

for element in sequence:

    # code block

Here, element is a temporary variable that takes the value of the next element in the sequence with each iteration.

6. What is a for-else in python?

Ans:

Python's for-else construct is a unique feature where the else block is executed after the for loop completes its iteration over the entire sequence. The else block does not execute if the loop is broken by a break statement. This can be useful for scenarios where you need to check if a break was caused in the loop. For example:

x = []

for i in x:

    print("in for loop")

else:

    print("in else block")

This will output "in else block" since the for loop has no elements to iterate over and completes its full iteration.

7. What are errors and exceptions in python programming?

Ans: In Python, errors are problems in a program that cause it to exit unexpectedly. On the other hand, exceptions are raised when some internal event disrupts the normal flow of the program. A syntax error, or parsing error, is an example of an error, which occurs when Python cannot understand what you are trying to say in your program. Exceptions, however, occur during the execution of a program, despite correct syntax, due to an unexpected situation, like attempting to divide by zero.

8. What is the key difference between list and tuple?

Ans: The primary difference between lists and tuples in Python is their mutability. Lists are uncertain, which means their elements can be altered, added, or removed. Tuples are immutable, meaning it cannot be modified once a tuple is created. This immutability makes tuples faster than lists and suitable for read-only data. For example, attempting to change an element of a tuple will result in a TypeError.

9. How to convert a string to a number in python?

Ans:

Ans: To change a string into a number within Python, you can use the function: built-in int() for integers or float() for floating-point numbers. It is generally used when you need to run mathematical actions on numbers initially read as strings from user input or a file. For example:

number = int("10")

print(number + 5)

This would output 15.

10. What is data cleaning?

Ans:  Data cleaning is the process of preparing raw data for analysis by changing or removing incorrect, incomplete, irrelevant, replicated, or improper data. This step is crucial in the data preparation process as it significantly impacts the quality of insights derived from the data.

11. What is data visualization and why is it important?

Ans: It shows data and information visually with the help of charts, graphs, and other visual formats. It is important because it translates complex data sets and abstract numbers into graphic designs that are easier to understand and interpret. Compelling data visualizations reveal patterns, trends, and insights that go unnoticed in text-based data.

PySpark Training Certification

• Master Your Craft
• Lifetime LMS & Faculty Access
• 24/7 online expert support
• Real-world & Project Based Learning

Explore Curriculum

12. What is Pyspark and explain its characteristics?

Ans: PySpark is the Python API for Apache Spark, an open-source, distributed computing system. It offers Python developers a way to parallelize their data-processing tasks across clusters of computers. PySpark's characteristics include:

• Its ability to handle batch and real-time data processing.
• Support for various data sources.
• Powerful cache and memory management capabilities.

13. Explain RDD and also state how you can create RDDs in Apache Spark.

Ans: Resilient Distributed Datasets (RDDs) are a fundamental data structure of Apache Spark. They are immutable distributed collections of objects, which can be processed in parallel across a Spark cluster. RDDs can be created in two ways: by parallelizing an existing collection in your driver program, or by referencing a dataset in an external storage system like HDFS, HBase, or a shared file system.

14. How do we create RDDs in Spark?

Ans:  RDDs in Spark can be created using the parallelize method of the SparkContext, which distributes a local Python collection to form an RDD, or by referencing external datasets. For example, creating an RDD from a local list would involve sc.parallelize([1,2,3,4,5]).

15. Name the components of Apache Spark?

Ans: Apache Spark consists of several components: Spark Core for basic functionality like task scheduling, memory management; Spark SQL for processing structured data; Spark Streaming for real-time data processing; MLlib for machine learning; and GraphX for graph processing.

16. How DAG functions in Spark?

Ans: In Spark, a Directed Acyclic Graph (DAG) represents a sequence of computations performed on data. When an action is called on an RDD, Spark creates a DAG of the RDD and its dependencies. The DAG Scheduler divides the graph into stages of tasks to be executed by the Task Scheduler. The stages are created based on transformations that produce new RDDs and are pipelined together.

17. What do you mean by Spark Streaming?

Ans: Spark Streaming is an addition of the core Spark API that allows scalable, high-throughput, fault-tolerant stream refining of real-time data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, and processed using complex algorithms expressed with high-level functions like map, reduce, join, and window.

                  Want to know more about Pyspark ,visit here Pyspark Tutorial !

18. What does MLlib do?

Ans: MLlib is a Spark's Machine Learning (ML) library. It aims to make practical ML scalable and easier. Further, it includes common learning algorithms and utilities along with clustering, classification, regression, collective filtering, as well as lower-level optimization primitives and higher-level pipeline APIs.

19. What are the different MLlib tools available in Spark?

Ans:  MLlib in Spark offers various tools including ML algorithms such as regression, classification, clustering, collective filtering, feature extraction, transformation, pipelines for building, evaluating, and tuning ML pipelines, and utilities for linear algebra, statistics, and data handling.

20. Explain the functions of SparkCore.

Ans:  SparkCore is the basic general execution engine helpful the Spark platform on which all other functionality is developed. It offers an in-memory caculation and referencing datasets in outside storage systems. Spark Core functions include memory management, fault recovery, interacting with storage systems, and scheduling and monitoring jobs on a cluster.

21. What is the module used to implement SQL in Spark? How does it work?

Ans:  Spark SQL is the module in Apache Spark for processing structured and semi-structured data. It provides a coding abstraction called DataFrames and can also act as an allocated SQL query engine. It enables querying data via SQL, as well as the Apache Hive variant of SQL, and it integrates with regular Python/Java/Scala code.

22. List the functions of Spark SQL.

Ans: Spark SQL functions include loading data from various structured sources, querying data using SQL, integrating with BI tools through JDBC/ODBC connectors, and providing a rich integration between SQL and regular Python/Java/Scala code.

Subscribe to our youtube channel to get new updates..!

Subscribe

23. What are the various algorithms supported in PySpark?

Ans:  PySpark supports various algorithms such as classification, regression, clustering, collaborative filtering, and others in its MLlib library. This includes support for feature extraction, transformation, and statistical operations.

24. Explain the purpose of serializations in PySpark?

Ans:  Serialization in PySpark is used to transfer data between different nodes in a Spark cluster. PySpark supports two serializers: the MarshalSerializer for a limited set of data types but faster performance, and the PickleSerializer which is slower but supports a broader range of Python object types.

25. What is PySpark StorageLevel?25. What is PySpark StorageLevel?

Ans:  The PySpark StorageLevel is used to define how an RDD should be stored. Options include storing RDDs in memory, on disk, or both, and configuring whether RDDs should be serialized and whether they should be replicated.

26. What is PySpark SparkContext?

Ans:  SparkContext is the entry point for Spark functionality in a PySpark application. It enables PySpark to connect to a Spark cluster and use its resources. The SparkContext uses the py4j library to launch a JVM and create JavaSparkContext.

27. What is PySpark SparkFiles?

Ans: SparkFiles is a feature in PySpark that allows you to upload files to Spark workers. This is useful for distributing large datasets or other files across the cluster. The addFile method of SparkContext is used to upload files, and SparkFiles can be used to locate the file on the workers.

28. Explain Spark Execution Engine?

Ans: The Spark Execution Engine is the heart of Apache Spark. It is responsible for scheduling and executing tasks, managing data across the cluster, and optimizing queries for performance. The engine works in memory, making it much faster than traditional disk-based engines for certain types of computations.

29. What do you mean by SparkConf in PySpark?

30. Name a few attributes of SparkConf.

Ans: Key attributes of SparkConf include set(key, value) for setting configuration properties, setAppName(value) for naming the Spark application, setMaster(value) for setting the master URL, and get(key, defaultValue=None) for retrieving the value of a configuration property.