What is Pyspark

What is Apache Spark?

Apache Spark is a specific Big Data analysis, storage, and data processing engine.It has many benefits over MapReduce: it is quicker, simpler to use, more easy, and can run just about anyhttps://moz.com/where. It has built-in tools for SQL, Machine Learning, and streaming, making this one of the most important and highest demanded tools in the IT business sector. Scala is the programming language used to create Spark. Although Apache Spark has APIs for Python, Scala, Java, and R, the former two are the most commonly used languages with Spark. 

What is Pyspark?

PySpark is a Python-based tool developed by the Apache Spark Community for use with Spark.It enables Python to work with RDDs (Resilient Distributed Datasets). It also includes PySpark Shell, which connects Python APIs to the Spark core in order to launch Spark Context. Spark is the name of the cluster computing engine, and PySpark is the Python library for using Spark.
Here some of the important features of pyspark. They are:

• It comes with real time processing computations and calculations.
• It works dynamically with RDDS.
• In order to process the bulk datasets of big data pyspark serves as the fastest framework when compared with others.
• One of the most attractive features of pyspark is the effective disk persistence and memory caching.
• Moreover pyspark is most compatible with other programming languages such as python, scala, java when processing large datasets.
  
  Become a Pyspark Certified professional by learning this HKR Pyspark Training !

Why Pyspark?

In order to perform the different operations on the big data, one needs to rely on different tools. But this is not  a good sign when dealing with bulk datasets processing.In the current market there are several flexible and scalable tools that deliver enormous results form the big data. One such tool is the pyspark which acts as an effective tool while dealing with big data. At present many data scientists, IT professionals prefer python as it has simple and neat user interface design.So many data analysts prefer this tool for performing data analysis, machine learning on big data. And the Apache spark community came up with a tool by combining both the spark and python i.e pyspark in order to deal with big datasets very easily.

Who can learn the Pyspark?

Python is quickly becoming a powerful language in data science and machine learning. One will be capable of working with Spark in Python using Py4j's library. Python is a programming language popularly used throughout machine learning and data science. Python allows for parallel computing.
The prerequisite to take this pyspark course are:

• Python programming knowledge
• Big data knowledge and framework.
• PySpark is a good fit for someone who wants to work with big data.

Installation and configuration of Pyspark

Just before installing the apache, you need to make sure that java and scala are installed on your system. If not install them first. Now you will walk through how to set up the pyspark environment.
Now we will walk through the installation steps on the Linux platform first then on windows as well.

Installation on Linux platform:

Step1:just download the updated version of the apache spark form the official website apache spark and try to locate it in the downloads folder.

Step2:Now extract the spark tar file

Step3: Immediately after the extraction of files is done, use the following commands to move them to the specific folder as they are placed in the downloads folder by default.
/usr/local/spark

$ su –

Password:

# cd /home/Hadoop/Downloads/

# mv sp

ark-2.4.0-bin-hadoop2.7 /usr/local/spark

# exit
Step4:Now set up the PATH for the pyspark.
export PATH = $PATH:/usr/local/spark/bin
Step5:Set up the environment for pyspart by using the following command.
$ source ~/.bashrc
Step6:You need to verify the pyspark installation with the help of the following command.
<pre">$ spark-shell
Output will be displayed showing successful installation of pyspark.
Step7: Invoke the pyspark shell by running the command in the spark directory as follows.
# ./bin/pyspark

PySpark Training Certification

• Master Your Craft
• Lifetime LMS & Faculty Access
• 24/7 online expert support
• Real-world & Project Based Learning

Explore Curriculum

Installation on Windows

In this section, we will learn how to install pyspark step by step on the Windows platform.

Step1:Download the latest version of spark from the official website.

Step2: Now extract the downloaded file into a new directory.

Step3: Now set the user and system  variables as follows.
User variables:

• Variable: SPARK_HOME
• Value: C:\Program Files (x86)\spark-2.4.0-bin-hadoop2.7\bin

System variables:

• VAriable:PATH
• Value: C:\Windows\System32;C:\Program Files (x86)\spark-2.4.0-bin-hadoop2.7\bin

Step4: Now download the Windows utilities by clicking here and move them to the C:\Program Files (x86)\spark-2.4.0-bin-hadoop2.7\bin.
Step5: Now you can start the spark shell by the following command.
Spark-shell
Step6:In order to start or begin the pyspark shell type the following command as follows.
Pyspark
Now your pyspark shell environment is ready and you need to learn about how to integrate and perform operations on the pyspark.
Before driving into the pyspark operations you need to take care of configuration settings that you need to take care.

Subscribe to our youtube channel to get new updates..!

Subscribe

SparkConf:

Well, before running any spark application you need to set some parameters and configurations and that can be done using the sparkconf.
Now we will discuss the most important attributes of the sparkconf while using the pyspark. They are:

• set(key, value): This attribute is used to configure a property.
• setMaster(value): The master URL is set using this attribute.
• setAppName(value): This attribute is used to specify the name of an application.
• get(key, defaultValue=None): This attribute is used to retrieve a key's configuration value.
• setSparkHome(value): This attribute is used to specify the location of the Spark installation.
  
                      Want to know more about Pyspark ,visit here Pyspark Tutorial !

Below is the code where some attributes of sparkconf are used mostly. 
>>> from pyspark.conf import SparkConf

>>> from pyspark.context import SparkContext

>>>conf = SparkConf().setAppName("PySpark App").setMaster("local[2]")

>>> conf.get("spark.master")

>>> conf.get("spark.app.name")
You have learned about how to set configurations using the sparkconf, next you need to learn about the sparkcontext.

SparkContext:

SparkContext is the portal by which any Spark-derived application or usability enters. It is perhaps the most important thing that happens when you run any Spark application. SparkContext is available as sc by default in PySpark, so creating a new SparkContext will result in an error.
Here is the list of sparkcontext parameters. They are:

• Master: The cluster's web address SparkContext establishes a connection with.
• AppName: The title of your position
• SparkHome: A directory for installing Spark
• Py Files: The.zip or.py files are sent to the cluster and then added to the PYTHONPATH environment variable.
• Environment: Variables affecting the environment of worker nodes.
• BatchSize: The number of Python objects that are represented in the batch. To disable batching, set the value to 1; to choose the batch size automatically based on the object size, set it to 0; and to use an unlimited batch size, set it to 1.
• Serializer : This parameter describes an RDD serializer.
• Conf: An LSparkConf object used to set all Spark properties
• profiler cls: A class of custom profilers used for profiling; however, the default one is pyspark.profiler.BasicProfiler.

Among all the parameters master and AppName are most widely used. And the basic initial code used for every pyspark application are:
from pyspark import SparkContext

sc = SparkContext("local", "First App")

SparkFiles and Class Methods:

When you use SparkContext.addfile to upload data to Apache Spark, you will use SparkFile (). SparkFiles contains two types of commands. They are: 

• get(Filename):When you need to specify the path of a file that you added using SparkContext.addfile() or sc.addFile(), use this class method ()
• getRootDirectory():Use this class method to specify the path of a file added with SparkContext.addfile() or sc.addFile() ()

Resilient Distributed Database(RDD):

Spark's RDD is one of its most important features. It is an abbreviation for Resilient Distributed Database. It is a group of items that are distributed across multiple nodes in a cluster in order to perform parallel processing. Faults can be recovered automatically by an RDD. Changes cannot be made to an RDD. However, you can create an RDD from an existing one by making the necessary changes, or you can perform various types of operations.
Here are the features of RDD. They are:

• Immutability: Once created, an RDD cannot be altered or reconfigured; however, if you want to make changes, you can create a new RDD from the existing one.
• Distributed: An RDD's data can exist on a cluster and be processed in parallel while parallel processing.
• Partitioned: More partitions distribute work among different clusters, but it also creates scheduling overhead.

Operations of RDDs:

Certain operations in Spark can be carried out on RDDs. These operations are, in essence, methods. RDDs can perform two types of operations: actions and transformations. Let us break them down individually with examples.
RDD is created using the following:
RDDName = sc.textFile(“ path of the file to be uploaded”)

Action Operations:

To perform certain computations, action operations are directly applied to datasets. The following are some examples of Action operations.

• take(n): This is one of the most commonly used RDD operations. It accepts a number as an argument and displays that many elements from the specified RDD.
• count() It returns the number of elements in the RDD.
• top(n): This operation also accepts a number, say n, as an argument and returns the top n elements.

Transformation Operations:

The set of operations used to create new RDDs, either by implementing an operation to an existing RDD or by creating an entirely new RDD, is referred to as transformation operations. Here are some examples of Transformation operations:

• Map Transformation: Use this operation to transform each element of an RDD by implementing the function to the entire element.
• Filter Transformation: Use this transformation operation to remove some elements from your dataset. These are known as stop words. You can create your own stop words.
  
  Top 30 frequently asked Pyspark Interview Questions !

Machine Learning(MLib) In Spark

Pyspark is a machine learning API, MLib that accommodates several types of algorithms.The different types of algorithms in pyspark MLib are listed below:

• mllib.classification. The spark. mllib package includes methods for performing binary classification, regression analysis, and multiclass classification. Naive Bayes, decision trees, and other algorithms are commonly used in classification.
• mllib.clustering: Clustering allows you to group subsets of entities based on similarities in the elements or entities.
• mllib.linalg: This algorithm provides MLlib utilities for linear algebra support.
• mllib.recommendation: This algorithm is used to fill in missing entries in any dataset by recommender systems.
• spark.mllib: This library supports collaborative filtering, in which Spark uses ALS (Alternating Least Squares) to predict missing entries in sets of user and product descriptions.

Pyspark in various industries:

Apache Spark is a widely used tool in a variety of industries. However this application is not limited to the IT industry, though it is most prevalent in that sector. Even the IT industry's big dogs, such as Oracle, Yahoo, Cisco, Netflix, and others, use Apache Spark to deal with Big Data.

• Finance: In the finance sector PySpark is used to extract the information related to the call recordings, emails, and social media profiles.
• E-commerce: In this industry, Apache Spark with Python can be used to obtain knowledge into real-time transactions. It can also be used to improve user suggestions based on new trends.
• Apache HealthCare Spark is used to analyze patients' medical records,as well as their prior medical history, and then predict the most likely health issues those patients will face in the future.
• Pyspark is widely used in the media industry as well. 

Conclusion

Pyspark is an industry benefited platform with enormous advantages.It supports the most general purpose and powerful programming languages like python. Python in combination with spark comes with advanced features, built in operations, building blocks that truly benefits the apache spark community to a great extent. Even if you don't have enough information I hope this blog post will help you a lot to get good data insights about the pyspark. 

Related Articles:

• PySpark Filter
• PySpark Join On Multiple Columns