What is Apache Pig

What is Apache Pig?

Pig is generally an abstraction over the Map reduce. It is used for analyzing the large volume of data sets and also represents the data flows and used with Hadoop. The Hadoop pig will allow us to perform the data manipulation related operations in Hadoop.

Pig is a platform that provides a high level language for the purpose of writing the data analysis programs. Pig Hadoop was developed by yahoo and this is specifically used for performing the data administration operations. To perform the data analysis, it has to be written using the Pig Latin. It includes several operators which are represented as personalized functions for reading, writing and processing the data that has been developed by the programmers or the developers.

Why Apache Pig?

I think you might have known that apache pig works with Hadoop which is based on the Java programming language. The Apache Pig came into existence when there was inconvenience for the programmers where they were not comfortable with Java and they were facing a lot of issues when working with Hadoop. This is specially when the map reduce tasks are being performed.

After the development of Apache Apache Pig and its implementation, it has become an easy task for the programmers to work with the map reduce task without any requirement of coding with the Java programming language.

The Apache pig utilizes the multi query approach which helps in reducing the length of the code and is also resulting in the reduced development time by at least 16 folds.

Pig Latin language is used for writing the scripts in pig. It is much similar to SQL and easy when compared with pig.

History of Apache Pig:

Apache Pig has been developed by Yahoo in 2006 as a research project that is used for creating and executing the mapreduce jobs that are available in the dataset. Later on in the year 2007, Apache Pig was open sourced through the Apache incubator. The Apache Pig was first released in the year 2010.

Apache Pig Architecture

The Architecture of Apache Pig is divided into two main components. They are represented below.

a. Pig Latin which is the language used for writing scripts.
b. A Runtime environment that is used for running the Pig Latin programs.

Let us discuss each of the components clearly.

A Pig Latin program includes the set of the transformations or the operations that are applied to a particular input data to deliver the output. These operations usually show the data flow which will be translated into an executable presentation by the Hadoop Pig execution environment. The results obtained from the transformations will be the jobs which the programmer is not really known for. Apache Pig allows the programmers to keep an eye and focus on the data instead of the nature of execution.

Pig Latin is a programming language which includes certain keywords that are used for the data processing like filter, group and join.

Become a Hadoop Certified professional by learning this HKR Hadoop Training 

Execution modes:

a.Local mode: The Hadoop Apache Pigk language will be running on a single Java Virtual Machine and we will also make use of a local file system in this mode. The local mode is used for the analysis of small volumes of data or data sets using Apache pig and Hadoop.

b.Map Reduce mode: All the queries will be translated into the map reduce jobs which will be running on the Hadoop cluster. This cluster is called the pseudo cluster or fully distributed cluster. The map reduce mode is used for running the large volume of data sets using Apache Pig in Hadoop.

How to download and install pig:

Let us know the download and install process of Hadoop pig. Before you actually start working with this process, you need to have Hadoop installed in your system. You need to change the user to 'hduser' which is nothing but the ID that is used for the configuration of hadoop, which also allows you to switch to the different user ID that has been used during the configuration of hadoop.

Step-1: The primary step is to download the latest Pig Hadoop release using the mirror sites that are available in the link below.

http://pig.apache.org/releases.html

You need to select tar.gz file to download.

Step-2: After the download is completed, you need to navigate to the directory that contains the tar file and move it to the location in which you want the Pig to be setup.In this case, you will be moving to /usr/local.

The next item is to move the directory that includes the Hadoop pig files.

Cd /usr/local

You need to extract the contents present in the tar file

sudo tar -xvf pig-0.12.1.tar.gz

Step-3: You need to use the ~/.bashrc to add the environment variables that are related to pig.

Open up the ~/.bashrc file in any of the text editor and make the below set of modifications.

export PIG_HOME=

export PATH=$PIG_HOME/bin:$HADOOP_HOME/bin:$PATH

Step-4: The next step is to source the environment using the command.

. ~/.bashrc

Step-5: Recompilation of PIG is required to support Hadoop 2.2.0. You can use the below step to do the recompilation.

Go to PIG home directory

cd $PIG_HOME

sudo apt-get install ant

With this, the download will start and requires certain time as per the internet speed.

sudo ant clean jar-all -Dhadoopversion=23

It is important for the system to be connected to internet as the recompilation process includes multiple components.

You can rerun the same command if the processing has been stuck at some point. You can use the Ctrl+C command and rerun the same command.

Step-6: Test the installation command now.

pig -help

Basic Types of Data Models in Pig:

There are 4 different types of data models in Pig. They are:

• Atom: An atom is a main atomic data value which can store both number and string. It can be represented in the form of a string.
• Tuple: A tuple is referred to as an ordered set of fields.
• Bag: Bag is referred to as the collection of tuples.
• Map: Map is referred to as the set of key/value pairs.

Features of Apache Pig:

There are many features offered by Apache Pig. Let us discuss some of the important features of Apache Pig.

•  Ease of programming: Writing the scripts in Pig Latin language is easy as Pig Latin language is similar to SQL.
•  All types of data handling: Apache pig is capable of storing and analyzing the large volumes of data -being structured or unstructured data and will store it in the   hadoop distributed file system.
•  In-built operators: Apache is capable of handling and performing different set of operators i.e the functionalities like filter, join, sort,etc.
•  Automatic optimization: Apache Pig is capable of optimizing the tasks automatically. With this, the programmers will focus on the semantics of the language.
•  Extensibility: extensibility is one of the key features that Apache Pig holds. It provides the flexibility to the users to develop their own set of functions to write,   read   or process the data by making use of the operators that are existing.
•  User-defined functions: This feature is a unique one which provides the users with the flexibility to create their own set of user defined functions.

Applications of Apache Pig

• Apache pig is capable of handling and processing large volumes of data.
• It is capable of providing its extensive support to prototyping and ad-hoc queries.
• It is capable of performing data processing in the search platforms.
• Apache pig is used in the telecom industries to identify the call data of the user.
• It helps in the processing of time-sensitive data loads.

Get ahead in your career with our Hadoop Tutorial!

Pig Commands:

Below are some of the frequently used commands in Apache pig.

• Load: The load function is used for reading the data from the particular system.
• Store: The store function is used for writing the data to the file system.
• Filter: It is used for applying the predicates and removes the set of the records for which the value returns to be false.
• Join: The join command is used for joining two or more inputs based on the key value.
• Order: The order command is used for sorting the records based on the key value.
• Distinct: The distinct command is used for removing the duplicate records.
• Union: The union command is used for merging the data sets.
• Limit: The limit command is used for setting up a limit for the number of the records.

Apache Pig Vs MapReduce:

Apache Pig is a high level, data flow language that uses the multi-query approach. Mapreduce is a data processing low level language that requires exposure to Java. Apache Pig does not require any compilation as every Pig operator will be converted to map reduce jobs. The map reduce jobs require a compilation process. Join operation is easy to use in Apache but it is difficult to use the join operation in MapReduce.

Apache Pig Vs SQL:

Apache Pig uses the Pig latin language which is a procedural language while SQL is a declarative language. Schema is required in SQL while it is optional in Apache Pig. Apache Pig uses the nested relational data model while SQL uses flat relational data model. There is limited opportunity to perform query optimization in Apache Pig while query optimization can be done easily in SQL.

Apache Pig Vs Hive:

Apache Pig uses the Pig Latin language while Hive uses the HiveQL or HQL language. Both the languages are different as HQL is a query processing language and pig latin is a data flow language. Hive is capable of handling structured data while Apache Pig is capable of handling all kinds of data –structured, unstructured or semi-structured data.

Conclusion:
In the current times, there is a high demand and scope for Hadoop and cloud computing. It is important to know the sub components in Hadoop like Hive, Pig, etc. With the increase in the demand there is an increase in the number of employment opportunities leading to betterment in the careers. Anyone who is willing to pursue their career in cloud computing, then Hadoop and Pig training will help you gain all the knowledge and expertise in the subject.

Related articles

• What is Hadoop
• Apache Spark Vs Hadoop
• Map Reduce in Hadoop
• What is Hive