Hadoop Tutorial

What is Hadoop?

Hadoop is an Apache product and this is nothing but a collection of open-source software framework utilities that provides a network of many computer devices to solve complex problems which involve a massive amount of data and perform computations. This offers a storage framework to process the big data modules with the help of MapReduce programs. This Hadoop also offers massive data storage for any kind of data, enormous data processing power, and allows users to handle enormous concurrent tasks or jobs. This Hadoop framework is written using programming languages like Java, but never makes use of OLAP or online analytical processing. This is used for batch or offline data processing. Hadoop is used by major companies like Facebook, Yahoo, Google, LinkedIn, and many more to store a large volume of data.

Why use Hadoop:

The following are the few advantages of using Hadoop technology:

1. One of the important advantages of Using Hadoop is its cost-effective nature when compared to other traditional database technologies in storing and performing data computations.

2. Hadoop comfortably accesses different kinds of business solution data and also proved its benefits in decision making.

3. This Hadoop also acts as an enabler for social media, log processing, data warehousing, emailing, and error detection.

4. By mapping the data sets wherever it is placed, Hadoop technology minimizes the time taken to unfolding any data sets. It can also work on a large number of petabytes of data on an hourly basis and makes it super-fast.

Big data Overview:

Now it’s time to know about one of the popular Hadoop technologies are Big data,

In the current tech market, we can see new technologies, communication media, devices, networking sites, and the amount of data production has been growing rapidly. The amount of data produced by us every day is about 20 gigabytes. So it’s always good to process, integrate, transfer, and produce error-free data. To overcome this hustle now we have Big data technology.

Let’s begin with the definition:

What do you know about big data?

Big data is nothing but a collection of large data sets that cannot be processed by using any traditional computing mechanisms. Big data is not a single technique or any type of tool, but it consists of various tools, techniques, and data frameworks to become a complete subject.

Big data composed of different devices and applications. The following are some of the major fields which come under the big data umbrella.

1. Black box data format: It is a kind of component mainly used in helicopters, jets, and airplanes. This tool helps to capture the voices of the flight crews, recording information of microphones, performance information of the aircraft device, and earphones.

2. Social media data format: The social media platforms like Facebook and Twitter contain information and millions of people view the post across the globe.

3. Stock exchange data format: The stock exchange data format consists of information only about “Sell” and “buy”. Here the decision is made on different companies' shares made by the customer.

4. Power grid data format: the power grid data consists of information that is consumed by a particular node with respect to base station methods.

5. Transport data format: The transport data composed of the model, distance, availability of the vehicle, and capacity information.

6. Search engine data format: the search engine helps to retrieve lots of data from multiple databases.

Big data offers many benefits like huge data volume storage, high velocity, and an extensive wide variety of data. These data can be differentiated into three different types:

1. Structured data – an example is Relational data.

2. Semi-structured data – an example is XML data.

3. Unstructured data – an example is Word, Text, notepad, PDF, and media logs.

Benefits of big data:

The following are the few benefits of big data:

1. Using information like social media networks such as Facebook, and the marketing agencies are learning about how to respond to their promotions, campaigns, and other advertising forums.

2. Using the various information like social media preferences and product perceptions of the respective customers, retail organizations, and product companies to plan their production.

3. Using the previous medical history such as patients’ details, hospitals are able to offer better and quick services.

Big data technologies:

Big data technologies are considered to be important in offering more accurate analysis, and this may lead to decision making which results in greater operational efficiency, reduction of cost, and also reduces the risk. There are various popular technologies like Amazon, IBM, and Microsoft, etc. to handle big data methods. Here we are going to differentiate into two technology classes:

1. Operational big data: In this section, the systems like MongoDB, offers real-time operational capabilities, workloads interactives to capture and store the data. NoSQL big data system is also designed to take advantage of cloud computing architecture and massive computations to run data inexpensively and efficiently.  This type of operational big data workloads are much easier to manage, faster implementations, and cheaper to use.

2. Analytical big data: These include systems like MPP or massively parallel processing database systems and Map-reduce that offer large analytical capabilities and complex analysis reductions. Whereas Map-reduce offers a new data analysis method that is complementary to the SQL data base capabilities.

The major challenges of big data:

1. Data capturing

2. Data curation

3. Storage

4. Searching

5. Sharing of data

6. Transfer a large volume of data

7. Analytical purpose

8. Presentation techniques.

Hadoop Administration Training

• Master Your Craft
• Lifetime LMS & Faculty Access
• 24/7 online expert support
• Real-world & Project Based Learning

Explore Curriculum

Big data solutions:

The following are the important approaches offered by Big data solutions such as;

1. Traditional approach:

In this traditional approach, an enterprise will consist of a computer which is used to store and process the data. For storage purposes, the developer or programmer will take the help of their database vendors like Oracle, IBM, and PeopleSoft. In this type of approach, usually, the user interacts with the device, which helps in data storage and analysis functions.

2. Google solution:

Google solved problems like storing and processing data using an algorithm called MapReduce. This Map Reduce divides the multiple tasks into smaller tasks and assigns these tasks to many computers, and collects the various results. These results are integrated into result databases.

3. Hadoop solution:

Hadoop solution is provided by Google, Doug cutting, and the team developed an open-source database project known as HADOOP.

This Hadoop runs the applications using an algorithm called Map Reduce, here data is processed in a parallel way. Hadoop is mainly used to develop various applications and helps to perform complete statistical analysis functions.

Hadoop Architecture Overview:

In this section, we are going to explain the basic functionalities, components, and work nature.

The following diagram explains the architecture overview of Hadoop:

Hadoop architecture offers reliable, scalable, and flexible distributed frameworks for a system cluster that offers efficient storage capacity and local computing power by commodity hardware resources. Hadoop architecture is the same as Master-slave architecture for data transformation and analyzing large volumes of data sets by using Map reduce Paradigm. There are three important Hadoop components available and they play different roles in the architecture are;

1. Hadoop distributed file system or HDFS – this is a type of pattern used in UNIX file systems.

2. Hadoop Map reduce components

3. Resource negotiator or YARN (Yet another resource negotiator).

When you start to learn about Hadoop architecture, every layer in the Hadoop architecture requires knowledge to understand various components. These components perform operations like designing the Hadoop cluster, performance tuning, and chain responsible to perform data processing. As I said earlier Hadoop architecture follows master-slave architecture design containing master node or name node, where master node is used for parallel processing of the data by using a job tracker.

Hadoop installation guide:

Follow the steps which are used to install 2.4.1 in pseudo-distributed mode.

Step 1: Setting up of Hadoop environment

Here you can start the Hadoop environment variable with the help of appending commands stored in the ~/.bashrc file.

Step 2: Here you can find all types of Hadoop configuration files in the location: “$HADOOP_HOME/ETC/Hadoop”. This file requires making any changes in the configuration files according to the Hadoop infrastructure:

$ cd $ HADOOP_HOME /etc/Hadoop  //Hadoop home page

 In order to develop any program in JAVA, first, you have to reset the java environment variables stored in Hadoop-env. sh, file just by replacing the JAVA_HOME value.

Export JAVA_HOME = /usr/local/jdk1.7.0_71.

Some of the important XML files will be used while installing Hadoop:

1. Yarn-site.xml

This file system is used to configure the Yarn into the Hadoop environment. First, you need to open the yarn

-site.xml file and add the properties like , tags.

2. Mapred-site.xml:

This file system is used to specify the Map-reduce framework. By default, we use Hadoop that contains a template of yarn-site.xml. First of all, it is necessary to copy the file from map red site.xml. Template to mapred.site.xml files with the help of the following command.

$ cp mapred-site.xml. Template mapred-site.xml.

Verifying Hadoop installation:

The following are the steps required to verify the Hadoop installation:

1. Name node setup:

Set up the name node using the below commands in “hdfs namenode –format” as follows:

$ cd ~

$ hdfs namenode –format

2. Verifying the Hadoop dfs:

The following are the commands used to get started dfs. Execute the below command to start the Hadoop file system:

$ Start –dfs. sh

3. Verifying the yarn script:

The following command is used to start the yarn script. Execute the following command to start YARN daemons.

$ Start –a yarn. sh

4. Accessing Hadoop on the browser:

The default port number to access any Hadoop system is 50070. Here you need to make use of the following URL to work with the Hadoop service.

5. Verify all the applications for the cluster:

Here the default port number used to access all kinds of applications is 8088. You should use the following URL to visit the service:

Hadoop- HDFS overview:

Hadoop file systems are specially developed by using distributed file system management design. This is usually run on commodity hardware. HDFS module is a highly fault-tolerant and low-cost hardware design. HDFS consists of a large volume of data and offers easier data access. To store a large amount of data, the files are used to store these data files across multiple machines. The HDFS file is usually stored in a redundant system to overcome possible data losses and reduce failure. HDFS also makes any application that is available for parallel processing.

Important features of HDFS:

The following are the important features of HDFS namely:

1. HDFS is suitable for distributed data processing and storage management.

2. Hadoop HDFS also provides a command interface to mainly interact with multiple systems.

3. The built-in servers like name node and data node help users to easily check the cluster status.

4. Offers data streaming to access various file system data.

5. HDFS module offers mechanisms like file permission and authentication service.

Goals of HDFS module:

The following are the important goals of the HDFS module:

1. Helps in fault detection and recovery: HDFS module consists of a large number of commodity hardware resources and components failures. So HDFS should support various mechanisms to quick defect, recovery, and automatic fault detections.

2. Helps to store huge data sets: HDFS consists of hundreds of node clusters, which are mainly used to manage the application database that holds huge data sets.

3. Offers hardware capabilities at data: Here a requested task will be done efficiently only during the time of computational tasks. This happens especially where large volumes of data sets are used; this reduces the network data traffic and increases them throughout the process.

Subscribe to our youtube channel to get new updates..!

Subscribe

Important operations of HDFS:

The following are the major operations of the HDFS module:

1. Starting with HDFS:

This is an initial stage, where you need to format the configured file system (HDFS file types), open name node cluster or HDFS server, and executes the below command:

$ Hadoop namenode   - format

This command formats the HDFS cluster node, and then it will start the distributed file system. The following command will start the name node in the cluster;

$ start – dfs. Sh

2. List the files in HDFS:

This step will occur, once you load the information into the server system, after this you are able to find the list of files in a record, or directory, a file status by using “ls”. The following syntax helps to pass the data to a directory or a file name in the argument.

$ $ HADOOP _ HOME /bin/hadoop fs   -ls

3. Inserting the data into HDFS:

Consider that we have data in the file system called file.txt in the local system directory and only saves the HDFS file system. The following are the important commands used to perform these operations:

$ $HADOOP _ HOME /bin/Hadoop fs  -mkdir /user/input

Now it’s time to transfer and store data files from the local systems to the HADOOP file systems by using the below command;

$ $ HADOOP_HOME / bin/Hadoop fs –put /home /file.txt /user/input

 To verify the file systems using the command:

$ $HADOOP_HOME /bin/hadoop fs –ls /user/input

4. Retrieving of data from HDFS:

Usually, a file in HDFS is called outfile. Below are the simple commands used to perform the operations in the Hadoop file system.

Firstly, view the data from HDFS by using the cat command:

$ $HADOOP _HOME /bin/ hadoop fs –cat /user/output/outfile

Now gets the file from HDFS to the local file system using the “get” command

$ $HADOOP _HOME /bin/hadoop fs –get/ user/output /   / home /hadoop_tp/

5. Shutting down the HDFS file system:

With the help of the below command you can shut down the file system:

$ Stop-dfs.sh

HADOOP module – YARN:

YARN is nothing but yet another resource manager that helps to take the programming to the next level and makes this programming application interact with another application Hbase, and SPARK, etc. One more important point to be considered here is that different YARN applications can exist on the same cluster for example Map Reduce, Hbase, and Spark run at the same time to offer manageability and clustering utilization benefits.

Important components of YARN:

1. Client: This component is used to submit the Map-reduce jobs.

2. Resource manager: this component is used to manage the resources across the cluster.

3. Node manager: used for launching and monitoring the computer containers on various machines in the cluster.

4. Map Reduce application master component: this will check the tasks which are running the Map-reduce job. The application master and the Map-reduceMap reduce tasks both will run in containers scheduled by the resource manager and managed by node managers.

In the previous Hadoop version, Job tracker and task tracker were used, which were used to handle resources and check the progress management. The latest version of Hadoop 2.0 consists of resource manager and Node manager to overcome the short fall of the Job tracker and task tracker components.

Benefits of YARN:

The following are the important benefits of YARN:

1. Offers high-level scalability: Map-reduce 1.0 offers scalability that consists of 4000 nodes and 40000 tasks, but here YARN is designed for 10,000 nodes and 1 lakh major tasks.

2. Better Utilization: here the node manager helps to manage a pool of hardware resources, other than fixing the number of designated tools so this increases the utilization.

3. Multitenancy: Different versions of Map-Reduce will run on YARN, this makes the upgrading process of Map Reduce.

Hadoop – command references:

There are many commands available in Hadoop which are located in the “$HADOOP_HOME/bin/hadoop   fs” file. Running this command will list all the commands with no additional arguments and all these arguments will be stored in the Fs Shell system.

Below are the few important commands and its description:

1. –ls  = this command will list all the contents of the directory which is specified by path, names, owner, size, permissions, and modified date for each individual entry.

2. –lsr = this command behaves like –ls command and displays the entries in all sub directories of the path.

3. –du = offers disk usage in terms of bytes, where this will consist of a path, and the file name is reported with the HDFS protocol prefix.

4. –dus = this command will print the summary of total disk usage of all file types and directories in the path.

5. –mv = this command helps to move the file or directory which is indicated by using src or dest in HDFS.

6. –cp = this command copies the file or directory specified by src to destination.

7. –rmr = this command removes the file or directory specified by path. And this will also help to delete the child entries.

8. –put = this command will copy the file or directory from the local file system which is identified by local src to a destination within the DFS.

9. –movefromlocal = this copies the file or directory from the local file system which is specified by localsrc to dest within the HDFS and that deletes the local copy files.

10. –get [-crc] = this command is used to copy the file or directory in HDFS module which is specified by src file to the local file system by localDest.

Hadoop – MapReduce:

MapReduce is just like the processing of techniques and programming models used for distributed computing based in the Java programming environment. Here the Map-reduce algorithm consists of two very important tasks; they are mapping and reducing the set of data and converting them into another set of data. Here the individual elements are broken down into tuples or (Key/value pair). The major advantage of Map Reduce is that it offers easy to scale data processing using multiple computing nodes. Under this Map Reduce model, the data processing data primitives are called mapping and reducing. The simple scalability attracts many programmers to make use of the Map-Reduce model.

The Map-Reduce algorithm:

The following are the key points of the Map-Reduce algorithm:

1. The Map-Reduce algorithm is developed on the basis of sending the computer to where the data sets reside.

2. In general, the Map-reduce algorithm can be divided into three stages such as map stage, reduce stage, and shuffle stage.

3. During the time of the Map-reduce job, the Hadoop program sends both the map and reduce the tasks to the appropriate cluster.

4. Here the framework manages all the appropriate details of data processing such as issuing tasks; verify the task completion, and data copying around the node cluster.

5. Most of the computing performance takes place on the data nodes that locate on local disk that will reduce the networking traffic.

6. Once you finish the given tasks, the server on the cluster node collects and reduces the data to form an appropriate result and sends them back to the Hadoop server.

The following diagram explains the Map Reduce Algorithm:

Map Reduce algorithm terminology:

1. Payload: here these applications implement both the reduce and map function, and form the core of the Hadoop job.

2. Mapper: This mapper maps the input key-value pairs to set the intermediate type of key-value pair.

3. Named Node: this node manages the HDFS or Hadoop distributed file system.

4. Data Node: This node processes the tasks where data is presented in advance.

5. Master Node: This is the node where Job tracker runs and accepts the job requests from respective clients.

6. Slave Node: This node helps to run Map and reduce programs.

7. Job tracker: This schedules the job and tracks to assign the relevant job to the task tracker.

8. Job: Job is a type of program which executes the mapper and reducer datasets across the cluster node.

9. Task: Task is an execution of a mapper or reducer in a slice of data sets.

10. Task attempt: A particular instance helps to attempt the execution of a given task on a slave node.

Conclusion:

When you hear the word Big data, your head turns and starts to think about it. I hope you got my point why I use these words because Big data technology is a popular product and developed with the help of Hadoop technology. Why do you need to learn Hadoop big data? The answer would be, to get appropriate and error-free data. As per the latest research, we generate almost 20 GigaBytes of data per day, so it’s always good to process the data to get accurate data sets. From this Hadoop tutorial, you will be able to learn, map-reduce, YARN, Big Data, and Algorithm  that makes you an expert with the technology. You can expect huge job openings for Big Data analysts across the globe.