Azure HDInsight

What is Azure HDInsight?

Azure HDInsight is a fully managed analytics service provided by Microsoft Azure. It helps in creating Hadoop clusters using Linux with Ubuntu. It was originally based on the Hortonworks Data Platform (HDP) Hadoop distribution on Microsoft Azure. Microsoft later announced its Hadoop distribution on July 21st, 2020, during the Microsoft Inspire event. Users won't know much of a difference, but the underlying Hadoop distribution is based on Apache open source components.

Azure HDInsight provides Hadoop as a service on top of the Azure platform. It helps in managing and analyzing big data. It comes with features to extract, transform, and load (ETL) huge amounts of data, data warehousing, IoT, and machine learning. The frameworks that we can install in HDInsight are Hadoop, Apache Spark, Apache Hive, Apache Storm, R, Apache Kafka, and many more.

       Interested in learning Azure Course ? Enroll in our Microsoft Azure Certification Training program now!

Features of HDInsight

Here are some of the features that Azure HDInsight provides. 

High availability

The applications run on thousands of cores and TBs of memory in Azure. It always makes sure that your applications are reliable by monitoring the health continuously. If it finds any failures at any point in time, it recovers from failures automatically.

End to end security

It provides enterprise-grade security and government compliance standards for data stored on clusters. You can even restrict access to sensitive data and applications for some users by defining role-based access policies. It also enables protection with encryption, Active Directory authentication. It has received more than 30 industry-standard certifications for protecting data. 

Global availability

HDInsight clusters can be deployed in more than 26 public regions and Government regions across the world. So it will be easier for users to deploy a cluster in a data center near them.

Open source ecosystem

HDInsight is based on open-source Apache frameworks. So if there is a new stable release of a framework in the market, users will be updated with the newest release of that framework.

Native integrations

It provides native integrations for a lot of Azure services like Synapse Analytics, Data Lake Storage, Blob Storage, Azure Cosmos DB, Event Hubs, and Data Factory. It also allows integrations with other big data certified applications with one-click deployment.

Monitoring

All the Hadoop related jobs generate log files that contain detailed information. They also include cluster configuration, error states, etc. To monitor all the logs of different clusters, users can integrate to Azure Log Analytics, where they can monitor all clusters through a single interface.

Scalable and Economical

Azure HDInsight is flexible to scale up and down as per need. We can reduce a cluster after a few weeks of use and expand in the peak business period. It also gives the economic freedom to pay as per use. Moreover, it's much easier to upgrade to the HDInsight version whenever needed, and it doesn't charge for unused resources.

Highly productive

Azure HDInsight provides many rich, productive tools to use. It provides these tools for Hadoop and Apache Spark to use in the preferred development environments. The development environments include Visual Studio, Eclipse, Python, R, VSCode, Java, and more. Therefore, it enhances developer productivity. 

Azure HDInsight Architecture

Before proceeding to use, it is good to know which architecture is better for Azure HDInsight. The following best practices will help us learn about Azure HDInsight Architecture.

• It is suggestible to move to Azure HDInsight from an on-premises Hadoop cluster through different workload clusters. It is instead of using a single cluster. The single cluster may be much more complex, and it may need to deal with the individual services to make all work together. It is suggested because using a large cluster volume can needlessly increase costs.
• Moreover, it is better to divide data storage into processing centers. Azure Data Lake and Azure Storage are computing and storage areas within the clusters. Therefore, it minimizes the huge storage costs. Also, it enables users to use temporary clusters, scale storage, share and replicate data throughout clusters, and evaluate freely.
• Here, on-demand temporary clusters are also used so that they can remove after the completion of the workload. It is because HDInsight clusters may only be in use for a short period. Therefore, it minimizes the resource costs as we rarely use Azure HDInsights. But while removing the clusters, we are not deleting the storage accounts or the meta-store. These are useful for rebuilding the cluster as per requirement.

Advantages of HDInsight

The following are the key advantages of Azure HDInsight.

• Users can easily run Hadoop related frameworks like Spark Kafka, Storm, etc., on the cloud.
• Azure HDinsight makes it easy to process huge amounts of data.
• Users can easily spin up clusters on Azure HDInsight within minutes. So, the users don’t have to worry about installations or managing infrastructure.
• Users can scale up or down the clusters whenever they want.
• Users can go ahead with the development in any language like Java, .Net, Python, R, C#, and Scala.
• Around 30 Hadoop and Spark applications are available in Azure Marketplace. Users can pick any application and deploy it directly in their cluster.
• Users can build applications in tools like Jupyter, Zeppelin, Visual Studio, Eclipse, and IntelliJ.
• HDInsight can integrate with other Azure services like Data Lake Storage, Data Factory, etc.
• Users can scale down the cluster whenever there is not much workload. So, the users can pay only for what they use.
• It can perform interactive queries over data in any format like structured or unstructured.
• Users can process streaming data from various devices in real-time.
• It offers high availability along with high scalability.
• In Azure HDInsight, it is easier to convert high-volume data. 
• HDInsight makes data management much simpler.
• It is very cost-effective, where it minimizes costs through developing on-demand clusters.
• Also, it offers a high-speed connection through a flat storage system between the blob storage and nodes.
• Further, it provides better performance and improves productivity by dividing the computing and storage services.

Types of clusters in HDInsight

Azure HDInsight offers several cluster types that serve different purposes. Users can customize the cluster according to their preference by adding components, features, languages, and utilities. Here are the available cluster types.

Apache Hadoop

Apache Hadoop is a fully managed Hadoop cluster on Azure. It consists of HDFS for storage and YARN for resource management. It also consists of MapReduce to process big data. MapReduce can be implemented in various languages like Java, C#, and Python.

Apache Spark

This cluster sources Apache Spark, which enables in-memory processing. This boosts the performance of big data analysis. These Spark clusters are compatible with Azure Storage and Azure Data Lake Storage. It also provides Jupyter and Apache Zeppelin notebooks for development.

Apache HBase

HBase is a NoSQL columnar database from Apache that can store huge amounts of unstructured and semi-structured data. Azure provides HBase as a managed cluster integrated into the Azure environment. It implements the scale-out architecture of HBase. The data stored through the Apache HBase cluster will directly get stored in Azure Storage. 

ML Services

This cluster provides on-demand access for scalable and distributed methods of analytics on HDInsight. It offers R based analytics on the data stored in either Azure Blob or Data Lake storage. It offers 8000+ open-source R packages. The nodes make it possible to run R scripts. The resultant predictions or models can be downloaded for local use.

Apache Storm

Apache Storm is a distributed and fault-tolerant computation system used to process streaming data in real-time. Apache Storm on HDInsight guarantees that every message is fully processed. Users can monitor and manage Storm topologies from the browser using the Storm UI.

Apache Interactive Query

The Interactive Query in Azure is also known as Apache Hive LLAP. It offers in-memory caching that makes the Apache Hive queries run faster. Users can use this cluster to run high-speed queries on data stored in Azure storage and Azure Data Lake Storage. This cluster only contains Hive service.

Apache Kafka

Apache Kafka is used for building data pipelines on streaming data. It offers message queue functionality where data can be published and subscribed. The backing store for Kafka is Azure Managed Disks. The cluster provides all the Kafka functionalities like partitioning, racks, replication, etc.

       Interested in learning Azure Course ? Enroll in our Microsoft Azure 500 Certification Training program now!

Creating an HDInsight cluster

Navigate to the Azure portal https://portal.azure.com/ and login to your account. Click on '+ Create a resource' from the top menu. Click on 'Analytics' and select the 'Azure HDInsight' option. The 'Create an HDInsight cluster' page will open. Select the 'Azure' for the subscription field. Select your resource group and give a unique name for the 'Cluster Name' field.

Select a region for the 'Location' field and choose a cluster type from the drop-down. Give the username and password for the cluster. Set 'sshuser' as the Secure Shell (SSH) username. Select the checkbox for 'Use cluster login password for SSH' field. Click on the 'Next: Storage >>' button. Select 'Azure Storage' from the drop-down for 'Primary storage type'. Choose the 'Use access key' radio button. Give your Storage account name and Access key. Click on 'Next: Security + networking >>' button.

All the options under the networking tab are optional. So if you want to configure anything, set those options, and click on 'Next: Configuration + pricing >>' button. Select the node size and number of nodes for each node type. Click on 'Review + create >>' button. You can view all the settings configured until now. Go through them once and click on 'Create'. It takes around 20 minutes to create the cluster. Once the process is complete, you will get a notification.

Install apps on HDInsight cluster

Navigate to the Azure portal https://portal.azure.com/ and login to your account. Click on '+ Create a resource' from the top menu. Click on 'Analytics' and select the 'Azure HDInsight' option. Select your HDInsight cluster from the list. Your cluster page will open where you can see all the information on your cluster.

Go to 'Settings' and select 'Applications' option. You will get a list of already installed applications. If you did not install any application, then the list will be empty. To add a new application, click on '+Add' from the menu. Select an application that you want to install in your cluster. Accept the terms and conditions. The installation status can be viewed on portal notifications. Once the application installation is complete, it will appear in the Installed Apps list.

Scenarios for using HDInsight

Batch processing (ETL)

ETL or Extract, Transform, Load is a process that involves the extraction of different types of data from diverse data sources. It may include various types of data. After this, the data is converted to a structured or required format and loaded into data storage. This modified data is useful for data warehousing or any other purpose.

Data warehousing

We use data warehousing to store large data volumes so that we can recover or analyze the data at any time. HDInsight is useful for executing various interactive queries over any data format, whether structured or unstructured. Entities manage these large-scale data warehouses to analyze them and make informed decisions based on the analysis. 

Internet of Things (IoT)

Today we can see many smart devices around us that make our life better and easier. These smart devices based on IoT help us create very small decisions easily related to our devices. IoT connects multiple devices to share data and analyze it for better communication. It needs processing and analysis of incoming data from various smart devices. This large volume of data is a major part of IoT and managing it is crucial for the proper functioning of smart devices. 

Azure HDInsights can efficiently process large data volumes coming from multiple IoT-enabled devices. 

 

Hybrid

The hybrid cloud environment combines private and public clouds useful for workflows. Business entities can benefit from flexibility, security, and scalability. HDInsights is useful to extend the enterprise's existing on-premise infra to cloud platforms. It gives better performance, processing, and analysis in the hybrid environment.

 

Thus, these are the top scenarios for using Azure HDInsight.

Azure HDInsight Pricing

The pricing system of using Azure HDInsight is based on the number of clusters and nodes we use. It also varies from region to region. Below are the pricing details:-

For Central India (pricing/hour):-

• For Hadoop, Spark, Interactive Query, Storm, and HBase, the pricing will be Base price/ node-hour + Rs. 0/core-hour.
• For Azure HDInsight ML Service, the pricing will be Base price/node-hour + Rs. 1.153/core-hour.
• For the Enterprise Security Package in HDInsights, the pricing system includes- Base price/node-hour + Rs. 0.721/core-hour.

For Central US (pricing/hour):-

• For Hadoop, Spark, Interactive Query, Storm, and HBase, the pricing will be Base price/ node-hour + $ 0/core-hour.
• For Azure HDInsight ML Service, the pricing will be Base price/node-hour + $ 0.016/core-hour.
• For the Enterprise Security Package in HDInsights, the pricing system includes- Base price/node-hour + $0.01/core-hour.

Conclusion

Azure HDInsight is the most popular Hadoop distribution on the cloud. LG, Roche, Virginia Tech, Blackball, Leeds Teaching Hospital, Fusionex, iTrend are some of the top companies that have already adopted HDInsight in their business. Through this post, you have learned how to create HDInsight clusters. Go ahead and try creating different clusters that suit your business needs. Azure provides Ambari UI to monitor and manage all the clusters.

Related articles :

Azure Load balancer

Aws vs Azure