ELK Stack Tutorial

What is ELK Stack and how is it used? 

Elastic Stack, formerly known as ELK Stack is used for log analysis and management. It is a collection of three open-source tools,

• E for Elasticsearch
• L for Logstash
• K for Kibana 

Although these tools are developed separately to serve different purposes, together they provide a complete end-to-end solution for log searching, visualizing and analyzing in real-time. It takes data in any format i.e structured or unstructured. Companies like Netflix, Fujitsu, Stackoverflow have already adopted Elastic Stack. Some of the important features are,

• Centralized store for data
• Near real-time search
• Create reports on top of the data

One more tool called Beats has been introduced recently into the stack by Elastic company. It is a light-weight tool used for shipping data from thousands of machines to either Logstash or Elasticsearch.

Learn new & advanced Architectures in ELK Stack with hkr's ELK Stack online course !

Importance of Log management

Log management is very important for any enterprise because we can monitor our systems 24/7. All the system events, transaction events, etc will be stored as logs. By reviewing these logs, we can know whether the system is functioning properly or not. We can observe if there are any unusual patterns in the logs. It will be helpful to quickly identify and analyze issues and take action accordingly. So we can stay ahead of any potential problems thereby preventing any disruptions to the systems involved.

ELK Stack Architecture 

Here is the architecture of the ELK Stack that shows the end-to-end flow of logs,

Image

Logs are the events generated by any system. The logs from various sources are collected and processed by Logstash. It will then send these logs to Elasticsearch which will store and analyze the data. Using kibana, the logs can be visualized and managed.

Become a Elasticsearch Certified professional  by learning Elasticsearch online course from hkrtrainings!

What is Elasticsearch? 

Elasticsearch is a distributed and powerful search engine built on top of Lucene. It is a document-oriented engine that stores, searches, and analyzes data in large quantities. When we want to store data in elastic search, we index them into documents and then they will be available for search. Elasticsearch reduces the results retrieval time and also provides full-text search.

Elasticsearch installation 

Download the latest installer file from the Elastic website and run the file. Follow the prompts to finish the installation. Elasticsearch will run on port 9200 by default.

For more details on Elasticsearch installation and usage, take a look at our Elasticsearch tutorial here - Elasticsearch tutorial

What is Logstash? 

Logstash is like a pipeline tool that is used for collecting data from multiple sources. It is often called as a data pipeline for Elasticsearch. It is also capable of integrating data from different sources before pushing it to Elasticsearch. Logstash is written in jRuby programming language. Some of the sources that it can collect data are mobile devices, social media, metrics, IoT, CRM, etc. It provides plugins for Github, heartbeat, iMac, etc.

Logstash installation 

First, make sure that Java 8 is installed on your machine. Go to downloads logstash and download the file that is suitable for your operating system. 

ELK STACK Training

• Master Your Craft
• Lifetime LMS & Faculty Access
• 24/7 online expert support
• Real-world & Project-Based Learning

Explore Curriculum

Windows

Download the logstash-7.8.0.zip file. Unzip the contents to a designated folder. Set the Lagstash path in the system environment variables. Open command prompt, go to the installation folder and run the bat file using the below commands,  

Top 30 ELK Stack interview questions and answers for 2020

cd

bin/logstash.bat

Linux

Download the logstash-7.8.0.tar.gz file. Open the terminal and extract the tar file using the below command,

$tar -xzf logstash-7.8.0.tar.gz

Go to the installation folder and run the below commands,

$cd /bin

$./logstash

The in the above commands would be the path of the installation folder.

On successful installation, Logstash runs on the port 9600 by default. To check if the Logstash is up and running or not, open a browser and run http://localhost:9600/ You should get a JSON response like below,

{

   "host":, 

   "version":"7.8.0",

   "http_address":"127.0.0.1:9600",

   "build_date":"2020-06-23T23:30:04+00:00",

   "build_sha":"2d8d6263dd09417793f2a0c6d5ee702063b5fada",

   "build_snapshot":false

}

What is Kibana? 

Kibana is a data visualization tool for the data stored in Elasticsearch. It is written in JavaScript. It provides a simple web-based interface, so the users can access data seamlessly based on Elasticsearch indices. We can perform operations on data using RESTful APIs through Kibana. Certain reports can also be generated like graphs, pie charts, etc, and save them to a dashboard. 

Want to get certified in kibana Learn from our experts and do excel in your career with hkr's kibana Online Course

Kibana installation 

Go to Download Kibana and download the zip file that suits your operating system. Unzip the file to your designated installation path and run the following command,

bin\kibana.bat

Kibana will run on port 5601 by default. For more details about Kibana installation and usage, check out our Elasticsearch tutorial here - Elasticsearch tutorial

Comparison between ELK Stack and Splunk

Here are some differences between ELK Stack and Splunk,

• Splunk is a proprietary software whereas ELK Stack is open-source
• The forwarders in Splunk will configure by default with a variety of sources. In ELK Stack, Logstash can set with data sources by using plugins
• The query syntax for ELK Stack will be Lucene query language which most of the people are familiar with, and it will be easier to work with ELK Stack. The Splunk query
• syntax is something new - SPL(Splunk Processing Language) which is based on MapReduce and developed by Splunk, so it has to learn from scratch
• Both Splunk and ELK Stack store indices in the form on flat files

Subscribe to our youtube channel to get new updates..!

Subscribe

Advantages and disadvantages of ELK Stack 

Advantages

• Logs from various sources will be converting into a single ELK instance
• Get insights into your data at a single location
• Open-source and can installed on-prem
• Easy to scale up vertically and horizontally
• It supports different language clients for Python, Ruby, Java, .Net, etc

Disadvantages

It might become difficult to handle the components when moved to a more complex setup with multiple nodes 
If the complex queries index size is larger than the allocated memory, it results in out of memory exceptions