Elastic Stack, formerly known as ELK Stack is used for log analysis and management. It is a collection of three open-source tools,
Although these tools are developed separately to serve different purposes, together they provide a complete end-to-end solution for log searching, visualizing and analyzing in real-time. It takes data in any format i.e structured or unstructured. Companies like Netflix, Fujitsu, Stackoverflow have already adopted Elastic Stack. Some of the important features are,
One more tool called Beats has been introduced recently into the stack by Elastic company. It is a light-weight tool used for shipping data from thousands of machines to either Logstash or Elasticsearch.
Learn new & advanced Architectures in ELK Stack with hkr's ELK Stack online course !
Log management is very important for any enterprise because we can monitor our systems 24/7. All the system events, transaction events, etc will be stored as logs. By reviewing these logs, we can know whether the system is functioning properly or not. We can observe if there are any unusual patterns in the logs. It will be helpful to quickly identify and analyze issues and take action accordingly. So we can stay ahead of any potential problems thereby preventing any disruptions to the systems involved.
Here is the architecture of the ELK Stack that shows the end-to-end flow of logs,
Logs are the events generated by any system. The logs from various sources are collected and processed by Logstash. It will then send these logs to Elasticsearch which will store and analyze the data. Using kibana, the logs can be visualized and managed.
Become a Elasticsearch Certified professional by learning Elasticsearch online course from hkrtrainings!
Elasticsearch is a distributed and powerful search engine built on top of Lucene. It is a document-oriented engine that stores, searches, and analyzes data in large quantities. When we want to store data in elastic search, we index them into documents and then they will be available for search. Elasticsearch reduces the results retrieval time and also provides full-text search.
Download the latest installer file from the Elastic website and run the file. Follow the prompts to finish the installation. Elasticsearch will run on port 9200 by default.
For more details on Elasticsearch installation and usage, take a look at our Elasticsearch tutorial here - Elasticsearch tutorial
Logstash is like a pipeline tool that is used for collecting data from multiple sources. It is often called as a data pipeline for Elasticsearch. It is also capable of integrating data from different sources before pushing it to Elasticsearch. Logstash is written in jRuby programming language. Some of the sources that it can collect data are mobile devices, social media, metrics, IoT, CRM, etc. It provides plugins for Github, heartbeat, iMac, etc.
Logstash installation
First, make sure that Java 8 is installed on your machine. Go to downloads logstash and download the file that is suitable for your operating system.
Windows
Download the logstash-7.8.0.zip file. Unzip the contents to a designated folder. Set the Lagstash path in the system environment variables. Open command prompt, go to the installation folder and run the bat file using the below commands,
Top 30 ELK Stack interview questions and answers for 2020
cd
bin/logstash.bat
Linux
Download the logstash-7.8.0.tar.gz file. Open the terminal and extract the tar file using the below command,
$tar -xzf logstash-7.8.0.tar.gz
Go to the installation folder and run the below commands,
$cd
$./logstash
The
On successful installation, Logstash runs on the port 9600 by default. To check if the Logstash is up and running or not, open a browser and run http://localhost:9600/ You should get a JSON response like below,
{
"host":,
"version":"7.8.0",
"http_address":"127.0.0.1:9600",
"build_date":"2020-06-23T23:30:04+00:00",
"build_sha":"2d8d6263dd09417793f2a0c6d5ee702063b5fada",
"build_snapshot":false
}
Kibana is a data visualization tool for the data stored in Elasticsearch. It is written in JavaScript. It provides a simple web-based interface, so the users can access data seamlessly based on Elasticsearch indices. We can perform operations on data using RESTful APIs through Kibana. Certain reports can also be generated like graphs, pie charts, etc, and save them to a dashboard.
Want to get certified in kibana Learn from our experts and do excel in your career with hkr's kibana Online Course
Kibana installation
Go to Download Kibana and download the zip file that suits your operating system. Unzip the file to your designated installation path and run the following command,
bin\kibana.bat
Kibana will run on port 5601 by default. For more details about Kibana installation and usage, check out our Elasticsearch tutorial here - Elasticsearch tutorial
Here are some differences between ELK Stack and Splunk,
Advantages
Disadvantages
It might become difficult to handle the components when moved to a more complex setup with multiple nodes
If the complex queries index size is larger than the allocated memory, it results in out of memory exceptions
5th April | 08:00 AM
150 Registered