Blog Detail

Covid-19 Tracker Ask Question

preview image DevOps
by Anurag Srivastava, Jan 31, 2018, 6:11:29 AM | 5 minutes |

Log analysis with Elastic stack

In this blog, I have explained how we can configure Elastic Stack and use it to monitor Apache logs. A slow application can trigger a series of escalating calls through an IT organization and whenever the application starts giving issues, we need to troubleshoot it and fix it as soon as possible. In the process, our log files are normally among the first places where we go when we start the troubleshooting process. A modern web application environment consists of multiple log sources, which collectively output thousands of log lines written in unintelligible machine language. If I talk about the LAMP stack set up, then we have PHP, Apache, and MySQL logs to go through. In addition to that, we have framework logs and system logs which collectively creates an endless pile of machine data. Now we have all the information scattered here and there but if we need to get information out of it we need to do the dirty work by using cat, grep, tail, etc.


Now let us talk about Elastic Stack and understand how it can solve the problem here. According to Elastic- Elastic Stack is a great tool to centralize logs from multiple sources, identify correlations, and perform the deep-data analysis. Elasticsearch is a search-and-analytics engine based on Apache Lucene that allows users to search and analyze large amounts of data in near real-time. Logstash can ingest and forward logs from anywhere to anywhere. Kibana is a dash-boarding tool with a user interface that allows us to query, visualize, and explore Elasticsearch data easily. I am not going to explain the installation process here but in the next article, I will try to cover them separately, and there I will try to explain the individual software starting from installation to implementation.


If you want to know the basics of Logstash then please refer to the "Introduction to Logstash" blog where I have explained the basics of Logstash.

The next step after installation is to set up a log pipeline into Elasticsearch for indexing and analysis using Kibana. There are various ways of forwarding data into Elasticsearch, but I am going to use Logstash. Logstash configuration files are written in JSON format under /etc/logstash/conf.d. The configuration consists of three sections: input, filter, and output. I am going to create a demo configuration file 'logs-apache.conf' for apache logs starting from the input section, the filter section is used to modify the input before sending through the output section. For now, I am skipping this part to make it simple to understand the topic. Next is the output section:


 We have created the Logstash configuration file with the input and output section. We need to start Logstash with the new configuration. I am using Ubuntu 17.04 here so run this command as per your Logstash setup in Operating System:

bin/logstash --path.settings /usr/share -f /etc/logstash/conf.d/logs-apache.conf

 



We can check the log data in Elasticsearch by accessing the created index through Logstash:

http://localhost:9200/logs_apache/_search



As we have the Apache logs in Elasticsearch, so the next thing is to display it in Kibana. I will show this process through a series of images.


First, configure an index pattern in Kibana by providing the name of the index then select the time filter field name and click on create.

The next screen shows the index with field and their data types. (I will explain the data types and other details on Elasticsearch in my next article.)



The discover tab of Kibana shows Apache log data with search capabilities.




Now try to search the keywords (as in the below image):



Now we have put everything in place lets play by accessing some local websites to push the Apache access logs. Logstash is already tailing this log, so these messages will be indexed into Elasticsearch and displayed in Kibana. Now play with the data in Kibana by analyzing it.

The Elastic Stack can be used with Beat to fetch files, network and system information, etc. It can be connected to an existing application to monitor the application performance as well as to create a great dashboard to monitor key performance indicators. We can also use it as a stand-alone system by pushing the data from any RDBMS or file-based data source. We can not only show or search data but can also perform analysis on top of that. In case of any confusion please do comment.

Other Blogs on Elastic Stack:
Introduction to Elasticsearch

Elasticsearch Installation and Configuration on Ubuntu 14.04
Log analysis with Elastic stack 
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Wildcard and Boolean Search in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch
Load CSV Data into Elasticsearch
Metrics Aggregation in Elasticsearch
Bucket Aggregation in Elasticsearch
How to create Elasticsearch Cluster

In case of any doubt please leave your comments. You can also follow me on Twitter: https://twitter.com/anu4udilse

If you found this article interesting, then you can explore “Mastering Kibana 6.0”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide” books to get more insight about Elastic Stack, how to perform data analysis, and how you can create dashboards for key performance indicators using Kibana.


Comments (2)

  • user image
    Anurag Srivastava
    Feb 1, 2018, 7:31:09 PM

    ELK is rocking.

  • user image
    Sarita Yadav
    Mar 17, 2018, 5:55:19 PM

    nice blog

Leave a comment

Related Blogs

Introduction to Cloud Computing

Sep 23, 2020, 2:38:00 PM | Anurag Srivastava

Monitoring website uptime using Kibana

Sep 6, 2020, 5:20:33 PM | Anurag Srivastava

Monitoring Application using Elastic APM

Jan 16, 2020, 11:43:29 AM | Anurag Srivastava

Top Blogs

Configure SonarQube Scanner with Jenkins

Jun 21, 2018, 4:58:11 AM | Anurag Srivastava

Execute Commands on Remote Machines using sshpass

Jul 16, 2018, 5:00:02 PM | Anurag Srivastava

Importing MongoDB data into Elasticsearch

Mar 9, 2019, 8:20:38 AM | Anurag Srivastava

Importing MySQL data into Elasticsearch

Feb 9, 2019, 12:06:18 PM | Anurag Srivastava

Configure Jenkins for Automated Code Deployment

Jun 13, 2018, 3:44:01 PM | Anurag Srivastava

Deploying Angular code using Python script

Jun 26, 2018, 4:50:18 PM | Anurag Srivastava