Now let us talk about Elastic Stack and understand how it can solve the problem here. According to Elastic- Elastic Stack is a great tool to centralize logs from multiple sources, identify correlations, and perform the deep-data analysis. Elasticsearch is a search-and-analytics engine based on Apache Lucene that allows users to search and analyze large amounts of data in near real-time. Logstash can ingest and forward logs from anywhere to anywhere. Kibana is a dash-boarding tool with a user interface that allows us to query, visualize, and explore Elasticsearch data easily. I am not going to explain the installation process here but in the next article, I will try to cover them separately, and there I will try to explain the individual software starting from installation to implementation.
If you want to know the basics of Logstash then please refer to the "Introduction to Logstash" blog where I have explained the basics of Logstash.
The next step after installation is to set up a log pipeline into Elasticsearch for indexing and analysis using Kibana. There are various ways of forwarding data into Elasticsearch, but I am going to use Logstash. Logstash configuration files are written in JSON format under /etc/
We have created the Logstash configuration file with the input and output sections. We need to start Logstash with the new configuration. I am using Ubuntu 17.04 here so run this command as per your Logstash setup in Operating System:
bin/logstash --path.settings /usr/share -f /etc/logstash/conf.d/logs-apache.conf
We can check the log data in Elasticsearch by accessing the created index through Logstash:
As we have the Apache logs in Elasticsearch, the next thing is to display it in Kibana. I will show this process through
The next screen shows the index with fields and their data types. (I will explain the data types and other details on Elasticsearch in my next article.)
The discover tab of Kibana shows Apache log data with search capabilities.
Now try to search the keywords (as in the below image):
Now we have put everything in place let's play by accessing some local websites to push the Apache access logs. Logstash is already tailing this log, so these messages will be indexed into Elasticsearch and displayed in Kibana. Now play with the data in Kibana by analyzing it.
The Elastic Stack can be used with Beat to fetch files, network and system information, etc. It can be connected to an existing application to monitor the application performance as well as to create a great dashboard to monitor key performance indicators. We can also use it as a
stand-alone system by pushing the data from any RDBMS or file-based
data source. We can not only show or search data but can also perform analysis on top of that. In case of any confusion please do comment.
Other Blogs on Elastic Stack:
Introduction to Elasticsearch
Elasticsearch Installation and Configuration on Ubuntu 14.04
Log analysis with Elastic stack
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Wildcard and Boolean Search in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch
Load CSV Data into Elasticsearch
Metrics Aggregation in Elasticsearch
Bucket Aggregation in Elasticsearch
How to create Elasticsearch Cluster
If you found this article interesting, then you can explore “Mastering Kibana 6.0”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide”
books to get more insight about Elastic Stack, how to perform data
analysis, and how you can create dashboards for key performance
indicators using Kibana.
You can also follow me on:
- LinkedIn: https://www.linkedin.com/in/anubioinfo/
- Twitter: https://twitter.com/anu4udilse
- Medium: https://anubioinfo.medium.com
Sep 23, 2020, 2:38:00 PM | Anurag Srivastava
Sep 6, 2020, 5:20:33 PM | Anurag Srivastava
Jan 16, 2020, 11:43:29 AM | Anurag Srivastava
Jan 14, 2020, 10:22:34 AM | Anurag Srivastava
Jan 7, 2020, 7:15:34 PM | Anurag Srivastava
Jan 6, 2020, 7:30:13 PM | Anurag Srivastava
Jun 21, 2018, 4:58:11 AM | Anurag Srivastava
Jul 16, 2018, 5:00:02 PM | Anurag Srivastava
Mar 9, 2019, 8:20:38 AM | Anurag Srivastava
Feb 9, 2019, 12:06:18 PM | Anurag Srivastava
Jun 13, 2018, 3:44:01 PM | Anurag Srivastava
Jun 26, 2018, 4:50:18 PM | Anurag Srivastava