Blog Detail

Covid-19 Tracker Ask Question

preview image DevOps
by Anurag Srivastava, Dec 20, 2019, 11:38:31 AM | 4 minutes |

Introduction to Logstash

Logstash is a data collection pipeline of Elastic Stack which is a utility to fetch data from different sources and send it to multiple sources. We can transform the data before sending it to the output. We can explain Logstash in the following points:

  • Data collection pipeline tool.
  • Gathers data from different sources.
  • A tool to collect, process, and forward events and log messages.
  • Data collection, enrichment, and transformation pipeline.
  • Configurable input and output plugins
    e.g. log file, event log, socket, Syslog, Redis, CSV, DB

So as mentioned in the above points, Logstash can be configured to gather data, transform it as per the requirement like convert unstructured data into structured data, and send it to multiple sources like Elasticsearch or any file, etc.

Use Cases:
We can use Logstash for the following purposes:

  • Analyze a large variety of structured/ unstructured data and events.
  • Connect with various types of input sources and platforms.
  • Transform the data and store it into different storage for analytics.

Logstash Pipeline:
A Logstash pipeline has three elements, input, filter, and output. Input and Output plugins are mandatory while the filter is an optional element. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination, please refer to the following diagram:



Input Plugins:
Input plugins can be configured for capturing data from different sources. Example of Logstash input plugins are as follows:

  • File
  • Stdin
  • Beats
  • MongoDB
  • Elasticsearch
  • JDBC
  • Redis
  • Kafka
  • Http

Filter Plugins:
Logstash filter plugins can be used to transform the input data as if we want to remove a certain field or want to convert the unstructured data into structured data. Example of Logstash filter plugins are as follows:

  • GeoIP
  • CSV
  • JDBC
  • Date
  • Grok
  • JSON
  • XML

Output Plugins:
Logstash output plugins are there to send data from Logstash to single or multiple sources. Like if we want to send RDBMS data to Elasticsearch then we can configure the Logstash output plugin. Example of Logstash output plugins are as follows:

  • File
  • Stdout
  • MongoDB
  • Elasticsearch
  • Redis
  • Kafka
  • Http
  • Nagios
  • S3

Structure of Logstash Configuration:
A Logstash configuration file contains input, output, and/or filter plugin.

input {
    ………
}
filter {
   ……….
}
output {
   ……...
}


Test Logstash Pipeline:
To test the Logstash pipeline we can run the following command after installing the Logstash.

bin/logstash -e 'input { stdin { } } output { stdout {} }'


Execute the Logstash Pipeline:
To execute the Logstash configuration file, run the following command:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/crimes.conf


Configuring Logstash to get data
:

Logstash is a data pipeline we can use to configure input to take data from multiple types of data sources, such as files, databases, CSV, or Kafka, and after taking the input, we can configure the output to send data on different sources, such as files, databases, Kafka, or Elasticsearch. Another important feature of Logstash filters, using which we can transform the input data before sending it to the output.

Fetch Apache log data:

For example, if we want to read the apache log data then it can be configured in the following way:

input
{
 file
 {
  path => "/var/log/apache.log"
  type => "apache-access"
  start_position => "beginning"
 }
}
filter
{
 grok
  {
       match => [ "message", "%{COMBINEDAPACHELOG}" ]
  }
}
output
{
 elasticsearch
 {
  hosts => ["localhost:9200"]
 }
}

This way we can configure Logstash to capture data from different sources and can send it to Elasticsearch or any other source destination.


In case of any query please leave a comment so that I can revert you back. After this introduction, you can use Logstash to push MySQL data into Elasticsearch, to load CSV Data into Elasticsearch, to send MongoDB data into Elasticsearch, or for Log analysis with Elastic stack.

Other Blogs on Elastic Stack:

Load CSV Data into Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch

How to create Elasticsearch Cluster
Bucket Aggregation in Elasticsearch

Metrics Aggregation in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch 
Wildcard and Boolean Search in Elasticsearch
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Log analysis with Elastic stack 
Elasticsearch Installation and Configuration on Ubuntu 14.04
Introduction to Elasticsearch


If you found this article interesting, then you can explore “Mastering Kibana 6.0”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide” books to get more insight about Elastic Stack, how to perform data analysis, and how you can create dashboards for key performance indicators using Kibana.


Comments (0)

Leave a comment

Related Blogs

Introduction to Cloud Computing

Sep 23, 2020, 2:38:00 PM | Anurag Srivastava

Monitoring website uptime using Kibana

Sep 6, 2020, 5:20:33 PM | Anurag Srivastava

Monitoring Application using Elastic APM

Jan 16, 2020, 11:43:29 AM | Anurag Srivastava

Configuring Django application with Elastic APM

Jan 14, 2020, 10:22:34 AM | Anurag Srivastava

Introduction to Elastic APM

Jan 7, 2020, 7:15:34 PM | Anurag Srivastava

Why monitoring is important?

Jan 6, 2020, 7:30:13 PM | Anurag Srivastava

Top Blogs

Configure SonarQube Scanner with Jenkins

Jun 21, 2018, 4:58:11 AM | Anurag Srivastava

Execute Commands on Remote Machines using sshpass

Jul 16, 2018, 5:00:02 PM | Anurag Srivastava

Configuring Logstash to send MongoDB data into Elasticsearch

Mar 9, 2019, 8:20:38 AM | Anurag Srivastava