Blog Detail

preview image DevOps
by Anurag Srivastava, Dec 20, 2019, 11:38:31 AM | 4 minutes

Introduction to Logstash

Logstash is a data collection pipeline of Elastic Stack which is a utility to fetch data from different sources and send it to multiple sources. We can transform the data before sending it to the output. We can explain Logstash in the following points:

  • Data collection pipeline tool.
  • Gathers data from different sources.
  • A tool to collect, process, and forward events and log messages.
  • Data collection, enrichment and transformation pipeline.
  • Configurable input and output plugins
    e.g. log file, event log, socket, Syslog, Redis, CSV, DB

So as mentioned in the above points, Logstash can be configured to gather data, transform it as per the requirement like convert unstructured data into structured data, and send it to multiple sources like Elasticsearch or any file, etc.

Use Cases:
We can use Logstash for the following purposes:

  • Analyze a large variety of structured/ unstructured data and events.
  • Connect with various types of input sources and platforms.
  • Transform the data and store it into different storage for analytics.

Logstash Pipeline:
A Logstash pipeline has three elements, input, filter, and output. Input and Output plugins are mandatory while the filter is an optional element. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination.


Input Plugins:
Input plugins can be configured for capturing data from different sources. Example of Logstash input plugins are as follows:

  • File
  • Stdin
  • Beats
  • MongoDB
  • Elasticsearch
  • JDBC
  • Redis
  • Kafka
  • Http

Filter Plugins:
Logstash filter plugins can be used to transform the input data as if we want to remove a certain field or want to convert the unstructured data into structured data. Example of Logstash filter plugins are as follows:

  • GeoIP
  • CSV
  • JDBC
  • Date
  • Grok
  • JSON
  • XML

Output Plugins:
Logstash output plugins are there to send data from Logstash to single or multiple sources. Like if we want to send RDBMS data to Elasticsearch then we can configure the Logstash output plugin. Example of Logstash output plugins are as follows:

  • File
  • Stdout
  • MongoDB
  • Elasticsearch
  • Redis
  • Kafka
  • Http
  • Nagios
  • S3

Structure of Logstash Configuration:
A Logstash configuration file contains input, output, and/or filter plugin.

input {
    ………
}
filter {
   ……….
}
output {
   ……...
}


Test Logstash Pipeline:
To test the Logstash pipeline we can run the following command after installing the Logstash.

bin/logstash -e 'input { stdin { } } output { stdout {} }'


Execute the Logstash Pipeline:
To execute the Logstash configuration file, run the following command:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/crimes.conf


Configuring Logstash to get data
:

Logstash is a data pipeline we can use to configure input to take data from multiple types of data sources, such as files, databases, CSV, or Kafka, and after taking the input, we can configure the output to send data on different sources, such as files, databases, Kafka, or Elasticsearch. Another important feature of Logstash filters, using which we can transform the input data before sending it to the output.

Fetch Apache log data:

For example, if we want to read the apache log data then it can be configured in the following way:

input
{
 file
 {
  path => "/var/log/apache.log"
  type => "apache-access"
  start_position => "beginning"
 }
}
filter
{
 grok
  {
       match => [ "message", "%{COMBINEDAPACHELOG}" ]
  }
}
output
{
 elasticsearch
 {
  hosts => ["localhost:9200"]
 }
}

This way we can configure Logstash to capture data from different sources and can send it to Elasticsearch or any other source destination.


In case of any query please leave a comment so that I can revert you back. After this introduction, you can use Logstash to push MySQL data into Elasticsearch, to load CSV Data into Elasticsearch, to send MongoDB data into Elasticsearch, or for Log analysis with Elastic stack.

Other Blogs on Elastic Stack:

How to create Elasticsearch Cluster
Bucket Aggregation in Elasticsearch

Metrics Aggregation in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch 
Wildcard and Boolean Search in Elasticsearch
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Log analysis with Elastic stack 
Elasticsearch Installation and Configuration on Ubuntu 14.04
Introduction to Elasticsearch


If you found this article interesting, then you can explore  "Mastering Kibana 6.0", "Kibana 7 Quick Start Guide", "Learning Kibana 7", and "Elasticsearch 7 Quick Start Guide" books to get more insight about Kibana and how we can configure ELK to create dashboards for key performance indicators. You can get the book links on the right pane of the page.

About Author

Anurag Srivastava

Author | Blogger | Tech Lead | Elastic Stack | Innovator |

View Profile

Comments (0)

Leave a comment

Related Blogs

Configuring Django application with Elastic APM

Jan 14, 2020, 10:22:34 AM | Anurag Srivastava

Monitoring Application using Elastic APM

Jan 16, 2020, 11:43:29 AM | Anurag Srivastava

MySQL Performance Tuning

Jan 31, 2018, 6:15:34 AM | Anurag Srivastava

Introduction to Elastic APM

Jan 7, 2020, 7:15:34 PM | Anurag Srivastava

Configure Logstash to send MongoDB data into Elasticsearch

Mar 9, 2019, 8:20:38 AM | Anurag Srivastava

Why monitoring is important?

Jan 6, 2020, 7:30:13 PM | Anurag Srivastava

Snapshot and Restore Elasticsearch Indices

Sep 16, 2019, 5:55:06 AM | Anurag Srivastava

Configure Logstash to push MySQL data into Elasticsearch

Feb 9, 2019, 12:06:18 PM | Anurag Srivastava

Log analysis with Elastic stack

Jan 31, 2018, 6:11:29 AM | Anurag Srivastava

Load csv Data into Elasticsearch

Feb 9, 2019, 6:34:22 PM | Anurag Srivastava

Top Blogs

Configure SonarQube Scanner with Jenkins

Jun 21, 2018, 4:58:11 AM | Anurag Srivastava

Build and deploy Angular code using Python

Jun 26, 2018, 4:50:18 PM | Anurag Srivastava

Configure Jenkins for Automated Code Deployment

Jun 13, 2018, 3:44:01 PM | Anurag Srivastava

SonarQube installation on Ubuntu

May 12, 2018, 4:47:07 PM | Anurag Srivastava

Execute Commands on Remote Machines using sshpass

Jul 16, 2018, 5:00:02 PM | Anurag Srivastava

Why SonarQube is important for IT projects ?

Apr 24, 2018, 2:52:28 PM | Anurag Srivastava

Wildcard and Boolean Search in Elasticsearch

Aug 10, 2018, 7:14:40 PM | Anurag Srivastava

Elasticsearch Rest API

Jul 31, 2018, 6:16:42 PM | Anurag Srivastava

Analyze your project with SonarQube

Jun 2, 2018, 10:49:54 AM | Anurag Srivastava

Configure Logstash to send MongoDB data into Elasticsearch

Mar 9, 2019, 8:20:38 AM | Anurag Srivastava