Blog Detail

preview image DevOps
by Anurag Srivastava, Apr 6, 2019, 8:41:41 PM | 3 minutes

How to create Elasticsearch Cluster

Many times people ask me the question about configuring an Elasticsearch cluster where they can distribute the load on different servers. There are too many other questions like how to decide on server configuration, which architecture to follow, how many shards to keep, etc, etc. I will explain these in other blogs but here I want to focus on just to configure an Elasticsearch cluster using three different servers on AWS running CentOS.

I am assuming that Elasticsearch is installed and running on all of the servers using which you want to create the cluster. If you have any doubt on that you may refer to my previous blog on installing Elasticsearch. So before creating the cluster you need to ensure that all of your Elasticsearch nodes are running on the same version in order to avoid any further version compatibility conflict. Here in three nodes, I want to create a master cum data node and two data nodes but you can create two nodes as the data node with one dedicated master node. It all depends upon the type of infrastructure you are having and the kind of requirement you are having.

Now let's configure the master node first and for that, we need to stop the Elasticsearch service on the server and open “elasticsearch.yml” file. You need to follow the same for all other nodes as well.

To stop the service execute the following command

sudo systemctl stop elasticsearch.service


To open the configuration file:

sudo vim /etc/elasticsearch/elasticsearch.yml


Now change the cluster name with a meaningful name and ensure that the same cluster name is configured in all of your Elasticsearch nodes:

cluster.name: bqcluster


Also, add node name and for each node, we need to provide different names through which we can identify it.
First node:

node.name: bqnode-1

For other nodes, we can give the node names as "bqnode-2" and "bqnode-3"


Now set the first node as master node using the following configuration:

node.master: true
node.data: true
node.ingest: true


In the other two nodes you can add the following configuration:

node.master: false
node.data: true
node.ingest: true


Now set the network host using the private IP of the server:

network.host: [your_private_ip_address]


Now set the initial list of hosts to perform discovery under discovery section on elasticsearch configuration file:

discovery.zen.ping.unicast.hosts: [node1_private_ip, node2_private_ip, node3_private_ip]]


After updating the “elasticsearch.yml” file you can start the elasticsearch service using the following command:

sudo systemctl start elasticsearch.service


After setting all three nodes we can check the status by executing the following command:

curl -XGET http://ip_address:9200/_cluster/health?pretty

Above command results the following output:

{
  "cluster_name" : "bqcluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 256,
  "active_shards" : 279,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 233,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 54.4921875
}


This way we can configure different nodes to work as an Elasticsearch cluster.

Other Blogs on Elastic Stack:
Introduction to Elasticsearch

Elasticsearch Installation and Configuration on Ubuntu 14.04
Log analysis with Elastic stack 
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Wildcard and Boolean Search in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch 
Metrics Aggregation in Elasticsearch
Bucket Aggregation in Elasticsearch
How to create Elasticsearch Cluster

If you found this article interesting, you can explore  "Mastering Kibana 6.0" and "Kibana 7 Quick Start Guide" to get more insight about Kibana and how we can configure ELK to create dashboards for key performance indicators.

About Author

Anurag Srivastava

Author | Blogger | Tech Lead | Elastic Stack | Innovator |

View Profile

Comments (0)

Leave a comment

Related Blogs

Snapshot and Restore Elasticsearch Indices

Sep 16, 2019, 5:55:06 AM | Anurag Srivastava

htop: An Interactive Process Viewer

Oct 13, 2018, 8:49:59 PM | Anurag Srivastava

Configure Logstash to push MySQL data into Elasticsearch

Feb 9, 2019, 12:06:18 PM | Anurag Srivastava

Configure Logstash to send MongoDB data into Elasticsearch

Mar 9, 2019, 8:20:38 AM | Anurag Srivastava

Introduction to Elasticsearch

Apr 14, 2018, 1:18:05 PM | Anurag Srivastava

Log analysis with Elastic stack

Jan 31, 2018, 6:11:29 AM | Anurag Srivastava

Load csv Data into Elasticsearch

Feb 9, 2019, 6:34:22 PM | Anurag Srivastava

Configure Jenkins for Automated Code Deployment

Jun 13, 2018, 3:44:01 PM | Anurag Srivastava

Why SonarQube is important for IT projects ?

Apr 24, 2018, 2:52:28 PM | Anurag Srivastava

SonarQube installation on Ubuntu

May 12, 2018, 4:47:07 PM | Anurag Srivastava

Top Blogs

Configure SonarQube Scanner with Jenkins

Jun 21, 2018, 4:58:11 AM | Anurag Srivastava

Build and deploy Angular code using Python

Jun 26, 2018, 4:50:18 PM | Anurag Srivastava

Configure Jenkins for Automated Code Deployment

Jun 13, 2018, 3:44:01 PM | Anurag Srivastava

SonarQube installation on Ubuntu

May 12, 2018, 4:47:07 PM | Anurag Srivastava

Execute Commands on Remote Machines using sshpass

Jul 16, 2018, 5:00:02 PM | Anurag Srivastava

Why SonarQube is important for IT projects ?

Apr 24, 2018, 2:52:28 PM | Anurag Srivastava

Elasticsearch Rest API

Jul 31, 2018, 6:16:42 PM | Anurag Srivastava

Analyze your project with SonarQube

Jun 2, 2018, 10:49:54 AM | Anurag Srivastava

Wildcard and Boolean Search in Elasticsearch

Aug 10, 2018, 7:14:40 PM | Anurag Srivastava

Install Jenkins on Ubuntu

May 26, 2018, 6:42:02 PM | Anurag Srivastava