How to manage Open Distro Elasticsearch cluster using REST API service call with Postman tool

Table of Contents

Test Environment

Fedora 32

Open Distro Elasticsearch

Open Distro Elasticsearch is used to analyze and index large datasets. Its primarily used for Log analytics, Real-time application monitoring, Clickstream analytics, Search backend.

Here in this article we will carry out some of the basic operation with the elasticsearch cluster using the REST API calls. In order to create and manage our REST API requests we will be using the ‘Postman’ an collaboration platform for API development.

If you are interested in watching the video. Here is the youtube video on the same step by step operations as shown below.

Step1: Download the Postman API client

As a first step download the Postman application tar file from the below mentioned URL.

URL - https://www.postman.com/downloads/
File - Postman-linux-x64-8.7.0.tar.gz

Step2: Extract the gunzip file

Once downloaded, lets extract the package to a particular folder.

[admin@fedser32 applications]$ tar -xzvf /home/admin/middleware/software/Postman-linux-x64-8.7.0.tar.gz -C .

Step3: Configure the Desktop icon by creating the below file

Now, lets create the below Postman.desktop file for creating a shortcut icon to launch the Postman application which we extract. Make sure to update the Exec line with the correct path where the Postman executable is extracted.

[admin@fedser32 applications]$ pwd
/home/admin/.local/share/applications
[admin@fedser32 applications]$ cat Postman.desktop 
[Desktop Entry]
Encoding=UTF-8
Name=Postman
Exec=/home/admin/middleware/Stack/Postman/app/Postman %U
Icon=/home/admin/middleware/Stack/Postman/app/resources/app/assets/icon.png
Terminal=false
Type=Application
Categories=Development;

Step4: Launch the Elasticsearch and Kibana services using the docker compose file

Here is the sample docker-compose file which you can get from the Open Distro Elasticsearch documentation and use it to launch the Elasticsearch and Kibana service.

[admin@fedser32 Kibana-Docker]$ cat docker-compose.yml
version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.12.0
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - /apps/elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:1.12.0
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node2
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - /apps/elasticsearch/data:/usr/share/elasticsearch/data
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.12.0
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net

#volumes:
#  odfe-data1:
#  odfe-data2:

networks:
  odfe-net:

[admin@fedser32 Kibana-Docker]$ docker-compose up -d
Starting odfe-node1  ... done
Starting odfe-kibana ... done
Starting odfe-node2  ... done

Now that we have our Elasticsearch service up and running, lets carry out the basic operation using the REST API calls from curl command or using the POSTMAN tool.

Step5: Get the elasticsearch cluster details

[admin@fedser32 Kibana-Docker]$ curl -X GET 'https://fedser32.stack.com:9200/' -u admin:admin@1234 --insecure
{
  "name" : "odfe-node1",
  "cluster_name" : "odfe-cluster",
  "cluster_uuid" : "5GOEtg12S6qM5eaBkmzUXg",
  "version" : {
    "number" : "7.10.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
    "build_date" : "2020-11-09T21:30:33.964949Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Step6: Get the elasticsearch node details

[admin@fedser32 Kibana-Docker]$ curl -X GET 'https://fedser32.stack.com:9200/_cat/nodes?v' -u admin:admin@1234 --insecure
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.18.0.2           13          97   5    0.51    0.51     0.45 dimr      -      odfe-node1
172.18.0.4            3          97   5    0.51    0.51     0.45 dimr      *      odfe-node2

Step7: Get the list of installed plugins in elasticsearch cluster

[admin@fedser32 Kibana-Docker]$ curl -X GET 'https://localhost:9200/_cat/plugins?v' -u admin:admin@1234 --insecure
name       component                       version
odfe-node1 opendistro-anomaly-detection    1.12.0.0
odfe-node1 opendistro-job-scheduler        1.12.0.0
odfe-node1 opendistro-knn                  1.12.0.0
odfe-node1 opendistro-reports-scheduler    1.12.0.0
odfe-node1 opendistro_alerting             1.12.0.2
odfe-node1 opendistro_index_management     1.12.0.1
odfe-node1 opendistro_performance_analyzer 1.12.0.0
odfe-node1 opendistro_security             1.12.0.0
odfe-node1 opendistro_sql                  1.12.0.0
odfe-node2 opendistro-anomaly-detection    1.12.0.0
odfe-node2 opendistro-job-scheduler        1.12.0.0
odfe-node2 opendistro-knn                  1.12.0.0
odfe-node2 opendistro-reports-scheduler    1.12.0.0
odfe-node2 opendistro_alerting             1.12.0.2
odfe-node2 opendistro_index_management     1.12.0.1
odfe-node2 opendistro_performance_analyzer 1.12.0.0
odfe-node2 opendistro_security             1.12.0.0
odfe-node2 opendistro_sql                  1.12.0.0

Step8: Index a document in elasticsearch cluster

[admin@fedser32 ELK]$ cat singledoc.json 
{
  "title": "The Wind Rises",
  "release_date": "2013-07-20"
}

[admin@fedser32 ELK]$ curl -X PUT -H 'Content-Type: application/json' 'https://fedser32.stack.com:9200/movies/_doc/3?pretty' -u admin:admin@1234 --insecure -d @singledoc.json 
{
  "_index" : "movies",
  "_type" : "_doc",
  "_id" : "3",
  "_version" : 2,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 12,
  "_primary_term" : 7
}

Step9: Search for the document

[admin@fedser32 ELK]$ curl -X GET 'https://fedser32.stack.com:9200/movies/_search?q=Wind&pretty' -u admin:admin@1234 --insecure
{
  "took" : 24,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.7361701,
    "hits" : [
      {
        "_index" : "movies",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 0.7361701,
        "_source" : {
          "title" : "The Wind Rises",
          "release_date" : "2013-07-20"
        }
      }
    ]
  }
}

Step10: Delete the document previously indexed

In this operation i have not used the pretty argument which actually outputs the JSON data in human readable format.

[admin@fedser32 ELK]$ curl -X DELETE 'https://fedser32.stack.com:9200/movies/_doc/3' -u admin:admin@1234 --insecure
{"_index":"movies","_type":"_doc","_id":"3","_version":3,"result":"deleted","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":13,"_primary_term":7}

Step11: Index bulk documents

In this operation i have not used the pretty argument which actually outputs the JSON data in human readable format.

[admin@fedser32 ELK]$ curl -X POST -H 'Content-Type: application/json' 'https://fedser32.stack.com:9200/data/_bulk' -u admin:admin@1234 --insecure --data-binary @bulkdoc.json 
{"took":8,"errors":false,"items":[{"index":{"_index":"movies","_type":"_doc","_id":"1","_version":7,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":16,"_primary_term":7,"status":200}},{"index":{"_index":"movies","_type":"_doc","_id":"2","_version":5,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":17,"_primary_term":7,"status":200}}]}

These are some of the basic operation that we can carry out using the REST API call on Elasticsearch cluster. You can go through the complete list of operations that are supported by the Elasticsearch REST API service at the following documentation reference – https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/rest-api-reference/

Hope you enjoyed reading this article. Thank you..

How to manage Open Distro Elasticsearch cluster using REST API service call with Postman tool